Strange DNS behavior in some containers including cert-manager

Environmental Info:
K3s Version: v1.22.5+k3s2 (for compatibility with rancher)

Node(s) CPU architecture, OS, and Version:

All nodes:

Architecture: AMD64
OS: Ubuntu 21.10 cloud-init image
Kernel: 5.13.0

Cluster Configuration:

3 control plane nodes with integrated etcd, 2 worker nodes. Feel free to look at the deployment scripts: https://gitlab.com/macklenc/ha-k3s-ansible-deployment (yes, the secrets are placeholders).
Describe the bug:

In some containers, e.g. cert-manager and ubuntu, the DNS seems to append my home networks domain name to every request (my net name is home.net). E.g. running an nslookup command against google will show that the address that was actually looked up was google.com.home.net, which works fine for http, but messes with SSL when trying to use e.g. Lets Encrypt. I realize from the linked issues below that there are some workarounds by either removing (in my case) home.net from /etc/resolv.conf inside the running container, or by changing ndots:5 back to the default of 1 in the same file. Or even appending a . to the domain name to force resolution as a FQDN. But all of these solutions are a bit hacky. Seems to me that deploying a fresh K3s cluster and deploying a container to e.g. curl an https domain should work out of the box.
Now strangely, using a busybox container doesn't seem to reproduce the issue even though the resolv.conf file in the container is identical to the problematic containers. See the reproduction section for examples.
Seemingly related issues.

https://github.com/k3s-io/k3s/issues/5044
https://github.com/k3s-io/k3s/issues/2801
https://github.com/k3s-io/k3s/issues/1527
https://stackoverflow.com/questions/62635732/k3s-rancher-debian-based-docker-images-not-resolving-dns-properly
https://stackoverflow.com/questions/63211706/update-default-search-domains-in-pods
https://stackoverflow.com/questions/54620711/kubernetes-appending-own-domain-name-to-all-requests

Steps To Reproduce:

Installed K3s: using the ansible script linked above. Noteworthy configs in a later section.
Run a container and try to e.g. curl a https site, the domain name gets re-written

As an FYI, I'm running pfSense for my router. I did try resetting it to factory defaults which didn't seem to help the issue.
Launching a network debugging test image based on Debian:
kubectl run tmp-shell --rm -i --tty --image nicolaka/netshoot -- /bin/bash
> nslookup google.com

will return:
Server:         <IP_ADDRESS>
Address:        <IP_ADDRESS>#53

Name:   google.com.home.net
Address: <IP_ADDRESS>

however, using the busybox image as recommended from the guide, the DNS resolves just fine:
kubectl run -it --rm --restart=Never busybox --image=busybox:1.28 -- nslookup google.com

will return:
Server:    <IP_ADDRESS>
Address 1: <IP_ADDRESS> kube-dns.kube-system.svc.cluster.local

Name:      google.com
Address 1: 2607:f8b0:400f:803::2004 den08s06-in-x04.1e100.net
Address 2: <IP_ADDRESS> den16s09-in-f4.1e100.net

Expected behavior:

pings/curls/nslookups for a domain e.g. google.com, should return results for google.com.
Actual behavior:

pings/curls/nslookups for a domain e.g. google.com is returning google.com.home.net
Additional context / logs:

Host OS resolv.conf (maintained by systemd-resolvd):
nameserver <IP_ADDRESS>
options edns0 trust-ad
search home.net

Control plane systemd service:
[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
After=network-online.target

[Service]
Type=notify
Environment=K3S_TOKEN=hXx9S1A

ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s server --data-dir /var/lib/rancher/k3s --cluster-init --node-taint CriticalAddonsOnly=true:NoExecute --tls-san <IP_ADDRESS> --disable servicelb --disable traefik
KillMode=process
Delegate=yes
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

Agent systemd service:
[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
After=network-online.target

[Service]
Type=notify
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s agent --server https://<IP_ADDRESS>:6443 --token K10b9f1d540b53277cbebbb8695322e4a65a5fcca92c697f4c96fd1e0a67c7c5c5b::server:hXx9S1A
KillMode=process
Delegate=yes
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

Backporting

[ ] Needs backporting to older releases

Just for some context, I'm still pretty new to K3s/Kubernetes so feel free to let me know if I left out an important config or details.

Could you check in what node is your coredns pod running? Then verify if nslookup works only on pods that are running in that same node

Sure thing. I had to get a little fancy since coredns was running on a control plane node. I created the following manifest:
---
apiVersion: v1
kind: Pod
metadata:
  name: netshoot
spec:
  restartPolicy: OnFailure
  nodeSelector:
    kubernetes.io/hostname: queen0
  tolerations:
    - key: CriticalAddonsOnly
      operator: Equal
      value: "true"
      effect: NoExecute
  containers:
    - name: netshoot
      image: nicolaka/netshoot
      imagePullPolicy: IfNotPresent
      command: ["nslookup"]
      args: ["google.com"]

Which resulted in the same .home.net getting appended:
~
❯ k -n kube-system get pods coredns-5cdc799f68-vfkd8 -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
coredns-5cdc799f68-vfkd8   1/1     Running   0          21h   <IP_ADDRESS>   queen0   <none>           <none>

~
❯ k get pods -o wide
NAME       READY   STATUS             RESTARTS      AGE   IP           NODE     NOMINATED NODE   READINESS GATES
netshoot   0/1     Completed          2 (18s ago)   34s   <IP_ADDRESS>   queen0   <none>           <none>

~
❯ k logs netshoot
Server:         <IP_ADDRESS>
Address:        <IP_ADDRESS>#53

Name:   google.com.home.net
Address: <IP_ADDRESS>

I don't know if it's relevent, but this showed up in the events:
  Type     Reason       Age              From               Message
  ----     ------       ----             ----               -------
Warning  FailedMount  1s (x3 over 3s)  kubelet            MountVolume.SetUp failed for volume "kube-api-access-wvxn9" : object "default"/"kube-root-ca.crt" not registered

Also, it may be worth noting that the IP resolved is for some website out on the net called home.net instead of resolving to my router's external IP. I tried adding a firewall rule to intercept external DNS queries and re-routing them to my network's DNS server, but it still resolves to the internet IP instead of my routers IP.

Sure thing. I had to get a little fancy since coredns was running on a control plane node. I created the following manifest:
---
apiVersion: v1
kind: Pod
metadata:
  name: netshoot
spec:
  restartPolicy: OnFailure
  nodeSelector:
    kubernetes.io/hostname: queen0
  tolerations:
    - key: CriticalAddonsOnly
      operator: Equal
      value: "true"
      effect: NoExecute
  containers:
    - name: netshoot
      image: nicolaka/netshoot
      imagePullPolicy: IfNotPresent
      command: ["nslookup"]
      args: ["google.com"]

Which resulted in the same .home.net getting appended:
~
❯ k -n kube-system get pods coredns-5cdc799f68-vfkd8 -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
coredns-5cdc799f68-vfkd8   1/1     Running   0          21h   <IP_ADDRESS>   queen0   <none>           <none>

~
❯ k get pods -o wide
NAME       READY   STATUS             RESTARTS      AGE   IP           NODE     NOMINATED NODE   READINESS GATES
netshoot   0/1     Completed          2 (18s ago)   34s   <IP_ADDRESS>   queen0   <none>           <none>

~
❯ k logs netshoot
Server:         <IP_ADDRESS>
Address:        <IP_ADDRESS>#53

Name:   google.com.home.net
Address: <IP_ADDRESS>

I don't know if it's relevent, but this showed up in the events:
  Type     Reason       Age              From               Message
  ----     ------       ----             ----               -------
Warning  FailedMount  1s (x3 over 3s)  kubelet            MountVolume.SetUp failed for volume "kube-api-access-wvxn9" : object "default"/"kube-root-ca.crt" not registered

Also, it may be worth noting that the IP resolved is for some website out on the net called home.net instead of resolving to my router's external IP. I tried adding a firewall rule to intercept external DNS queries and re-routing them to my network's DNS server, but it still resolves to the internet IP instead of my routers IP.

Could you run cat /etc/resolv.conf in both the "working" pod and the non-working pod?

 Done
E: The repository 'http://security.ubuntu.com/ubuntu focal-security Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'http://archive.ubuntu.com/ubuntu focal Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'http://archive.ubuntu.com/ubuntu focal-updates Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'http://archive.ubuntu.com/ubuntu focal-backports Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.

When you lookup google.com, it will try google.com.default.svc.cluster.local, google.com.svc.cluster.local, google.com.cluster.local, google.com.home.net and google.com. The first one that comes with an answer is the chosen one. For strange reasons google.com.home.net exists but you are not seeing a wrong behaviour

I recommend you to use the tool dig instead of nslookup. When using dig, if you don't set the flag +search, it will just search for what you provided without trying to build FQDN based on your resolv.conf

I'm not sure what you mean by home.net is my hostname. I double checked that the VMs running the cluster have different names, and the containers created have the same hostname as the name of the container when I created them. E.g. netshoot.
Out of curiosity if that is normal behavior, why do busybox and alpine images seem to work while the distroless coredns and debian-based images do not? You're correct that dig works as expected bypassing the resolv.conf, but that unfortunately doesn't solve my problem. I tried adding a domain override to my router to point home.net to itself, as well as choosing a network name that I own, and both of them still tried to append the new network names even though they don't actually return.
E.g. overriding home.net in my DNS:
❯ kubectl run -it --rm --restart=Never netshoot --image=nicolaka/netshoot -- nslookup google.com
Server:         <IP_ADDRESS>
Address:        <IP_ADDRESS>#53

Non-authoritative answer:
*** Can't find google.com.home.net: No answer

pod "netshoot" deleted

And using a domain I own, but has no DNS records (A, CNAME, or otherwise):
❯ kubectl run -it --rm --restart=Never netshoot --image=nicolaka/netshoot -- nslookup google.com
Server:         <IP_ADDRESS>
Address:        <IP_ADDRESS>#53

Non-authoritative answer:
*** Can't find google.com.continuumtek.com: No answer

pod "netshoot" deleted

This problem doesn't seem to exist on any other host on my network, bare metal, VM, docker, etc. Just when running on my new k3s cluster.

Sorry, I should not have used the word hostname, ignore that. Let me go through the basics (sorry if you already knew this).
The resolv.conf of your pods is injected by kubelet. You can get more information about it here: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/. By default, kubelet keeps the search domains that it finds in the node's /etc/resolv.conf and adds some extra related to the deployment (e.g. svc.cluster.local). Note that this is configurable as you can read in the previous link.
In your case, you must have search home.net in the resolv.conf file of your node. By looking at the resolv.conf files of your pods, it seems kubelet is working correctly and injecting the correct resolv.conf file. Therefore, in my opinion, k3s is working correctly.
Now, regarding google.com.home.net. I am not sure how each OS is implementing the read on resolv.conf. According to its manual: https://www.man7.org/linux/man-pages/man5/resolv.conf.5.html, it will always try the different search domains before testing google.com alone (i.e. an initial absolute query) because it thinks that it is not a complete FQDN (unless it had 5 dots). This is because of the kubelet's default option ndots5 that you can see in the resolv.conf of the pod. This link has very good information about it: https://pracucci.com/kubernetes-dns-resolution-ndots-options-and-why-it-may-affect-application-performances.html and a couple of potential solutions you can apply.
One thing that surprises me is that you get problems when running apt. I think it is just bad luck that google.com.home.net exists but I'd be surprised if you got also a "collision" with other urls. Could you show me an output of what you get when using apt?
In any case, due to what I explained, I don't think this is a problem of K3s or Kubernetes, in my opinion, what you are seeing is the expected behaviour

I'm not sure what you mean by home.net is my hostname. I double checked that the VMs running the cluster have different names, and the containers created have the same hostname as the name of the container when I created them. E.g. netshoot.
Out of curiosity if that is normal behavior, why do busybox and alpine images seem to work while the distroless coredns and debian-based images do not? You're correct that dig works as expected bypassing the resolv.conf, but that unfortunately doesn't solve my problem (since e.g. apt still uses resolv.conf). I tried adding a domain override to my router to point home.net to itself, as well as choosing a network name that I own, and both of them still tried to append the new network names even though they don't actually return.
E.g. overriding home.net in my DNS:
❯ kubectl run -it --rm --restart=Never netshoot --image=nicolaka/netshoot -- nslookup google.com
Server:         <IP_ADDRESS>
Address:        <IP_ADDRESS>#53

Non-authoritative answer:
*** Can't find google.com.home.net: No answer

pod "netshoot" deleted

And using a domain I own, but has no DNS records (A, CNAME, or otherwise):
❯ kubectl run -it --rm --restart=Never netshoot --image=nicolaka/netshoot -- nslookup google.com
Server:         <IP_ADDRESS>
Address:        <IP_ADDRESS>#53

Non-authoritative answer:
*** Can't find google.com.continuumtek.com: No answer

pod "netshoot" deleted

This problem doesn't seem to exist on any other host on my network, bare metal, VM, docker, etc. Just when running on my new k3s cluster.

It is really weird that the image netshoot never tries to look up the "initial absolute query", i.e. google.com. If I understand correctly, according to https://www.man7.org/linux/man-pages/man5/resolv.conf.5.html, at some point it must try with google.com without adding any extra domain (what tries first, will depend on ndots config)

I always appreciate setting a good foundation, so thanks for that! I had actually found that ndots article earlier which was super helpful. I was able to "recreate" what I'm seeing by setting ndots:5 on a bare-metal host. Do you know why 5 is the default with kubelet and if that can be changed? Seems no matter how hard I try to block DNS traffic for the internal network name I choose, it's still able to get through and find e.g. that home.net IP address.
I have experimented with setting ndots to 1 and removing the home.net search path, and both have fixed the problem, so it seems like ndots:5 isn't the best default from my point of view. Unfortunately removing the search path home.net from the host nodes does mess with the ability of the nodes to be able to pull down images to run. Not sure if I'm messing up a k3s config or what, but it's a pretty vanilla install. Maybe I'll give k8s a try with kubespray to see if I get the same behavior. I have also tried resetting my pfSense router back to factory defaults, which doesn't seem to fix the issue either.
I appreciate the help, I think I see how this isn't necessarily a k3s problem now but I'm at a total loss on how to fix this or where to seek help at this point.

Here is the reasoning for that ==> https://github.com/kubernetes/kubernetes/issues/33554#issuecomment-266251056
But apart from the google.com problem, do you get problems with other urls?

Thanks for the link, I'll take a look.
Yeah, that issue comes up with every URL I've tried. It gets really fun when accessing internal resources, e.g. pv0.home.net turns into pve0.home.net.home.net.
Oh... Interesting update ubuntu's apt update is working now when using my domain name instead of home.net. And I was mistaken, looks like netshoot is alpine based, not debian based (which still isn't resolving as expected).

I really appreciate your assistance. I wasn't able to get home.net working as my home network name, but after I removed my host from using cloudflare back to google domains that domain started to work. Super weird that the resolv DNS is able to bypass my local DNS overrides, but at least it works now.

Once again, thanks for your help. Unless you have further input, I'll go ahead and close the issue. If anyone else comes along in the future, the problem seems to be related to how the unbound service in pfSense handles transparent traffic. If you want the external domain to be blocked you can set the system domain local zone type to static to block the outbound request, though I'm not sure what the side effects are, here's the relevant doc:
static
                 If  there  is a match from local data, the query is answered.
                 Otherwise, the query is answered  with  nodata  or  nxdomain.
                 For  a  negative  answer  a  SOA is included in the answer if
                 present as local-data for the zone apex domain.

It seems that the general recommendation is to choose an internal TLD that doesn't exist on the internet, e.g. home.lan instead.

Figured I'd follow up with a better solution than what I implemented if anyone else has this issue: https://forums.lawrencesystems.com/t/google-domains-vs-cloudflare-dns-with-ndots-5-in-resolv-conf/12887/3?u=macklenc

@macklenc Thank you so much. Your research fixed bunch of my issues.
adding this to pfsense dns config made it easy
server:
local-zone: "example.com" static


Update all dependencies to their latest versions

It's been nearly one and a half year since I've created this project. Most of the dependencies must be outdated. After the updates, some of the functions used in the project might be deprecated. These also should be fixed.

I'll pick this

@ghost did you update this app?


[Error at Build] Memory leak detected

Hello, I'm trying to build gyro from master  (90af164477936d3c2cb4e93b6ca29b5fc9025fd2) using also zig from master (0.8.0-dev.1860+1fada3746) and I've had this happen while building gyro with gyro bootstrapped. I'm pretty sure it happened because I'm using zig from master, so I'll just the release instead. Just wanted to give you guys a heads up that it might break gyro in future updates.
Also, what's the supported zig version to build gyro?

Hello! Gyro is meant to be built by zig master, it looks like you've hit some sort of DNS error when grabbing packages from astrolabe, and the memory leak is from one of the dependencies (I'll look into that for sure).
Do you get the same error when using the prebuilt gyro? you could use that to do the bootstrap build just to check. And are you able to access https://astrolabe.pm?

Yes, I get the same error using the prebuilt gyro. This error happens when I try to build gyro with gyro (bootstrapped and prebuilt). Also, I am able to access astrolabe on my browser.

I'd like to say that I'm getting this on my desktop, but not my laptop. I'll try changing my DNS server...

Maybe it's the port? Like previously said, I can acess astrolabe using my browser, but not through gyro.

Oops, closed it by mistake

It could be the port. I've been able to access it on other networks, so perhaps the PM is doing something that the router has blocked ports for? Then again, I don't recall having set up any firewall...

That could be related to the hosting service. I suspect my Internet Provider blocks any incoming SSH connections to my machine, so something of similar effect might be happing on the PM side? I'm not sure.

It does all the requests over the HTTPS port, maybe the issue is in zig-network, you could try to point zfetch's tests at astrolabe to see if they work correctly.
It might also be some configuration thing I need to do on the nameserver side of things, but I don't know what that might be

Interestingly, it has started working again for me.

I'm going to close this, gyro has migrated from zfetch to libcurl, so it has a different networking stack


Call insertAtomicBlockAsync to upload image but ends up to insert to top of editor

I tried to customize a image uploader and insert the image to the editor. I found that when I tried to insert in a specific line, the image was inserted to the top of the editor after insertAtomicBlockAsync was called.

my code:
const uploadImage = (url: string) => {
    return new Promise<TAsyncAtomicBlockResponse>(async (resolve, reject) => {
      if (!url) {
        console.log("no url");
        reject();
        return;
      }
      resolve({
        data: {
          url,
          type: "image",
          width: 400,
          height: 200,
        },
      });
    });
  };

  const onImageUpload = async (result: CroppedImage) => {
    await ref.current?.insertAtomicBlockAsync(
      "IMAGE",
      uploadImage(result.cropped_src),
      "Uploading..."
    );
  };

Hey, have you found a workaround for this? I am also having issues with the insertAtomicBlockAsync api. My image doesn't load, and my code is the same as yours. The images don't go to the top of the file though.


Publish Public Roadmap

Create and organize GitHub issues outlining planned features and improvements. Organize them into milestones or categories to provide a clear picture of the development roadmap.

We've successfully created and organized a public backlog that outlines our planned features and improvements. This backlog will serve as our dynamic roadmap, and we'll be revisiting it weekly to ensure it stays aligned with our development goals and community feedback. This issue has been addressed with the establishment of the backlog.
You can view the backlog here for ongoing updates: AgenticJS Public Backlog.
Closing this issue as completed. Thank you, everyone, for your contributions!
Original idea from @valdo99 Thanks ✌


Coverage fixes

I was thinking it might be a good idea to require 100% coverage (with select 'no cover' lines), but codecov might be too buggy. We can already see some of the issues with codecov https://codecov.io/gh/Quansight/ndindex/compare/9bd57a3c93d56bb9ee8f7b3984145d772ab8b2d7...51edc3bad7667c84309c1e03c7e57324cdb1a09f/src/ndindex/tuple.py?before=ndindex/tuple.py. I guess we can require the coverage in Travis, and just use codecov as a way of seeing the coverage.

I think codecov is definitely too buggy for that

I have it so it just fails on Travis, and codecov is just there so you can look at it. Except I can't figure out how to make codecov set a status on GitHub when the tests fail.

codecov just sets the status randomly. I give up. I guess if it sets it great, otherwise, we can just look for the URL in the log, or use the normal coverage output.

We will also probably end up with some flakiness because of hypothesis randomly not covering some lines. If it happens, we can add some @examples to force them to always be tested.

If this ends up being a pain we can remove it, but I think this is a good goal to have for now.


Add Knox

@Treydone - thank you very much for the high quality contribution! I'm not exactly sure why travis is failing, but I'm looking into it now.
I'll get this merged soon and will make it part of the 0.1.10 release. I'm hoping to hear back on the couple of open issues before I cut that release, as I'd like to include any fixes that may come about there. If I don't hear anything in a couple days, I'll go ahead and release.

@Treydone - I just commented on the patch. I released 0.1.9 just yesterday, sorry about that. If you can update the pom to 0.1.10-SNAPSHOT, I think that will fix the issue. Thanks!

Ok, seems to be good now!

Coverage decreased (-1.9%) to 88.143% when pulling c7a9232ecf94dda047f19f5ec3d8a3bc1e5ebe0f on Treydone:master into 59511ed1e4352a9f2c45bb232e7baf6e8342bd63 on sakserv:master.

LGTM, thanks again!

These changes have been included in 0.1.10 which was just released. Thanks again!


keras.Model.fit cannot handle variable epoch sizes

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): ubuntu 18.04
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): v2.2.0-0-g2b96f3662b 2.2.0
Python version: 3.6.9

Describe the current behavior
keras.Model.fit with finite-length dataset inputs can run without steps_per_epoch. However, if the number of batches in the epoch changes between epochs, this can lead to:

truncation of epoch if subsequence epochs are longer than the initial epoch; and
truncation of training if subsequence epochsa re shorter than the initial epoch.

While not a common occurance, this happens e.g. in graph neural network applications where it makes more sense to batch according to some maximum number of edges/nodes in a graph as opposed to a fixed batch size. A fixed number of graphs in a dataset can result in a variable number of batches when these are shuffled.
Describe the expected behavior
Each epoch of fit should iterate over a dataset in it's entirety when steps_per_epoch is not provided, and run to the defined number of epochs (unless terminated prior with callbacks), even if the number of steps varies between epochs. Alternatively, a special steps_per_epoch value (say, -1) should be able to be provided to specify a dynamic epoch size.
Standalone code to reproduce the issue
Colab
Other info / logs
From colab above, when subsequence epochs are shorter than the first:
WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches (in this case, 30 batches). You may need to use the repeat() function when building your dataset.

Was able to reproduce the issue with TF v2.2 and TF-nightly. Please find the gist of it here. Thanks!

@jackd Thanks for the issue! I'd recommend using a Dataset that contains every batch you want to train on, and setting steps_per_epoch as an arbitrary number, so that epoch boundaries are just a way to check on the progress of training in your case. If that doesn't work, then you can always write a custom training loop for your use case: https://keras.io/guides/writing_a_training_loop_from_scratch/


Add a functional test against live origins

This ties in with https://github.com/getlantern/tlsmasq/issues/30 and came out of a meeting with @hwh33 (context here).
We'll need a list of origins and a functional test capable of running expected input and asserting the output against those origins. This'll uncover a lot of potential bugs in the system.

Related to https://github.com/getlantern/lantern-internal/issues/4723


Difference in sum of tests done, as total sum of tests done in india is not same as cumulative sum of tests done by each state.

Describe the bug
Difference in data, as tests done differ when we see tests done in india and total sum the tests done adding up all states.
Expected behavior
There should not be difference between total tests done in india and cumulative sum of tests done by states.
Screenshots
As it is showing that tests done in india is approx 8.3 Lakhs but by adding separately all states sum of test done is coming approx 9.9 Lakhs.

This is not an issue on data collection or the front end. This is an issue with the govt agencies reporting different numbers.


site-packages\keras\callbacks.py:1090: RuntimeWarning: invalid value encountered in less   self.monitor_op = lambda a, b: np.less(a, b - self.min_delta)

2020-02-22 17:19:04.316272: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:971]      0
2020-02-22 17:19:04.318202: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:984] 0:   N
2020-02-22 17:19:04.320169: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6392 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)
Create YOLOv3 model with 9 anchors and 5 classes.
Load weights C:\Users\Dawei Chen\Desktop\Vince\hotel\m41\trained_weights_final.h5.
Freeze the first 185 layers of total 252 layers.
Train on 2200 samples, val on 549 samples, with batch size 8.
Epoch 1/100
275/275 [==============================] - 104s 378ms/step - loss: 24.3365 - val_loss: 22.9203
Epoch 2/100
275/275 [==============================] - 93s 339ms/step - loss: nan - val_loss: nan
C:\Users\Dawei Chen.conda\envs\vince\lib\site-packages\keras\callbacks.py:1090: RuntimeWarning: invalid value encountered in less
self.monitor_op = lambda a, b: np.less(a, b - self.min_delta)
C:\Users\Dawei Chen.conda\envs\vince\lib\site-packages\keras\callbacks.py:543: RuntimeWarning: invalid value encountered in less
if self.monitor_op(current - self.min_delta, self.best):
Epoch 3/100
135/275 [=============>................] - ETA: 37s - loss: nan
(vince) C:\Users\Dawei Chen\Desktop\Vince\hotel\keras-yolo3-master>

Is there anyone else facing this problem? I guess my dataset has bad samples. I trained YOLO3 before with one dataset. The result is pretty good. But this time, when I train it with another dataset, there's always some nan/inf loss.

Early Sttoping callback and LROnPlateau use val_loss as a reference. If val_loss is a a not numeric value, these callbacks won't work. Now, the messages showed on console mean that the callbacks are not working properly because of nan values.
The solution of this problem is usually de .names file, if you pass a wrong .names file, the Dataset object will be corrupted and the train will return nan values.
Another solution is to check tfrecord files. VoTT use to export some corrupted files.

Thank you for the suggestion, dear Piero. In fact, the inf/nan train loss
appears within one epoch, which means it doesn't reach the validation
process. Am I right?
Best,
Wen
Piero Casusol<EMAIL_ADDRESS>于2020年3月4日周三 下午1:14写道：

Early Sttoping callback and LROnPlateau use val_loss as a reference. If
val_loss is a a not numeric value, these callbacks won't work. Now, the
messages showed on console mean that the callbacks are not working properly
because of nan values.
The solution of this problem is usually de .names file, if you pass a
wrong .names file, the Dataset object will be corrupted and the train will
return nan values.
Another solution is to check tfrecord files. VoTT use to export some
corrupted files.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/zzh8829/yolov3-tf2/issues/186?email_source=notifications&email_token=AIGXRW7J2JGF7Y3FGJBP7CLRF2SCVA5CNFSM4KZVRIF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENZWMRQ#issuecomment-594765382,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AIGXRWZCNYEPYTXT5RUJLBLRF2SCVANCNFSM4KZVRIFQ
.

--
Wen Xie, Ph.D. Student
Department of Electrical and Computer Engineering
W319, Engineering Building 2
4800 Calhoun Road
University of Houston
Houston, TX 77004-4005
Tele:<PHONE_NUMBER>
<EMAIL_ADDRESS><EMAIL_ADDRESS>B.ENG.  Electronic Information Engineering, UESTCB.A. Finance, UESTC


Search on a static site

Will this have to be done client-side?

I guess I could run a little server-side dealie that consumes a the-whole-site JSON?

It might not be pretty, but it works.

I just stumbled on your blog and was surprised to see a search function. That's a very neat solution, would not have thought about it! Thanks for the tip :-)


Integrate 'Copy Citation' Button with Archive Snapshot Functionality

Issue Description: Users currently cannot easily cite and archive page snapshots.
Proposed Solution: Add a 'Copy Citation' button, integrated with WayBack Machine functionality or similar, to allow users to cite and save snapshots of the page.

https://docs.github.com/de/actions/security-guides/using-secrets-in-github-actions


Use shared memory lock for threads generated from same module

Multiple threads should use the same lock to protect atomic operations. Fix #1958
Before this PR, each thread was using a different lock to protect atomic operations (e.g. atomic add), making the lock ineffective.

#1958

LGTM


Changes the default skybox to avoid the blue shade

Old:

New:

The blue shade is still a little bit present but is a lot more subtle.
Closes #18226

Looks like the horizon transition towards the ground is very hard, maybe it can be made a bit softer?

Looks like the horizon transition towards the ground is very hard, maybe it can be made a bit softer?

Maybe, but the transition between the Sky and the ground is not interpolated in the code (see line 188 of scene/resources/sky_box.cpp). The little softness seen might be due to texture sampling.
To soften the transition, I tried choosing two close colors for ground horizon and sky horizon, but it looked weirder than it does now. Or I could add a parameter to ProceduralSky to soften the transition.

Yeah, but as explained it looks weird:

For me this is too spooky.

I think the hard transition looks better

I often just desaturate the sky color and tighten the sun when doing 3D projects for that reason. Thanks to this I might not have to do it again :)

I never understood this, why is the sky contribution 1.0 by default? Shouldnt this be changed instead of the sky color? In real life I can have a strong blue sky, but white objects are still white, not blue, unless they are reflective/mettalic. shouldnt this behave the same way?

@nunodonato In real life, our eyes adapt to different types of lighting so we see clearly.
Perhaps this can be improved further when we get the Vulkan renderer, however, for now I think this is a good change. We should also compare to other engines to see how they look.

Can this be cherry picked for stable?

Bit late but shouldn't the ground color be something brighter as well? Maybe some greens?

I think a neutral color like gray is better than green, since it would be reflected all around the scene.

Feel free to experiment and post screenshots here, so we can decide what should be the default by a vote or something like that.

For me the problem isn't so much the color but the fact that I just can't drop mesh instances into a scene and start messing around with ideas. Everything gets smushed together visually most of the time and it kills any flow.

@photex i think u should open a new issue for that.


Add tests for routing

I really need help with testing. I welcome PRs for tests.

@RogerCreasy  i can work on that

@andradedev Great! I am not very good with tests. I may have questions for you. Keep in mind that if you are doing this for Hacktoberfest, PRs have to be made between October 1 and 31 in order to count.

What routing do you exactly mean? The Route-class in lib/Framework?
If yes, unit testing this class would need some small refactorings.

@dangenendt yes. But, @andradedev is working on this one. Perhaps we can discuss the steps/changes needed here, and we can all work on it?

@RogerCreasy Yes, i saw that @andradedev is working on that. I just where interested in that one :)
thanks for the info

@dangenendt we need other tests; I am not good with testing....If you are, would you be willing to help with others where you see a need? Feel free to open issues. If you do, and you want me to tag them as Hacktoberfest, let me know.

I'm pretty new to ChassisPHP. What important functionality should be tested except the routing? What would you recommend?

I just took a quick look through the project, and with my limited knowledge of testing, I don't see much.
I am working on PSR-7 implementation, which will add a need for testing responses and requests on the container. Middleware, logging, and authentication testing will come soon too.
Sorry. There will be more soon,
THanks so much for your willingness to help!

@RogerCreasy
No problem.
If the code is ready, feel free to mention me.
I can take a look on it then.


plot_sky scatter labels for single target in time

When doing something like
from astroplan.plots import plot_sky
from astroplan import time_grid_from_range
import astropy.units as u

start_time = Time("2024-01-01 00:00:00")
time_grid = time_grid_from_range([start_time, start_time+30*u.day*12], time_resolution=30*u.day)

plot_sky(target_source, observer, time_grid, ax=None, style_kwargs=None, north_to_east_ccw=False, grid=True, az_label_offset=0*u.deg, warn_below_horizon=False, style_sheet=None)

the timing information is missing from the plot (in this case it would be a small number from 1 to 12 months, but in the case of an arbitrary amount of time it would be a finer array)
I did not find a simple way from astroplan API to do it, so I would propose to do it in the form of a (optional) colorbar which with a time scalar mappable maps time to color.

in case of multiple targets pone can always use different markers


Word Wrap Issue !

Does this issue occur when all extensions are disabled?: Yes/No

VS Code Version:
OS Version:

Can anyone tell me how to solve this word wrap issue in line number 42 , 43 , 44 ??

Duplicate of #32856


feat: Export Handlers (req, resp, https)

This is usufull when handlers will be dynamicaly created and removed.

Can you please explain why you can't use the existing methods to manipulate the handlers?
Making a private field public is risky, as it cannot be undone.

@elazarl in current implementation, Handlers are manipulated by .Do(, .HijackConnect(, .HandleConnect( and only with an = append(.
It makes impossible to remove elements or to know what are elements inside.
What I need is to Add / List and Remove handlers to be able to do it dynamically (at run-time).

@elazarl an usage example: https://github.com/guilhem/egress-proxy-operator/blob/master/controllers/request_controller.go#L126

I don't see why you need indirection. Making them public should be good enough.
Slice is already a pointer.
Also not sure how you handle concurrency.

I don't see why you need indirection. Making them public should be good enough.
Slice is already a pointer.
Also not sure how you handle concurrency.

I totally agree with you.
But I faced a lot of problems with slices and make it working with pointer. 🤷♂
I need to do more tests.
I also find this article, not sure if applicable: https://medium.com/swlh/golang-tips-why-pointers-to-slices-are-useful-and-how-ignoring-them-can-lead-to-tricky-bugs-cac90f72e77b

@elazarl I did more tests and slice is not updated if it's not a pointer.
Having a pointer also have some interest, it's possible to update content atomically. Just create a new slice with right content and update pointer in 1 step.

@guilhem what I suggest is, add a single handler that has internal handlers, and handle concurrency inside
We'll add this as an extension.
What do you think?

@guilhem what I suggest is, add a single handler that has internal handlers, and handle concurrency inside
We'll add this as an extension.
What do you think?

I will try to do this solution :)


flutter apk build graddle fail

Gradle build failed to produce an Android package.
tried clean project, reinstalling fresh stable flutter and dart sdks.
using gradle 3.3.0 as described at https://flutter.dev/docs/development/packages-and-plugins/androidx-compatibility
 classpath 'com.android.tools.build:gradle:3.3.0'

android.enableJetifier=true
android.useAndroidX=true
org.gradle.jvmargs=-Xmx1536M

distributionUrl=https\://services.gradle.org/distributions/gradle-5.1.1-all.zip

[flutter] flutter doctor
Doctor summary (to see all details, run flutter doctor -v):
[√] Flutter (Channel stable, v1.7.8+hotfix.4, on Microsoft Windows [Version 10.0.17763.678], locale en-US)
[√] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
[√] Android Studio (version 3.5)
[√] VS Code (version 1.37.1)
[√] Connected device (1 available)

**Gradle build failed to produce an Android package.**

#0      throwToolExit (package:flutter_tools/src/base/common.dart:28:3)
#1      _buildGradleProjectV2 (package:flutter_tools/src/android/gradle.dart:514:7)
<asynchronous suspension>
#2      buildGradleProject (package:flutter_tools/src/android/gradle.dart:346:14)
<asynchronous suspension>
#3      buildApk (package:flutter_tools/src/android/apk.dart:34:9)
<asynchronous suspension>
#4      BuildApkCommand.runCommand (package:flutter_tools/src/commands/build_apk.dart:79:11)
<asynchronous suspension>
#5      FlutterCommand.verifyThenRunCommand (package:flutter_tools/src/runner/flutter_command.dart:478:18)
<asynchronous suspension>
#6      FlutterCommand.run.<anonymous closure> (package:flutter_tools/src/runner/flutter_command.dart:383:33)
<asynchronous suspension>
#7      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:153:29)
<asynchronous suspension>
#8      _rootRun (dart:async/zone.dart:1124:13)
#9      _CustomZone.run (dart:async/zone.dart:1021:19)
#10     _runZoned (dart:async/zone.dart:1516:10)
#11     runZoned (dart:async/zone.dart:1463:12)
#12     AppContext.run (package:flutter_tools/src/base/context.dart:152:18)
<asynchronous suspension>
#13     FlutterCommand.run (package:flutter_tools/src/runner/flutter_command.dart:375:20)
#14     CommandRunner.runCommand (package:args/command_runner.dart:197:27)
<asynchronous suspension>
#15     FlutterCommandRunner.runCommand.<anonymous closure>
(package:flutter_tools/src/runner/flutter_command_runner.dart:396:21)
<asynchronous suspension>
#16     AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:153:29)
<asynchronous suspension>
#17     _rootRun (dart:async/zone.dart:1124:13)
#18     _CustomZone.run (dart:async/zone.dart:1021:19)
#19     _runZoned (dart:async/zone.dart:1516:10)
#20     runZoned (dart:async/zone.dart:1463:12)
#21     AppContext.run (package:flutter_tools/src/base/context.dart:152:18)
<asynchronous suspension>
#22     FlutterCommandRunner.runCommand (package:flutter_tools/src/runner/flutter_command_runner.dart:356:19)
<asynchronous suspension>
#23     CommandRunner.run.<anonymous closure> (package:args/command_runner.dart:112:25)
#24     new Future.sync (dart:async/future.dart:224:31)
#25     CommandRunner.run (package:args/command_runner.dart:112:14)
#26     FlutterCommandRunner.run (package:flutter_tools/src/runner/flutter_command_runner.dart:242:18)
#27     run.<anonymous closure>.<anonymous closure> (package:flutter_tools/runner.dart:62:22)
<asynchronous suspension>
#28     _rootRun (dart:async/zone.dart:1124:13)
#29     _CustomZone.run (dart:async/zone.dart:1021:19)
#30     _runZoned (dart:async/zone.dart:1516:10)
#31     runZoned (dart:async/zone.dart:1500:12)
#32     run.<anonymous closure> (package:flutter_tools/runner.dart:60:18)
<asynchronous suspension>
#33     AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:153:29)
<asynchronous suspension>
#34     _rootRun (dart:async/zone.dart:1124:13)
#35     _CustomZone.run (dart:async/zone.dart:1021:19)
#36     _runZoned (dart:async/zone.dart:1516:10)
#37     runZoned (dart:async/zone.dart:1463:12)
#38     AppContext.run (package:flutter_tools/src/base/context.dart:152:18)
<asynchronous suspension>
#39     runInContext (package:flutter_tools/src/context_runner.dart:56:24)
<asynchronous suspension>
#40     run (package:flutter_tools/runner.dart:51:10)
#41     main (package:flutter_tools/executable.dart:62:9)
<asynchronous suspension>
#42     main (file:///C:/D/SDK/flutter/packages/flutter_tools/bin/flutter_tools.dart:8:3)
#43     _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:299:32)
#44     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:172:12)

Hello there @Xgamefactory,
Could you try some commands, so we are sure there is nothing out of place and we can have a better hint of what is happening?
Run in your project folder :
flutter pub get
flutter build apk --debug
There is no problem if the second command still fails.
Enter in the $project/android folder and run
./gradlew assembleDebug
Check if the output of the last command and post here.

Hello @Xgamefactory !
Without additional information, we are unfortunately not sure how to resolve this issue. We are therefore reluctantly going to close this bug for now. Please don't hesitate to comment on the issue, if you have any more information for us; we will reopen it right away!
Thank you for your contribution.
P.S. If anyone still experience the same problem, please file a new issue with the information required in the template.
All system setups can be slightly different so it’s always better to open new issues and reference related issues.


esp32-at with esp-idf v3.2

Hello,
Almost 1 month ago, a new version of esp-idf was released: v3.2
So is there a release plan for the esp32-at?
If yes, when it will be released?
Thanks!

Hi,
Sorry for that. We found a bug which blocks esp32-at release. We cannot release it before the new IDF which fixes this bug is released.

Thanks for the quick response.
And you can give an approximate date about when is estimated to be released (1month ahead, 3 months ahead...)
Thanks!

Sorry, it depends on the time IDF v3.2.1 releases. Maybe about a month.

Hello again,
Now that the release has been launched: https://github.com/espressif/esp32-at/tree/v<IP_ADDRESS>
It can be seen that the version of esp-idf is still v3.2.0, is that ok?

Update version number to ESP-IDF v3.2.0

Furthermore, in the esp32-at repo, the commit for the esp-idf submodule is not a commit from a release, it's just a commit in the branch (commit ddbce78):

So, this commit is the one that may be used for esp32-at project? Or it could be better to use an stable one (tag 3.2, 3.2.1 or 3.2.2).
Thank you

Hello,
anyone knows the answer?
Thank you!

@carlessole
Hi, we have test all the AT function, it works well, because esp-idf always delays, we cannot wait it every time. So it may not be the release tag.

thanks @xcguang
So I understand that the commit that must be used is that one = ddbce78.
Thanks!


When using “Publish all artifacts in repo”, deleted contents in repo are not deleted in APIM.

Release version
APIOps Toolkit for Azure APIM v4.7.0
Describe the bug
When using “Publish all artifacts in repo”, deleted contents in repo are not deleted in APIM.
Let’s say we have 2 apis in repo.
We deployed to APIM using “Publish all artifacts in repo”.
Now, we deleted 1 api in repo and deployed using “Publish all artifacts in repo”.
Now, the deleted API is not removed from APIM.
After going through documentation, looks like deleted artifacts in repo will be removed from apim when using "publish artifacts in last commit".
Can we pass any parameters in publish pipeline to make the delete functionality work for “Publish all artifacts in repo” ?
Expected behavior
Repo contents should match APIM.
Actual behavior
Deleted contents in Repo are not removed from APIM when using "Publish all artifacts in repo"
Reproduction Steps
Delete an api in repo and deploy using "Publish all artifacts in repo".
Check if the deleted api is removed from apim.

@guythetechie whatzare your thoughts on this?

Why don't you simply trigger the publisher against a specific commit ID in that case?

@harshith79 - One of the principles of APIOps is indeed to use git as the single source of truth. However, it does not do reconciliation between the repo and the APIM automatically. It relies on git diff and commits to do the reconciliation. When you do publish all, the publisher won't know what was deleted without explicitly passing the commitid. This is unlike a PUT operation where it does not matter since it will either create new or overwrite existing artifacts. This is expected behavior. @guythetechie , @waelkdouh - I think this delete model works as intended unless. I dont see why we need to introduce complexities which compares the existing APIM snapshot with the artifacts in git, which is an overkill for a much simpler solution using the commit id.


Styled-components doesn't work with Material UI

Styled-components doesn't work with Material UI
I'm trying to style material-ui component using styled components and doesn't work
Example of code:
import Button from 'material-ui/Button';
const StyledButton = styled(Button)`
    background: red;
    border: 1px solid red;
    font-size: 2rem;
    font-weight: bold;
`;

class ButtonTemplate extends Component {
    render() {
        return(


Org and Team commands remember the last used org/team

resolves: #1231
Last used org/team is stored in a 'server'-preferences.yml (localhost / cloud.appcelerator.io)
to test
$ make build-cli
$ amp cluster create --tag=local
$ amp org create --org=test<EMAIL_ADDRESS>-> this will remember the org name now
$ amp org member ls
[user jgrahamjones @ localhost:50101]
organization name: test <- automatically entered
MEMBER         ROLE
jgrahamjones   ORGANIZATION_OWNER
$ amp org remove test
[user jgrahamjones @ localhost:50101]
test <- check the server-preferences.yml to see that the org is blank

There are a number of commands that this can be tested with.

Also, the preferences file should be removed when the user is deleted.

@bquenin @neha-viswanathan What's the status of this PR?

@bquenin @neha-viswanathan ^^

@JosephGJ status?

The PR is finished.

@subfuzion I reviewed it yesterday. It looks good. Just need to address the file deletion on user rm and it's good to go.

@JosephGJ Can you work with @bquenin and @neha-viswanathan to get this merged or is it going to be put on the icebox along with org support?

@subfuzion I can get this finished, I'll just add bertrands requested changes.


Pytorch 1.7 update API changes

Description
Changes related to Pytorch 1.7 update
Fixes (https://github.com/NVIDIA/TRTorch/issues/207)
Type of change

New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Checklist:

[x] My code follows the style guidelines of this project (You can use the linters)
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas and hacks
[ ] I have made corresponding changes to the documentation
[ ] I have added tests to verify my fix or my feature
[ ] New and existing unit tests pass locally with my changes

Bump this to 0.2.0 https://github.com/NVIDIA/TRTorch/blob/3ecf5d09fffce278c1df011878693aa21123ecad/cpp/api/include/trtorch/macros.h#L23

Bump this to 0.2.0a0 https://github.com/NVIDIA/TRTorch/blob/4ee6c2023ec547ca579af53754fa2e1381189a10/py/setup.py#L19

https://github.com/NVIDIA/TRTorch/blob/master/README.md#dependencies Can you update the dependency info here

Can you bump up to the latest bazel version? https://github.com/NVIDIA/TRTorch/blob/master/.bazelversion

https://github.com/NVIDIA/TRTorch/blob/pytorch_update/tests/py/requirements.txt bump to latest torchvision

https://github.com/NVIDIA/TRTorch/blob/pytorch_update/py/requirements.txt bump to latest torch

@peri044 was there anything outstanding on your side or is this ready for a final review and merge?

@narendasan  Updated to pytorch 1.7.1 and torchvision 0.8.2


RuntimeError: 'weight' must be 2-D while training Flan-T5 models with stage 3

I am using Huggingface Seq2SeqTrainer for training Flan-T5-xl model with deepspeed stage 3.
trainer = Seq2SeqTrainer(
                #model_init = self.model_init,
                model=self.model,
                args=training_args,
                train_dataset=train_ds,
                eval_dataset=val_ds,
                tokenizer = self.tokenizer,
                data_collator=self.data_collator,
                compute_metrics=self.compute_metrics,
            )

trainer.train()

I am stuck on below error:
  File "/users/snannawa/.conda/envs/sn_torch/lib/python3.10/site-packages/transformers/trainer.py", line 1527, in train
    return inner_training_loop(
  File "/users/snannawa/.conda/envs/sn_torch/lib/python3.10/site-packages/transformers/trainer.py", line 1773, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/users/snannawa/.conda/envs/sn_torch/lib/python3.10/site-packages/transformers/trainer.py", line 2523, in training_step
    loss = self.compute_loss(model, inputs)
  File "/users/snannawa/.conda/envs/sn_torch/lib/python3.10/site-packages/transformers/trainer.py", line 2555, in compute_loss
    outputs = model(**inputs)
  File "/users/snannawa/.conda/envs/sn_torch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1488, in _call_impl
    return forward_call(*args, **kwargs)
  File "/users/snannawa/.conda/envs/sn_torch/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1158, in forward
    output = self._run_ddp_forward(*inputs, **kwargs)
  File "/users/snannawa/.conda/envs/sn_torch/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1111, in _run_ddp_forward
    return module_to_run(*inputs[0], **kwargs[0])  # type: ignore[index]
  File "/users/snannawa/.conda/envs/sn_torch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1488, in _call_impl
    return forward_call(*args, **kwargs)
  File "/users/snannawa/.conda/envs/sn_torch/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1611, in forward
    encoder_outputs = self.encoder(
  File "/users/snannawa/.conda/envs/sn_torch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1488, in _call_impl
    return forward_call(*args, **kwargs)
  File "/users/snannawa/.conda/envs/sn_torch/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 941, in forward
    inputs_embeds = self.embed_tokens(input_ids)
  File "/users/snannawa/.conda/envs/sn_torch/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1488, in _call_impl
    return forward_call(*args, **kwargs)
  File "/users/snannawa/.conda/envs/sn_torch/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 162, in forward
    return F.embedding(
  File "/users/snannawa/.conda/envs/sn_torch/lib/python3.10/site-packages/torch/nn/functional.py", line 2210, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: 'weight' must be 2-D

The code works with Zero2 config but not working with Zero 3. I have tried a couple of settings but no luck.
{
    "fp16": {
        "enabled": true,
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "initial_scale_power": 16,
        "hysteresis": 2,
        "min_loss_scale": 1
    },

    "zero_optimization": {
        "stage": 3,
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": true
        },
        "offload_param": {
            "device": "cpu",
            "pin_memory": true
        },
        "overlap_comm": true,
        "contiguous_gradients": true,
        "sub_group_size": 1e9,
        "reduce_bucket_size": "auto",
        "stage3_prefetch_bucket_size": "auto",
        "stage3_param_persistence_threshold": "auto",
        "stage3_max_live_parameters": 1e9,
        "stage3_max_reuse_distance": 1e9,
        "stage3_gather_fp16_weights_on_model_save": true
    },

    "gradient_accumulation_steps": 8,
    "gradient_clipping": "auto",
    "steps_per_print": 10,
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto",
    "wall_clock_breakdown": false
}

Any help would be appreciated.

Looks like same error popped up in diffusers using zero stage3 :) https://github.com/huggingface/diffusers/issues/1865

Don't know if this helps, but I get the same 2-D error with stage3 in a weird way: I use the datasets map function with the method of a class that contains a SentenceTranformer model. Basically, I want to augment my dataset before training, and when used with deepspeed it gives the 2-D error in the sentence transformer that has nothing to do with the model I'm actually training. Stage 2 seems to work okay.
I'm just beginning with deepspeed and probably don't understand how to use if fully, but maybe it helps with this issue.

Hello @smitanannaware, thank you for reporting.
According to the documentation of HuggingFace, you need to pass your deepspeed config file to TrainingArgument. Can you try the setting?
training_args = Seq2SeqTrainingArguments(
...
    deepspeed="ds_config.json"
)

I tried to train Flan-T5 using the code on this article.
The training diverged with FP16 as suggested in the article, but I didn't see the error with stage 3.

+1, getting same error

@djaym7 Thank you for your report!
Can you give us more details? Did you pass deepspeed argument to Seq2SeqTrainingArguments as shown in my comment?
Is it possible to share the entire code?

config is loaded from https://github.com/philschmid/deep-learning-pytorch-huggingface/blob/main/training/configs/ds_flan_t5_z3_config.json
training_args = TrainingArguments(
output_dir=f"./results/{question_name}_{output_dir_suffix}",
learning_rate=lr,
per_device_train_batch_size=batch_size,
per_device_eval_batch_size=batch_size,
# auto_find_batch_size=True,
num_train_epochs=epochs,
weight_decay=0.02,
warmup_steps=warmup_steps, #1epoch  = 1530/16-- 95 steps
lr_scheduler_type= 'linear',
optim='adamw_torch',
evaluation_strategy='epoch',
# save_strategy='epoch',save_steps=eval_steps,
logging_steps=eval_steps,
eval_steps=eval_steps,
gradient_checkpointing=gradient_checkpointing,
# do_eval=False,
save_total_limit=2,
# load_best_model_at_end=True,
fp16=fp16,
# metric_for_best_model='f1',
gradient_accumulation_steps = gradient_accumulation_steps,
dataloader_num_workers = dataloader_num_workers,
sharded_ddp=sharded_ddp,
)
if deepspeed:
training_args.deepspeed = deepspeed_dict
    from transformers.deepspeed import HfTrainerDeepSpeedConfig
    training_args.hf_deepspeed_config = HfTrainerDeepSpeedConfig(deepspeed_dict)

@djaym7 @woodyx218
Can you try the complete example at philschmid's blog.
He showed the complete code to train Flan-T5 using DeepSpeed. The code successfully worked in my environment.

Error is when using PEFT with Flan..

Error is when using PEFT with Flan..

I have the same problem, how did you fix it?

havent, using regular inference without deepspeed

Hi @djaym7,
I apologize for the delayed response.
I have tried to reproduce the problem using both deepspeed and PEFT (prefix tuning) but haven't seen the same error.
My code is available at https://github.com/tohtana/ds_repro_2746
You can set up the dataset using prepare_dataset.py and then run run_t5_ds_peft.sh.
I came across the error that you mentioned at https://github.com/huggingface/peft/issues/168.
However, I found that the error happened regardless of whether I used deepspeed or not.
I could resolve it by setting False to both args.gradient_checkpointing and use_cache as you mentioned in the thread of the issue.
I didn't see an error after making these changes.
Can you let me know if I missed something?
The versions of peft, transformers, deepspeed were:

peft                          0.3.0.dev0
deepspeed                     0.8.3
transformers                  4.28.0.dev0

See https://github.com/huggingface/diffusers/pull/3076
Please carefully read the OP of the PR for details.

Hi @djaym7, I apologize for the delayed response.
I have tried to reproduce the problem using both deepspeed and PEFT (prefix tuning) but haven't seen the same error. My code is available at https://github.com/tohtana/ds_repro_2746 You can set up the dataset using prepare_dataset.py and then run run_t5_ds_peft.sh.
I came across the error that you mentioned at huggingface/peft#168. However, I found that the error happened regardless of whether I used deepspeed or not. I could resolve it by setting False to both args.gradient_checkpointing and use_cache as you mentioned in the thread of the issue.
I didn't see an error after making these changes. Can you let me know if I missed something?
The versions of peft, transformers, deepspeed were:

peft                          0.3.0.dev0
deepspeed                     0.8.3
transformers                  4.28.0.dev0

There's no error in training, error is in inference ..  add following after training and there'll be error
for batch in tqdm(data_loader):
# need to push the data to device
with torch.no_grad():
outs = model.generate(input_ids=batch['input_ids'].to(device),
attention_mask=batch['attention_mask'].to(device),
max_new_tokens=128)  # num_beams=8, early_stopping=True)

Hi @djaym7,
I added the folllowing code after trainer.train() in this example but didn't see the error.
Is it possible to share your code?
    device = torch.device("cuda")
    loader = torch.utils.data.DataLoader(eval_dataset, batch_size=args.per_device_eval_batch_size, shuffle=False, collate_fn=data_collator)
    for batch in loader:
        with torch.no_grad():
            outputs = model.generate(input_ids=batch['input_ids'].to(device),
                attention_mask=batch['attention_mask'].to(device), max_new_tokens=128) # num_beams=8, early_stopping=True)
            print(tokenizer.batch_decode(outputs, skip_special_tokens=True))

Try, error is in LoraConfig and gradient_checkpointng on. If that doesn’t reproduce, i'll cleanup and share the file.

Actually, it comes from using bertScore

def evaluate(data_loader, model,tokenizer,print_samples=False,metric='bertscore_complex',device=None,**kwargs):
    """
    Compute scores given the predictions and gold labels
    """

    if device is not None:
        model = model.to(device)

    inputs,outputs, targets = [], [], []

    inputs_dat,outputs_dat = [], []

    for batch in tqdm(data_loader):
        # need to push the data to device
        if device is not None:
            batch['input_ids']=batch['input_ids'].to(model.device)
            batch['attention_mask']=batch['attention_mask'].to(model.device)

        with torch.no_grad():
            outs = model.generate(input_ids=batch['input_ids'], #
                                            attention_mask=batch['attention_mask'],
                                            max_new_tokens=128,**kwargs)  # num_beams=8, early_stopping=True)

        dec = [tokenizer.decode(ids, skip_special_tokens=True) for ids in outs]
        labels = batch['labels']
        labels[labels==-100] = tokenizer.pad_token_id
        target = [tokenizer.decode(ids, skip_special_tokens=True) for ids in batch["labels"]]
        inp = [tokenizer.decode(ids, skip_special_tokens=True) for ids in batch["input_ids"]]

        inputs.extend(inp)
        outputs.extend(dec)
        targets.extend(target)

    if print_samples:
        print("\nPrint some results to check the sanity of generation method:", '\n', '-'*30)
        for i in [1, 5, 25, 42, 50, 4, 10, 35]:
            try:
                print(f'>>Input     : {inputs[i]}')
                print(f'>>Target    : {targets[i]}')
                print(f'>>Generation: {outputs[i]}\n\n')
            except UnicodeEncodeError:
                print('Unable to print due to the coding error')

        if 'input_ids_dat' in batch:
            print('\n\n On TARGET DOMAIN')

            for i in [1, 5, 25, 42, 50, 4, 10, 35]:
                try:
                    print(f'>>Input     : {inputs_dat[i]}')
                    print(f'>>Generation: {outputs_dat[i]}\n\n')
                except UnicodeEncodeError:
                    print('Unable to print due to the coding error')
            print()

    scores, all_labels, all_preds = compute_scores(outputs, targets,metric=metric)
    # results = {'scores': scores, 'labels': all_labels, 'preds': all_preds}

    scores['refs'] = all_labels
    scores['preds'] = all_preds

    scores['exact_match_metrics'] = compute_f1_scores(outputs,targets)
    return scores#, all_labels, all_preds

def compute_f1_scores(pred_pt, gold_pt):
    """
    Function to compute F1 scores with pred and gold quads
    The input needs to be already processed
    """
    # number of true postive, gold standard, predictions
    accuracies = []
    for p,r in zip(pred_pt,gold_pt):
        if p==r:
            accuracies.append(1)
        else:
            accuracies.append(0)

    return {'accuracy':np.mean(accuracies)}

def compute_scores(pred_seqs, gold_seqs, metric='bertscore_complex'):
    """
    Compute model performance
    """
    scores={}
    assert len(pred_seqs) == len(gold_seqs)
    if 'bertscore' in metric and 'complex' in metric:
        bert_score = load_metric('bertscore')
        scores.update(bert_score.compute(predictions=pred_seqs, references=gold_seqs,model_type='bert-base-uncased' ))
        for sim in [0.5,0.6,0.7,0.8,0.9]:
            scores['accuracy_'+ str(sim)] = [1 if i>=sim else 0 for i in scores['f1'] ]
            scores['accuracy_'+ str(sim)+'_mean'] = np.mean(scores['accuracy_'+ str(sim)])

        scores['class_metrics'] = class_wise_metrics(scores,pred_seqs,gold_seqs)
        scores['class_length'] = Counter(gold_seqs)
        new_scores = {}
    if 'bertscore' in metric and 'simple' in metric:
        bert_score = load_metric('bertscore')
        scores.update(bert_score.compute(predictions=pred_seqs, references=gold_seqs,model_type='bert-base-uncased' ))
        for sim in [0.5,0.6,0.7,0.8,0.9]:
            scores['accuracy_'+ str(sim)] = [1 if i>=sim else 0 for i in scores['f1'] ]
            scores['accuracy_'+ str(sim)+'_mean'] = np.mean(scores['accuracy_'+ str(sim)])
    if 'rouge' in metric:
        bert_score = load_metric('rouge')
        scores.update(bert_score.compute(predictions=pred_seqs, references=gold_seqs ))

    return scores, gold_seqs, pred_seqs

@djaym7 Can we clarify what errors you have now? I see several different errors regarding this issue.

The error posted by the author of this issue was about training. Have you encountered the error?
The error you mentioned at https://github.com/huggingface/peft/issues/168 is about both training and inference. Do you still have the errors?
Your latest error is in computing metrics. Do you have no issue for training and inference(generation) now?

It would be helpful if you could give us the entire reproducing code.

The error you mentioned at https://github.com/huggingface/peft/issues/168 is about both training and inference. Do you still have the errors? YES
Your latest error is in computing metrics. Do you have no issue for training and inference(generation) now? YES
To reproduce, add the evaluate function shared above after training the model. Error is posted above as well.

@djaym7

The error you mentioned at https://github.com/huggingface/peft/issues/168 is about both training and inference. Do you still have the errors? YES

I am a bit confused. You wrote "There's no error in training, error is in inference" at https://github.com/microsoft/DeepSpeed/issues/2746#issuecomment-1505844709. Do you have an error with training or not?
I wrote an example of training/generation using DS and PEFT. I didn't fully test it but at least it didn't throw the error. What is the difference with your code?

To reproduce, add the evaluate function shared above after training the model. Error is posted above as well.

I think we need to make sure that we are doing the same for training/generation before further investigation.

Closing because we have no additional information.
Please feel free to reopen if the problem still exists.

Faced the same issue when run inference using T5ForConditionalGeneration.from_pretrained() to load a pre-trained model.
Solution: use trainer.save_model() instead of model.save_pretrained() to save the pre-trained model.

Facing same issue when using torch.jit.trace(model, example_inputs=dummy, strict=False) to save cerebras/Cerebras-GPT-111M pretrained model from huggingface. I dont see this error when not using deepspeed.

I had the same issue. Thankfully, it went away when I upgraded to Deepspeed 0.9.5.

Facing the same issue with DeepSpeed 0.13.4.
Training with PEFT: QLora + DeepSpeed Zero Stage 3, offload param and optimizer to CPU.
Model: LLaMA2
Training is fine.
After training, we merge_and_unload to model and perform inference, once we do inference we got this error:
  File "/root/code_sft/sft_main.py", line 461, in main
    test_pfm = evaluate(args, test_dataloader, model, mix_precision=mix_precision, tokenizer=tokenizer,
  File "/root/code_sft/sft_main.py", line 223, in evaluate
    generated_ids = module.generate(input_ids=feature["input_ids"],
  File "/miniconda/envs/py310/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/miniconda/envs/py310/lib/python3.10/site-packages/transformers/generation/utils.py", line 1474, in generate
  File "/miniconda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/miniconda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/miniconda/envs/py310/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1183, in forward
    outputs = self.model(
  File "/miniconda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/miniconda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/miniconda/envs/py310/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1027, in forward
    inputs_embeds = self.embed_tokens(input_ids)
  File "/miniconda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/miniconda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/miniconda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 162, in forward
    return F.embedding(
  File "/miniconda/envs/py310/lib/python3.10/site-packages/torch/nn/functional.py", line 2233, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: 'weight' must be 2-D


Eigenvalue computation changes in recent version

Dear Cellrank team,
Thank you for your tremendous help on my previous question https://github.com/theislab/cellrank/issues/297 and I'm sorry to trouble you again.
I was previously working with v1.0.0rc0 or rc1 I believe where I ran the basic workflow successfully, then ran most of the advanced workflow successfully, up until the previous question. From the basic workflow, I saw that my sample had four terminal states. When I ran the advanced workflow with backward = False and computed the eigenvalues, I plotted them via g.plot_spectrum(real_only = True) and the eigengap in the picture separated out four eigenvalues. If I'm not mistaken, the four eigenvalues should correspond to the four terminal states from the basic workflow and is the correct setting for the parameters use = 4 in plot_schur() and n_states = 4 in compute_metastable_states(). Please let me know if I have conceptually misunderstood this. I inferred this from the fact that the workflow showed 3 eigenvalues past the eigengap and the rest of the parameters were set to 3. Also, the basic workflow recovered 3 terminal states (Alpha, Beta, Epsilon).

However, now that I have upgraded to the newest version, v1.0.0rc6, I noticed that the eigengap only separates out a single eigenvalue. I am wondering what was changed between the versions as far as computing the eigenvalues that might result in such a significant decrease? Moreover, I am concerned because this sample I am analyzing does not have any knockdown or other biological significance that would indicate biologically only a single final state.

Finally, before I realized the number of eigenvalues decreased, I had still set use = 4 and n_states = 4 and it seemed the code would still run, produce figures, and states. I'm guessing if we ask for more eigenvalues beyond the eigengap, the program will still generate the statistics for them, but they may not be as "significant"?
Thank you again.

Hi @timlai4, between rc1 and rc6, we changed the way in which we compute the transition matrix. Before, we used scvelo's velocity graph whereas now, we compute this ourselves in the VelocityKernel. Try setting softmax_scale=None in kernel.compute_transition_matrix in the advanced workflow, and let me know whether this recovers the old spectrum (4 eigenvalues before the eigengap).
@michalk8 , we should make softmax_scale=None the default value - changing this can have quite significant consequences.

Re your other question, the eigengap is an indication of what may be a "good" number of final states. However, you may choose any number you like. As you said correctly, choosing more states may just mean that some of them are not very significant of meaningful. More precisely, these will be less stable states, so it may include intermediate, metastable rather than final states.

Hi @timlai4 , any news on this?

Thank you for the clarification. I added this to my transition matrix computation and the new eigenvalue picture looks like:

I have two eigenvalues before the eigengap now, but it's still not quite the same as the original picture. The eigengap is before the fourth eigenvalue and it seems a fifth eigenvalue is close to the magnitudes of the third and fourth as well. I noticed some of the previously real eigenvalues are colored as complex, but when plotting in the complex plane, I've noticed all the eigenvalues are basically on the real line, so I doubt this is significant beyond indicating that the computation has at least numerically changed.

thanks for trying this. Could you please post the code you use to compute the transition matrix here?

one more question, how many cells are we looking at?

scv.tl.velocity_graph(adata, mode_neighbors='connectivities', compute_uncertainties=True)
vk = VelocityKernel(adata)
vk.compute_transition_matrix(scale_by_variance = True, softmax_scale=None)
g = GPCCA(vk)
g.compute_schur(n_components=20)
g.plot_spectrum(real_only = True)

one more question, how many cells are we looking at?

>>> print(adata)
AnnData object with n_obs × n_vars = 3488 × 2999

Looks like 3,488 cells.

Hi @timlai4, let's try the following:

please update to the most recent version of cellrank
we simplified our interface a bit, among other changes, it does not matter any longer how you compute the velocity graph in scvelo becasue we internally re-compute it in cellrank (so you don't need to compute it at all in scvelo). Also, softmax scale is the default now. So simply run:

vk = VelocityKernel(adata)
vk.compute_transition_matrix()
g = GPCCA(vk)
g.compute_schur(n_components=20)
g.plot_spectrum(real_only = True)

please, set the verbosity level to 4 (cr.settings.verbosity = 4) and post all output that the above commands give you here.

@timlai4 , I got it - I understand why you see a different spectrum now. Basically, we changed the way in which we propagate uncertainty into the transition probabilities. The old way was to do it via scale_by_variance = True. We got rid of this in favour for a more rigorous approach that propagates uncertainty using first principles, either through Monte Carlo simulations or though an analytical second order Taylor approximation. This is now controlled via the mode parameter, which can be monte_carlo for the former and stochastic for the latter.
So in short: we made a backwards breaking change to the way in which we treat uncertainty in the velocity vectors. Sorry about that, but these things happen in early development. I can only reassure you that the new way to uncertainty is more rigorous than the old and the results are more trustworthy. Feel free to choose a greater number of sates than suggested by the eigengap - 4 or 5 looks reasonable for your data.
@michalk8, this should clear things up.


No AMDRMCLI.exe

Hi, i just instaling the ASRock Motherboard Utility but there is no exe called "AMDRMCLI.exe"
i already tried other version but no luck, thank you

Hi, can you please tell me with which components are you trying to follow this guide? Only CPU and Mobo needed

"After installing it, you have to create a .bat file with the command to disable PROCHOT. In order to do this, you just need to paste this in a text file: "PATH\OF\INSTALLATION\ASRock Utility\A-Tuning\Bin\AMDRMCLI.exe" -a DisablePROCHOT. And save it with whatever name you want but make sure it has .bat as its extension. Place it in a comfy location as it will have to stay there and never be touched again."
No AMDRMCLI.exe so i can't execute the bat

I meant: what CPU are you using? On which Motherboard is it installed?

ohh sorry
R5 2600
AsRock b450m steel legend


Colour accessibility

Tasks

[x] understand the usage intent for the colours found on the activity page
[x] de dupe / rationalise number of colours if possible (not currently possible)
[x] refactor sass variables based on understanding of usage:

[x] $colour-activity-page-text becomes $colour-text-primary
[x] $colour-activity-feed-teal-one becomes $colour-brand
[x] Create new variable $colour-text-secondary: $colour-brand
[x] Text uses of old $colour-activity-feed-teal-one should use $colour-text-secondary
[x] Non-text uses of old $colour-activity-feed-teal-one should use $colour-brand
[x] $colour-activity-feed-toggle becomes $colour-accent
[x] $colour-activity-page-background becomes $colour-page-background
[x] $colour-activity-feed-responses-background becomes $colour-component-background
[x] $colour-article-border becomes $colour-furniture
[x] $colour-overlay becomes $colour-avatar-overlay
[x] Remove $colour-activity-feed-teal-two, and remove the borders of the responses buttons (the only place that it's used).

[x] First pass: check accessibility of relevant background / foreground colour combinations using https://contrastchecker.com/ (See results attached https://github.com/sciety/sciety/issues/531#issuecomment-790556171)

[x] against a background of $colour-page-background

[x] check $colour-text-primary
[x] check $colour-text-secondary
[x] check $colour-accent

[x] against a background of $colour-component-background

[x] check $colour-text-primary
[x] check $colour-text-secondary
[x] check $colour-accent

[x] against a background of $colour-brand

[x] check white

[x] against a white background

[x] check $colour-text-primary
[x] check $colour-text-secondary
[x] check $colour-accent

[x] against a background of $colour-accent:

[x] check white

[x] make adjustments to improve colour contrasts where necessary.

[x] darken $colour-text-secondary to rgb(75, 96, 106) to pass at small font sizes
[x] darken $colour-text-primary to rgb(52, 67, 74) to complement secondary text colour change
[x] darken $colour-accent to rgb(206, 72, 26) to enable pass against white. Note that this does not pass against $colour-page-background nor $colour-component-background so do not it in these combinations

[ ] Second pass: rerun checks as a batch with new colours & attach generated PDF to this ticket.

my-wcag-color-samples.pdf
Accessibility results (in reverse order)

Accessibility results for primary, secondary and accent against FFFFFF, page-background and component-background (results go from bottom to top).
https://contrastchecker.com/?swatch_session=1614944872937
my-wcag-color-samples.pdf


Error Connection Refused

Hello My Name is Galuh Boy H.
Im From Indonesia
I read your paper and try to replicate your code, for purposing of my homework in collage,
but i got stuck when i try to connect from instance party to instance server.
it says connection refused. (screenshoots attach)
I alredy do every steps, from video and paper, include install all requirement software form requirement.txt
what should i do for this error?
regards,
Galuh Boy H.
University of Indonesia
Depok, West Java.

Dear Galuh,
I have checked the code, and it worked properly. However, I have a few suggestions that may help:

Regarding the error "No such file or directory," I recommend cloning the entire project from GitHub using the command git clone https://github.com/shokri-matin/Fed-eGlass.git instead of manually copying files and folders. This error suggests that some files were not transferred correctly.

Create a virtual environment and ensure all the requirements and dependencies are installed by using the command pip install -r requirements.txt.

Ensure that the server is running before starting the clients, or double-check that the server is stable. The clients need to connect to the server for everything to work smoothly.

Hai,
i alredy try your @shokri-matin suggestions, i make new instance in AWS Lightsail, then i copy entire project using git, and i do pip install the requirement.txt. but in process i have trouble install the Tensorflow, then i try intall Tensorflow with separate command line ("pip install tensorflow==2.9.1 --no-cache-dir")
i make some assumption maybe Tensorflow is somehow broken or something? because if i use command ("pip install tensorflow==2.9.1") the instalation just says "killed" in the end of line status, and can't continue
and in the end, i still got connection refuse. I alredy check "manage" server using static IP, and add rule UDP and TCP and set 12345
please help
regards,
galuh boy h

hai .
i alredy done the code, thanks for help


[List Suggestion] Thread Crowd malware list

It's a small,community driven malware list.
PROS:
Frequently  updated(info is a bit outdated,would recommend monitoring first)
As of today,intersection: 94 domains only
CONS:
Might be a pain to report false-positives,if any.
No License present

https://www.threatcrowd.org/feeds/domains.txt
Steve @StevenBlack your call 👍

info ->
we removed it from nextdns as it is no longer up to date

we removed it from nextdns as it is no longer up to date

You are NOT a member or associated in any way to NextDNS so please...spare us all that....
No one here is even close to be impressed by you so far...if ever ❗

Nobody ever said they were part of nextdns; I'm saying that I know this list since I had it included in the threat intelligence section.
Source ->
https://github.com/nextdns/metadata/issues/136
PS ->
I don't care about your personal aspects

Hey that’s correct that list is no longer updated.
If you’re looking at blocking malware domains, I’d suggest you take a look at
https://github.com/Neo23x0/signature-base/tree/master/threatintel

@threatcrowd thanks for the prompt response 👍
Steve @StevenBlack i'm closing this. Reopen if needed.


When trying to close a non-selected template from the resource nav tabs, it actually closes the selected one

Also, there is no way to close a selected template from the resource nav tabs.

Not being able to close the selected tab from nav is intentional; if @astridu disagrees, we can change the design.

That is expected behavior.


Missing application_properties in BatchMessage fixes

https://github.com/Azure/azure-event-hubs-python/issues/97

Hi @yunhaoling - I updated pylint as it was throwing errors due a version conflict, however this has resulted in some new formatting errors. Could you also fix these? There's not too many:
https://travis-ci.org/Azure/azure-uamqp-python/jobs/511066675

@annatisch Sure, I'll investigate into that.

Hi @yunhaoling - I would revert your last two commits and simply remove Python 3.4 from the travis config:
https://github.com/Azure/azure-uamqp-python/blob/master/.travis.yml#L14
We don't plan to support Python 3.4 from the next release anyway :)

Hey @annatisch
I've done the rollback and removed python 3.4 from travis config.


TypeError: Cannot read property 'schema' of undefined

I am trying to pass a document in to autoForm via a template helper. The helper is defined as such:
'document': function () {
    // Get Home ID from template data context
    // convert the 'String' object to a string for the DB query
    var documentId = this.toString();

    // Get home document
    var document = Documents.findOne(documentId);
    console.log(document);
    return document;
}

The document is logged to the console, but autoForm fails to render with the following error:
TypeError: Cannot read property 'schema' of undefined

I have also tried passing the document as a template data context and constructing the AutoForm as such:
{{#autoForm id="editDocument" type=update collection=Documents doc=this  }}
...

When using this in the template, it behaves as expected:
{{ this.title }} <!-- displays the document title in the template -->

Related issues
I have found the following issues, which may be related:

https://github.com/aldeed/meteor-autoform/issues/790
https://github.com/aldeed/meteor-autoform/issues/753
https://github.com/aldeed/meteor-autoform/issues/350

I am running AutoForm 5.5.0. I have also tried passing the document as a template data context and constructing the AutoForm as such:
{{#autoForm id="editDocument" type=update collection=Documents doc=this  }}
...

When using this in the template, it behaves as expected:
{{ this.title }} <!-- displays the document title in the template -->

Here is the output of meteor list
...
aldeed:autoform                5.5.0
aldeed:autoform-select2        1.0.5*
aldeed:collection2             2.5.0
aldeed:simple-schema           1.3.3
...

It has nothing to do with your document. It's complaining about collection=Documents. Probably you do not have a helper named "Documents", so you meant to do collection="Documents"

I was getting the same error when my schema was formatted as such :
var Schemas = {};
Schemas.Room = new SimpleSchema({
    title: {
        type: String,
        label: "Title",
        max: 200
    }
});

When I changed my schema to :
Rooms.attachSchema(new SimpleSchema({
  title: {
    type: String,
    label: "Title",
    max: 200
  }
}));

It worked...


Forum methods

Add fourm.py forum_group.py forum_category.py forum_thread.py forum_post.py to the library.
Enable users to perform actions like get/create/edit/destroy categories/threads/posts in sites' forums.

@azureact
Thank you for your contribution!
This set of features is necessary, but it has been put off for a while.
I'm traveling right now, so I'll check it out tomorrow or the day after.


Is CosmosDB supported?

Microsoft CosmosDB has support for a Cassandra API. Adding it to CI might be difficult, however it might be interesting testing it in order to understand if there is any blocker issue.

Hey @bettio, no, it's not officially supported. It's the first time I hear about it. I dont think we should support it unless we can test it reliably on CI, so that would be the first step. As you said, some testing to see if anything breaks would be interesting too.
Closing the issue as the question is answered. Thanks!


Adding Canvas support

Obsidian Canvas Support
Since the newest updates of Obsidian it added a new node type named Canvas.
Its awesome to create moodboard, collection of images or mindmaps.
Currently Perlite does not support this filetype and does not show it in the vault.
Obsidian Canvas

Yes, I would like to have that too, but looking at the canvas source code, I have no idea how to achieve that at the moment.
If anyone has a good js library or an idea how to render this, please let me know.
Example
{
	"nodes":[
		{"id":"fbf016a2366d4d5b","x":-1241,"y":-533,"width":250,"height":60,"type":"text","text":"Start of Enum"},
		{"id":"49f6f3de90eab0ac","x":-774,"y":-593,"width":250,"height":60,"type":"text","text":"test 2"},
		{"id":"a47a5494d9e717cb","x":-561,"y":-343,"width":250,"height":60,"color":"3","type":"text","text":"what can i doe here?"},
		{"id":"a0076932c1b056bc","x":-1316,"y":-398,"width":400,"height":400,"type":"file","file":"OFFSEC Notes/Network Services/OCSP.md"}
	],
	"edges":[
		{"id":"f6d63d5748f282a3","fromNode":"fbf016a2366d4d5b","fromSide":"top","toNode":"49f6f3de90eab0ac","toSide":"left"},
		{"id":"89d6375ccd9600e0","fromNode":"49f6f3de90eab0ac","fromSide":"bottom","toNode":"a47a5494d9e717cb","toSide":"top"}
	]
}

little update to this, I found two workarounds to get at least a html / pdf out of the canvas.
Best option so far is to use the obsidian-webpage-export plugin and manually export the desired canvas files (self-contained-documents), this html file can then be placed into the webserver and is directly accessible. Its not nice because its not included in perlite and also needs a manuell export but at least its a quick way to get your canvas into the browser.
Iam thinking about to write some script to extract the canvas part only from the exported file and store it into some custom file format, which then will be recognized and displayed by Perlite. But not sure if this is really the only and best option. Another idea would be to create a fork of the plugin and adjust it to export canvas files only and perlite "ready".
Also found another option by using this script https://forum.obsidian.md/t/exporting-canvas-to-html-and-pdf/60331 to export the canvas as a html and then create a pdf out of it and include it into perlite.


[AutoDiff] Minor activity analysis changes.

Hoist activity marking visited value set out of loop over bbs.
This is safe because bbs directly start with dominator bbs's active values.
Visit bb arguments for activity marking. This was accidentally deleted in
https://github.com/apple/swift/pull/28225. Re-adding the logic doesn't seem to affect any tests.

@swift-ci Please test tensorflow


Add seal shard api

Similar to create shard.
Add test to create shard, seal and put blob and restart and verify shard is still sealed and put blob fails.

Codecov Report
Attention: 104 lines in your changes are missing coverage. Please review.

Comparison is base (7b11031) 83.93% compared to head (358918d) 82.24%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #81      +/-   ##
==========================================
- Coverage   83.93%   82.24%   -1.69%
==========================================
  Files          28       31       +3
  Lines         809     1267     +458
  Branches       58      105      +47
==========================================
+ Hits          679     1042     +363
- Misses         98      169      +71
- Partials       32       56      +24

Files
Coverage Δ

src/include/homeobject/blob_manager.hpp
100.00% <100.00%> (+50.00%)
:arrow_up:

src/include/homeobject/common.hpp
100.00% <ø> (ø)

src/lib/blob_route.hpp
54.54% <100.00%> (+25.97%)
:arrow_up:

src/lib/homeobject_impl.hpp
100.00% <ø> (ø)

src/lib/homestore_backend/tests/bits_generator.hpp
100.00% <100.00%> (ø)

src/lib/memory_backend/mem_blob_manager.cpp
69.23% <100.00%> (ø)

src/lib/memory_backend/mem_homeobject.hpp
100.00% <ø> (ø)

src/lib/memory_backend/mem_shard_manager.cpp
81.81% <100.00%> (ø)

src/lib/shard_manager.cpp
87.17% <100.00%> (ø)

src/lib/blob_manager.cpp
81.81% <83.33%> (ø)

... and 9 more

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

lgtm，it will be more clear once put/get blob pr is merged and rebase on that.

lgtm，it will be more clear once put/get blob pr is merged and rebase on that.

Rebased to latest. Please review.


DMD bug: Array comparison mismatch for immutable[] vs []

See https://github.com/bpfkorea/agora/blob/9d2c958fc34f8cd4fcd3d4ef95e28b62aa8f4f36/source/agora/common/Deserializer.d#L114-L131

Now that we have a deserializer that is capable to deserialize to const/immutable it is less of an issue, but should still be reported upstream.

@AndrejMitrovic : Do we have a tracker ID for this ?

I'm not sure I ever filed this one. But it's got to be a known bug somewhere..

This needs to get merged into #561 once we get a tracker ID.

Got it: https://issues.dlang.org/show_bug.cgi?id=21012
Moving to #561


Upgrade meshkit

Description
This PR fixes breaking CustomResourceDefinition component.
Notes for Reviewers
Signed commits

[ ] Yes, I signed my commits.

@leecalcote  We would need to upgrade the same in meshery-extensions.


ModuleNotFoundError: No module named 'riffusion.audio'

When running Google Colab https://colab.research.google.com/drive/1FhH3HlN8Ps_Pr9OR6Qcfbfz7utDvICl0?usp=sharing
In line:
from riffusion.audio import wav_bytes_from_spectrogram_image, spectrogram_from_waveform
I get:
ModuleNotFoundError: No module named 'riffusion.audio'

This is because things were refactored in the new release. Either the colab needs to be updated or the version checked out needs to be tag v0.2.0
https://github.com/riffusion/riffusion/releases/tag/v0.3.0

This is because things were refactored in the new release. Either the colab needs to be updated or the version checked out needs to be tag v0.2.0
https://github.com/riffusion/riffusion/releases/tag/v0.3.0

this didn't fix for me. I used the latest version and the audio subpackage is still missing.

I'm also looking for that missing from riffusion.audio import spectrogram_from_waveform to be able to load a audio file to run img2img.  I've loaded the  -b v0.3.0 version and it's not in there, and searched through the github code and branches for the function with no luck.  Any advice?  Looking forward to the next version too, thanks.


Refactor to avoid fragmentation

Primary goal: avoid table fragmentation from having 80 threads inserting at once.
Secondary goal: improve performance when running in single thread mode.
Can now insert consistently at 8.4k rows/second with one thread. Increasing threads now results in insane performance (4 threads can reach 16k rows/second), but the higher you go, the more table fragmentation you incur. Might still be useful for CI etc. though. Might need tuning if we want to push threads too high as it currently consumed a lot of mysql connections when populating beatmap attribute caches.
Have not tested watch mode yet. Just PRing for initial visibility, as always, already testing on production.

Do you intend to address the remaining review comments?


Error while ES6 importing 'prettier-plugin-rust' in JavaScript file.

Bug Report
I am using prettier and its plugins (prettier-plugin-java, @prettier/plugin-php) in Node.js to format each file based on its inferred parser. Java & PHP are working fine with ES6 imports but I am getting the error while importing prettier-plugin-rust. It is working fine in Common JS modules though.
💻 Input
// index.js
import prettier from "prettier";
import javaPlugin from "prettier-plugin-java";
import phpPligin from "@prettier/plugin-php";
import rustPugin from "prettier-plugin-rust";

🔴 Output
node:internal/process/esm_loader:94
    internalBinding('errors').triggerUncaughtException(
                              ^

Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/mnt/Data/code/formatter_test/node_modules/prettier/doc' imported from /mnt/Data/code/formatter_test/node_modules/prettier-plugin-rust/index.js
Did you mean to import prettier/doc.js?

    at new NodeError (node:internal/errors:372:5)
    at finalizeResolution (node:internal/modules/esm/resolve:437:11)
    at moduleResolve (node:internal/modules/esm/resolve:1009:10)
    at defaultResolve (node:internal/modules/esm/resolve:1218:11)
    at ESMLoader.resolve (node:internal/modules/esm/loader:580:30)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:294:18)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:80:40)
    at link (node:internal/modules/esm/module_job:78:36) {
  code: 'ERR_MODULE_NOT_FOUND'
}

Common JS imports are working fine though.
const prettier = require("prettier");
const javaPlugin = require("prettier-plugin-java");
const phpPligin = require("@prettier/plugin-php");
const rustPlugin = require("prettier-plugin-rust");

I just published 0.1.5, should be good~

I updated the package and it is working fine now.
import prettier from "prettier";
import javaPlugin from "prettier-plugin-java";
import phpPligin from "@prettier/plugin-php";
import * as rustPlugin from "prettier-plugin-rust";

const plugins = [javaPlugin, phpPligin, rustPlugin];

Thank you!


Support node and pod metrics

We should support gathering the following metrics:

how many nodes are in the cluster per week, month, and year
how many pods there are in the cluster per week, month, and year

Is there a way to detect the average simultaneously running data, versus aggregate?
eg if pods spin up and down, we might show 1,000,0000 unique pods. But in reality only 20,000 might be running simultaneously (and require CrowdStrike subscription).

Closing this as this is out of scope for the operator's workload to perform.


PR from Prithvi's repo to this repo for Jupyter Apptainer App

I'm merging this PR to bring fork as a branch in this repo.


Use Helia for fetching files

We should explore using Helia for getting files from IPFS without depending on gateways.

@bjesus you can check out helia-verified-fetch if you just need to fetch content. It is supposed to make fetching content easier and more hands-off than using Helia directly if you don't need to do much node management
feel free to join our Helia WG meetings if you want to chat with us or get any help :)