[SPARK-5681][Streaming] Add tracker status and stop to receive messages when stopping tracker

Related to #4364.
Sometimes the receiver will be registered into tracker after ssc.stop() is called. Especially when stop() is called immediately after start(). So the receiver doesn't get the StopReceiver message from the tracker. In this case, when you call stop() in graceful mode, stop() would get stuck indefinitely.
This pr adds a status to ReceiverTracker and asks ReceiverTracker stop to receive messages when stopping.
This also adds a timeout check to ReceiverLauncher.stop.

cc @tdas.

Test build #27091 has started for   PR 4467 at commit 77983f3.

This patch merges cleanly.

Test build #27091 has finished for   PR 4467 at commit 77983f3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27091/
Test PASSed.

@tdas Please take a look of this when you have time.

I realized that this is a tricky thing to fix while maintaining the stopGracefully semantics. Stop gracefully must ensure that if there are receivers that have already started, they must be stopped and all the received data processed before stopping completely. But what happens to the receivers that are still starting and have not registered yet? We have to wait for them to all be started, because if we dont, they may have started and pull data already, which may lead to loosing data. This is not good.
So to solve this correctly. We probably need a Starting state as well. And stopGracefully must stasrt the stoppign process only after the system has reached Started state. So it has to wait for all the receivers to have started, otherwise it is hard to guarantee that all the receivers are correctly stopped.
Also, this behavior must be properly unit tested with different state transitions, etc. Even before that, I would like to see what is the ideal state behavior -

if state = X, then allow register,
if state = Y do not start stopping,
etc .

The state behavior should be:

If state = Initialized, then disallow all actions. (actor is not initialized yet)
If state = Started, then allow all actions {register, addblock, report, deregister}.
If state = Stopping, then disallow {register, addblock}, allow {report, deregister}.
If state = Stopped, then disallow all actions. (actor is destroyed)

For the receivers that are still starting and have not registered yet, we have two options.

As you said, we have to wait for them to all be started.
We ignore their register messages and let them timeout. So they simply fail their starting process and don't process any data.

I think both options guarantee no data would be lost. I was thinking using option 2 in this pr. Because it  should be more simple and, semantically we should not allow receivers to register and then process data after stop is called.
I just realized that the current implementation of ReceiverSupervisor calls receiver.onStart before sending RegisterReceiver message. Thus it is possible the receiver are already started and process data. I think it is incorrect behavior. Correct one should be getting registered first and then begin to start the receiver. Otherwise, the receiver may process data and store data before it has registered successfully. It might cause some problems hard to detect. We should change the order so it register with tracer first then begin starting procedure.
The important reason I think we don't choose option 1 to wait for receivers to all be started is, from the tracker's aspect, it has not idea what receivers are started or not. It just asynchronously waits for them to register and deregister. The receivers are visible to the tracker only when they are registered with it. When it is going to stop, because it doesn't know if there are receivers started but not registered yet, so it doesn't know how longer it should wait for them. Thus it is safer to make sure that the receivers must register before they start.

It should accept addblock even it is stopping because there might be receivers processing data.
Modified state behavior should be:

If state = Initialized, then disallow all actions. (actor is not initialized yet)
If state = Started, then allow all actions {register, addblock, report, deregister}.
If state = Stopping, then disallow {register}, allow {addblock, report, deregister}.
If state = Stopped, then disallow all actions. (actor is destroyed)

Test build #27273 has started for   PR 4467 at commit 3d568e8.

This patch merges cleanly.

Test build #27273 has finished for   PR 4467 at commit 3d568e8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class TaskCommitDenied(
class CommitDeniedException(
  class OutputCommitCoordinatorActor(outputCommitCoordinator: OutputCommitCoordinator)

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27273/
Test FAILed.

Test build #27277 has started for   PR 4467 at commit 355f9ce.

This patch merges cleanly.

Test build #27277 has finished for   PR 4467 at commit 355f9ce.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class TaskCommitDenied(
class CommitDeniedException(
  class OutputCommitCoordinatorActor(outputCommitCoordinator: OutputCommitCoordinator)
case class StreamingListenerReceiverRegistered(receiverInfo: ReceiverInfo)

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27277/
Test PASSed.

I dont think that correct. If the state becomes Stopping before a receiver
has registered, it may so happen that the receiver starts up and receives
some data and puts the data in its buffer/block manager. Since it was not
registered, it wont get a stop receiver message. So it will continue
receiving messages and will never be stopped. Isnt it?
That's why I think option 1 is cleanest. Wait for everything to start up,
so that we cleanly and correctly stop all of them.
On Tue, Feb 10, 2015 at 11:45 PM, UCB AMPLab<EMAIL_ADDRESS>wrote:

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27277/
Test PASSed.
—
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/4467#issuecomment-73843764.

Let's analyze it clearly. The following is a simplified status transformation of the problem:
         tracker       receivers

t = 1      started       registered:{A, B}; starting, not registered: {C}
t = 2      stopping    got stop msg:{A, B}; starting, not registered: {C}
t = 3      stopping    stopped:{A, B}; registered: {C}
The above causes potential data loss. We want to avoid that. I agree.
If we implement option 1, now the status transformation:
         tracker       receivers

t = 1      started       registered:{A, B}; starting, not registered: {C}
t = 2      stopping    got stop msg:{A, B}; starting, not registered: {C}
*we are going to wait for receivers that are started but not registered yet.
*suppose we wait a fixed time period n.
*however, we can't guarantee when the receiver C will be registered.
*so, after waiting time n, the system status can be:
t = n+2  stopping    stopped:{A, B}; registered: {C}
*or still
t = n+2  stopping    stopped:{A, B}; starting, not registered: {C}
As you see, there will still be possible status that we have unregistered receiver C that processes data.
This pr implements another approach. The receivers register first then do starting process:
         tracker       receivers

t = 1      started       registered, started:{A, B}; registered, starting: {C}
t = 2      stopping    got stop msg:{A, B, C}; **D wants to register -> timeout
t = 3      stopping    stopped:{A, B, C}

@tdas, Do you have time to take a look of the analysis and the current implementation?

Sorry I am bit tied with stuff. I will definitely take a look as soon as i
get a chance. :)
On Fri, Feb 13, 2015 at 8:24 AM, Liang-Chi Hsieh<EMAIL_ADDRESS>wrote:

@tdas https://github.com/tdas, Do you have time to take a look of the
analysis and the current implementation?
—
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/4467#issuecomment-74280513.

Hey ... lets continue the discussion. I took a quick look at the logic, sounds good. Let me think a bit more and look at the code.

@tdas Any updated ideas or comments?

/cc @tdas Still busy?

@viirya Sorry for slacking on this, been busy. I think understand your explanation. But I also spent some more time thinking about this ground up.
Correct me if I am wrong, but the thing was getting stuck because of this line
https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala#L248
What happens in this line? The receiverInfo should be empty because whatever had registered, would deregister and make the map empty. So that cannot be the reason of the hang. So the only reason it will stay stuck in that line is because running = true. running is set to false only if the job that runs the receiver completes (this line). The fact that it is stuck indefinitely that means this job never completes.
That can happen only if the receiver (C in your examples) that had not registered by the time "stop gracefully" was called are somehow still running indefinitely. Ideally, it should have never registered and started at all! That is, the ReceiverTracker should prevent any further registration as soon as it gets a stop signal. In that case the sequence of the events should be.
time | tracker | receivers
t = 1 | started | registered:{A, B}; starting, not registered: {C}
t = 2 | stopping | got stop msg:{A, B}; starting, not registered: {C}
t = 3 | stopping | stopped:{A, B}; attempts to register but denied by tracker, never starts itself: {C}
Since C is not allowed to start, its stops itself and the task completes. The jobs completes, running = false, tracker stops.
t = 4 | stopped | stopped:{A, B}; stopped: {C}
Isnt this a viable solution? If so, I think this is simpler than introducing another state.

Currently, Receiver.onStart() is called before it sends RegisterReceiver message to ReceiverSupervisor for registration.
In Receiver.onStart(), it already begins to pull data. So in your cases, receiver {C} is possibly to lose its data.
We know that the thing was getting stuck because sometimes receiver will register after tracker is going to stop. So the receiver will not get stop message properly.
As you said, we can disallow its registration. However, because it already pulls data, it may lose received data.
This solution I proposed, also disallows receiver to register. Besides, it moves registration ahead of starting. So a receiver is only going to pull data after it is registered with the tracker. By doing that, we guarantee no data loss for receivers.

Test build #29785 has started for   PR 4467 at commit 9e1a760.

Test build #29785 has finished for   PR 4467 at commit 9e1a760.

This patch fails Scala style tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

case class StreamingListenerReceiverRegistered(receiverInfo: ReceiverInfo)

This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29785/
Test FAILed.

Test build #29786 has started for   PR 4467 at commit c419677.

Test build #29786 has finished for   PR 4467 at commit c419677.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

case class StreamingListenerReceiverRegistered(receiverInfo: ReceiverInfo)

This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29786/
Test PASSed.

Could you resolve the conflicts?

Merged build triggered.

Merged build started.

Test build #31869 has started for   PR 4467 at commit 34c18dc.

Test build #31869 has finished for   PR 4467 at commit 34c18dc.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class StreamingListenerReceiverRegistered(receiverInfo: ReceiverInfo)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31869/
Test PASSed.

Merged build triggered.

Merged build started.

Test build #32342 has started for   PR 4467 at commit e0ef72a.

Test build #32342 has finished for   PR 4467 at commit e0ef72a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class StreamingListenerReceiverRegistered(receiverInfo: ReceiverInfo)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32342/
Test PASSed.


time-wait handler should take precedence over OnFailedToDispatchPacket

I think the order in quic_dispatcher could be improved, here
This block allows an overridden dispatcher to catch a packet that doesn't match a connection ID and perform an alternative action; I'm thinking about using this in Envoy to facilitate hot restart, wherein we would want to pass new connections to the new instance, but continue to pass packets for existing connections to their respective sessions.
With the code as it is now, this would mostly work, but in the case of a connection being in time-wait state, we would incorrectly pass the packet to the new instance.
If I'm understanding things correctly, it seems to me that a packet which can be associated with a time-wait connection is correctly dispatched by doing what time_wait_list_manager_ does with it, so any packet which could take that branch should not fall into the OnFailedToDispatchPacket handler.
Switching the order of the two blocks would achieve this.

If that would be considered too dangerous of a change since it's possible (though unlikely) that someone is relying on the behavior being how it currently is, adding OnFailedToDispatchPacket2 to OnUnrecognizedConnectionId or something like that, before StatelesslyTerminateConnection would also address the problem.

@yangfanud

Discussed with @yangfanud offline, forwarding packet at the current call site of OnFailedToDispatchPacket() is probably okay. Skipping time wait list checking should only affect closed connections whose CONNECTION_CLOSE frame gets lost, which will be retransmitted by the time wait list upon receiving more packets from the peer. Forwarding those packets to the child process should be fine given they are short header packets and will be dropped anyway.


Template resource length limit exceeded

Is this a request for help?: Yes

Is this an ISSUE or FEATURE REQUEST? (choose one): ISSUE

What version of acs-engine?: PR #1143

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm) Kubernetes
What happened:
trying to deploy a default cluster with using the default cluster.json with the modifications
      "kubernetesConfig": {
        "enableRbac": true,
        "networkPolicy": "calico"
      }

FATA[0014] resources.DeploymentsClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="InvalidTemplate" Message="Deployment template validation failed: 'The template resource 'k8s-master-42902998-0' at line '1' and column '27924' is not valid: The language expression length limit exceeded. Limit: '24576' and actual: '25975'.. Please see https://aka.ms/arm-template-expressions for usage details.'."
I believe this is happening on the customData field on the agentPool
What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know:

#18

I think as an interim solution I can try collapsing all the manifests for a single application into one yaml list. Eg. tiller-deploy and tiller-svc can become simply tiller.yaml. Thoughts?

Couple of thoughts on my end.

Provision a blob, per cluster place the file in there and download it onto disk
Investigate some sort of compression?
Increase the max field size on ARM

This is arm length limit:
“customData”: “[concat(variables(‘part1’), variables(‘part2’), variables(‘part3’)]”
Also, we were diligent early on about gzipping all content, so we should just make sure content is still gzipped.

confirmed that it's all gzipped.

#1162 for temp workaround

I am running into this issue on 0.7.0 release (k8s 1.7.5). Tried deploying exact config that works previously on k8s 1.7.2. Is it possible for the ARM limit to be increased ? as we have more customizations we have not yet applied.

Got the same, trying to deploy calico+maxpods+rbac+aadprofile+customvnet..using the latest commit (2cd215dd) and k8s 1.8

Can confirm with k8s 1.8 + calico + rbac + custom vnet with 5b57309 as well.

@lachie83
Hi,
Any updates please? I've got the same issue with acs-engine v0.8.
Thanks.

@lachie83
This is template that I have used:
{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorRelease": "1.7",
      "kubernetesConfig": {
         "enableRbac": true,
         "networkPolicy": "calico",
         "clusterSubnet": "<IP_ADDRESS>/23",
         "maxPods": 200,
         "serviceCidr": "<IP_ADDRESS>/23",
         "dnsServiceIP": "<IP_ADDRESS>"
      }
    },
    "masterProfile": {
      "count": 3,
      "dnsPrefix": "k8sdev8",
      "vmSize": "Standard_D2",
      "vnetSubnetId": "/subscriptions/b234e268-db65-4037-ab3e-052665760654....../resourceGroups/k8s-armv8/providers/Microsoft.Network/virtualNetworks/k8s-vnet-test/subnets/masters",
      "firstConsecutiveStaticIP": "<IP_ADDRESS>"
    },
    "agentPoolProfiles": [
      {
        "name": "aparmv0",
        "count": 2,
        "vmSize": "Standard_DS2",
        "availabilityProfile": "AvailabilitySet",
        "dnsPrefix": "",
        "vnetSubnetId": "/subscriptions/b234e268-db65-4037-ab3e-052665760654....../resourceGroups/k8s-armv8/providers/Microsoft.Network/virtualNetworks/k8s-vnet-test/subnets/pods"
      }
    ],
    "linuxProfile": {
      "adminUsername": "k8s",
      "ssh": {
        "publicKeys": [
          {
            "keyData": "ssh-rsa xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
          }
        ]
      }
    },
    "servicePrincipalProfile": {
      "clientId": "38d27248-...............",
      "secret": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    }
  }
}

Thanks.

cc @anhowe

cc @jchauncey

Is there a working workaround? #1162 is not working in 0.8.0

So the RBAC is what seems to push this over the limit. Custom VNETs + Calico are perfectly ok

az group deployment create --name "kube-prod2" --resource-group "ContainerProd" --template-file "./_output/kube-prod2/azuredeploy.json" --parameters "./_output/kube-prod2/azuredeploy.parameters.json"
Deployment failed. Deployment template validation failed: 'The template resource 'k8s-master-23453153-0' at line '1' and column '96087' is not valid: The language expression length limit exceeded. Limit: '24576' and actual: '24759'.. Please see https://aka.ms/arm-template-expressions for usage details.'.
Doing acs-engine generate using template below
{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "kubernetesConfig": {
        "networkPolicy": "calico"
      }
    },
    "masterProfile": {
      "count": 3,
      "dnsPrefix": "kube-prod2",
      "vmSize": "Standard_D2_v2"
    },
    "agentPoolProfiles": [
      {
        "name": "agentpool1",
        "count": 10,
        "vmSize": "Standard_D3_v2",
        "availabilityProfile": "AvailabilitySet"
      }
    ],
    "linuxProfile": {
      "adminUsername": "xxxx",
      "ssh": {
        "publicKeys": [
          {
            "keyData": "ssh-rsa xxx"
          }
        ]
      }
    },
    "servicePrincipalProfile": {
      "clientId": "da5e64b1-xxxx-xxx-xxxx-xxxxxxxxxxxx",
      "secret": "dadd8745-xxxx-xxx-xxxx-xxxxxxxxxxxx"
    }
  }
}

@jchauncey
Last I heard this was supposed to be increased on 11/10. I'm still unable to provision an instance with RBAC enabled.
What is the new ETA for the template length restriction to be increased?

So the change was supposed to finish roll out today by 2pm PST. Let me know if you have any issues after that

@jchauncey I provisioned a cluster today, 11/15, at 11:07AM using a template I generated yesterday at 9AM. This template was once working, but is no longer working.
Looks like I'm still having the same issue.
kubectl get nodes
Error from server: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader

etcdctl cluster-health
member 33b0bb17e22c066a is unhealthy: got unhealthy result from http://<IP_ADDRESS>:2379

kubectl cluster-info
Kubernetes master is running at https://mov-dev-we-m.westeurope.cloudapp.azure.com

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

kubectl cluster-info dump
Error from server: client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://<IP_ADDRESS>:2379 has no leader

journalctl -xe
Nov 15 19:59:27 k8s-master-48084675-0 docker[7056]: W1115 19:59:27.712637    8227 status_manager.go:431] Failed to get status for pod
Nov 15 19:59:27 k8s-master-48084675-0 etcd[3353]: etcdserver: request timed out, possibly due to connection lost
Nov 15 19:59:29 k8s-master-48084675-0 etcd[3353]: etcdserver: request timed out, possibly due to connection lost
Nov 15 19:59:30 k8s-master-48084675-0 docker[7056]: I1115 19:59:30.045913    8227 kubelet_node_status.go:503] Using Node Hostname fro
Nov 15 19:59:30 k8s-master-48084675-0 docker[7056]: W1115 19:59:30.648711    8227 cni.go:196] Unable to update cni config: No network
Nov 15 19:59:30 k8s-master-48084675-0 docker[7056]: E1115 19:59:30.648803    8227 kubelet.go:2095] Container runtime network not read
Nov 15 19:59:32 k8s-master-48084675-0 etcd[3353]: failed to reach the peerURL(http://<IP_ADDRESS>:2380) of member 7f03ea887d8488f0 (G
Nov 15 19:59:32 k8s-master-48084675-0 etcd[3353]: cannot get the version of member 7f03ea887d8488f0 (Get http://<IP_ADDRESS>:2380/ver
Nov 15 19:59:33 k8s-master-48084675-0 etcd[3353]: etcdserver: request timed out, possibly due to connection lost
Nov 15 19:59:33 k8s-master-48084675-0 etcd[3353]: failed to reach the peerURL(http://<IP_ADDRESS>:2380) of member a4058da44ae0514c (G
Nov 15 19:59:33 k8s-master-48084675-0 etcd[3353]: cannot get the version of member a4058da44ae0514c (Get http://<IP_ADDRESS>:2380/ver
Nov 15 19:59:34 k8s-master-48084675-0 etcd[3353]: etcdserver: request timed out, possibly due to connection lost
Nov 15 19:59:34 k8s-master-48084675-0 docker[7056]: W1115 19:59:34.714484    8227 status_manager.go:431] Failed to get status for pod
Nov 15 19:59:34 k8s-master-48084675-0 etcd[3353]: etcdserver: request timed out, possibly due to connection lost
Nov 15 19:59:35 k8s-master-48084675-0 etcd[3353]: etcdserver: request timed out, possibly due to connection lost
Nov 15 19:59:35 k8s-master-48084675-0 docker[7056]: W1115 19:59:35.649701    8227 cni.go:196] Unable to update cni config: No network
Nov 15 19:59:35 k8s-master-48084675-0 docker[7056]: E1115 19:59:35.650205    8227 kubelet.go:2095] Container runtime network not read
Nov 15 19:59:37 k8s-master-48084675-0 etcd[3353]: etcdserver: request timed out, possibly due to connection lost
Nov 15 19:59:37 k8s-master-48084675-0 docker[7056]: E1115 19:59:37.050694    8227 kubelet_node_status.go:390] Error updating node sta
Nov 15 19:59:39 k8s-master-48084675-0 etcd[3353]: etcdserver: request timed out, possibly due to connection lost
Nov 15 19:59:40 k8s-master-48084675-0 etcd[3353]: etcdserver: request timed out, possibly due to connection lost
Nov 15 19:59:40 k8s-master-48084675-0 etcd[3353]: failed to reach the peerURL(http://<IP_ADDRESS>:2380) of member 7f03ea887d8488f0 (G
Nov 15 19:59:40 k8s-master-48084675-0 etcd[3353]: cannot get the version of member 7f03ea887d8488f0 (Get http://<IP_ADDRESS>:2380/ver
Nov 15 19:59:40 k8s-master-48084675-0 docker[7056]: W1115 19:59:40.651057    8227 cni.go:196] Unable to update cni config: No network
Nov 15 19:59:40 k8s-master-48084675-0 docker[7056]: E1115 19:59:40.651277    8227 kubelet.go:2095] Container runtime network not read
Nov 15 19:59:41 k8s-master-48084675-0 etcd[3353]: etcdserver: request timed out, possibly due to connection lost

@zimmertr this is no longer related to the length issue and is tracked at the existing 1621.  Closing this issue since the limit has been lifted.

Confirmed the language expression length limit has been bumped from 24576 to 81920:
I also tested deployment with the inputs specified in lachie83's and mikhail-manuilov's comments and can not reproduce this issue.


Merge branch

updated models for klsa, lm and our interfaces
cleaned up
added markdown-documentation

just updated the pull-request:
now with markdown documentation
corrections and update on ecore-models
cleaned up


Missing images on Twibbyn select screen

Some of the images appear to be missing on the Twibbyn selection screen.

Related? https://github.com/coders-for-labour/dashboard-for-labour/issues/5

Yes, this is the same as issue #5.


Update install_test.yaml

Pull Request Description (replace this text with your description)
Closes #xxxx

After creating the pull request: in order to pass the release_notes_updated check you will need to update the "Future Release" section of docs/source/release_notes.rst to include this pull request.

No linked issues found. Please add the corresponding issues in the pull request description.  Use GitHub automation to close the issue when a PR is merged


Is it possible to control the empty fill thickness?

I am trying to get the empty fill thickness a bit smaller than the fill thickness - is there an elegant solution to this?
And also, this library is the king!

Thanks for good words!
Unfortunately it's not so easy now, but it's possible. Here, I've made a snippet.

But this is a good feature request for the future version. I will implement it

Super cool, thanks a lot - keep coding! ;-)

I will, just need to get few evenings for this :)

Amazing ...
but I have a little problem,
This can be made for the countdown?
I had a problem with setInterval,
I hope the result is the value of your plugin is updated every second.
but it did not succeed.
see this example,
Can you provide a solution?

@kinzania - here I've updated your example

Wow you are really the best.
yesterday I have tried in my own way (before you reply to my message).
and this is the result.
Do you want to help how to synchronize plugin value jquery-circle-progress with jQuery countdown
I was glad once with your response so quickly.
thank you

Hey Any update on this request to control thickness of the empty Fill ?
Also any way to change the center blank color ?

@AbhaySoningra - not yet, sorry.

No Worries Mate, anyway Thanks for this wonderful Library :)


[nfs-provisioner] Running outside kubernetes (standalone)

Hello!
I'm trying to understand how the StorageClass/kubernetes will point to the nfs-provisioner (and the nfs server) running outside of the cluster (standalone or containerized).
Example:
sudo ./nfs-provisioner -provisioner=example.com/nfs
-kubeconfig=$HOME/.kube/config
-run-server=false
-use-ganesha=false
-enable-xfs-quota=true
I'm not sure how to set up, inside the kubernetes cluster, the ip address/hostname to the pointing to the nfs-provisioner.
Thanks in advance!

The kubeconfig integrates the nfs-provisioner with the Kubernetes Cluster (OKD in my case).
My setup was like this:
/usr/local/bin/nfs-provisioner -provisioner=/nfs -master=https://:8443 -kubeconfig=/root/.kube/config -use-ganesha=false -run-server=false -server-hostname= -enable-xfs-quota=true -export-subnet=
Then i deployed the ServiceAccount:
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-provisioner

And the RBAC: https://github.com/kubernetes-incubator/external-storage/blob/master/nfs/deploy/kubernetes/rbac.yaml
After that, downloaded the kubeconfig of the SA nfs-provisioner and pointed the nfs-provisioner command line to load it.


chore: add CODEOWNERS

:tada: This PR is included in version 1.9.3 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Support pushing other files to AUR repository

I would like to publish other files (for example, .gitignore, .install script, patches, other assets...) to the AUR repository along with the PKGBUILD and .SRCINFO. I think this could be done by adding another input to the action (maybe assets) that accepts a list of files (maybe also whole directories or glob patterns), copies them to /tmp/local-repo and commits them along with existing files.

Let's just use assets key and glob pattern. However, I am busy right now, so I'll be accepting pull requests.


Four node contraction gives inconsistent results

I have a series of four tensors (two matrix product operators) that I want to contract in a slightly non standard way. The index conventions and contraction I want to perform is shown in the attached image.

However, I get inconsistent results between iterations when comparing the contracted tensor. I've attached a MWE below. I'm somewhat at a loss for why this could be happening> I know it has nothing to do with the edge_order, as the ordering should be well defined in the call to get_all_dangling(), and in any case I tested by hardcoding an edge order.
Would appreciate any help!
import numpy as np
import tensornetwork as tn
a_shape = [(2, 1, 1, 2), (2, 2, 2, 2)]
lambda_shape = [(2, 1, 1, 2), (2, 1, 2, 4)]
A1 = [np.random.rand(*shape) for shape in a_shape]
Lambda = [np.random.rand(*shape) for shape in lambda_shape]

def process_mpos(mpo):
    tn_mpo = [tn.Node(i) for i in mpo]
    for i in range(len(mpo)-1):
        tn_mpo[i][-1] ^ tn_mpo[i+1][2]
    return tn_mpo

def mpo_on_mpo(mpoL, mpoR):
    L = len(mpoL)
    mpoL[0][1] ^ mpoR[0][2]
    for i in range(L-1):
        mpoL[i+1][1] ^ mpoR[i][0]
    mpoL[L-1][3] ^ mpoR[L-1][0]

    edge_list = []
    for i in range(L):
        edge_list.extend(mpoL[i].get_all_dangling())
        edge_list.extend(mpoR[i].get_all_dangling())
    output = tn.contractors.auto(tn.reachable(mpoL[0]), edge_list)
    return output

for i in range(10):
    A1_tn, Lambda_tn = process_mpos(A1), process_mpos(Lambda)
    out = mpo_on_mpo(A1_tn, Lambda_tn)
    A1_tn, Lambda_tn = process_mpos(A1), process_mpos(Lambda)
    out1 = mpo_on_mpo(A1_tn, Lambda_tn)
    print(np.allclose(out1.tensor, out.tensor))

get_all_dangling returns a set, so actually no there is no order guarantee at all.
https://tensornetwork.readthedocs.io/en/latest/tensornetwork.html#tensornetwork.get_all_dangling

I'm a bit puzzled by this, since I tried replacing the loop adding to edge_list with the line edge_list = [mpoL[0][0], mpoL[0][2], mpoL[1][0], mpoR[0][1], mpoR[1][1], mpoR[1][3]], and still get the same results. If the problem were the ordering being inconsistent, would this not fix the issue?

You can make your own get_all_dangling that would guarantee an order though.
def custom_get_all_dangling(nodes):
  edges = []
  for node in nodes:
    for edge in node.edges:
      if edge.is_dangling():
        edges.append(edge)
  return edges

Yes, that fixes it. Thanks. Out of curiosity, why are the dangling edges returned as a set, and not in (say) the order they're in the list of all edges?

Because the only constraint we set on the input of the function is that it is iterable. I don't think we'll change this API but I'm not against adding a get_dangling_edges_ordered which would just be the above code.


Fix/docs

Hello, I've opened the above pull request to add important information into the ReadMe about breaking changes in v0.2.7 regarding shallowCopy/the new deepCopy method. Thank you!


fix multiline descriptions

Few students had issues this morning with multiline descriptions, maybe from accidentally hitting enter when writing it?
Anyway this does what it seems like it does, uses triple quotes with multiline strings and single quotes with single line strings. the extra backslashes just strip the leading and trailing whitespace.
So with a multiline description

we get this
[project]
name = "my_project"

version = "0.1.0"

description = """\
sup description

anotherline

third\
"""
authors = [
    { name = "me", email = "me" },
]

which parses like
>>> with open('pyproject.toml', 'rb') as tfile:
>>>     project = tomllib.load(tfile)
>>>
>>> data['project']['description']
'sup description\n\nanotherline\n\nthird'

And for a single line description we get, as expected
[project]
name = "my_project"

version = "0.1.0"

description = "description single line"
authors = [
    { name = "me", email = "me" },
]

Could have sworn i already closed this in favor of https://github.com/pyOpenSci/pyos-package-template/pull/51


wrong color for certain variable names

Steps to cause the bug to occur

install & enable the python extension
open a python file and type "forcing"

Actual behavior
"for" and "in" in the variable name is colored as if they are keywords in a for loop

Expected behavior
All letters are colored the same within same variable name

Your Jupyter and/or Python environment
Please provide as much info as you readily know

Jupyter server running: Local | Remote | N/A
Extension version: 20YY.MM.#####-xxx
VS Code version: 1.46.0-insider
Setting python.jediEnabled: true | false
Setting python.languageServer: Jedi | Microsoft | None
Python and/or Anaconda version: python 3.7
OS: Windows | Mac | Linux (distro): Ubuntu 20.04
Virtual environment: conda | venv | virtualenv | N/A | ...

Developer Tools Console Output

Microsoft Data Science for VS Code Engineering Team: @rchiodo, @IanMatthewHuff, @DavidKutu, @DonJayamanne, @greazer, @joyceerhl

We don't control syntax highlighting for Python files. That comes from VS Code which gets it from MagicPython. It's best to report an issue there.


fixing #4 - setting a default frequency

If options are passed in without a frequency, the default frequency is used.

fixes #4

Interesting; might be because this isn't initialized yet in the default args.
Thanks!

I think it's because when the RealtimeConversation is instantiated without any options params, it gets the default. But if we only supply a single arg, like { debug: false }, then frequency remains undefined since the constructor options now contain a valid object.
let conversation = new RealtimeConversation(); // the frequency is now set with the default
conversation = new RealtimeConversation({ debug: false }); // frequency is now undefined

Anyways, thanks for merging!


CSS

CSS
User Story
As a bootcamp student
I want the prework notes to be structured on a webpage
So that I can easily find and read the information
Acceptance Criteria
GIVEN a prework studyguide website
WHEN I view the study guide
THEN I see a dark blue header and a footer, and four boxes with a shadow

finished this CSS issue


Fix custom attributes in mustache templates.

The immediate motivation for this was from me testing the slurm provider at NERSC where a custom attribute specifying node type is mandatory.
With this PR I can write:
JobAttributes(custom_attributes=
  {"slurm":[
    {"key": "constraint", "value": "knl"},
    {"key":"foo", "value":"bar"}]
  }
)

resulting in these lines:
#SBATCH --constraint="knl"
#SBATCH --foo="bar"

into my slurm job submission file.
That key/value syntax is ugly but it is enough for me to move me along in my testing/development.

Looks like I can write this too:
JobAttributes(custom_attributes={"slurm.constraint":"knl", "slurm.another":"othervalue"}),

for a better key value experience.

some short documentation and / or a small test would be great. Or is there a ticket to reference? I am not sure what this PR about I'm afraid...

Sorry, my bad. Basically, what Ben says. This is a follow-up to a discussion on slack. The slurm template contains this snippet:
    {{#custom_attributes.slurm}}
#SBATCH --{{key}}="{{value}}"
    {{/custom_attributes.slurm}}

It does not work with the way custom attributes are meant to be specified in the spec/lib. This PR makes the above snippet work correctly by setting the relevant keys in the template context dictionary and in the relevant form.

This might be a PSI/J spec thing but I wonder if it might be better to let users specify the entire string? E.g. "#SBATCH --foo=bar rather than{"foo": "bar". They're already going to need to be familiar with the batch system, and I think making users provide the entire string might be a little more transparent and simpler. I can see a lot of users mistakenly providing {"--foo": "bar". Or wanting to provide options that take a single dash rather than two: "#SBATCH -fbar.

Thanks for the context all - I missed out of that slack discussion.

but I wonder if it might be better to let users specify the entire string?

That would break the abstraction a bit: the user would need to change job description when moving from, say, Slurm to PBS.  Well, then again the key is unlikely to be valid on the PBS machine anyway - but in general I would tend to stay away from backend specifica as much as possible.  No strong opinion, just my $0.05 on this point...

I wonder if it might be better to let users specify the entire string? E.g. "#SBATCH --foo=bar rather than{"foo": "bar".

That would break the abstraction a bit: the user would need to change job description when moving from, say, Slurm to PBS.

An executor implementation may choose to not use submit scripts but pass these as command line arguments to, e.g., sbatch, in which case the option would not have the "#SBATCH " part. So we may want this to come with some implicit information.

Or wanting to provide options that take a single dash rather than two: "#SBATCH -fbar

Good point. I do believe in most cases the short options are a subset of the long options, but it's still a possibility. It's also possible to have a flag, such as --x, without the =value part, which is not something that is handled right now. The spec says these are key value pairs, but otherwise it's entirely up to the executor implementation what it does with them. So maybe we can find something that could cover more scenarios. But maybe that's a separate issue.

Well, I guess the idea of different SLURM implementations being able to recognize the same options is noble, but somewhat contradicts "it's entirely up to the executor implementation what it does with them."

What about only making the #SBATCH implicit so that the user would have to write "--foo=bar"?

In contexts that I can think of at the moment (pytest/conf, gettopt) the API tends to treat the dashes as a command-line only occurrence. For example, when you want to get the value corresponding to "--foo" on the command line, you say config.getoption("foo"), or when you specify the options to getopt-long, you exclude the dashes. It's somewhat similar in argparse, in that you get the values without the dashes, but when you specify the options you need the full form, since argparse allows you to specify and distinguishes (maybe?) between "-foo" and "--foo". There are reasons for this, which is that the command line does not otherwise have enough structure to distinguish between positional and named arguments, a problem that does not generally exist in an API.
So maybe let's leave it as is for now.

That works for me.

With my user support hat on, my main concern with any form of passing options onto an underlying system is this scenario:
A user (who knows about Science, and nothing about execution) and an HPC sysadmin (who knows about the local execution system, and nothing more) conspire to execute a program.
The HPC sysadmin sees an error and gives the solution "add #SLURM --constraints=baz" to your sbatch submission file.
The user (who knows nothing about execution) is unable to make sense of this meaningless instruction: they don't even know what an sbatch is, let alone whether it has a file or not.
Really the important thing for me in that use case is that it needs to be very clear how either of the intelligent-but-ignorant participants above can understand how to translate such an instruction (or other similar documentation) into the appropriate psi-j incantation.
My feeling from ops experience is that this is a documentation issue, rather than a debate-API-forms issue.

Thanks Ben.

My feeling from ops experience is that this is a documentation issue, rather than a debate-API-forms issue.

We'll have to carefully document the various aspects of this, and that's beyond the API docs that we are somewhat having now.
But we should also make parallel efforts to address these complexities: making things "intuitive", things like having repositories of configurations, etc.


Eli/editions with edition

https://hackmd.io/RG2QdiRqTTOxSMdigvRz7A

[ ] Add summary to README
[ ] Editions specification page (similar to marketplace & auction examples)


test release

Signed-off-by: kranurag78<EMAIL_ADDRESS>Purpose of PR?:
Fixes #
Does this PR introduce a breaking change?
If the changes in this PR are manually verified, list down the scenarios covered::
Additional information for reviewer? :
Mention if this PR is part of any design or a continuation of previous PRs
Checklist:

[ ] Bug fix. Fixes #
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update
[ ] PR Title follows the convention of  <type>(<scope>): <subject>
[ ] Commit has unit tests
[ ] Commit has integration tests

Not intended, this was meant for personal repo. gh tricked me here. :)


Consider separating CLI.

Something like flow-typed and flow-typed-cli.
This would expose the API and make it easier to test.

Hey! I see you are really motivated to get this thing going, but I am not in charge for major decisions... I am also a little bit lost here, since I don't know what the flow team is cooking :-)
Since there is no huge benefit (and since every maintainer is used to this repo design), we will keep it like that for now.
If you feel confident in contributing, I could offer you to try #104, since this task would bring a lot of value :-)

One of the reasons I would advise this is to incentivise testing and documentation.  The lack of tests is holding up #490; to write tests, I would either have to expose the API anyway, or test the CLI.  The lack of documentation is forcing the create-react-app team to use flow-typed in a very round-about way.

Okay got ya... maybe let's continue discussion on the create-react-app issue

@ryyppy the discussion is pretty long already and this sub-discussion is not central to the feature I'm integrating. We don't need clear CLI separation but that'll be surely a nice to have in the long term. I don't want you to rush a design decision on this. Take your time, when available and documented, we'll reintegrate that into CRA.

I've been thinking about this a bit myself as well and I think it might make sense. I'll look into what it will take to split into another repo

Would you like a PR making flow-typed a lerna monorepo?  Then you could separate it into flow-typed and flow-typed-cli or something.


Context is displayed when jumping to netrw

Description
When a context is shown and switching to netrw (by running :Ex), the context is persisted in the netrw buffer, as if the cursor was still on the previous buffer.
Neovim version
NVIM v0.9.0 Build type: Release LuaJIT 2.1.0-beta3     system vimrc file: "$VIM/sysinit.vim"   fall-back for $VIM: "/usr/local/Cellar/neovim/0.9.0/share/nvim"  Run :checkhealth for more info
Expected behavior
The context shouldn't be shown in the netrw buffer.
Actual behavior
Inside a file showing context:

Run :Ex:

Minimal config
local plugins = {
  ts         = 'https://github.com/nvim-treesitter/nvim-treesitter',
  ts_context = 'https://github.com/nvim-treesitter/nvim-treesitter-context',
  -- ADD ADDITIONAL PLUGINS THAT ARE _NECESSARY_ TO REPRODUCE THE ISSUE
}

for name, url in pairs(plugins) do
  local install_path = '/tmp/nvim/site/'..name
  if vim.fn.isdirectory(install_path) == 0 then
    vim.fn.system { 'git', 'clone', '--depth=1', url, install_path }
  end
  vim.o.runtimepath = install_path..','..vim.o.runtimepath
end

-- ADD INIT.LUA SETTINGS THAT IS _NECESSARY_ FOR REPRODUCING THE ISSUE

Steps to reproduce

nvim --clean -u minimal.lua
Open a file and place the cursor in a line that shows a context at the top.
Run :Ex

I have the same issue with dirbuf.

PR welcome


[nothing to see here] lib: Add functionInfo

Description of changes
Adds a powerful functionInfo function, giving you more information about your functions!
functionInfo builtins.add ->
  {
    type = "builtin";
  }

functionInfo (arg: arg + 1) ->
  {
    type = "argument";
    argumentName = "arg";
  }

functionInfo (arg@{ a }: a + 1) ->
  {
    type = "attributes";
    argumentName = "arg";
    attributes = { a = false; };
    ellipsis = false;
  }
functionInfo ({ a ? 0, ...}: a + 1) ->
  {
    type = "attributes";
    argumentName = null;
    attributes = { a = true; };
    ellipsis = true;
  }

PS: Don't look at the implementation
PPS: This PR is only half-serious, motivation came from https://github.com/NixOS/nixpkgs/pull/194514#discussion_r990477041, where this PR would actually have a good use case (cc @roberth)
Things done

argumentName doesn't seem useful.
ellipsis : bool is slightly more useful, but I don't think it leads to an improvement. lazyFunction does have a "problem" with ..., which is that those don't end up in the @ param. However, we simply can't put anything in ... without figuring out the attribute names of the argument.
Interesting PR... :D


Move "Known Issues" Section

Dear windows11 Maintainers,

I moved this section up, so it will be more noticeable, and hopefully will result in less repeat issues made on the repo. I also changes the header type, to fit where it was put.
EDIT: I have to fix something on it, so it will be in 2 commits...

Best Wishes,
Lukas Batema

Why does this bot spam that message, tho???

It would be an exaageration to say that firefox blur issue is the cause for repeated issues. Infact there is only one.

It would be an exaageration to say that firefox blur issue is the cause for repeated issues. Infact there is only one.

I realise that, but if you, in the long run, put more known issues, listed there, it would be more noticeable. This would be helpful in the long run, not just short-term.

Readme was for introducng the project. I added the firefox blur issue just for people starting out and it will be removed eventually.

Readme was for introducng the project. I added the firefox blur issue just for people starting out and it will be removed eventually.

Ah, ok. I do think having a section of known “major” issues in the readme file, near the top, would be a good idea. It’s up to you though :)

I would rather include them in default issue template than readme. lol

I would rather include them in default issue template than readme. lol

Understood :) I will close this then ;)


KeeperErrorCode = ConnectionLoss for /mesos

Hi,
With 0.2.0, I retried multiple times, both provision and destroy+up, exactly like yesterday, when it used to work.
TASK: [mesos | create zookeeper acl] ******************************************
failed: [default] => {"changed": true, "cmd": "docker run --rm -e ZK_AUTH=super:eLkJBbOEyKNwouGBicxn4SMv5ETT+SvD7OgjZ8c5a2Hm -e ZNODE=/mesos -e ZNODE_ACL=world:anyone:r,digest:mesos:3KXBhDZ0nQFsKCjQnlXOItGdKF4=:cdraw -e ZK_SERVER=zookeeper.service.consul:2181 --entrypoint /usr/local/bin/setacl.sh asteris/zookeeper:latest", "delta": "0:00:02.044643", "end": "2015-05-19 13:12:31.649043", "rc": 1, "start": "2015-05-19 13:12:29.604400", "warnings": []}
stderr: Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /mesos
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:695)
at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:588)
at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:360)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:323)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)
stdout: Connecting to zookeeper.service.consul:2181
2015-05-19 13:12:30,509 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2015-05-19 13:12:30,516 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=885dd5140911
2015-05-19 13:12:30,517 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.7.0_75
2015-05-19 13:12:30,522 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2015-05-19 13:12:30,522 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.7-openjdk/jre
2015-05-19 13:12:30,522 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/opt/zookeeper/bin/../build/classes:/opt/zookeeper/bin/../build/lib/.jar:/opt/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/opt/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/opt/zookeeper/bin/../lib/log4j-1.2.16.jar:/opt/zookeeper/bin/../lib/jline-0.9.94.jar:/opt/zookeeper/bin/../zookeeper-3.4.6.jar:/opt/zookeeper/bin/../src/java/lib/.jar:/opt/zookeeper/bin/../conf:
2015-05-19 13:12:30,522 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2015-05-19 13:12:30,523 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2015-05-19 13:12:30,523 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=
2015-05-19 13:12:30,523 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2015-05-19 13:12:30,523 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2015-05-19 13:12:30,523 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-229.1.2.el7.x86_64
2015-05-19 13:12:30,524 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
2015-05-19 13:12:30,524 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
2015-05-19 13:12:30,524 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/opt/zookeeper
2015-05-19 13:12:30,526 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=zookeeper.service.consul:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@630ed791
Welcome to ZooKeeper!
2015-05-19 13:12:30,570 [myid:] - INFO  [main-SendThread(<IP_ADDRESS>:2181):ClientCnxn$SendThread@975] - Opening socket connection to server <IP_ADDRESS>/<IP_ADDRESS>:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2015-05-19 13:12:30,585 [myid:] - INFO  [main-SendThread(<IP_ADDRESS>:2181):ClientCnxn$SendThread@852] - Socket connection established to <IP_ADDRESS>/<IP_ADDRESS>:2181, initiating session
2015-05-19 13:12:30,590 [myid:] - INFO  [main-SendThread(<IP_ADDRESS>:2181):ClientCnxn$SendThread@1098] - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
[zk: zookeeper.service.consul:2181(CONNECTING) 0] addauth digest super:eLkJBbOEyKNwouGBicxn4SMv5ETT+SvD7OgjZ8c5a2Hm
[zk: zookeeper.service.consul:2181(CONNECTING) 1]
[zk: zookeeper.service.consul:2181(CONNECTING) 1] create /mesos ""
FATAL: all hosts have already failed -- aborting

0.2.0


Kp, F10.7, IMF forecasts

Extend OMNI, F10.7, and Kp routines to download forecasted values as well as observed or measured values.

Last 30 days of Daily Kp: https://services.swpc.noaa.gov/text/daily-geomagnetic-indices.txt
Next 3 days of Kp Forecast: https://services.swpc.noaa.gov/text/3-day-geomag-forecast.txt
Current F10.7: https://services.swpc.noaa.gov/text/27-day-outlook.txt
Next 3 days of F10.7: https://services.swpc.noaa.gov/text/3-day-solar-geomag-predictions.txt
45-day forecasted kp/ap: https://services.swpc.noaa.gov/text/45-day-ap-forecast.txt

According to my meeting notes this should be on my action item list, correct? I'll work on this today unless I hear otherwise.

Yes, it is.
Tricky thing that came up when discussing usage with Sarah:
People may want to archive these forecasts so that future work can be replicated.  I think pysat can do this easily.  We'll just need to make sure we include the datetime that the forecast was retrieved as well as the datetime the forecast was made for.
Cheers,
Angeline
From: Russell Stoneback<EMAIL_ADDRESS>Reply-To: rstoneback/pysat<EMAIL_ADDRESS>Date: Tuesday, October 9, 2018 at 11:34
To: rstoneback/pysat<EMAIL_ADDRESS>Cc: "Burrell, Angeline"<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [rstoneback/pysat] Kp, F10.7, IMF forecasts (#83)
According to my meeting notes this should be on my action item list, correct? I'll work on this today unless I hear otherwise.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/rstoneback/pysat/issues/83#issuecomment-428240031, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AGuC_pFkqgdLuaW6eMkylDXlUoKVRzvAks5ujMHggaJpZM4W_rw8.

Just pushed my first commits for Kp forecast support. Files are stored based upon the date of the forecast. Loading a forecast (by generation date) loads the three days of forecast Kp into pysat and a DataFrame.

I looked around and couldn't find any IMF forecast products, so I think it's just F10.7, kp, and ap.

See pull #95

I believe this is complete.


fix panic issue on update avatar email (#4580)

fix #4580
back port PR for release/v1.5,  refer to #4581

LGTM


Include pairwise LD information in a track?

Is there a way in Gviz to include anything like a plot of which SNPs are in pairwise LD with each other?  Something like this, for example:

I haven't managed to figure out a way to include something like this, but it would be a nice addition.
Gviz is fantastic, by the way; thank you for all your work on it.

Hi @pbreheny,
Unfortunately this is currently not possible but I would be happy to implement it. I was thinking about a "rotated" heatmap instead of those lines. What is the data format in which you store the LD, is it a matrix or sparse version of it?

I typically use the ld() function in the snpStats package to compute linkage disequilibrium.  It has a depth argument that controls the maximum lag between SNPs, and returns a sparse matrix (specifically, a dgCMatrix) with zeros for all pairs outside depth.
A rotated heatmap would be fantastic, especially if it incorporates that sparsity/depth.  For example, in the above figure, you probably wouldn't want a heatmap of the entire region, but a heatmap of the central band, rotated, would be great.  I think the main appeal of the lines is that it's easy to restrict them to a certain depth, but if you can accomplish that with a heatmap, even better.


Can't setup the integration anymore

Hello,
First I want to thank you for developing and maintaining this integration. I've been a long user and it usually worked like a charm.
Today, a technician from Securitas performed some maintenance on my alarm system and removed (without notice) the local account I use for this integration.
Therefore, I had to do the setup process all over again.
I removed the integration, stopped HA, updated this integration to the latest tag, restarted HA and went to setup the integration.
Unfortunately, it failed to setup after having set the SMS from 2FA. I received this new kind of error : I guess Securitas changed the JWT token format, since I can read Invalid token type :
Logger: aiohttp.server
Source: /usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py:421
First occurred: 13:55:03 (1 occurrences)
Last logged: 13:55:03

Error handling request
Traceback (most recent call last):
  File "/config/custom_components/securitas/securitas_direct_new_api/apimanager.py", line 338, in login
    token = jwt.decode(
            ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/jwt/api_jwt.py", line 210, in decode
    decoded = self.decode_complete(
              ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/jwt/api_jwt.py", line 151, in decode_complete
    decoded = api_jws.decode_complete(
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/jwt/api_jws.py", line 198, in decode_complete
    payload, signing_input, header, signature = self._load(jwt)
                                                ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/jwt/api_jws.py", line 254, in _load
    raise DecodeError(f"Invalid token type. Token must be a {bytes}")
jwt.exceptions.DecodeError: Invalid token type. Token must be a <class 'bytes'>

File "/config/custom_components/securitas/config_flow.py", line 103, in async_step_otp_challenge
    return await self.finish_setup()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/securitas/config_flow.py", line 107, in finish_setup
    await self.securitas.login()
  File "/config/custom_components/securitas/__init__.py", line 353, in login
    await self.session.login()
  File "/config/custom_components/securitas/securitas_direct_new_api/apimanager.py", line 344, in login
    raise SecuritasDirectError(
custom_components.securitas.securitas_direct_new_api.exceptions.SecuritasDirectError: Failed to decode authentication token None

Let me know if you need more information or how I can help to help you fix this issue.
Thanks in advance !

did you try a second time? Sometimes these errors resolve themselves. I don't think they've changed the format of the token (or the integration would stop working for most of us).

I’ll try a new time in a few hours, but I did retry some time later and it was still not working. From what I inspected in my web browser inspector, the token format hasn’t changed : I’ve been able to decode the payload.
I think it’s more in the order of the requests that it may have changed, or the token is not repeated after the 2FA validation in the login resquest. I guess the code expects it that way, but the token really is not set at that time.
I didn’t spend enough time to understand where the code is not working for me.

When you configure the integration try leaving unchecked the "Use 2FA" checkbox. It shouldn't make a difference, but maybe worth a try.

I just did some tries, and I'm still blocked at the same place. :(
I did try to leave the checkbox "Use 2FA" unchecked, but I got an error "An error occurred" within the dialog box : I must say I'm not surprised as 2FA is required in France from several months now.
I'm still blocked with the missing token in the login function call error in the logs  :
File "/config/custom_components/securitas/config_flow.py", line 153, in async_step_user
    return await self.finish_setup()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/securitas/config_flow.py", line 107, in finish_setup
    await self.securitas.login()
  File "/config/custom_components/securitas/__init__.py", line 353, in login
    await self.session.login()
  File "/config/custom_components/securitas/securitas_direct_new_api/apimanager.py", line 344, in login
    raise SecuritasDirectError(
custom_components.securitas.securitas_direct_new_api.exceptions.SecuritasDirectError: Failed to decode authentication token None

I'll try to debug a bit, because maybe I already have the token and may not need to set it in the login function.

Did you know that you can enable debug logs in the integration so you can see the API output as well?

No, I didn't know ! Can you direct me as to where it is explained how to enable it ?

You have to do thing like this in your configuration.yaml
logger:
  default: error
  logs:
    custom_components.securitas: debug

Having enabled debug logs, I can see the login request result (the message below means "User correctly validated"), and as "expected" the hash is not set :
{"data":{"xSLoginToken":{"__typename":"XSLoginToken","res":"OK","msg":"Utilisateur correctement validé","hash":null,"refreshToken":null,"legals":false,"changePassword":true,"needDeviceAuthorization":null,"mainUser":false}}}

When I log into the website in my browser, the hash is set in the "Validate Device" request, and from there, no login request is made.
When I try to setup the integration, the debug logs for the "Validate Device" don't show the hash, like for the login request:
{"data":{"xSValidateDevice":{"res":"OK","msg":"Demande effectuée correctement","hash":null,"refreshToken":null,"legals":null}}}

,"mainUser":false
Looks like you're using a secondary account. It should work, but could you try with your main account?

Good catch, you just made my day ! Problem is solved, let me explain (for you guys and possibly other people some day).
I did create a dedicated user for this integration, to separate my usage from the "normal" one (my phone, my web browser etc.). I gave it the access the proper rights (no admin rights for example), but I didn't went to the website with this new account (always used my normal account, as it has admin rights). Your question made me want to connect to securitas website with this new restricted account. Doing so made me "finish the setup of the account" : I realised that it was not fully trusted by securitas yet, and notably I had to change my password, validate it through 2FA and connect again.
After doing all theses steps, I have been able to have my account properly validated from securitas POV, and I'm now able to add the integration properly into HA.
Once again, thank you guys for the support, I wouldn't have find it without you.
And of course thank you too for the development of this integration :)

I'm glad that it worked.


ack and nack for specific messages

This can be a capability interface that a consumer has to implement, and we can check for this capability when using it in runtime.
Suggested by @shyam-king

Depends on #12
This capability is not possible out of the box with Kafka due to its intricacies, but possible with Pulsar. So a Pulsar Consumer can implement this 'capability' interface (just like how the Pulsar Producer implements the DelayedMessages capability interface).
https://pulsar.apache.org/docs/3.0.x/concepts-messaging/#messages is a good place to start.
We also use batch receive policy, so keep that in mind.


type conversion before tl.dot fails compilation

Context: we are trying to upgrade trition pin in pytorch and start encountering test failure for python test/inductor/test_pattern_matcher.py -k test_mixed_mm
Doing a type conversion for an input tensor to tl.dot starts to fail. It works in triton commit: e6216047b8b0aef1fe8da6ca8667a3ad0a016411 , but fail on a recent commit<PHONE_NUMBER>ddcad54aa4c403276b478a36379b90 .
Here is a standalone repro w.py:  https://gist.github.com/shunting314/3a3b8ce1ccee7b51b8ee0d9a2d24dd3d
Running python w.py will report the following error in the new commit:
loc("/tmp/torchinductor_shunting/bm/cbm7qsh5esh6xdkdddmv7l2ilel4kdbfwgy2luolzmme62njagrb.py":64:17): error: LLVM Translation failed for operation: builtin.unrealized_conversion_cast
Failed to emit LLVM IR
Translate to LLVM IR failedLLVM ERROR: Failed to translate TritonGPU to LLVM IR.
Aborted (core dumped)

Here is a same repro but with inductor dependencies removed: https://gist.github.com/shunting314/4eb4a6a7d8cbfc6396726518196e26d1

cc @ptillet @Jokeren any idea how we can unblock the triton version upgrade in pytorch?

A similar testing program https://gist.github.com/shunting314/2eb4dd1c12ec0ac90df42d0d2d6efd3a  with slight different input types also start fail on the new commit with error:
unimplemented code path
UNREACHABLE executed at /home/shunting/ws/triton/lib/Conversion/TritonGPUToLLVM/ElementwiseOpToLLVM.cpp:254!
Aborted (core dumped)

Can you provide a standalone reproducer without the inductor dependency? I'm getting errors like

TypeError: instance_descriptor.new() got an unexpected keyword argument 'ids_of_folded_args'

@Jokeren  oh, that's because without my other PR ( https://github.com/pytorch/pytorch/pull/107722 ) , inductor uses a stale definition of instance_descriptor. This script https://gist.github.com/shunting314/3a3b8ce1ccee7b51b8ee0d9a2d24dd3d should avoid the issue you mentioned. I can also spend more time to remove inductor dependencies if that helps

@Jokeren  BTW, here is another repro with inductor dependencies removed: https://gist.github.com/shunting314/4eb4a6a7d8cbfc6396726518196e26d1

Thanks, I can reproduce the bug now

Looks like different combination of dtypes surface different issues. Here are a summary of them

bfloat16 -> float32

repro https://gist.github.com/shunting314/4eb4a6a7d8cbfc6396726518196e26d1
status: fixed by #2162

int8 -> float32

repro https://gist.github.com/shunting314/d164f74f9830239fbe04b1913b690fa0
status: fixed by #2184

int8 -> bfloat16

repro https://gist.github.com/shunting314/724d60b5a4f834234c6e922a7567679d
status: *** NEED HELP ***
check error mesage 1 below

uint8 -> float16

this is similar to the case above. Hopefully fixing one also fixes the other. But here is the repro: https://gist.github.com/shunting314/9f47293d653582808bc0edd1c4db4e04

Error message 1:
python: /home/shunting/.triton/llvm/llvm+mlir-17.0.0-x86_64-linux-gnu-ubuntu-18.04-release/include/llvm/ADT/ArrayRef.h:255: const T& llvm::ArrayRef<T>::operator[](size_t) const [with T = mlir::Type; size_t = long unsigned int]: Assertion `Index < Length && "Invalid index!"' failed.```

@Jokeren  could you also look like the 8bit -> 16bit conversion case? I tried to report as much as I can. But these are the error I see after the others are fixed. Thanks

@Jokeren could you also look like the 8bit -> 16bit conversion case? I tried to report as much as I can. But these are the error I see after the others are fixed. Thanks

Sure, I'll probably take look early next week

Sure, I'll probably take look early next week

Sounds good. Thank you for all your efforts unblocking us upgrading triton version in pytorch!

I haven't figured out the fix yet, still investigating

@shunting314 I actually left a comment here https://github.com/openai/triton/blob/1465b573e8d8e4c707d579092001bcff0f1523ed/python/triton/language/semantic.py#L1280
It's better to use a blocking size greater or equal to 16x32 for i8. It's tricky in the mixed precision case because we don't know the operands were converted from other precisions so the frontend didn't report an error.

And I think the check is insufficient because 16x16 fma should work.

I'm checking with @ptillet to see if we need to support small matrix sizes as other precisions or improve the IR verification in the backend.
Even if none of the proposed changes finally landed, you can add some constraints at the inductor end.

@Jokeren  I'm a bit confused here about the block size restriction for int8

with the old triton pin, small block size for int8 actually works. It the restriction added recently?
the 'int8 -> float32' case also create small block for int8 but has been fixed. Why this one works with the 'int8 -> bfloat16' case doesn't?

On the other hand, I'm trying to use larger block size to confirm

with the old triton pin, small block size for int8 actually works. Is the restriction added recently?

hmm, I'm not sure about why that worked.

the 'int8 -> float32' case also create small block for int8 but has been fixed. Why that one works while the 'int8 -> bfloat16' case doesn't?

The previous one was for 16x16 with blocked layout (non-tensor core path).

@Jokeren Just FYI, we work around the issue regarding int8 by make sure the BLOCK SIZE is at least 32.  In case you figure out why it works in previous commit later, let us know :)

I am using commit id: 5812d970a89d47d28e5717bdd73deec7e6829246
I have test on T4,nvidia driver  Driver Version: 450.80.02    CUDA Version: 11.0
and A100, Driver Version: 470.82.01    CUDA Version: 11.8
there is same issue, call stack:
#0  0x00007ffff71188af in raise () from /usr/lib64/libc.so.6
#1  0x00007ffff711a4aa in abort () from /usr/lib64/libc.so.6
#2  0x00007ffff7110d37 in __assert_fail_base () from /usr/lib64/libc.so.6
#3  0x00007ffff7110de2 in __assert_fail () from /usr/lib64/libc.so.6
#4  0x00007fff3d51c483 in llvm::ArrayRef<mlir::Value>::operator[] (Index=<optimized out>, this=<optimized out>) at /root/.triton/llvm/llvm+mlir-17.0.0-x86_64-linux-gnu-ubuntu-18.04-release/include/llvm/ADT/ArrayRef.h:254
#5  0x00007fff3d52353f in llvm::ArrayRef<mlir::Value>::operator[] (Index=<optimized out>, this=<optimized out>) at /root/.triton/llvm/llvm+mlir-17.0.0-x86_64-linux-gnu-ubuntu-18.04-release/include/llvm/ADT/ArrayRef.h:254
#6  operator() (__closure=<optimized out>, __closure=<optimized out>, idx=<optimized out>) at /root/python/torch-triton/lib/Conversion/TritonGPUToLLVM/ConvertLayoutOpToLLVM/SharedToDotOperandMMAv2.cpp:299
#7  MMA16816SmemLoader::loadX4 (this=this@entry=0x7fffffff56e0, mat0=<optimized out>, mat1=<optimized out>, ptrs=..., matTy=..., shemPtrTy=...) at /root/python/torch-triton/lib/Conversion/TritonGPUToLLVM/ConvertLayoutOpToLLVM/SharedToDotOperandMMAv2.cpp:300
#8  0x00007fff3d525d76 in operator() (__closure=0x6517f290, a=0, b=2) at /root/.triton/llvm/llvm+mlir-17.0.0-x86_64-linux-gnu-ubuntu-18.04-release/include/llvm/ADT/SmallVector.h:291
#9  0x00007fff3d52271a in std::function<void (int, int)>::operator()(int, int) const (__args#1=2, __args#0=0, this=0x7fffffff5950) at /usr/include/c++/10/bits/std_function.h:617
#10 loadArg (rewriter=..., loc=..., tensor=..., encoding=..., smemObj=..., typeConverter=0x7fffffff76f0, thread=..., isA=true) at /root/python/torch-triton/lib/Conversion/TritonGPUToLLVM/ConvertLayoutOpToLLVM/SharedToDotOperandMMAv2.cpp:616
#11 0x00007fff3d4864bf in ConvertLayoutOpConversion::lowerSharedToDotOperandMMA (this=this@entry=0x64eb9a50, op=..., adaptor=..., rewriter=..., mmaLayout=..., dotOperandLayout=..., isOuter=isOuter@entry=false)

I'm seeing similar issues here with uint8 → float16 (or float32). Using nightly with an A6000. The application is quantized matrix multiplication. I've found that basically only a block size of 16 works, while anything larger results in either "Invalid index!" for f16 or "unimplemented code path" and segfault for f32.
Just curious if we have an idea of what's causing this, or if there are any hints for choosing the right block size. Thanks!


Add best-effort warning header

Fixes #118.
Because we modify index.hbs, we should occasionally compare with
mdbook's own copy and rebase our changes onto it.

cc @steveklabnik Since this relates to my PR on TRPL, and you merged that.

Thanks!
I will get to your other PRs either later today or tomorrow :)


Add kextex and kextex_all script references to setup.py

I believe this was already covered in #52, but thanks for the PR!


Type mismatch: inferred type is String? but String was expected | Build failed

Hello team,
I recently upgraded Flutter to v3.3.2 and as soon as I started building my project, This error popped up :
/Users/divyamdhadwal/Documents/flutter/.pub-cache/hosted/pub.dartlang.org/truecaller_sdk-0.1.1/android/src/main/kotlin/com/truecallersdk/TruecallerSdkPlugin.kt: (146, 38): Type mismatch: inferred type is String? but String was expected

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':truecaller_sdk:compileDebugKotlin'.
> Compilation error. See log for more details

Seems like an issue within the TruecallerSdkPlugin.kt file on line 146 (REQUEST_VERIFICATION block). Any idea how to resolve this ? Our app relies heavily on Truecaller for day to day auth needs.
I'm using  Truecaller plugin's v0.1.1 (latest). Any help would be really appreciated.

Hi @divyamdhz
We acknowledge the issue reported above, and a fix is already in progress. This would be fixed in the next version of Truecaller SDK flutter plugin that we'll release soon post testing.
We did some investigation at our end and were able to conclude that the SDK implementation caused the issue mentioned when the flutter stable channel was greater than 2.8.0. Therefore, as an interim solution you can also downgrade the flutter version to 2.8.0 or wait for the next release which ideally should be live by later half of the next week.
Please feel free to write to us on our support channel(https://developer.truecaller.com/contact) for faster and dedicated response.

Facing the same issue. Please release a fix.

Hello team,
I recently upgraded Flutter to v3.3.2 and as soon as I started building my project, This error popped up :
/Users/divyamdhadwal/Documents/flutter/.pub-cache/hosted/pub.dartlang.org/truecaller_sdk-0.1.1/android/src/main/kotlin/com/truecallersdk/TruecallerSdkPlugin.kt: (146, 38): Type mismatch: inferred type is String? but String was expected

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':truecaller_sdk:compileDebugKotlin'.
> Compilation error. See log for more details

Seems like an issue within the TruecallerSdkPlugin.kt file on line 146 (REQUEST_VERIFICATION block). Any idea how to resolve this ? Our app relies heavily on Truecaller for day to day auth needs.
I'm using Truecaller plugin's v0.1.1 (latest). Any help would be really appreciated.

well in my case i have gone to this path "~/.pub-cache/hosted/pub.dartlang.org/truecaller_sdk-0.1.1/android/src/main/kotlin/com/truecallersdk/TruecallerSdkPlugin.kt" and commented line no 146 and it worked, it was asking for a safe cast, what I did was just a hack but will do for now.


Driver installer hangs

Does not complete and just hangs, acts like the KMDF portion has a dialog that is waiting for user response but since it is installed in the background with no UI the dialog is inaccessible and the window just dings when attempting to click anything on the installer window (as applications do when there is a child "dialog" window open).
Windows 10 14393

I get such complaints from time to time.
Please search my Forum for similar cases.


android [nfc]: Factor TextCompression logic out of React module

This will let us straightforwardly re-use this logic from within the Android-side code, for #5117 .
We start by converting the file to Kotlin and making some cleanups that that enables. Then the last commit is the payload.

LGTM, thanks! Merged.


Run lint-staged for FE apps

Run lint-staged for frontend apps

Tested in https://github.com/MOV-AI/frontend-npm-launcher/actions/runs/11436455055/job/31814444276

Lots of console errors on the test though, see https://github.com/MOV-AI/frontend-npm-launcher/actions/runs/11436455055/job/31814444276#step:17:12

The tests need a review :disappointed:


Create Data Api endpoint

I had a lot of issues with other APIs but fixed everything and tested all the API.
Merged to main.
@jordanzyx  - close issue 78


Invalid frequency passed to pd.Grouper

Invalid frequency string passed to pd.Grouper.
The group_yearly_df function passes either "YE" or "Y" to pd.Grouper, depending on the pandas version. Using pandas version '2.1.4', the frequency passed to pd.Grouper is invalid. The correct string is "Y", but "YE" is passed.
The following error was caused:

File ~/.local/lib/python3.10/site-packages/spei/utils.py:85, in group_yearly_df(series)
83 grs = {}
84 freq = "YE" if pd_version >= "2.1.0" else "Y"
---> 85 for year_timestamp, gry in series.groupby(Grouper(freq=freq)):
86     index = validate_index(gry.index)
87     gry.index = to_datetime(
88         "2000-" + index.strftime(strfstr), format="%Y-" + strfstr
89     )
File /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.10/lib/python3.10/site-packages/pandas/core/resample.py:2046, in TimeGrouper.init(self, freq, closed, label, how, axis, fill_method, limit, kind, convention, origin, offset, group_keys, **kwargs)
2043 if convention not in {None, "start", "end", "e", "s"}:
2044     raise ValueError(f"Unsupported value {convention} for convention")
-> 2046 freq = to_offset(freq)
2048 end_types = {"M", "A", "Q", "BM", "BA", "BQ", "W"}
2049 rule = freq.rule_code
File offsets.pyx:4460, in pandas._libs.tslibs.offsets.to_offset()
File offsets.pyx:4557, in pandas._libs.tslibs.offsets.to_offset()
ValueError: Invalid frequency: YE

Hi @mathilde-ritman,
I pushed bdba99c to the dev branch. Does that solve your issue?

Thanks! Works great.


After-job hook

This PR adds a function to the pool that gets executed each time a job is finished. Such function can be useful for monitoring.
I thought about adding a full-fledged middleware chains first, but it would add a lot of complexity without too many benefits. This implementation gets a function which can accept a Job via (Pool).SetAfterFunc() and runs it each time a job is finished regardless of its status, passing the job to the function.
Also, I moved the job.finished assgnment before the error checks (worker.go:85); this way the timings are saved even when the job returned an error.
(resubmitted, previous PR: #29)

@albrow didn't know about that trick. Neat!
Yeah, that's probably not needed then.

Looks great. I'm going to merge this and include it in a new release, hopefully tonight. I'll likely make this 0.4.0 and push back features planned for 0.4.0 to 0.5.0.


Empties transform doesn't get saved in tours

Empties (nodes with no "content" but that might have children; eg. the "Lights" node).
I think the relevant code is in CVSnaphots.ts:66,114: Only models and lights gets added. It's more or less a feature given that the features doesn't specify transform as a save target but the more specific lightsand models. However it would be practical to be able to move a bunch of nodes together in a tour.
Regarding a possible fix, aside from finding a way to map this back to the features array, I couldn't get a sense of how to filter the CVNode instances accordingly.

After digging in a bit, I came with a prototype:
@@ -79,16 +86,24 @@ export default class CVSnapshots extends CTweenMachine

         const models = this.getGraphComponents(CVModel2);
         models.forEach(model => {
-            this.updateComponentTarget(model.transform, !!features["models"]);
             this.updateComponentTarget(model, !!features["models"]);
         });

         const lights = this.getGraphComponents(CLight);
         lights.forEach(light => {
-            this.updateComponentTarget(light.transform, !!features["lights"]);
             this.updateComponentTarget(light, !!features["lights"])
         });

+
+        const transforms = this.getGraphComponents(CVNode);
+        for(let transform of transforms){
+            let node = transform.node
+            if(!isNVNode(node)) continue;
+            if(node.camera) continue;
+
+            this.updateComponentTarget(transform, !!features["transforms"]);
+        }
+

The downside is: this introduces another tour "feature" users have to keep in mind, which I very much dislike. Especially since the concept of "transform" being a thing of its own, more or less separated from the underlying data object, while sound from a technical perspective (everybody does it internally) is not intuitive at all for non-technical users.
Leaving this here for later reference but I think I might be able to propose something better in the near future : I'm trying to optimize how the states array gets stored because it gets stupidly large on scenes with multiple objects, a large features set and many tour steps, which is very much the types of scenes I am working on.


always require login when using subdomain

I start my reader servise on port 5833
in nginx conf, I mapping xxx.xxx.com to <IP_ADDRESS>:5833
I can do everything using ip_address:5833
but when i using xxx.xxx.com to add feed, always require login .
how to solve this ?
I'm running in my vps

more :    when i using subdomain to delete one feed, it's warning: something broke!

more info， I setup stringes using ip:port , then configure nginx proxy set
server {
listen 80;
server_name xxx.xxx.com;
location / {
proxy_pass http://<IP_ADDRESS>:5833;
}
}

I ran into this problem as well, @ericyue. I found that adding proxy_set_header Host $host; fixed this.
server {
        listen 80;
        server_name xxx.xxx.com;
        location / {
                proxy_pass http://<IP_ADDRESS>:5833/;
                proxy_set_header Host $host;
        }
}

Hope that helps!

thanks malantonio! It works!


There are summaries that summarize too many ops, indicating we are not summarizing often enough

I do not remember if we use default settings, or SPO actually overwrites these with other values.
I'll assume we use these defaults. Even if we do not (and the issue is in settings, not actual behavior), the following results suggest such settings needs to be changed.
First, default settings:
// Consider idle 5s of no activity. And snapshot if a minute has gone by with no snapshot.
const IdleDetectionTime = 5000;

const DefaultSummaryConfiguration: ISummaryConfiguration = {
    idleTime: IdleDetectionTime,

    maxTime: IdleDetectionTime * 12,

    // Snapshot if 1000 ops received since last snapshot.
    maxOps: 1000,

    // Wait 2 minutes for summary ack
    // this is less than maxSummarizeAckWaitTime
    // the min of the two will be chosen
    maxAckWaitTime: 120000,
};

Query:

union Office_Fluid_FluidRuntime_*
| where Data_eventName contains "fluid:telemetry:Summarizer:Running:Summarize_generate"
| where Data_opsSinceLastSummary > 200
| where Data_timeSinceLastAttempt > 120000
| extend First = Data_summarizeCount == 1
| summarize count() by Data_eventName, Data_summarizeReason, First

Data:

Data_eventName
Data_summarizeReason
First
count_

fluid:telemetry:Summarizer:Running:Summarize_generate
maxTime
true
2,297

fluid:telemetry:Summarizer:Running:Summarize_generate
maxTime
false
37

fluid:telemetry:Summarizer:Running:Summarize_generate
idle
true
22

fluid:telemetry:Summarizer:Running:Summarize_generate
idle
false
17

fluid:telemetry:Summarizer:Running:Summarize_generate
maxOps
true
2

My intuition says - we should not see that many hits (ideally zero). I.e. we should summarize at 1 minute mark.
There will be some hits due to races - that's expected. But extra minute - it's a lot! We should look at couple sessions closer.
But the main point of this bug - maxOps = 1000 is too big of a number, in a world where it affects so much perf and functional characteristics of our products.
I think we have to change it to be below current throttling limits (even though that's mostly ODSP specific).
Currently, clients would get throttled at 1000 ops / min.
I'd personally move it to 100-250 as a starter, and observe.
At the moment, there are very few summaries that hit "maxOps" reason, so we should test how low we can go before it starts impacting number of such summaries in significant way.

Another angle:

union Office_Fluid_FluidRuntime_*
| where Data_eventName contains "fluid:telemetry:Summarizer:Running:Summarize_generate"
| extend First = Data_summarizeCount == 1
| summarize count(), Seconds=toint(avg(Data_timeSinceLastAttempt) / 1000), toint(avg(Data_opsSinceLastSummary))
by Data_summarizeReason, First

Data_summarizeReason
First
count_
Seconds
avg_Data_opsSinceLastSummary

maxTime
false
1,014,729
254
19

idle
false
440,739
36
49

idle
true
832,518
6
18

maxTime
true
259,471
161,346
48

lastSummary
false
145
40
121

maxOps
false
897
28
1,123

maxOps
true
641
-25
1,201

lastSummary
true
414
20
154

Notice that maxTime does not really work - we have 254 seconds for First = false, and 161K seconds for First = true.
The former should be accurate (it measures time between summaries from same client), and thus it should be below 60 seconds, right?
That said, First = false is not important (other than it seems like it does not work how we think it works), as average number of summarized ops is small. But First = true, we have 48 ops on average, and that's more concerning.

And this is what matters the most - 95th percentile:

union Office_Fluid_FluidRuntime_*
| where Data_eventName contains "fluid:telemetry:Summarizer:Running:Summarize_generate"
| extend First = Data_summarizeCount == 1
| summarize count(), Seconds=toint(percentile(Data_timeSinceLastAttempt, 95) / 1000), toint(percentile(Data_opsSinceLastSummary, 95))
by Data_summarizeReason, First

Data_summarizeReason
First
count_
Seconds
percentile_Data_opsSinceLastSummary_95

idle
false
440,783
68
166

idle
true
832,588
38
67

maxTime
true
259,494
600,658
208

maxTime
false
1,014,807
303
149

lastSummary
true
414
55
357

maxOps
true
641
55
1,927

lastSummary
false
145
63
244

maxOps
false
897
56
1,598

I'd say that every line where last column (ops) is bigger than 100 is a concern. And the higher in the table (more hits) - the more important it is.
reason = "maxTime" seems like big problem, but it's a problem because we have high maxOps value - once it is reduced, we will start summarizing sooner.

I started to look at this data and I suppose we need to understand why we have so many elevated values for Data_timeSinceLastAttempt  - (specially where First == 1)  - that seems to be affecting the analysis.  That is a bug that needs to be fixed asap and I will investigate it.
For now, I'm trying this query assuming that if the Data_timeSinceLastAttempt when  First == 1 is greater than the difference between the first load event for that docId, then it is wrong and I'm forcing it to be the difference between the 2 timestamps instead. I need to do some more validation but wanted to hear your feedback:
let toUnixTimeMS = (dt:datetime)
{
((dt - datetime(1970-01-01)) / 1s ) * 1000
};
Office_Fluid_FluidRuntime_Performance
| where Data_eventName contains "fluid:telemetry:Summarizer:Running:Summarize_generate"
| where Data_timeSinceLastAttempt > 0
| extend First = Data_summarizeCount == 1
| project Event_Time, Data_eventName, Data_timeSinceLastAttempt, Data_fetchLatest,  Data_opsSinceLastSummary, Data_loaderId,Data_summarizeReason,First, Data_docId
|lookup kind = inner (Office_Fluid_FluidRuntime_Performance
| where Data_eventName contains "fluid:telemetry:Container:Load_start" // not sure if this is the best event but should be enough
| where Data_clientType == "interactive"
| project Event_Time, Data_loaderId, Data_docId
| summarize arg_min(Event_Time, *) by Data_docId
| order by Event_Time
) on Data_docId
| where Event_Time1 < Event_Time
| project Event_Time, Event_Time1, Data_timeSinceLastAttempt, Data_fetchLatest,  Data_opsSinceLastSummary, Data_loaderId, Data_summarizeReason, Data_docId, First
| extend DiffBetweenEvents = round(toUnixTimeMS(Event_Time) - toUnixTimeMS(Event_Time1))
| extend ActualTimeSinceLastAttempt = case( (First == 1 and (Data_timeSinceLastAttempt >  DiffBetweenEvents)), DiffBetweenEvents, Data_timeSinceLastAttempt)
| summarize count(), Seconds=toint(avg(ActualTimeSinceLastAttempt) / 1000), toint(avg(Data_opsSinceLastSummary))
by Data_summarizeReason, First

Data_summarizeReason
First
count_
Seconds
avg_Data_opsSinceLastSummary

maxTime
1
244246
78118
39

maxTime
0
916819
259
16

maxOps
1
390
28
1234

maxOps
0
823
27
1124

lastSummary
0
105
40
117

lastSummary
1
274
5
156

idle
0
387937
36
47

idle
1
712802
23
16

	</HTML>

Maybe we should split this issue in two? I think data suggest that we should change maxOps.
Investigating other things may be lower priority.

Looking at more telemetry, I think we should go as low as maxOps = 100.
There are various ways to look at summaries (total count, number of last summaries), but across the board that looks like will result in 10% summaries, which is not that large impact, but at the same time will substantially move P95 across the board. Having maxOps = 200 impacts almost 5X less summaries, so impact of such change would be small

@vladsud   I started looking at the code and it seems we get the MaxOps from the server.
connection = await docService.connectToDeltaStream(this.client);
async connectCore(connectionMode?: ConnectionMode): Promise {(connectionManager.ts:463)
connect(connectionMode?: ConnectionMode) { (connectionManager.ts:408)
this.connectionManager.connect(args.mode); (deltaManager.ts:436)
connectToDeltaStream(args: IConnectionArgs) { (container.ts:1096)
load (container.ts:1134)
await in load (async)
...
load (container.ts:248)
loadContainer (loader.ts:490)
resolveCore (loader.ts:426)
...
Basically our local settings get overridden by the server.
private static detailsFromConnection(connection: IDocumentDeltaConnection): IConnectionDetails {
return {
claims: connection.claims,
clientId: connection.clientId,
existing: connection.existing,
checkpointSequenceNumber: connection.checkpointSequenceNumber,
get initialClients() { return connection.initialClients; },
mode: connection.mode,
serviceConfiguration: connection.serviceConfiguration,
version: connection.version,
};
}

private get summaryConfiguration() {
return {
// the defaults
... DefaultSummaryConfiguration,
// the server provided values
... this.context?.serviceConfiguration?.summary,
// the runtime configuration overrides
... this.runtimeOptions.summaryOptions?.summaryConfigOverrides,
};
}
Tony did some changes to allow the override from the runtimeOptions, mainly for testing purposes.
We could ignore the server settings but I don't suppose it is the right solution.  How should we proceed ?  I will ask Marcelo just in case.

Right, see https://github.com/microsoft/FluidFramework/issues/9948.
I've chatted offline with Gary and he was not against the idea.
I believe settings will be very specific to worklaods, so there is no place for centralized settings.
I.e. I expect even in "document space" they will differ, i.e. Whiteboard vs. Chapter1 will use different settings.
I think service should clearly articulate only higher level goals, like no more than 10K unsummarized ops (very important setting that is active, but not exposed to client in any way), but not describe small details how client operates.

So, for this specific bug, should we take o Dr and dictate ourselves the max ops ? Runtime can still override …
Btw - I had chatted with Marcelo and he filed:
https://onedrive.visualstudio.com/SPIN/_workitems/edit/1336325/
Should we hold it off ?

@NicholasCouri , I believe you have PR to change maxOps to 100.
Please either get it in (and close this issue), or port it to ADO.
Thanks!

@vladsud   done - There was a PR from Kian to decrease it to 100 - that is why I had hold off on this one - it seems his PR never went through.


modern elevators

Before reporting, I have confirmed that

[ ] This bug does not appear to be reported on GitHub Issues before.
[X] This bug still exists in the latest version of the Minecraft Transit Railway mod.
[X] I have tested this in the official version of Minecraft Transit Railway, not a modified, custom, or unofficial build.

Describe the Bug
При блокировке дверей лифта на ней появляется отсутствующая текстура

Reproduction Steps
Блокировка дверей лифта кистью
In-game Log and Crash Report
2024-09-25-1.log.gz
Mod Loader
Fabric
Minecraft Version
1.16.5
Minecraft Transit Railway Version
3.1.6
Operating System
win 11
List of Mods or Modpack You Were Using

Extra Information
No response

Modern Elevator is no longer supported, please install Minecraft Transit Railway 3.2.2 which is fully compatible with Modern Elevator, and should have the issue fixed.

Minecraft Transit Railway

Могут ли произойти проблемы с моим миром?


Tooltip content is not visible from top

If I am clicking on very first element of the page and tooltip content is more then it is opening at top & it's top content is hiding inside of window.
Please refer below link, and click on first anchor link 'Heading with a tooltip'
http://jsfiddle.net/54h4eh4g/28/
You will see top content of tooltip is not visible.
Thanks

Hi, it's an old version you're using. Try Tooltipster v4. If you want a CDN, it's on jsDelivr (http://www.jsdelivr.com/projects/jquery.tooltipster).

Can you please provide me unminified version file link, actually updated version is not working with my code base so I need to compare both files.

Pull from the repo for unminifed files or use those of the documentation.
There is a migration guide, check the open issues.

I'll close this, feel free to open a new issue if you think there is something wrong with v4 after you finish the upgrade. Thanks


Save Test State

Enables us to save and recover from a saved test state.

Codecov Report

Merging #640 (2c005e0) into main (19b89c3) will decrease coverage by 0.55%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main     #640      +/-   ##
==========================================
- Coverage   33.24%   32.69%   -0.55%
==========================================
  Files           6        6
  Lines         358      364       +6
==========================================
  Hits          119      119
- Misses        229      235       +6
  Partials       10       10

Impacted Files
Coverage Δ

client/mockserver.go
31.48% <0.00%> (-1.86%)
:arrow_down:


shouldn't the functions be displayed?

Hi, I just discovered this, it looks super useful!
However,  I navigated through https://spinkney.github.io/helpful_stan_functions/, but the functions are never displayed, just the name and the description. Shouldn't the Stan code be displayed?

Maybe? Normally the documentation doesn't contain the function internals. Think of R documentation or even Stan.
I can see how you'd want this to learn though. However, I hope that the examples and repo is enough. I'm using the documentation more for the math and references component.

ok, I see, but a link to the "source" would be very useful at least. Otherwise one needs to navigate back and forth between the website and the github repo.

Yea, that makes sense and would be good to have 👍

Luckily, this is going to be easy.
Here's one example: https://spinkney.github.io/helpful_stan_functions/group__logical__array.html.
I plan to add the link to the stan file and example(s). If there are multiple languages then they'll each get a link in the documentation.

great!! thank you!

I found out that it's easy to include the source code. All the files have been updated. This is better in case I ever decide to reorganize the structure or if the github links change.