GlusterFS PV failed after kubelet restart

Hello,
we have created a Cluster with Rancher (infos below) and installed a GlusterFS Cluster with heketi described here (https://github.com/heketi/heketi/wiki/Kubernetes-Integration/832a65e365b4644a1c64ac47601893a3fdb52daf)

Three GluterFS Servers inside the same K8S Cluster
Heketi as Provisioner

When upgrading Rancher from 2.1.3 to 2.1.8 all kubelet processes are restarted, after the process restart pods beginning to fail because they could not read/write to their existing Gluster volume mounts.
It looks like when restarting kubelet all existing mounts are getting terminated, which is OK if a single kubelet process restart on one GlusterFS Node occurred, but when on all three GlusterFS Nodes the process is restarted in a short period of time (which happened on a rancher update/or cluster upgrade), all clients with a PVC will fail.
We could reproduce the issue when we tried to add an additional kubelet parameter to our cluster.yaml, which also forces a kubelet restart on all nodes.
The kubelet log show errors like this:
32067 kubelet.go:1616] Unable to mount volumes for pod \"XXXXXX(c3751cf0-51c9-11e9-9d82-005056897a94)\": timeout expired waiting for volumes to attach or mount for pod \"XXX\"/\"XXXX\". list of unmounted volumes=[volume]. list of unattached volumes=[volume default-token-qlvbv]; skipping pod\n","stream":"stderr","time":"2019-04-02T06:16:17.54248644Z"}

{"log":"E0402 06:08:09.838966   32067 nestedpendingoperations.go:267] Operation for \"\\\"kubernetes.io/glusterfs/c3751cf0-51c9-11e9-9d82-005056897a94-pvc-e62d65b3-ed76-11e8-98e8-00508-005056897a94: transport endpoint is not connected\"\n","stream":"stderr","time":"2019-04-02T06:08:09.839246077Z"}

What kind of request is this (question/bug/enhancement/feature request):
question/bug
Steps to reproduce (least amount of steps as possible):
Result:
Other details that may be helpful:
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI):
Rancher: v2.1.8 UI: v2.1.21
Installation option (single install/HA):
single

Cluster information

Cluster type (Hosted/Infrastructure Provider/Custom/Imported):
Provider Custom v1.11.3-rancher1-1
Network Provider:
Canal
Machine type (cloud/VM/metal) and specifications (CPU/memory):
VM (CentOS 7.5.1804 3.10.0-862.14.4.el7.x86_64)
Kubernetes version (use kubectl version):

Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

Docker version (use docker version):

Client:
 Version:       17.12.1-ce
 API version:   1.35
 Go version:    go1.9.4
 Git commit:    7390fc6
 Built: Tue Feb 27 22:15:20 2018
 OS/Arch:       linux/amd64

Server:
 Engine:
  Version:      17.12.1-ce
  API version:  1.35 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   7390fc6
  Built:        Tue Feb 27 22:17:54 2018
  OS/Arch:      linux/amd64
  Experimental: false

Cluster Yaml:
addon_job_timeout: 30
authentication:
  strategy: "x509"
bastion_host:
  ssh_agent_auth: false
ignore_docker_version: true
#
#   # Currently only nginx ingress provider is supported.
#   # To disable ingress controller, set `provider: none`
#   # To enable ingress on specific nodes, use the node_selector, eg:
#      provider: nginx
#      node_selector:
#        app: ingress
#
ingress:
  provider: "none"
kubernetes_version: "v1.11.3-rancher1-1"
monitoring:
  provider: "metrics-server"
#
#   # If you are using calico on AWS
#
#      network:
#        plugin: calico
#        calico_network_provider:
#          cloud_provider: aws
#
#   # To specify flannel interface
#
#      network:
#        plugin: flannel
#        flannel_network_provider:
#          iface: eth1
#
#   # To specify flannel interface for canal plugin
#
#      network:
#        plugin: canal
#        canal_network_provider:
#          iface: eth1
#
network:
  options:
    flannel_backend_type: "vxlan"
  plugin: "canal"
#
#      services:
#        kube_api:
#          service_cluster_ip_range: <IP_ADDRESS>/16
#        kube_controller:
#          cluster_cidr: <IP_ADDRESS>/16
#          service_cluster_ip_range: <IP_ADDRESS>/16
#        kubelet:
#          cluster_domain: cluster.local
#          cluster_dns_server: <IP_ADDRESS>
#
services:
  etcd:
    creation: "12h"
    extra_args:
      election-timeout: "5000"
      heartbeat-interval: "500"
    retention: "72h"
    snapshot: false
  kube-api:
    pod_security_policy: false
    service_node_port_range: "30000-32767"
  kubelet:
    fail_swap_on: false
ssh_agent_auth: false

I have encountered the same issue, today and every time I upgraded Kubernetes, though I don't use Heketi. This time it was Rancher 2.2.2 (single), after upgrading cluster from k8s 1.11.6 to 1.13.5.

We are facing the same issue and have no solution or workaround yet. Our environment is:

Rancher 2.2.3 HA
Rancher launched cluster with Kubernetes v1.13.5
Docker 18.6.3
Gluster Server 4.1.8 (tested with version 3.8.15 too to have a similar version to the gluster client in kubelet 3.8.8, with no success)
Heketi 8.0.0

The problem is reproducible by creating a PVC, backed by the gluster storage class, mounting the volume into a pod and restarting kubelet ( docker restart kubelet ) on the respective node.
This causes the glusterfs mount to become stale and with this the pod can no longer be removed by normal means.
A force deletion (--force --grace-period=0) of the old pod is necessary to allow the workload to be redeployed.
The problem not only occurs on kubernetes updates, but at seemingly random times during the runtime of the cluster. Probably because kubelet is restarted at certain occassions. I do not have enough knowledge of the internal works of rancher, maybe this restarting of kubelet is intended anyway.
We have a suspicion that with the restart, the kubelet container is killed too fast and not given enough time to clean things up. However, we are not sure how to test this theory by giving the kubelet more time.

Are there any updates on this issue, which is open for two months now? If this is not a rancher related issue (what I don't believe as more users are having the same problems) I would move on and migrate to another storage provider.
Any kind of help/reaction would be beneficial.

Same issue with:

Rancher 2.2.7 HA
Rancher launched cluster with Kubernetes v1.14.5
Imported cluster with Kubernetes v1.14.5
Docker 18.6.3
Gluster Server 4.1.8
Heketi 8.0.0

Since we do have a support contract I asked for help with this issue. Seems a node restart will help as a workaround for all workloads. I also asked if GlusterFS/Heketi should be the way to go with shared volumes. Update follows as soon as I got news on this.

@vourd Thank you for this update... Hope they find out something or could provide a real workaround, cause rebooting the machines did not help in a few cases. Some Volumes go in a kind of error state which could only be resolved by restarting them in gluster. This is really a pain in ....... ;)

I can relate. NFS seems to be unaffected by this issue. Discussing now with support pros/cons for NFS vs GlusterFS/Heketi.

@vourd Thanks again for this update. To be honest I'm would be really disappointed if they do not find any workaround this issue because Gluster is a really nice and performant concept and used by a lot of people/companies. We extra moved from fixed gluster servers into k8s to get the flexibility which k8s is providing and the failover mechanism which gluster supports out of the box.

@vourd have you had a chance to solve this with the support?

Unfortunately not. We're evaluating to switch to NFS, since it's only our shared data store. For fast/big files we're using a different storage provider.

@janeczku Could this be enabled by default?

@janeczku Will try that... Really really appreciate your help!

We face the same issue with gluster mounts and the extra bind doesnt seem to fix this.
When restarting the kubelet we still see this on our prometheus pod:

E0926 13:37:24.929715   21838 nestedpendingoperations.go:267] Operation for ""kubernetes.io/glusterfs/d817a129-e059-11e9-8ca7-005056b8b600-pvc-prometheus-operator-prometheus-0" ("d817a129-e059-11e9-8ca7-005056b8b600")" failed. No retries permitted until 2019-09-26 13:39:26.929653783 +0000 UTC m=+3677.172965618 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume "pvc-prometheus-operator-prometheus-0" (UniqueName: "kubernetes.io/glusterfs/d817a129-e059-11e9-8ca7-005056b8b600-pvc-prometheus-operator-prometheus-0") pod "prometheus-prometheus-operator-prometheus-0" (UID: "d817a129-e059-11e9-8ca7-005056b8b600") : stat /var/lib/kubelet/pods/d817a129-e059-11e9-8ca7-005056b8b600/volumes/kubernetes.io~glusterfs/pvc-prometheus-operator-prometheus-0: transport endpoint is not connected"

Redeploying the pod does the trick.

I have just upgraded Rancher to 2.2.9 and then Kubernetes to 1.15.5, and - having this extra bind before the upgrade - for the first time the issue did NOT happen :)
Now, I can't prove it's thanks to this workaround, or just some improvement in Rancher, Kubernetes or Gluster (which was upgraded from 4.0 to 6.5 prior to Rancher upgrade), so it would be great if everyone shared their experiences with this hack.

We upgraded our sandbox Rancher to version 2.2.9 and that alone didn't fix the issue. After the upgrade to k8s 1.15.5 the volume remained mounted after a kubelet restart. We'll test this further on some other clusters but at first sight it seems to be working.
For completion: We're running gluster 3.12

Thanks for sharing the outcome, please include OS (cat /etc/os-release) and kernel (uname -r) when sharing.

@santascsi Have you had this extra bind configured before the upgrade? Or just the upgrade alone somehow did the trick?

@adampl I had to configure the extra bind. I removed the bind and upgraded to 1.15.5 but the issue resurfaced once I restarted a kubelet. I then added the bind again and no longer experienced the issue after a kubelet restart.
As far as I can see, the bind + 1.15.5 fixed the issue. I'm still waiting for a go to upgrade our other clusters and see whether this sticks.

Any information that people get from testing, please share. I have run some tests on a cluster with this bind and adding some hosts to it, and the results are:

Ubuntu 16 and CentOS7 that have the binary will trigger correct behavior
Ubuntu 18's binary is too new and won't work in the hyperkube image due to library issues, it will try to run the binary on start and fail, not triggering correct behavior
Nodes without the binary will get a directory created for the mounted binary (default Docker behavior)

Occurrences of this issue that I've found are almost all related to glusterfs, and there doesn't seem any traction on the issues reported upstream. Based on the smaller scope of this issue and the "success" rate of adding the default bindings, I would recommend to add this to the documentation so users can read up on the limitations in this area.
I'd love to hear opinions if you don't agree or if you have other ideas.

I agree on the documentation.
As long as there is no definitive fix, users should be made aware of the limitations of this setup before it's put into production. It will lead to better visibility to those who encounter the issue or mitigate it entirely when the user checks the docs first.

https://rancher.com/docs/rancher/v2.x/en/cluster-admin/volumes-and-storage/#glusterfs-volumes-with-rancher-launched-kubernetes-clusters

Updating the docs doesn't solve the issue, it only makes people aware of it, so I suggest to reopen it.

This has not been resolved. Why close it? @deniseschannon


low mIOU in train with coco and ResNet as backbone

hi, thank you for your code, I trained the network use coco dataset and ResNet as backbone with 40 epoches, the problem is that Val mIOUs are same value and low (about 0.03x). I just run train_coco.py, are there any wrong on my work? Thanks.

I met the same problem， do you have any idea?

I believe the issue is with the way images are padded in the RandomScaleCrop data augmentation. As is, the code pads with 0s, but 0 is one of the MS COCO classes evaluated on. You should pad with say 255, and ignore 255 when calculating the cross entropy loss. Also, I believe the starting learning rate of 0.01 is too high - try 0.001. Hope this helps

@dfan  Thanks for your plan, I also encountered the same problem, can you provide more detailed ?


Plugin components face undesired mounting when parent state changes or there is a rerender

Description
When a plugin component is rendered it faces unwanted mounting whenever there is a rerender caused by its parent component
Impact
The plugin component is in the middle of a state change and suddenly it gets unmounted and remounted with all the state lost
This I believe is occurring because we use plugin components as inline components in our component render cycle and each time there is a rerender the inline component function signature changes and because of reacts shallow comparison on function(which are objects) the component gets unmounted completely
One solution that could prevent this would be we could use useMemo to render the plugin components this way we prevent the unnecessary rerenders as the component will be computed only when the dependency array things change
But according to the docs on useMemo it says you should not completely rely on useMemo your component should work without the need for it and useMemo should be used only when you are trying to optimize, and thus a solution without useMemo is something we need right now.
The other thing is according to redux docs we should not store components in our store but we have to store it because redux store is the only way our plugin can communicate with the app world, the question is can there be some alternate approach through which we can prevent this?
http://web.archive.org/web/20150419023006/http://facebook.github.io/react/docs/interactivity-and-dynamic-uis.html#what-shouldnt-go-in-state

cc: @joaquimrocha @illume

@joaquimrocha we still need to look for a concrete solution to prevent the unnecessary rendering

I think we need to change the way we do plugin registration.
As @ashu8912 mentions, we should not store functions inside the redux store.
Components in react have a life cycle, and (un)mount and renders and re-renders are something that should be allowed to happen. It's good that we don't re-render and un(mount) unnecessarily, but components should work despite that. Plugins should really use the react component life cycle.


Connection lost without any trigger

I am using two different simulators at the same time once the pusher is initialized and subscribe to a private channel it works perfectly  but after one minute or more it stops listing without any trigger and sometimes I get an error and after some time it auto-reconnect what causes the problem and how I can resolve this
onError: Pong reply not received code: 4201 exception: <PusherSwift.PusherError: 0x600000d73db0>
first, it will work perfectly and after 30 or 40 seconds it stops listing for new events and does not give any state change event

but after some time I got a new console like

it automatically connects and disconnect again and again until I reset pusher

Here is my configuration code for the pusher
export const Connect = async () => {

 return await pusher.init({
    apiKey: PUSHER_API_KEY,
    cluster: PUSHER_CLUSTER,
    useTLS: false,
    onConnectionStateChange,
    onAuthorizer: onAuthorizer,
    onError,
    onEvent,
    onSubscriptionSucceeded,
    onSubscriptionError,
    onSubscriptionCount,
    onDecryptionFailure,
    onMemberAdded,
    onMemberRemoved,
    maxReconnectionAttempts: 100,
    maxReconnectGapInSeconds: 0.01,
  })
}

 const initPusher = async () => {
    if (pusher.channels.size > 0) {
      const promises: any[] = [];

      pusher.channels.forEach(({channelName}) =>
        promises.push(pusher.unsubscribe({channelName})),
      );

      await Promise.allSettled(promises);
    }
    try {
      await Connect();
      if (pusher?.connectionState !== 'CONNECTED') {
        await pusher.subscribe({
          channelName: `private-chatify.${current_user_id}`,
          onEvent: (event: PusherEvent) => {
            console.log(event.eventName, event);
            if (event?.eventName === 'messaging') {
              setMessages(pre => [JSON.parse(event.data)?.api_message, ...pre]);
            }
          },
        });
        await pusher.connect();
      }
    } catch (error) {
      console.log(error);
    }
  };

useFocusEffect(
    useCallback(() => {
     initPusher();
   }, []),
  );

@pusher/pusher-websocket-react-native": "^1.3.0"
react native : "0.72.5"
x code Version 15.2

The error you are receieving (pong reply not received from server) would indicate that the client has been unable to check in with the server and so has terminated the connection. This would normally be expected when the network conditions have degraded, and certainly wouldn't be expected with as much frequency as you are experiencing.
It could be that there is a a proxy, or other network device, that is closing the connection uncleanly. Often this is because such devices detect the connection as being idle. If this is the case you can use the https://github.com/pusher/pusher-websocket-react-native?tab=readme-ov-file#activitytimeout-double option to ensure the connection is active. As you are experiencing the disconnection after 30-40 seconds it may be prudent to try a value of 20 seconds and see if this resolves the issue.

The error you are receieving (pong reply not received from server) would indicate that the client has been unable to check in with the server and so has terminated the connection. This would normally be expected when the network conditions have degraded, and certainly wouldn't be expected with as much frequency as you are experiencing. It could be that there is a a proxy, or other network device, that is closing the connection uncleanly. Often this is because such devices detect the connection as being idle. If this is the case you can use the https://github.com/pusher/pusher-websocket-react-native?tab=readme-ov-file#activitytimeout-double option to ensure the connection is active. As you are experiencing the disconnection after 30-40 seconds it may be prudent to try a value of 20 seconds and see if this resolves the issue.

thank you for your response let me check if is it working in my case or not.


Swap Drupal make with Pantheon

Hi.
I am a looking to "migrate" from XAMP to Vagrant so I am looking for ready-to-use solutions and I like the idea of this project but I am working with Pantheon so I would need Vagrant to fetch the configured environment instead of Drupal. Would it be possible to add support for Pantheon?

@ivanjaros - At this point, I would like to make Drupal VM flexible enough to emulate most any environment via config.yml, but I don't want to add specific platform support to fetch configurations (e.g. Patheon, Acquia Cloud, Platform.sh, etc.) since that would be unmaintainable in the long run (especially considering I don't even use any of those platforms for my own projects).
However, if you or someone else can come up with a shell script or something similar to help with pulling down Pantheon configs, I would gladly link to them in the wiki.


how to get cookies and put it in default request?

I got the following situation: I receive cookies using your library, and then I need to send cookies in a regular request, for example, via request.get(), but when I do this these cookies do not pass. maybe i need to do something with jar? even though I tried...

Maybe this helps
https://github.com/puppeteer/puppeteer/blob/v6.0.0/docs/api.md#pagecookiesurls


Port 5156 Not Found

Describe the bug
No Ports Opened
To Reproduce
Steps to reproduce the behavior:

Click Gitpod button to spin up Gitpod
Go to Ports tab -> State = 'Not Served'
Go to Terminal tab ->
`/workspace/mockpass/lib/express/oidc/v2-ndi.js:237
const { profile, nonce } = lookUpByAuthCode(authCode)
^

TypeError: Cannot destructure property 'profile' of 'lookUpByAuthCode(...)' as it is undefined.
at /workspace/mockpass/lib/express/oidc/v2-ndi.js:237:17
at processTicksAndRejections (node:internal/process/task_queues:96:5)
[nodemon] app crashed - waiting for file changes before starting...`
Expected behavior

Ports tab -> port 5156 is found
Terminal tab -> Mockpass listening on port 5156

Screenshots
https://prnt.sc/NB2vGW_86xQz)
Desktop (please complete the following information):

macOS Ventura 13.3 (22E252)
Chrome Version 111.0.5563.146 (Official Build) (arm64)

Additional context
Add any other context about the problem here.

Re-opening issue as occuring again

Can restart nodemon in Terminal tab with rs


Put wrapper container in its own stacking context

This would put the wrapper and its content in "isolation" so their z-index wouldn't overflow to outside of the container.
Before (black pane is non react flow UI, which is not interactable because the cursor is also the grabbing hand moving the canvas behind):

After (black pane is interactable and the controls are now behind it):

Thanks @neo
I don't understand which issue gets fixed with this PR. Why are the controls no inside the flow pane in the upper screenshot? Could you create a codesandbox or do you have a repo?

here's a simple repro: https://codesandbox.io/s/quizzical-panka-s4j76c?file=/App.js:296-344
you can read more from one of these links if you want:

https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Positioning/Understanding_z_index/The_stacking_context
https://web.dev/learn/css/z-index/
https://www.joshwcomeau.com/css/stacking-contexts/


"Run Tests" button breaks after clicking Cat Photo link

Challenge Waypoint: Link to External Pages with Anchor Elements has an issue.
User Agent is: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36.
Please describe how to reproduce this issue, and include links to screenshots if possible.
To reproduce this bug on this waypoint, do the following:

Copy code below and paste in workspace.
Result: Code is populated in workspace.

Hover over the newly created anchor link "cat photos" on the right side of page (inside the fake iPhone screen).
Result: cursor changes in order to show a link.

Click the "cat photos" anchor text.
Result: After clicking the link, nothing happens.

Click the green "Run Tests" button.
Result: The list of tests will disappear and the page seemingly does nothing. No matter how many times you click the "Run Tests" button you cannot proceed.

Workaround: Refresh the page (your tests will be validated), and then click "Run Tests" to continue.

It might be more accurate to say that clicking a link in the "phone" causes the tests to break.
I did confirm this.  I believe this will be fixed with the new instance of the test infrastructure.  For the moment, don't click the links.

Agreed. Changed the name of the title accordingly. Clicking the link in the phone is unimportant and not needed for this exercise. I just wanted to make everyone aware of a possible scenario a new camper (like myself) might encounter. Thanks for validating and giving the suggestion on the title!

I'm going to leave this open so other campers can see it.  As I say, we've got a complete revamp of the testing environment coming "Real Soon Now"(:tm:).

I'm not able to reproduce this anymore. Closing. @joshoates thanks for letting us know and for excellent formed issue description!


Log external authentication groups on failed login.

@jrafanie @bdunne Please review.
From a diagnostic point of view, when group membership is the reason for failure, it's impossible to know why, but as soon as you see the incoming group names, it's easy to solve.  This moves that logging to info and audit from debug.  This also changes the debug to not print one-per-line which is overkill.

@jrafanie Updated.


Text passed to fillText is not adequately escaped when rendered as javascript

To reproduce:

run the following

#!/usr/bin/env stack
{- stack --resolver lts-11.22 runghc
         --package static-canvas-<IP_ADDRESS>
-}
{-# LANGUAGE OverloadedStrings #-}

module Main where

import Graphics.Static

main :: IO ()
main = writeCanvasDoc "out.html" 600 400 $ fillText "some text\\" 150 100

Open out.html in a browser
Open dev tools and you should see an error
Look at the generated javascript. The string 'some text' has no closing quote since it has been escaped.

Thanks for posting the issue. I'm not sure when I will be able to get to this. If you would like to submit a pull request it would be appreciated


Paths never get remapped on Windows

Tried using this module on Linux, and it seems to do a great job.
Though on a Windows environment it seems that the resolve for root here breaks the local descendant output due to generating the root in a different format. Originally, the root passed to configResolved is in the format of /Users/..., but the resolve() changes it to the format of C:\Users\....
The resolveId's importer variable follows the root variable's path scheme, so the isLocalDescendant function will never be able to correctly validate that the path starts with root (as it compares format of /Users/... to C:\Users\...). It also additionally uses sep after the resolve, which seems to be always / for importer's paths (instead of sep's value \ on Windows) from my testing.
Changing the mentioned line to root = (opts.root || root) + '/'; seems to fix the issue on both OSes, though I don't know if this is a good fix due to opts.root being configurable by the user and it could follow the C:\Users\... format too due to usage of e.g. process.cwd() or path.join().

Fixed in 2.0.0-alpha.4. Thanks for the detailed report!

Fixed in 2.0.0-alpha.4. Thanks for the detailed report!


Support CJSX

This may be out of scope, but would be handy!
https://github.com/jsdf/coffee-react

This wouldn't be too hard if there was an alternative CoffeeScript parser that treated these as part of the parse tree, but it looks like it is simply a pre-processor. That makes it pretty much a non-starter, unfortunately.

too bad. i have a medium-sized codebase i’d like to convert which is full of JSX literals.
maybe we could hack something up using the CJSX preprocessor and a babel transform which does React.createElement(...) → <...></...>

hi, so i created babel-plugin-transform-react-createelement-to-jsx. that’s quite a mouthful, but actually would allow us to do like the README says:

CJSX syntax → coffee-react-transform → decaffeinate → react-createelement-to-jsx → JSX syntax

@flying-sheep thats pretty sweet. +1 to that.

damn, didn’t expect michaelficarra/CoffeeScriptRedux#325


Author Attribution

Add the author names in addition to Development Seed for the Citation.

Thanks for pointing this out @wildintellect it's been addressed in PR #13.


Not working on VirtualBox 5

I installed VirtualBox 5.1.6 and is not working on that version. I'm getting this message

It would be great if you fix it!!
Have a nice day!

Hi @marionava,
Unfortunately, I can't reproduce your problem. Could you provide a debug log?

Maybe it's the same error as #9

Looks like it. Please reopen the issue if it happens again.


WIP, DO NOT MERGE! Creates a retry filter using reactor-extra

/cc @dsyer @ryanjbaxter
@smaldini & @simonbasle I want to make sure I'm not abusing Repeat.
I'd also like some advice on retrying POST which would involve buffering the request body.
fixes gh-94

Codecov Report

Merging #195 into master will increase coverage by 0.12%.
The diff coverage is 91.66%.

@@           Coverage Diff            @@
##           master   #195      +/-   ##
========================================
+ Coverage   71.88%    72%   +0.12%
========================================
  Files          87     88       +1
  Lines        1860   1872      +12
  Branches      119    120       +1
========================================
+ Hits         1337   1348      +11
- Misses        459    460       +1
  Partials       64     64

Impacted Files
Coverage Δ

...cloud/gateway/route/builder/GatewayFilterSpec.java
32.75% <0%> (-0.58%)
:arrow_down:

...cloud/gateway/config/GatewayAutoConfiguration.java
100% <100%> (ø)
:arrow_up:

...eway/filter/factory/RetryGatewayFilterFactory.java
100% <100%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d8fde99...0352589. Read the comment docs.

Looks good.  I am sure requests for backoff, timeout, etc will come quick ;)
As far as retrying POST requests the hard part, I think, is the potential memory impact of keeping the body in memory.  In Netflix we talked about making it configurable but we ultimately didnt do that.


Getting exception while running the app

Hi,
I tried all the steps mentioned in the readme.
Also uploaded to my own docker hub repo. But running the dotnet command gives me container not found error. I
I have update Program.cs in Aspire.Contribs.AppHost
line 9 : ContainerImageName = "sohdasgu/aspire-spring-maven-sample:latest"
Exception:
Unhandled exception. System.InvalidOperationException: The resource 'containerapp' does not have a container image specified. Use WithImage to specify the container image and tag.
at Aspire.Hosting.ContainerResourceBuilderExtensions.ThrowResourceIsNotContainer[T](IResourceBuilder1 builder) in /_/src/Aspire.Hosting/ContainerResourceBuilderExtensions.cs:line 225    at Aspire.Hosting.ContainerResourceBuilderExtensions.WithImageRegistry[T](IResourceBuilder1 builder, String registry) in /_/src/Aspire.Hosting/ContainerResourceBuilderExtensions.cs:line 129
at Aspire.Contribs.Hosting.Java.JavaAppHostingExtension.AddJavaApp(IDistributedApplicationBuilder builder, String name, JavaAppContainerResourceOptions options) in C:\Users\sohadasgupta\IdeaProjects\aspire-contribs\src\Aspire.Contribs.Hosting.Java\JavaAppHostingExtension.cs:line 33
at Aspire.Contribs.Hosting.Java.JavaAppHostingExtension.AddSpringApp(IDistributedApplicationBuilder builder, String name, JavaAppContainerResourceOptions options) in C:\Users\sohadasgupta\IdeaProjects\aspire-contribs\src\Aspire.Contribs.Hosting.Java\JavaAppHostingExtension.cs:line 58
at Program.$(String[] args) in C:\Users\sohadasgupta\IdeaProjects\aspire-contribs\src\Aspire.Contribs.AppHost\Program.cs:line 6
What am I missing here?
Also, does it have to refer a docker hub image? Can I not use a local image?

@sohamda Oh, I also found that.
https://github.com/aliencube/aspire-contribs/blob/d9d33648566ce958757cefa72a1f7c2d682be473/src/Aspire.Contribs.Hosting.Java/JavaAppContainerResourceOptions.cs#L11
From this line, simply replace docker.io with null and try again, in the meantime.
I'll update the code.

Thanks a lot. Yes it works now.
Was wondering (and pardon my less knowledge on Aspire) whether the code is looking for an image on local docker repo, so the upload the docker hub or any other registry is not needed to run this code, right?
Secondly, if I want to refer to an external registry, how do I control the property? By setting JavaAppContainerResourceOptions.ContainerRegistry property?

If you set this property, ContainerRegistry, to null, it basically looks for your local machine.
If you set this property, ContainerRegistry, to your desired container registry like Azure Container Registry or Docker Hub or something else, then, you'll be able to get the container image from there.

thanks a lot.
I am closing the issue as well.


Documentation conflicts on whether 8 characters may be removed in shortening

In the shorten method documentation / https://github.com/google/open-location-code/blob/a323e80159d6d6170158cc22497250cb86bf3583/API.txt

Passed a valid full Open Location Code and a latitude and longitude this removes as many digits as possible (up to a maximum of eight) ...

In the Plus Codes specification https://github.com/google/open-location-code/blob/main/docs/specification.md#short-codes

Short codes have at least two and a maximum of six digits removed from the beginning of the code

This is fixed in https://github.com/google/open-location-code/pull/463
Preference is given to "Short codes have at least two and a maximum of six digits removed from the beginning of the code" and all other conflicting notes are obliterated.


Patch version identifier and Account display mailAddress

This pull request is adding in patchVersion so that the releases like 3.1.1 will display the full release in the header. It will work with 3.2 or 3.2.1 versions style.
This also has a change to display the mailAddress under account.

Please divide this pull request into Patch version identifier and Account display mailAddress.
About displaying mail address, I assume users should be possible to select whether or not to display their own e-mail address like GitHub. At least, mail address should be displayed for only logged-in users.


Course Summaries download CSV

We need to move the "Download all assignment data" functionality from the assignments list to the course summaries page.
Also, we need to add both spreadsheet total marks and marking scheme weights to the downloaded CSV.
@dannyreis @PatGuan

Hey David,
Sorry but for the spreadsheet total marks column, did you want the total marks for A1-A4 summed up as they are on the CSV or did you want them summed up but with their corresponding weights? I'm guessing the latter?

Yeah, definitely the latter.

Fixed in #2216.


Add Biblia Koshodo no Jiken Techo to list in README.md

Thank you your choice! It's a book that looks interesting :)


node v5 is not available

λ nvm list available

Showing the 15 latest available releases.

      STABLE   |    UNSTABLE
   ---------------------------
       v4.2.1  |    v0.11.16
       v4.2.0  |    v0.11.15
      v0.12.7  |    v0.11.14
      v0.12.6  |    v0.11.13
      v0.12.5  |    v0.11.12
      v0.12.4  |    v0.11.11
      v0.12.3  |    v0.11.10
      v0.12.2  |     v0.11.9
      v0.12.1  |     v0.11.8
      v0.12.0  |     v0.11.7
     v0.10.40  |     v0.11.6
     v0.10.39  |     v0.11.5
     v0.10.38  |     v0.11.4
     v0.10.37  |     v0.11.3
     v0.10.36  |     v0.11.2

For a complete list, visit http://coreybutler.github.io/nodedistro

installing v5 works though..

NVM for Windows currently shows the old release pattern. I made a snap decision to only list LTS releases in stable, but that doesn't mean the other versions (like v5) are unstable. I'll need to come up with a better way to display this info... maybe an LTS and non-LTS column.

yeah.. that would be great!

or maybe.. nvm list available-lts and nvm list available

However I find 4.2.2 is not listed as above but it is listed on http://coreybutler.github.io/nodedistro/ which si referenced in the output of nvm list available

PS the fact the above list prefixes with v is a little confusing. The v  must not be specified when installing  or using

I think in nvm list available there should be another column, beside STABLE and UNSTABLE, showing LTS releases.
Specifically, the STABLE column should be renamed to LTS, while the new STABLE column should contain 5.x releases.

Hello all
on a Windows 7 x64 machine, nvm 1.1.0, as of today no node 5.x version is listed as available:
Showing the 15 latest available releases.

      STABLE   |    UNSTABLE
   ---------------------------
       v4.3.0  |    v0.11.16
       v4.2.6  |    v0.11.15
       v4.2.5  |    v0.11.14
       v4.2.4  |    v0.11.13
       v4.2.3  |    v0.11.12
       v4.2.2  |    v0.11.11
       v4.2.1  |    v0.11.10
       v4.2.0  |     v0.11.9
     v0.12.10  |     v0.11.8
      v0.12.9  |     v0.11.7
      v0.12.8  |     v0.11.6
      v0.12.7  |     v0.11.5
      v0.12.6  |     v0.11.4
      v0.12.5  |     v0.11.3
      v0.12.4  |     v0.11.2

but then trying to install 5.0.0 or 5.6.0 works fine. And 5.6.0 is dubbed as stable on nodejs.org. Is this behaviour expected? Thanks!

@BrainCrumbz - yes. See my comment above. The 5.x series is stable, but not LTS.

Thanks for feedback, got it. BTW, @MaxArt2501 suggestion seems sensible.

It could have been sensible two months ago, now it sounds a little outdated as there's no more "stable" and "unstable", but rather "LTS" and "stable".
I think a little extra work can be done if le list of the versions is pulled from https://nodejs.org/download/release/index.json instead of here (maybe it's a list maintained/generated by Corey himself?). Identifying LTS versions is easy, as they're the ones with the lts property not set to false.
I don't know enough Go to change the code, compile it and make a pull request after some tests, but that's what I'd do.

@MaxArt2501 - The node release page is the list I intend to implement in the next version. It didn't exist when I first created the utility (pre node foundation/io.js). The second link you mention (nodedistro) is a list I maintain. I have a script that runs every hour to look for a new release. Not ideal.
I do like your suggestion for naming. The only thing I'm still thinking about is the naming of LTS lines and whether they'll even be a factor (right now I don't think they will, but I'm trying to confirm that). For example, "Argon" is the current LTS line, but the name will change when the next LTS line comes out.

@MaxArt2501 Yep, I meant: among the various suggestions, I favored the one having separate columns with proper (up to date) headers.

We are using node version 5.5.0 in our build environment. I'm using nvm to switch versions.
Today I tried to setup  a new machine and I'm getting this error.
nvm install 5.5.0
Node.js v5.5.0 is not yet released or available.

@RobMoloney I'm not sure when it happened, but the node distribution list got a little tangled up. I fixed it, so it should work now. For future reference, version issues can be corrected here.

Great. Thanks for the quick reply.

I am trying to install a listed node version. What could be the possible reasons for this behavior?

how to resolve it?

You will having old version of "nvm". Update to new version of nvm will solve this issue. Download latest nvm for windows from here and then try again to install node again.

Yep, upgrading nvm worked for me. (the issue of it saying "not yet released or available" for a version listed by the "list available" command)


Kdump: support testing RHEL6/7 on WS2012

Extra kdump config is required on WS2012, while previous extra config
does not work for WS2012 host.
Test-NetConnection does not work for WS2012, use function
WaitForVMToStartSSH in TCUtils instead.

This patch might still have some problem, I need more time to double check, closing this PR.
Sorry for the trouble

The kdump cases are not stable, refetch ipv4 of VM after config reboot and crash reboot in case a new ipv4 allocated to the VM.
We have a known issue that kvp would not work after kdump on WS2016 - RHEL6. This bug has been confirmed won't be fixed, so I had to reboot the VM again after kdump so that new ip address could be retrieved by kvp.
https://bugzilla.redhat.com/show_bug.cgi?id=1383037

Fixed, thank you!
Also fixed a stability issue that crash on multiple core would abort with timeout sometimes.


code is only considered a block if indent is maintained

Absolutely love this package, but... Hydrogen will run this block from the first line:
x = (
    "a",
    "b",
    "c")

but not this one:
x = (
    "a",
    "b",
    "c"
)

This is problematic since the second example is the default behavior in atom (type x = (<enter> and the closing bracket moves to that location)

Yup, unfortunately this is a side effect of how Hydrogen is finding blocks
at the moment. If Atom doesn't think something is foldable, Hydrogen uses
the indentation.
After I wrap up my thesis (in the next couple of weeks) I'll work on
implementing an API for language-specific plugins that will be able to do a
much better job of this.
On Fri, Jan 22, 2016 at 12:01 AM Jeremiah Lowin<EMAIL_ADDRESS>wrote:

Hydrogen will run this block from the first line:
x = (
"a",
"b",
"c")
but not this one:
x = (
"a",
"b",
"c"
)
This is problematic since the second example is the default behavior in
atom (type x = ( and the closing bracket moves to that location)
—
Reply to this email directly or view it on GitHub
https://github.com/willwhitney/hydrogen/issues/155.

Got it! Sounds like you have far more important things to deal with than my desire to not have to highlight all my text before executing it inline. :) Hydrogen is fantastic, well done.

@jlowin Since we still rely heavily  on Atom's folding. You could open a issue for this at atom/language-python.


No connection recovery on broker restart

Affects Version(s): 2.2.0
We've encountered a production outage that was triggered by a restart of a broker.
The sequence of events:

Broker restarted.
Log from producer: Channel shutdown: connection error; protocol method: #method<connection.close>(reply-code=320, reply-text=CONNECTION_FORCED - broker forced connection closure with reason 'shutdown', class-id=0, method-id=0)
Log from producer: org.springframework.amqp.AmqpIOException: java.net.SocketTimeoutException: connect timed out

It's interesting that one of our producer hosts actually did reconnect automatically. I can see Created new connection: SpringAMQP#34e35055:2/SimpleConnection@14fc7b94 [delegate=amqp://ge@<IP_ADDRESS>:5672/ge, localPort= 48358] log from that host right after Channel shutdown. However, other hosts did not reconnect and recovered only after we redeployed the application.
This was our RabbitTemplate configuration:
        CachingConnectionFactory connectionFactory =
                new CachingConnectionFactory();
        connectionFactory.setAddresses(AWSUtils.resolveDnsName(dnsName));
        connectionFactory.setUsername(user);
        connectionFactory.setPassword(password);
        connectionFactory.setVirtualHost(virtualHost);
        connectionFactory.setPublisherConfirms(true);
        connectionFactory.setRequestedHeartBeat(30);

        RabbitTemplate template = new RabbitTemplate(connectionFactory);
        template.setExchange(exchange);
        return template;

And we only use the send method in RabbitTemplate.
The question is: is there anything we can change in our client configuration to avoid such a problem in the future?
We consider adding connectionFactory.getRabbitConnectionFactory().setAutomaticRecoveryEnabled(true); but I see that documentation discourages this: https://docs.spring.io/spring-amqp/reference/html/#_connection_factory_changes
Would this help us? And what are the downsides exactly?

recovered only after we redeployed the application.

This is most likely an application problem.
If these are producer only applications, we won't try to reconnect until the next RabbitTemplate.send() operation.
If it didn't reconnect, it means your app didn't perform any more publishing operations after the failure.
If your application stops sending, using an auto-recovered connection won't help; you'll just have an idle connection.
Applications with consumers will reconnect automatically.

@garyrussell wow, thanks for your swift reply!
I can see that we were attempting to send messages after Channel shutdown and this is when we encountered timeouts.
But I feel like I figured out why it happened: broker instances weren't just restarted, they were terminated and then respawned (AWS ECS). Their IPs changed, and producer could not know about the new IP addresses, because this is how it discovered brokers: connectionFactory.setAddresses(AWSUtils.resolveDnsName(dnsName));
You may close the issue. Now I'm wondering if Rabbit provides any dynamic broker discovery capabilities (Rabbit newbie here). 🙇♂

Best place to ask questions like that is on the rabbitmq-users Google group.


Implement more best arm identification algorithms

Have you thought of implementing more BAI algorithms? I can do it, but I'll need some time to get familiar with your environment.

No, I never though of doing so, but it could be a very valuable addition!!
The entire API is oriented toward the action/reward loop, which is also followed by the BAI task, isn't it?
But the Evaluator class is more oriented towards evaluating the reward maximization task.

I'm also interested in collaborative best arm identification! Some simple meta-aggregate algorithms like [1, 2] might be a good place to start.
https://arxiv.org/abs/1311.0800
https://arxiv.org/abs/1811.07763

Hi @choltz95, thanks for your comment!
I am familiar with the second work you quoted, not the first one, thanks for the interesting reference!
I no longer have much time to work on SMPyBandits, and cannot take the time to implement BAI algorithms / benchmarks, unfortunately.
However, I will, of course, do my best to help if you want to take a chance at it!
Regards.


[C-1521] Signin page opens even after authentication

crowd.dev edition
Cloud (hosted)
Version
No response
What happened?
Steps to reproduce

First Sign-in/signup
Now when you are on the home page, try to go back
It will open sign-in page again.

Ideally it should not allow user to open sign-in page after authentication
https://github.com/CrowdDotDev/crowd.dev/assets/67850763/f66ea925-7fa2-4f2e-bee1-5dca28e3ef9f
Command / Request
No response
Relevant log output
No response
C-1521

Thanks for sharing! We are checking on this.


Using latest commit for bevy_main branch

Would it be alright if the bevy_main branch of this library just always tracked the latest git commit (bevy = { git = "https://github.com/bevyengine/bevy", default-features = false }). It would make it easier for those of use who also keep track of the latest Bevy git commit, instead of having to use a specific revision (as typically, you want to follow the development branch of Bevy since you wan't the latest features all the time).

I am honestly not sure about this. Personally, I prefer to update (read: get breaking changes) when I want to and not whenever I compile the next time after commits were pushed to Bevy.
If we do the proposed change, there can be times the bevy_maser branch does not compile. I would prefer having it compile at all times and would accept being behind a few commits now and then.

I see where you're coming from. The only (admittedly minor) issue for me is having to recompile all of Bevy twice whenever I use the repo. Could we maybe have two branches that are identical, with the difference being that one follows the latest Bevy always, while the other always follows a stable version of Bevy git (perhaps by renaming bevy_main to bevy_main_stable and then having a bevy_main branch that truly follows it). If not by the way, that's completely fine.

Since I don't think this is the working mode of many users, I'd prefer to keep this in forks.
If there are more people commenting here, that they would like such a branch, we can of course talk about it again.

Alright, that's totally fine. Thank you for considering!


Optimize image layers with project code

Description
Current templates in kedro-docker create 3 layers with user code for every build (due to separate copy->chown->chmod stages). However, during image build we can specify what should be the target owner of the files. Also, there is no need to run chmod, as file/dirs permission is preserved.
Development notes
Tested locally by building the docker image and examining the filesystem. Layers optimization reduce both build time and image size.
Checklist

[ ] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
[ ] Updated the documentation to reflect the code changes
[x] Added a description of this change in the relevant RELEASE.md file
[ ] Added tests to cover my changes

@szczeles Thank you for the PR. There are couple test failings. Could you also show some stats or how big  is the new image compare to the old one.

@noklam Yea, I've seen the failures but didn't have time to look into these yet. Anyway, I've collected a few ideas on kedro-docker templates improvements (for both build speed and size) and will push them as next PR, closing this one.
Regarding the image size changes - it depends. If your codebase of Kedro project (including data/) is 100KB, then image will be smaller by 200KB. But if it is 100MB, image size will drop by 200MB.


eask: 0.9.9 -> 0.10.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/emacs-eask/cli/releases.
meta.description for eask is: CLI for building, runing, testing, and managing your Emacs Lisp dependencies
meta.homepage for eask is: https://emacs-eask.github.io/
meta.changelog for eask is: https://github.com/emacs-eask/cli/blob/0.10.0/CHANGELOG.md
Updates performed

NPM version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.10.0 with grep in /nix/store/f9ixnkxwym88jsaa3r9zpimk8fsi0y3f-eask-0.10.0
found 0.10.0 in filename of file in /nix/store/f9ixnkxwym88jsaa3r9zpimk8fsi0y3f-eask-0.10.0

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath

eask

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/f9ixnkxwym88jsaa3r9zpimk8fsi0y3f-eask-0.10.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A eask https://github.com/r-ryantm/nixpkgs/archive/fff0dd2e0c0be1ca008861b28d871ac32abe0ce4.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/fff0dd2e0c0be1ca008861b28d871ac32abe0ce4#eask

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/f9ixnkxwym88jsaa3r9zpimk8fsi0y3f-eask-0.10.0
ls -la /nix/store/f9ixnkxwym88jsaa3r9zpimk8fsi0y3f-eask-0.10.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review --extra-nixpkgs-config '{ allowInsecurePredicate = x: true; }' run on x86_64-linux 1

1 package built:

eask

Maintainer pings
cc @jcs090218 for testing.

Confirmed.


Update Documentation for v1.0

This is really a placeholder ticket for all documentation updates needed to support the 1.0 release.
We can use this ticket as an umbrella ticket, and create breakout tickets for individual tasks.
Some example areas likely requiring updates:

Examples
Tutorials
Client READMEs

Client readmes have been reviewed and updated recently, so should only need editing to reflect further updates/API changes.
We could do with more examples, but also the chemical similarity section in particular needs to be substantially rewritten.
The architecture docs page is nearly empty, I think that page should have more content or be removed.
https://github.com/pilosa/www/issues/227


rust ci

closes https://github.com/lambdaclass/libtorrent-rs/issues/9

thanks for this! the team will check it and merge it.

Quick question, why are the changed targets using nightly?

I would recommend against it. Using nightly for the lints could randomly break the workflows.

I'll push changes soon.

removed nightly


Please merge 2c304d9 into dev channel

commit(s) to merge: 2c304d9
merge instructions: clean merge
reason: high priority customer issue; low risk fix. Cherry pick was discussed here: https://github.com/dart-lang/sdk/issues/36372#issuecomment-485029830
/cc @dgrove @kevmoo @mit-mit @whesse @athomas

Merged to Version 2.3.0-dev.0.2


MLAT with docker-piaware using docker-readsb-protobuf not working(?)

I'm using  docker-readsb-protobuf, docker-piaware and other SDR Enthusiasts feeders and am unable to get MLAT working. I've followed the readmes and have also tried using themlathub method but none seem to work. Co-ordinates are set accuratley. I'm in a very populated area with lots of air traffic (near LHR airport) and would expect to synchronization with quite a few MLAT sites.
My setup:
RPi3 B+ (with official Pi PSU) running Raspberry Pi OS (64bit) and Docker version 20.10.5+dfsg1 and a FlightAware Pro Stick Plus
$ uname -a
Linux rpi3-iot 5.15.32-v8+ #1538 SMP PREEMPT Thu Mar 31 19:40:39 BST 2022 aarch64 GNU/Linux

$ lscpu
Architecture:                    aarch64
Byte Order:                      Little Endian
CPU(s):                          4
On-line CPU(s) list:             0-3
Thread(s) per core:              1
Core(s) per socket:              4
Socket(s):                       1
Vendor ID:                       ARM
Model:                           4
Model name:                      Cortex-A53
Stepping:                        r0p4
CPU max MHz:                     1400.0000
CPU min MHz:                     600.0000
BogoMIPS:                        38.40
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Not affected
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
Vulnerability Spectre v2:        Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fp asimd evtstrm crc32 cpuid

$ lsusb
Bus 001 Device 004: ID 0bda:2832 Realtek Semiconductor Corp. RTL2832U DVB-T
Bus 001 Device 005: ID 0424:7800 Microchip Technology, Inc. (formerly SMSC)
Bus 001 Device 003: ID 0424:2514 Microchip Technology, Inc. (formerly SMSC) USB 2.0 Hub
Bus 001 Device 002: ID 0424:2514 Microchip Technology, Inc. (formerly SMSC) USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

$ free
               total        used        free      shared  buff/cache   available
Mem:          931588      293208       84288       12768      554092      579532
Swap:         102396       38144       64252

$ docker info
Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 9
  Running: 9
  Paused: 0
  Stopped: 0
 Images: 34
 Server Version: 20.10.5+dfsg1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: local
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1.4.13~ds1-1~deb11u1
 runc version: 1.0.0~rc93+ds1-5
 init version:
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.32-v8+
 Operating System: Raspbian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 909.8MiB
 Name: rpi3-iot
 ID: 6RQR:F6X7:WQ6C:RPQ5:BF57:HRZL:XEF4:GBHW:ICPI:7WDE:ROS3:5SQQ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  <IP_ADDRESS>/8
 Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support
WARNING: Support for cgroup v2 is experimental

$ cat docker-compose.yaml
version: '3.6'

networks:
  adsbnet:

volumes:
  readsbpb_rrd:
  readsbpb_autogain:

services:

  readsb:
    image: mikenye/readsb-protobuf:latest
    tty: true
    container_name: readsb
    hostname: readsb
    restart: always
    devices:
      - ${DEV_1}
    environment:
      - TZ=${TZ}
      - READSB_DCFILTER=true
      - READSB_DEVICE_TYPE=rtlsdr
      - READSB_FIX=true
      - READSB_GAIN=autogain
      - READSB_LAT=${LAT}
      - READSB_LON=${LONG}
      - READSB_MAX_RANGE=600
      - READSB_MODEAC=true
      - READSB_RX_LOCATION_ACCURACY=2
      - READSB_STATS_RANGE=true
      - READSB_NET_ENABLE=true
      - READSB_NET_CONNECTOR=mlathub,30105,beast_in
#      - DISABLE_PERFORMANCE_GRAPHS=true
#      - DISABLE_WEBAPP=true

    volumes:
      - ${LOC_TZ}
      - readsbpb_rrd:/run/collectd
      - readsbpb_autogain:/run/autogain
    tmpfs:
      - /run/readsb:size=64M
      - /var/log:size=32M
    networks:
      adsbnet:
    ports:
      - 8888:8080
      - 30003:30003
      - 30005:30005
    dns:
      - ${DNS_1}
      - ${DNS_2}

  mlathub:
    image: mikenye/readsb-protobuf:latest
    tty: true
    container_name: mlathub
    restart: always
    environment:
      - TZ=${TZ}
      - DISABLE_PERFORMANCE_GRAPHS=true
      - DISABLE_WEBAPP=true
      - READSB_NET_ENABLE=true
      - READSB_NET_ONLY=true
      - READSB_FORWARD_MLAT=true
      - READSB_NET_CONNECTOR=piaware,30105,beast_in;rbfeeder,30105,beast_in
      - READSB_NET_BEAST_OUTPUT_PORT=30105
      - READSB_NET_BEAST_INPUT_PORT=30104
    volumes:
      - ${LOC_TZ}
    tmpfs:
      - /run/readsb
      - /var/log
    networks:
      adsbnet:
    ports:
      - 30105:30105
    dns:
      - ${DNS_1}
      - ${DNS_2}

  piaware:
    image: mikenye/piaware:latest
    tty: true
    container_name: piaware
    restart: always
    environment:
      - TZ=${TZ}
      - LAT=${LAT}
      - LONG=${LONG}
      - RECEIVER_TYPE=relay
      - BEASTHOST=readsb
      - BEASTPORT=30005
      - ALLOW_MLAT=yes
      - MLAT_RESULTS=yes
      - MLAT_RESULTS_BEASTHOST=mlathub
      - MLAT_RESULTS_BEASTPORT=30104
      - USERNAME=${PiA_USERNAME}
      - PASSWORD=${PiA_PASSWORD}
      - FEEDER_ID=${PiA_FEEDER_ID}
    volumes:
      - ${LOC_TZ}
    tmpfs:
      - /run:exec,size=64M
      - /var/log
    networks:
      adsbnet:
    ports:
      - 8080:80/tcp
    dns:
      - ${DNS_1}
      - ${DNS_2}
    depends_on:
      - readsb
      - mlathub

  fr24feed:
    image: mikenye/fr24feed:latest
    container_name: fr24feed
...

$ docker logs piaware | tail
[piaware] 2022/04/25 09:26:40 69820 msgs recv'd from dump1090 (1299 in last 5m); 69739 msgs sent to FlightAware
[piaware] 2022/04/25 09:31:40 71117 msgs recv'd from dump1090 (1297 in last 5m); 71036 msgs sent to FlightAware
[piaware] 2022/04/25 09:36:40 72363 msgs recv'd from dump1090 (1246 in last 5m); 72282 msgs sent to FlightAware
[piaware] 2022/04/25 09:39:46 mlat-client(2922): Receiver status: connected
[piaware] 2022/04/25 09:39:46 mlat-client(2922): Server status:   clock unstable
[piaware] 2022/04/25 09:39:46 mlat-client(2922): Receiver:  484.2 msg/s received       82.1 msg/s processed (17%)
[piaware] 2022/04/25 09:39:46 mlat-client(2922): Server:      0.0 kB/s from server    0.0kB/s TCP to server     1.0kB/s UDP to server
[piaware] 2022/04/25 09:39:46 mlat-client(2922): Aircraft: 7 of 11 Mode S, 27 of 30 ADS-B used
[piaware] 2022/04/25 09:41:40 73562 msgs recv'd from dump1090 (1199 in last 5m); 73481 msgs sent to FlightAware
[piaware] 2022/04/25 09:46:40 74457 msgs recv'd from dump1090 (895 in last 5m); 74376 msgs sent to FlightAware

Message on my FlightWare stats page for my site:
This feeder is not being used for multilateration because its timing information appears to be unreliable. This can be caused by the site location being incorrect, or because your Pi is running out of free CPU.
The readsb-protobuf performance graphs show at most 2-3 MLAT messages in total directly after startup but never increment further.
Does anyone have this working in a similar setup? Am I doing something incorrectly that could be causing this.

Anyone solved this issue in any other way? I am having the same problem, but I am on a fast Intel cpu.
I am using the latest ultrafeeder instead of readsb-protobuf.
Thanks

@ebaschiera check in the docker container with netstat if port 30105 is available. (docker exec piaware netstat)
Then check utltrafeeder if something goes wrong with the connection of piaware. (docker logs ultrafeeder | grep piaware).
I noticed 2 things:

port 30105 on piaware was refusing connections, redeploy of the stack solved this
port 30105 on piaware simply did not excist: my stupid mistake, wrong feeder ID (did not copy the first character)

Thank you for your reply, but I gave up and switched to https://github.com/dirkhh/adsb-feeder-image on a small dedicated PC.
No more issues!


error or issue running dotnet cmd

Problem encountered on https://dotnet.microsoft.com/learn/aspnet/hello-world-tutorial/install
Operating System: windows
Provide details about the problem you're experiencing. Include your operating system version, exact error message, code sample, and anything else that is relevant.
command prompt message

dotnet

Usage: dotnet [options]
Usage: dotnet [path-to-application]
Options:
-h|--help         Display help.
--info            Display .NET Core information.
--list-sdks       Display the installed SDKs.
--list-runtimes   Display the installed runtimes.
path-to-application:
The path to an application .dll file to execute.
************

That looks like the expected result. Have you continued the tutorial @Malphabetagama?

yes I have begun Thanks a Bunch mairaw 👍

Awesome! I plan to add the expected output to make the tutorial a bit more clear. Let us know if you run into other issues.


Quick and Practical example of Hexagonal Architecture in Java

Pull request: Hexagonal Architecture in Java

closing the request


Requesting Manifest, Vocab and SP-Model for KsponSpeech

❓ Questions & Help
I am requesting Manifest, Vocab and SP-Model files for experiments with KsponSpeech dataset. I did request files on the link in README but I didn't get responses yet. Could any guys can help me? If it is okay, can you send them to my email<EMAIL_ADDRESS>

Hi! Sorry. I must have missed the mail. Can you send it again? If you send me the approval details of KsponSpeech, I will approve it right away.

@sooftware
Thank you for your quick response. I just sent a permission email by<EMAIL_ADDRESS>Do I need to request it separately for each Manifest, Vocab, and SP-Model files? I just sent one email to<EMAIL_ADDRESS>If needed further actions, please let me know.

@sooftware
I just got two files. Any other Vocab or SP-Model files are not needed for training?

@jun-danieloh  I just approved another one. Did they all get approved?

@sooftware Just these two files, kspon_character_manifest.txt and kspon_subword_manifest.txt are approved. Are they enough to run training?

@sooftware Also, what is the path for dataset.test_manifest_dir??
This is my current training script. Can you please tell me whether I am doing right?
python ./openspeech_cli/hydra_train.py \
        dataset=ksponspeech \
        dataset.dataset_path= /mnt/dataset/KsponSpeech/kspon/kspon \
        dataset.manifest_file_path= /mnt/dataset/KsponSpeech/kspon/kspon_character_manifest.txt \
        dataset.test_dataset_path= /mnt/dataset/KsponSpeech/kspon/kspon_eval \
        dataset.test_manifest_dir= \
        tokenizer=kspon_character \
        model=joint_ctc_listen_attend_spell \
        audio=melspectrogram \
        lr_scheduler=warmup_reduce_lr_on_plateau \
        trainer=gpu \
        criterion=joint_ctc_cross_entropy

My kspon directory:
jun1.oh@elf-desktop:/mnt/dataset/KsponSpeech/kspon/kspon$ ls
KsponSpeech_01  KsponSpeech_02  KsponSpeech_03  KsponSpeech_04  KsponSpeech_05

My kspon_eval directory:
jun1.oh@elf-desktop:/mnt/dataset/KsponSpeech/kspon/kspon_eval$ ls
eval_clean  eval_other  KsponSpeech_eval.zip

Specify test_manifest_dir, and if test_manifest_path is not valid, will automatically preprocess and create it.


Remove inline style.

Closes #8.
crispEdges and fill can be expressed as attributes in SVG. By doing it this way CSP headers which ban inline style attributes won't be upset by the SVG.

Thanks for doing this!

Bump. It'd be great to get this out.


Figure out what do to about abandonware async arango clients

We're currently using https://github.com/mirrorrim/aioarango for ArangoDB comms, but that repo hasn't been updated in almost 2 years. Questions about that are going unanswered: https://github.com/mirrorrim/aioarango/issues/3 https://github.com/mirrorrim/aioarango/issues/5
The main client repo has no plans to support async/await: https://github.com/ArangoDB-Community/python-arango/issues/95
The only other async lib I can find also appears to be abandonware: https://github.com/getflaps/aioarangodb
The whole service is structured around async/await and undoing that would be painful and also probably lead to a performance hit on longer queries.
Currently it's not causing any problems, but some library dependencies are locked to specific major versions and so we won't be able to update those packages once they do a major release:
aioarango==1.0.0
  - httpx [required: >=0.18.2,<0.19.0, installed: 0.18.2]
    - certifi [required: Any, installed: 2022.12.7]
    - httpcore [required: >=0.13.3,<0.14.0, installed: 0.13.7]
      - anyio [required: ==3.*, installed: 3.6.2]
        - idna [required: >=2.8, installed: 3.4]
        - sniffio [required: >=1.1, installed: 1.3.0]
      - h11 [required: >=0.11,<0.13, installed: 0.12.0]
      - sniffio [required: ==1.*, installed: 1.3.0]
    - rfc3986 [required: >=1.3,<2, installed: 1.5.0]
    - sniffio [required: Any, installed: 1.3.0]
  - PyJWT [required: >=2.1.0,<3.0.0, installed: 2.6.0]
  - requests-toolbelt [required: >=0.9.1,<0.10.0, installed: 0.9.1]
    - requests [required: >=2.0.1,<3.0.0, installed: 2.28.2]
      - certifi [required: >=2017.4.17, installed: 2022.12.7]
      - charset-normalizer [required: >=2,<4, installed: 2.1.1]
      - idna [required: >=2.5,<4, installed: 3.4]
      - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.14]

For example, httpx is now on version 0.23.3 (https://github.com/mirrorrim/aioarango/issues/7)

I suppose on the dependency front we could go back to pip... meh

another nasty option would be to pull out the code we need from aioarango (which is probably a relatively small subset) into this repo. Then we can set the dependency levels to whatever we want & update code as necessary to make them work.


Funnels won't extract / insert items from / to contraption inventory if being steady

Describe the Bug
Seems the funnel won't work if being steady when the contraption moves.
Reproduction Steps

Place a mechanical bearing facing any direction.
Attach a chest to it.
Insert something into the chest.
Attach a funnel at the contraption's rotating axis facing the same axis.
Rotate the contraption.

Expected Result
The funnel should extract the items from the chest.
Screenshots and Videos
https://user-images.githubusercontent.com/50518519/174303534-6f4b79d1-b368-4381-8b1e-205d87831517.mp4
Crash Report or Log
No response
Operating System
Windows 10
Mod Version
0.4.1
Minecraft Version
1.18.2
Other Mods
No response
Additional Context
No response

Alright, I found this bug being also present in the forge version, opened an issue there.
https://github.com/Creators-of-Create/Create/issues/3125

Issues that occur on Forge as well are no longer being tracked on this repo. Please follow the respective issue on the Forge repo instead: https://github.com/Creators-of-Create/Create/issues/3125


feat: allow for overriding how we check if a type matches a specifier

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#406  👈
#405
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @RebeccaStevens and the rest of your teammates on  Graphite

Merge activity

Apr 15, 12:52 AM EDT: @RebeccaStevens started a stack merge that includes this pull request via Graphite.


About Japanese correspondence

Hi, Eric.
Thank you for the useful tool.
I'm verifying ActivitiesCache.db on my PC.
However, when WxTCmd is used, all the Japanese part in the created tsv file becomes "?".
Are there any good methods or plans for improvement?
If possible, it will be helpful if you can correspond to Japanese.
(I am using Google translation because I can not speak English.)

私は日本語を話さないので、これもgoogle翻訳です
私はCSVファイルをUnicodeエンコーディングに変更したので、v<IP_ADDRESS>を試してみてください
I do not speak Japanese, so this is also google translate
Try v<IP_ADDRESS> as I changed the CSV files to Unicode encoding


feat: add custom_query param to OpenSearch retrievers

Related Issues

fixes support for custom OpenSearch queries

Proposed Changes:

add custom_query param
we support str and dict as custom_query: the former is easier to use with YAML, the latter is easier to use with ~python~JSON

How did you test it?

added tests

Notes for the reviewer

What do I mean with easier with YAML? See how a already properly formated OpenSearch query in JSON can be simply copied into yaml under https://docs.cloud.deepset.ai/docs/opensearch-queries-for-boosting-retrieval#used-in-a-pipeline

Checklist

I have read the contributors guidelines and the code of conduct
I have updated the related issue with new insights and changes
I added unit tests and updated the docstrings
I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.

Let's just use Dict[str, Any] for custom_query and scrap support for str.

@silvanocerza I'd normally also like the idea to keep it simple, but you usually interact with OpenSearch using JSON, e.g. via OpenSearch dev tools. Don't you think this makes it quite cumbersome to set custom queries as you always have to convert them to however you want to interact with the retrievers here (e.g. from JSON to python dicts, or to YAML if you use YAML marshalling)?

Let's just use Dict[str, Any] for custom_query and scrap support for str.

@silvanocerza I'd normally also like the idea to keep it simple, but you usually interact with OpenSearch using JSON, e.g. via OpenSearch dev tools or postman. Don't you think this makes it quite cumbersome to set custom queries as they can get quite complex and you always have to convert them to however you want to interact with the retrievers here (e.g. from JSON to python dicts, or to YAML if you use YAML marshalling)?

Ah anywas, we can readd it if we see it's not working.

@tstadel correct JSON is almost correct Python too, the only difference being null instead of None. Copy and paste wouldn't be a great issue.
I agree that it might be annoying handling converting that to YAML for a user, though you'd get a validation that you wouldn't get if it's a string.
My main concern is that using a string could be error prone and hard to notice. The fact that most of the docstrings are not valid Python nor JSON is pretty telling. :)
Also, as you said, we can add this later if we see the need.


Update werkzeug imports

https://trello.com/c/aQbWtees/1419-resolve-pyup-prs
The new version of Werkzeug introduces breaking changes for direct imports. We had been using import werkzeug to give us explicit namespacing to differentiate between werkzeug.exceptions and requests.exceptions, but given we've still got the latter, I think it's fine.
The tests now run 100% green with no warnings 😸

WFM. Wish we didn't have to feel such urgency to jump on top of just-released versions all the time.

WFM. Wish we didn't have to feel such urgency to jump on top of just-released versions all the time.


load data to memsql table from Hadoop (HDFS) ORC files

Hello Team,
I have a external HIVE table with billions of records in Hive and the underlying files are ORC format for this table. Is there a way I can use any memsql pipeline concept or any other memsql approach I can read data from HDFS ORC files and load to memsql table ?
we have memsql and hadoop on two different machine in the same network.
We tried using memsql-spark connector, but data loads very slow. Trying other options to load the data quickly.

Hello,
It is possible to achieve pretty good performance with memsql-spark connector.
Connector creates one LOAD DATA query per RDD partition and executes them in parallel.
Also, it is load balancing these queries between dmlEndpoints.
These two things (number of RDD partitions and aggregators in the cluster) can improve the performance.

Closing this issue for now - please reopen if you are unable to improve performance via increasing the number of partitions in the RDD you are writing to the database.  Also ensure that you have more than one aggregator listed in the dmlEndpoints configuration of the connector.


Done and Cancel buttons don't work

Describe the bug
I can't dismiss this picker at all, Done button does nothing and neither does Cancel.
To Reproduce
func textFieldDidBeginEditing(_ textField: UITextField) {
        if textField.tag == 4 {
            let min = Date().addingTimeInterval(-60 * 60 * 24 * 4)
            let max = Date().addingTimeInterval(60 * 60 * 24 * 4)
            picker = DateTimePicker.create(minimumDate: min, maximumDate: max)
            picker.frame = CGRect(x: 0, y: 250, width: picker.frame.size.width, height: picker.frame.size.height)
            picker.is12HourFormat = false
            picker.dateFormat = "dd/MM/YYYY HH:mm"
            self.view.addSubview(picker)
        }
       }

Hi, in order to get callback for Done and Cancel button, please make sure that you implement dismissHandler and completionHandler.

Still don't quite understand. I don't get why dismissView() and donePicking() just aren't called. The buttons have touch gestures to trigger these functions? After adding picker, I only expected to have to code logic for "send date selected here". I have no way to reference the cancel and done buttons, calling dismissHandler does nothing.

Please look again at the demo project, there's sample code there. If you want to be notified when picker is dismissed, use this:
       picker.dismissHandler = {
            picker.removeFromSuperview()
        }

For donePicking, do this:
       picker.completionHandler = { date in
            let formatter = DateFormatter()
            formatter.dateFormat = "hh:mm aa dd/MM/YYYY"
            self.title = formatter.string(from: date)
        }

If you just want to be notified for selected dates, make sure to set delegate of the picker:
picker.delegate = self
And implement the delegate function:
    func dateTimePicker(_ picker: DateTimePicker, didSelectDate: Date) {
        title = picker.selectedDateString
    }

Got it working now. I completely missed the demo project, hence the confusion. Thank you!


Documentation Improved

Status
READY
Description
Added complete installation instructions line 140
Fixed formatting line 145, 148, 1180
Changed Grammar line 889
Fixed λ display error line 894
Todos

[ ] Tests - None required
[ ] Documentation - README.md and doc/en-US/user_guide.md have been changed

Deploy Notes
No working components have been changed. Only documentation has received changes
Impacted Areas in Application
Files changed = 2
README.md
doc/en-US/user_guide.md

@DigvijayBhosale1729 Resolve conflicts

Resolved conflicts.
Number of conflicts = 1
Conflicts resolved = 1
File where conflict occured = user-guide.md
Conflicting Statement -
It simply represents the number of times you want to ignore the possibility of an attack. In other words, it is the extent to which IDS will not bother to inform you about the attack. Once it crosses the limit (here threshold), it will start notifying you about the possible attack. The lower the number is, the more sensitive IDS is, and may also give rise to false alarms. The higher the number is, the less sensitive IDS is, it may give rise to less false positives but at the same time choosing a very high number is not suggested either. Choose a mid range number within (10-100) to be on the safer side while keeping alarms of false positives to the minimal.

@rejahrehim sir,
Improved Documentation was included in PR #302 .
That PR is now merged. This implies that this PR is no longer Neccessary.
Please deal with this as you see fit.


Rename document_type to elasticsearch_type

This PR aims to avoid confusion about the concept "document type".

The word document_type is used by the content store to describe the type of a content item. See RFC 41 for the background.

But "document type" is also used by this application to refer to the elasticsearch type of the document.

To avoid confusion and allow us to use document_type consistently across GOV.UK this commit renames document_type to elasticsearch_type where applicable.
This should not change anything for clients of this API. There are 2 todo's after this PR:

We currently allow clients to filter on document_type. This needs to be removed or renamed to elasticsearch_type.
We currently return document_type in the payload by default. This needs to be removed or renamed too.

How to review

[ ] Read through the entire diff to see if the change makes sense and I haven't made a typo
[ ] Check out the branch and see if the remaining occurrences of document_type make sense

Trello: https://trello.com/c/C8A9bnuO

This rename makes sense to me, and the only places I can see where the old name is still being is used are:

The message queue consumer via lib/indexer/message_processor.rb
This is only when logging the payload of a message. It doesn't look like we use document_type from incoming messages, so maybe we should stop the publishing api from sending it?
The result presenter (has a TODO)
Filter params (has a TODO)

I think whatever we do with document_type should be consistent across the API. The endpoints for indexing include the document types as well so rummager doesn't hide that detail even if we dropped the filter/result field. I think a rename to elasticsearch_type would be the simplest thing for now.
I'm assuming that when we index from the publishing api the two things (elasticsearch_type and document_type/format) would converge anyway?

I think whatever we do with document_type should be consistent across the API. The endpoints for indexing include the document types as well so rummager doesn't hide that detail even if we dropped the filter/result field. I think a rename to elasticsearch_type would be the simplest thing for now.

Not sure that I get this, can you elaborate?

I'm assuming that when we index from the publishing api the two things (elasticsearch_type and document_type/format) would converge anyway?

I'd actually expect elasticsearch_type and schema_name to converge, since they both describe the data.

I'd actually expect elasticsearch_type and schema_name to converge, since they both describe the data.

Oh yeah, that makes more sense.

Not sure that I get this, can you elaborate?

I just mean that the way that an API user deals with that field should be consistent across endpoints, and the concept is exposed in a few different places at the moment.
POST or DELETE /:index/documents supports a _type field
GET /content/?link=/a-link returns _type
GET /search has filter_document_type and returns document_type

Good point, we'll update those as well!


Add help and version text

Add texts for --help and --version to serverbot.

Added in https://github.com/nozel-org/freebsd-serverbot/commit/066fc9525af8daa444ba45648c61a5a450609002.


[Automation] Generate Fluent Lite from healthcareapis#package-2023-11

[Automation] Generate Fluent Lite from healthcareapis#package-2023-11

API change check
APIView has identified API level changes in this PR and created following API reviews.
com.azure.resourcemanager:azure-resourcemanager-healthcareapis


x86_64-apple-darwin: unrecognized command line option ‘-arch’

When I attempted to cross-compile cxx from linux, I got the following emitted from link-cplus-plus dependency, which uses cc-rs:
   Compiling link-cplusplus v1.0.6 (/work/depends/link-cplusplus)
The following warnings were emitted during compilation:

warning: c++: error: x86_64: No such file or directory
warning: c++: error: unrecognized command line option '-arch'; did you mean '-march='?

error: failed to run custom build command for `link-cplusplus v1.0.6 (/work/depends/link-cplusplus)`

Caused by:
  process didn't exit successfully: `/work/depends/link-cplusplus/target/debug/build/link-cplusplus-8998b4e36463b047/build-script-build` (exit status: 1)
  --- stdout
  TARGET = Some("x86_64-apple-darwin")
  OPT_LEVEL = Some("0")
  HOST = Some("x86_64-unknown-linux-gnu")
  CXX_x86_64-apple-darwin = None
  CXX_x86_64_apple_darwin = None
  TARGET_CXX = None
  CXX = None
  CROSS_COMPILE = None
  CXXFLAGS_x86_64-apple-darwin = None
  CXXFLAGS_x86_64_apple_darwin = None
  TARGET_CXXFLAGS = None
  CXXFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("true")
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2,sse3,ssse3")
  running: "c++" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-arch" "x86_64" "-Wall" "-Wextra" "-o" "/work/depends/link-cplusplus/target/x86_64-apple-darwin/debug/build/link-cplusplus-a777ebbad1615f40/out/dummy.o" "-c" "/work/depends/link-cplusplus/target/x86_64-apple-darwin/debug/build/link-cplusplus-a777ebbad1615f40/out/dummy.cc"
  cargo:warning=c++: error: x86_64: No such file or directory
  cargo:warning=c++: error: unrecognized command line option '-arch'; did you mean '-march='?
  exit status: 1

  --- stderr

  error occurred: Command "c++" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-arch" "x86_64" "-Wall" "-Wextra" "-o" "/work/depends/link-cplusplus/target/x86_64-apple-darwin/debug/build/link-cplusplus-a777ebbad1615f40/out/dummy.o" "-c" "/work/depends/link-cplusplus/target/x86_64-apple-darwin/debug/build/link-cplusplus-a777ebbad1615f40/out/dummy.cc" with args "c++" did not execute successfully (status code exit status: 1).

Instead of -arch x86-64, the argument should be -march=x86-64 for darwin target (-march instead of -arch as key-value parameter and Apple target is dashed x86-64 instead of x86_64).
If there's some way to override or work around this, then I'd happily go for that until this issue can be fixed.

Can you confirm that you can cross-compile the simplest hello-world Rust snippet on Linux for MacOS? Not that I know it can't ever work, but if I simply add x86_64-apple-darwin target and try to compile "hello, world", I get error messages similar to yours. This suggests that you might have also attempted simply adding the target. And from what I do know, those error message are not unexpected. And even if I tweak my environment to bypass the -arch error, I'll face even more serious problems later on.
As a general rule of thumb. In most common, all non-bare-metal really, cases it's user's responsibility to provide working cross-compiling environment that actually meets specific expectations. Even for compiling pure Rust code, let alone mixed one. This is because even pure Rust relies on C toolchain to link the final application.

Researching this a bit, GCC does have support for the -arch flag, it's unfortunately not in the docs for Darwin-specific flags, but it is stated in the options summary, and the code can be found in the source.
I guess we could use -march, but it would only be possible for a few architectures - for example, what should we emit when the target architecture is Aarch64?
In any case, it seems like the support for -arch in GCC has been available since 2015, earliest release being GCC 6.1, so I'm inclined to not want to support it.
See related https://github.com/golang/go/issues/59544


test

Version of Advanced Rocketry
Have you verified this is an issue in the latest unstable build

[x] Y
[x] N

Version of LibVulpes
Version of Minecraft
Does this occur without other mods installed

[x] Y
[x] N

If Y, what is the MINIMUM set of mods required.

Crash report and log, or VisualVM snapshot (if applicable)

http://pastebin.com is a good place to put them
crash reports that are put in the issue itself are impossible to read

If it's an issue involving lag, please submit a VisualVM sampler snapshot.  Mods like lag goggles cannot provide all of the information needed to debug lag in any useful capacity.
For more information on how to create a snapshot, please see "How to Run CPU Sampling" here.  Then export the snapshot as an .nps.  You may have to zip it since github is weird, then please upload it here.

Description of the problem

Lock this issue please. We don't want any more comments after this.


You probably don't need get_user_model

Description
Import User directly, rather than using get_user_model.
Rationale
get_user_model is meant for reusable apps, while it is my understanding this project is targeted more towards creating websites than packages. Especially within the users app it doesn't make any sense to use it (are we expecting users to create a new user model in a custom app but then keep the users app in their project?) , and switching can prevent new users from assuming they have to use it in all their custom apps (which may or may not be what happened to me). For a more in-depth explanation of why it's an anti-pattern, read this blog post.

Thank you for bringing this to our attention, the blog post is super clear and makes a great case. I noticed a few times that the function didn't play nice with type checker, that makes sense now.
Do you want to send us a pull request?


Colorless Water in JEI

When looking at a recipe that uses water in JEI, the water has no color:

Oh... that's annoying. It's because 1.12 uses a blue texture and 1.13+ uses a white texture. This is also the reason for needing all the biome color overrides.
Thanks for the report; not entirely sure how I'll fix this one.


feat: bump SOPS for age support

Would be nice to support newest version of SOPS, it has support for https://github.com/mozilla/sops#encrypting-using-age, its now supereasy to encrpyt/deprypt files without PGP.
Thanks

Resolved via #56 and #58. Thanks for your help @lukasmrtvy!


Best Practice: Agency.txt has id, lang, phone, fare_url, email fields (WARNING)

agency.txt has valid fields for

agency_id
agency_lang
agency_phone
agency_fare_url
agency_email

Already done. All fields are validated.


typescript type imports fail

pretty quick fails when we use typescript Type-Only Imports
e.g.
ROOT\node_modules\prettier\index.js:9750
      throw error;
      ^
SyntaxError: '=' expected. (9:13)
  87 | import { Public } from '../../common/decorators/public.decorator';
>  9 | import type { IUploadDataObject, IUploadDev, IVallenSetup, IVallenUploadCommandsObject } from './types/types';
     |             ^

pretty-quck-version: 2.0.1

upps - sorry that was meant for the prettier project - but this will just work when updating prettier to 2.x (which supports typescript 3.8)


CompressionUtils performance improvement ask

FileUtils, needs (on or around line 234), a BufferedOutputStream instead of a FilterOutputStream (we think.)  Apologies for not submitting a proper patch.
Affected Version
0.14.2-incubating
Description
When we compress a 20mb smoosh folder today into a zip.  It takes about 15 seconds, whereas the previous implementation of Compression Utils takes about a second.  This makes the hand-off compression much slower.
The software is amazing, thank you.

what was the version where same zipping operation was taking 1 second ?
If you use "local" deep storage then I "guess" it could be due to explicit "fsync" that was enabled sometime last year and could be really responsible for this time. but that was done to fix corruption issues.

Yes, local deep storage, to a disk drive.  The "atomic" update yes, which was enabled last year, made it 20x slower.  You can still do the atomic update, but need to use the Buffered stream, not FilterOutput stream.

I think https://github.com/apache/incubator-druid/pull/6677 is the PR in question. that PR  didn't change anything wrt use[or no use] of buffered output steam, only change introduced is to write to temp file instead of final file and do an fsync . so not sure why that specific change caused perf to go 20x slower (if fsync() is not the problem)
can you point to the change that removed usage of Buffered OutputStream that is causing the slowness ?

It's possible the change from FileOutputStream to FileChannel on https://github.com/apache/incubator-druid/pull/6677/files#diff-8cbeaa1874f0c7db8634387441433351L81 had some effect. Maybe FileOutputStream does buffering differently from the OutputStream you get by doing Channels.newOutputStream(fileChannel)?

#5344 seems to have it?  Should be using something other than, FilterOutputStream?
https://github.com/apache/incubator-druid/pull/5344#pullrequestreview-257485887
https://stackoverflow.com/questions/47721200/javas-filteroutputstream-adds-a-performance-penalty
Hope this helps, and we meet one day at an event, terrific software, ty!

@crodier thanks . ok, I see now.
In this case, Performance degradation is not related to BufferedOutputStream which we never had in that context but due to the use of FilterOutputStream  . I did some tests and https://github.com/apache/incubator-druid/pull/8024 should fix the regression.

Thanks for the investigation @himanshug & @crodier!!


keyboard must be run as root in linux

Replace keyboard with pynput so that scripts can be run as non-root user on linux systems.

Fixed with pynput.


Dynamic configuration in Azure Functions using primary and secondary Azure AppConfiguration stores?

Following Resiliency and disaster recovery recommendations we should configure two Azure AppConfiguration stores. I'm trying to apply the same approach to Azure Functions. But as it is shown in this example IConfigurationRefresher configured for one store. What would be the right way to provide dynamic configuration in this case? Configure separate refreshers for both of stores?
Is there a way of using IOptionsMonitor to control changes?

@sslepova Yes, you could configure refresh for both the stores. I have shared a sample code snippet below.
The IConfigurationRefresher instances can be used to trigger on-demand refresh. This operation is not supported through IOptionsMonitor. In order to make the refresher instances available through dependency injection, you could inject the IConfiguration instance to the service collection, and then call the AddAzureAppConfiguration(IServiceCollection) method. This will make an instance of IConfigurationRefresherProvider available through dependency injection, and its Refreshers property will provide the refresher instances for each configured store.
public static IHostBuilder CreateHostBuilder(string[] args)
{
    return Host.CreateDefaultBuilder(args)
        .ConfigureWebHostDefaults(webBuilder =>
        {
            webBuilder.ConfigureAppConfiguration((hostingContext, config) =>
            {
                var settings = config.Build();

                Action<AzureAppConfigurationOptions> optionsInitializer = (options) =>
                {
                    options.Select("Settings:*")
                            .ConfigureRefresh(refresh =>
                            {
                                refresh.Register("Settings:BackgroundColor");
                            });
                };

                config.AddAzureAppConfiguration(options =>
                        {
                            options.Connect(settings["ConnectionString_SecondaryStore"]);
                            optionsInitializer.Invoke(options);
                        }, optional: true)
                        .AddAzureAppConfiguration(options =>
                        {
                            options.Connect(settings["ConnectionString_PrimaryStore"]);
                            optionsInitializer.Invoke(options);
                        }, optional: true);
            });

            webBuilder.UseStartup<Startup>();
        });
}

API Reference
AddAzureAppConfiguration(IServiceCollection)
IConfigurationRefresherProvider.Refreshers

Thanks for the response, @abhilasharora. Just to clarify: I'm talking about Azure Functions, so by default, app doesn't create host builder explicitly. According to this one should add a startup class inherited from FunctionsStartup, where ConfigurationBuilder.AddAzureAppConfiguration and IConfigurationRefresher are involved. Refresher then configured as singleton service.
So, if I combine these two approaches (to configure two stores and get a refresher from options) don't I end up with two refreshers?) Or should I firstly create AzureAppConfigurationOptions instance, use it to configure both AppConfiguration stores, and get a refresher? If that's possible could you please share an example.

@sslepova I had a chance to discuss your scenario with the team. In case the primary AppConfiguration instance goes down, the goal of disaster recovery handling is to ensure the following.

Function instances that are currently running continue to use the last available cached configuration
New function instances can load initial configuration from secondary App Configuration on startup or restart

During the period when the primary App Configuration is down, there won't be any changes to the configuration in either the primary or the secondary App Configuration instance. For the above scenarios, it should be enough to configure refresh only for the primary App Configuration instance. Here is an example of how the code might look like for an Azure Function.
class Startup : FunctionsStartup
{
    public override void Configure(IFunctionsHostBuilder builder)
    {
        IConfigurationRefresher configurationRefresher = null;
        ConfigurationBuilder configurationBuilder = new ConfigurationBuilder();

        configurationBuilder.AddAzureAppConfiguration(options =>
        {
            options.Connect(Environment.GetEnvironmentVariable("ConnectionString_SecondaryStore"));
        }, optional: true);

        configurationBuilder.AddAzureAppConfiguration(options =>
        {
            options.Connect(Environment.GetEnvironmentVariable("ConnectionString_PrimaryStore"))
                   .Select("TestApp:*")
                   .ConfigureRefresh(refreshOptions => refreshOptions.Register("TestApp:Settings:Sentinel", refreshAll: true))
                   .UseFeatureFlags();

            configurationRefresher = options.GetRefresher();
        }, optional: true);

        IConfiguration configuration = configurationBuilder.Build();
        builder.Services.Configure<Settings>(configuration.GetSection("TestApp:Settings"));
        builder.Services.AddFeatureManagement(configuration);
        builder.Services.AddSingleton<IConfigurationRefresher>(configurationRefresher);
    }
}

@abhilasharora I think what you said in the last response can be useful guidance to other customers as well. Do you mind to update our BCDR doc?


The snap package hangs when calling asciidoctor

Introduced in https://github.com/gohugoio/hugo/pull/11101 due to an invalid "strict confinement" test. Transitioning to core22 bumped ruby from 2.7 to 3.0. Snap debugging indicates gemspec permission requirement that was not present with ruby 2.7:
= AppArmor =
Time: Jun 15 12:54:59
Log: apparmor="DENIED" operation="file_lock" class="file" profile="snap.hugo.hugo" name="/snap/hugo/x1/usr/lib/ruby/gems/3.0.0/specifications/default/abbrev-0.1.0.gemspec" pid=95381 comm="ruby" requested_mask="w" denied_mask="w" fsuid=1000 ouid=0
File: /snap/hugo/x1/usr/lib/ruby/gems/3.0.0/specifications/default/abbrev-0.1.0.gemspec (write)
Suggestion:
* adjust program to not write to $SNAP

Instead of staging the asciidoctor package, consider staging the ruby snap and install the gem.

OK. I'm not doing any release until over the weekend. I have been burned pretty badly when doing Friday releases before.

Although it may be possible to include Asciidoctor with core22 under strict confinement, I have explored several avenues and cannot get past the Apparmor denials. Conclusions:

Revert https://github.com/gohugoio/hugo/pull/11101.
Transition to classic confinement (separate issue, no rush).


Wide table

supersedes #10243
closes #6245
Here is the code as it stands now.  Appending MultiIndex axes now works.  The one test that still fails is  test_append_frame_column_oriented: writing the DateTimeIndex as columns fails because the index is converted to an array of timestamps so np.array creates an object array.  If it could be passed down unmolested np.array() would do the right thing.  I don't yet have a good idea how to fix that.
I added a test for reading the old format, let me know if you think it's sufficient coverage.  I've been testing the reading more broadly against my own data.

so, seeing that we are now generating legacy files for hdf5, I think its time to expand this generator to do it; you can see how we do this for msgpack/pickle. need to expand this to write several types of files for hdf5 (e.g. Series,DataFrame,Panel), with varying datatypes (you can prob use a lot of what is there now).
Then we just import this to compare, makes creation of legacy files real easy.

@MJuddBooth can you update

Yes, I re-factored a bit before the holidays and all the tests looked good but year-end work stopped me before I could finish and push.  I can get back to it today or tomorrow.

closing, but pls reopen if you'd like to finish this up. Its a nice change.

@MJuddBooth Any chance you would be able to finish this up? I know it has been two years...

Oh my.  I need to upgrade my fork to 0.23 so I will try to revisit this week.

Yay :champagne: ! I am happy to help by the way.


Finish rename of "LoginHandler" to "VerifyHandler"

Complete transition from "Login" to "Verify" which makes more sense in the MFA context.

It's green on builds. Are you okay with just merging this @robbieaverill


ci: Setup pipeline

What
This PR adds the configuration of the Github Actions in order to provide a pipeline to the project. It also configures the release-it to generate the CHANGELOG.md. Plus, bumps the version of the package and pushes the tag to the repository.
How
The pipeline consists of the following steps:

Validate

Runs the validations such the prettier, eslint (to be created);
Builds the project to ensure there's no problem.

Release

It's triggered manually. The user must choose the bump type [image 1];
Reuses the validate workflow;
Runs the release-it which will generate the changelog and bump the version;
Publish the package to NPM.

image 1

Thanks, @fsmaia for the review and suggestions! @evertontavares-ifood we must create a GitHub Token in order to generate the release. It must be called RELEASE_IT_TOKEN as described at https://github.com/release-it/release-it/blob/master/docs/github-releases.md#automated. In this previous link, there's a link to generate the token as required.


How to change date format

I want to change date format yyyy-MM-dd to dd-MM-yyyy can you help me.

hi @jswebtech, version 1.0.0 supports this