failed to "StartContainer" for "nginx-ingress-controller" with CrashLoopBackOff

hi, I am using the newest ingress yaml, but still got below error in kubelet log, I have checked the readinessProbe and livenessProbe config, they are all using /healthz as path and 10254 as port. I don't know what's wrong else:
E0226 22:15:39.289364    2536 pod_workers.go:184] Error syncing pod dcee08ac-fc5e-11e6-87bb-080027e0776f, skipping: failed to "StartContainer" for "nginx-ingress-controller" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=nginx-ingress-controller pod=nginx-ingress-controller-2478719372-sst0p_kube-system(dcee08ac-fc5e-11e6-87bb-080027e0776f)"
Below is my yaml config output:
apiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/created-by: |
{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"kube-system","name":"nginx-ingress-controller-2478719372","uid":"dcebdf9e-fc5e-11e6-87bb-080027e0776f","apiVersion":"extensions","resourceVersion":"1130706"}}
creationTimestamp: 2017-02-26T20:05:04Z
generateName: nginx-ingress-controller-2478719372-
labels:
k8s-app: nginx-ingress-controller
pod-template-hash: "2478719372"
name: nginx-ingress-controller-2478719372-sst0p
namespace: kube-system
ownerReferences:

apiVersion: extensions/v1beta1
controller: true
kind: ReplicaSet
name: nginx-ingress-controller-2478719372
uid: dcebdf9e-fc5e-11e6-87bb-080027e0776f
resourceVersion: "1165296"
selfLink: /api/v1/namespaces/kube-system/pods/nginx-ingress-controller-2478719372-sst0p
uid: dcee08ac-fc5e-11e6-87bb-080027e0776f
spec:
affinity: {}
containers:
args:

/nginx-ingress-controller
--default-backend-service=$(POD_NAMESPACE)/default-http-backend
env:
name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.2
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: nginx-ingress-controller
ports:
containerPort: 80
hostPort: 80
protocol: TCP
containerPort: 443
hostPort: 443
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources: {}
terminationMessagePath: /dev/termination-log
dnsPolicy: ClusterFirst
nodeName: nodep1124
restartPolicy: Always
securityContext: {}
terminationGracePeriodSeconds: 60
status:
conditions:

lastProbeTime: null
lastTransitionTime: 2017-02-26T20:05:04Z
status: "True"
type: Initialized
lastProbeTime: null
lastTransitionTime: 2017-02-26T20:05:04Z
message: 'containers with unready status: [nginx-ingress-controller]'
reason: ContainersNotReady
status: "False"
type: Ready
lastProbeTime: null
lastTransitionTime: 2017-02-26T20:05:04Z
status: "True"
type: PodScheduled
containerStatuses:
containerID: docker://3c957d7f3849a5721cbff0ade13c2230104b4ae6878cb21c7c1767b3ce707c2d
image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.2
imageID: docker-pullable://gcr.io/google_containers/nginx-ingress-controller@sha256:977a68f887e1621fb30e80939b3a8f875cbb20c549af1e42d12f2fef272b8e9b
lastState:
terminated:
containerID: docker://3c957d7f3849a5721cbff0ade13c2230104b4ae6878cb21c7c1767b3ce707c2d
exitCode: 255
finishedAt: 2017-02-27T03:15:51Z
reason: Error
startedAt: 2017-02-27T03:15:51Z
name: nginx-ingress-controller
ready: false
restartCount: 89
state:
waiting:
message: Back-off 5m0s restarting failed container=nginx-ingress-controller
pod=nginx-ingress-controller-2478719372-sst0p_kube-system(dcee08ac-fc5e-11e6-87bb-080027e0776f)
reason: CrashLoopBackOff
hostIP: <IP_ADDRESS>
phase: Running
podIP: <IP_ADDRESS>
startTime: 2017-02-26T20:05:04Z

Can you please remove the 'healthz' check and check if the error persists?

@rikatz Thanks I just fix this issue by adding 'apiserver-host' when starting ingress controller. I don't know why there is no any guide or instructions mentioned these so important thing. Maybe the document failed to be updated in time?

@mmzhou you need to check the pod logs. If there's any issue with the connection to the api server you will see a reference to the troubleshooting doc

@mmzhou please reopen if you still have issues (also include the logs of the ingress pod)

I can reproduce this issue:
Steps to reproduce:

setup a cluster using kubernetes-anywhere on vshpere.
clone this repo
cd ingress/examples/static-ip/nginx
kubectl create -f ../../deployment/nginx/default-backend.yaml
add namespace: kube-system to both all the yaml files as default-backend example is in kube-system namespace
kubectl create -f static-ip-svc.yaml
kubectl create nginx-ingress-controller.yaml

$ kubectl get pods -n kube-system
nginx-ingress-controller-2275227678-n7b22   0/1       CrashLoopBackOff   6          7m

$ kubectl logs nginx-ingress-controller-2275227678-n7b22 -n kube-system
I0329 16:39:47.098483       5 launch.go:96] &{NGINX 0.9.0-beta.3 git-3dd7461 git@github.com:ixdy/kubernetes-ingress.git}
I0329 16:39:47.098693       5 launch.go:99] Watching for ingress class: nginx
I0329 16:39:47.099228       5 launch.go:245] Creating API server client for https://<IP_ADDRESS>:443
I0329 16:39:47.100962       5 nginx.go:127] starting NGINX process...
I0329 16:39:47.169809       5 launch.go:115] validated kube-system/default-http-backend as the default backend
F0329 16:39:47.171747       5 launch.go:125] service kube-system/nginx-ingress-lb does not (yet) have ingress points

sorry the pod name is not the same, I tried delete and create again.
$ kubectl describe pods nginx-ingress-controller-2275227678-9q00c -n kube-system
Name:		nginx-ingress-controller-2275227678-9q00c
Namespace:	kube-system
Node:		node3/<IP_ADDRESS>
Start Time:	Wed, 29 Mar 2017 17:43:58 +0100
Labels:		k8s-app=nginx-ingress-controller
		pod-template-hash=2275227678
Status:		Running
IP:		<IP_ADDRESS>
Controllers:	ReplicaSet/nginx-ingress-controller-2275227678
Containers:
  nginx-ingress-controller:
    Container ID:	docker://f06b2fcaffd3792ec78587787021ed5d7104155dc784af4d677f93c4c06d2e7c
    Image:		gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.3
    Image ID:		docker://sha256:383e5ec1f5f90a9d43b4c65392af8d918afe8bf8e6764f41facd21ed7e038d35
    Ports:		80/TCP, 443/TCP
    Args:
      /nginx-ingress-controller
      --default-backend-service=$(POD_NAMESPACE)/default-http-backend
      --publish-service=$(POD_NAMESPACE)/nginx-ingress-lb
    State:		Waiting
      Reason:		CrashLoopBackOff
    Last State:		Terminated
      Reason:		Error
      Exit Code:	255
      Started:		Wed, 29 Mar 2017 17:46:53 +0100
      Finished:		Wed, 29 Mar 2017 17:46:53 +0100
    Ready:		False
    Restart Count:	5
    Liveness:		http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:		http-get http://:10254/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Volume Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-hcnbb (ro)
    Environment Variables:
      POD_NAME:		nginx-ingress-controller-2275227678-9q00c (v1:metadata.name)
      POD_NAMESPACE:	kube-system (v1:metadata.namespace)
Conditions:
  Type		Status
  Initialized 	True
  Ready 	False
  PodScheduled 	True
Volumes:
  default-token-hcnbb:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	default-token-hcnbb
QoS Class:	BestEffort
Tolerations:	<none>
Events:
  FirstSeen	LastSeen	Count	From			SubObjectPath					Type		Reason		Message
  ---------	--------	-----	----			-------------					--------	------		-------
  4m		4m		1	{default-scheduler }							Normal		Scheduled	Successfully assigned nginx-ingress-controller-2275227678-9q00c to node3
  4m		4m		1	{kubelet node3}		spec.containers{nginx-ingress-controller}	Normal		Created		Created container with docker id 21ac0a1295dd; Security:[seccomp=unconfined]
  4m		4m		1	{kubelet node3}		spec.containers{nginx-ingress-controller}	Normal		Started		Started container with docker id 21ac0a1295dd
  4m		4m		1	{kubelet node3}		spec.containers{nginx-ingress-controller}	Normal		Created		Created container with docker id 7ea74ce5aef3; Security:[seccomp=unconfined]
  4m		4m		1	{kubelet node3}		spec.containers{nginx-ingress-controller}	Normal		Started		Started container with docker id 7ea74ce5aef3
  4m		4m		3	{kubelet node3}								Warning		FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "nginx-ingress-controller" with CrashLoopBackOff: "Back-off 10s restarting failed container=nginx-ingress-controller pod=nginx-ingress-controller-2275227678-9q00c_kube-system(e7d0c4cd-149e-11e7-81d4-00505685528b)"

  4m	4m	1	{kubelet node3}	spec.containers{nginx-ingress-controller}	Normal	Created		Created container with docker id 19e419914c9a; Security:[seccomp=unconfined]
  4m	4m	1	{kubelet node3}	spec.containers{nginx-ingress-controller}	Normal	Started		Started container with docker id 19e419914c9a
  4m	4m	3	{kubelet node3}							Warning	FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "nginx-ingress-controller" with CrashLoopBackOff: "Back-off 20s restarting failed container=nginx-ingress-controller pod=nginx-ingress-controller-2275227678-9q00c_kube-system(e7d0c4cd-149e-11e7-81d4-00505685528b)"

  3m	3m	1	{kubelet node3}	spec.containers{nginx-ingress-controller}	Normal	Created		Created container with docker id 8a31b9e1ea87; Security:[seccomp=unconfined]
  3m	3m	1	{kubelet node3}	spec.containers{nginx-ingress-controller}	Normal	Started		Started container with docker id 8a31b9e1ea87
  3m	3m	4	{kubelet node3}							Warning	FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "nginx-ingress-controller" with CrashLoopBackOff: "Back-off 40s restarting failed container=nginx-ingress-controller pod=nginx-ingress-controller-2275227678-9q00c_kube-system(e7d0c4cd-149e-11e7-81d4-00505685528b)"

  3m	3m	1	{kubelet node3}	spec.containers{nginx-ingress-controller}	Normal	Started		Started container with docker id 652f48600bf1
  3m	3m	1	{kubelet node3}	spec.containers{nginx-ingress-controller}	Normal	Created		Created container with docker id 652f48600bf1; Security:[seccomp=unconfined]
  3m	1m	7	{kubelet node3}							Warning	FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "nginx-ingress-controller" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=nginx-ingress-controller pod=nginx-ingress-controller-2275227678-9q00c_kube-system(e7d0c4cd-149e-11e7-81d4-00505685528b)"

  4m	1m	6	{kubelet node3}	spec.containers{nginx-ingress-controller}	Normal	Pulled		Container image "gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.3" already present on machine
  1m	1m	1	{kubelet node3}	spec.containers{nginx-ingress-controller}	Normal	Created		Created container with docker id f06b2fcaffd3; Security:[seccomp=unconfined]
  1m	1m	1	{kubelet node3}	spec.containers{nginx-ingress-controller}	Normal	Started		Started container with docker id f06b2fcaffd3
  4m	9s	26	{kubelet node3}	spec.containers{nginx-ingress-controller}	Warning	BackOff		Back-off restarting failed docker container
  1m	9s	9	{kubelet node3}							Warning	FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "nginx-ingress-controller" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=nginx-ingress-controller pod=nginx-ingress-controller-2275227678-9q00c_kube-system(e7d0c4cd-149e-11e7-81d4-00505685528b)"

I am having the same issue as well. It happens when I set type: NodePort for the --publish-service service. It does not have load balancer associated which triggers this code:
		if len(svc.Status.LoadBalancer.Ingress) == 0 {
			// We could poll here, but we instead just exit and rely on k8s to restart us
			glog.Fatalf("service %s does not (yet) have ingress points", *publishSvc)
		}

I was wondering if nginx controller even supports NodePort services?

The solution for me was to remove --publish-service altogether

Same for me. I am creating an nginx ingress controller using methods outlined in a number of guides, but using service type NodePort in front of the nginx controller instead of load-balancer. I kept getting:
⇒  kubectl get pods | grep nginx
ingress-nginx-1785899290-wccg8                  0/1       CrashLoopBackOff   1          8s
with the pod logs showing
⇒  kubectl logs -f ingress-nginx-1785899290-wccg8 I0621 19:08:46.331218       5 launch.go:101] &{NGINX 0.9.0-beta.8 git-245e6b0 https://github.com/kubernetes/ingress} I0621 19:08:46.331322       5 launch.go:104] Watching for ingress class: nginx I0621 19:08:46.331741       5 launch.go:257] Creating API server client for https://<IP_ADDRESS>:443 I0621 19:08:46.393209       5 nginx.go:185] starting NGINX process... I0621 19:08:46.479777       5 launch.go:120] validated default/default-http-backend as the default backend F0621 19:08:46.483025       5 launch.go:130] service default/ingress-nginx does not (yet) have ingress points
I also removed the --publish-service param and it is finally staying up. Not sure if it will work yet as haven't finished testing, but I found this when Googling the above error.

I had to add to do the following to get it to work:
--- nginx-ingress.orig/templates/controller-deployment.yaml     1969-12-31 16:00:00.000000000 -0800
+++ nginx-ingress/templates/controller-deployment.yaml  2017-06-20 11:51:38.539392152 -0700
@@ -29,7 +29,7 @@
args:
- /nginx-ingress-controller
- --default-backend-service={{ if .Values.defaultBackend.enabled }}{{ .Release.Namespace }}/{{ template "defaultBackend.fullname" . }}{{ else }}{{ .Values.controller.defaultBackendService }}{{ end }}

       - --nginx-configmap={{ .Release.Namespace }}/{{ template "controller.fullname" . }}

       - --configmap={{ .Release.Namespace }}/{{ template "controller.fullname" . }}
       - --tcp-services-configmap={{ .Release.Namespace }}/{{ template "fullname" . }}-tcp
       - --udp-services-configmap={{ .Release.Namespace }}/{{ template "fullname" . }}-udp
     {{- range $key, $value := .Values.controller.extraArgs }}

@kfox1111 that flag was changed in 0.9-beta.1 https://github.com/kubernetes/ingress/releases/tag/nginx-ingress-controller-0.9-beta.1

yup. and by default the charts pointing at the 0.9 series now:
https://github.com/kubernetes/charts/blob/master/stable/nginx-ingress/values.yaml#L8
So the stable charts broken out of the box when I tried it a couple days ago.

@andreychernih removing the --publish-service worked for me.
What is the impact of removing it?

What is the impact of removing it?

The status you see in the ingress rules will show the IP of the node/s where the ingress controller is running and not the FQDN of the load balancer.

@vendrov I did not notice any in my use-case


In the following scenario，When the array data is changed responsively,  the view cannot be updated normally

Version
2.6.11
Reproduction link
https://codesandbox.io/s/youthful-khayyam-4kid1?fontsize=14&hidenavigation=1&theme=dark
Steps to reproduce
step1: I defined a contextMenu component，in this component，i append the $el element to body in mounted hook。
<template>

I am ContextMenu.

</template>

step2： In App.vue，i render a list，and the list data is obtained asynchronously。
<template>

      {{ item.id }}

    <ContextMenu />

</template>

step3: the question is，when  i change the list data,  the view is not update yet.
What is expected?
I hope to see is:  When the list data is updated, the view is updated。
What is actually happening?
the view is not update yet.

I try some methods to avoid this question, such as:

wrap  ContextMenu or list in a  div element.
init the list at least one data.

but I don't kown why,  and I really hope you can tell me why.
thank you very much!

You cannot manually manipulate the HTML that is being rendered by Vue because it wouldn't be able to keep the changes up to date. So you can't  document.body.appendChild(this.$el);. You should use https://portal-vue.linusb.org/ for that purpose


4.2.1 Site link goes to a list of records for the same species at the selected site.

Already in place.


RkTabSet and RkTab not found in version 3.1.2

Issue type
I'm submitting a ...  (check one with "x")

[x] bug report
[ ] feature request

Issue description
RkTabSet and RkTab not found in version 3.1.2

Hi @yaibait
Some components included in framework were not released to npm.
Just like RkCalendar (#294), RkTabSet was not included in latest release. To use it you can install framework directly from github:
npm i react-native-ui-kitten akveo/react-native-ui-kitten

@artyorsh  Thanks you 👍


"Roșiori,Mihai Bravu, Vaida" are villages in "Roșiori" commune, "BIHOR" county

Description
When an user search a location, this appear in the drop-down menu
according to the law
Precondition
The website ( https://peviitor-ro.github.io/ui.orase/ ) is up an running.
Empty the search bar after every step

Steps 1
Write "Roșiori"
Expected result
In the drop down menu you find "Sat Roșiori, BIHOR(Roșiori)"
Test Data
Roșiori

Steps 2
Write "Mihai Bravu"
Expected result
In the drop down menu you find "Sat Mihai Bravu, BIHOR(Roșiori)"
Test Data
Mihai Bravu

Steps 3
Write "Vaida"
Expected result
In the drop down menu you find "Sat Vaida, BIHOR(Roșiori)"
Test Data
Vaida


Merge to 1.1.x: Make Pub/Sub thread pools Spring-managed beans (#1505)

The two thread pools are hardcoded in autoconfiguration, so they fall through the cracks when a Spring application shuts down:

Spring does not manage them because they are not context-managed beans.
gRPC stub does not manage them because FixedExecutorProvider.shouldAutoClose() is false (also, we don't call SubscriberStub.shutdown(), which we arguably should do, as well.

This does not manifest on the publisher side because that thread pool is never started, but the subscriber thread pool is started by way of subscriber stub creation when PubSubSubscriberTemplate is instantiated.
Cherrypicked from #1505.
Fixes #1503.
Addresses the Pub/Sub part of #1506.

Codecov Report

Merging #1518 into 1.1.x will increase coverage by 0.03%.
The diff coverage is 100%.

@@             Coverage Diff              @@
##              1.1.x    #1518      +/-   ##
============================================
+ Coverage     68.29%   68.33%   +0.03%
- Complexity     1429     1431       +2
============================================
  Files           203      203
  Lines          5728     5735       +7
  Branches        578      578
============================================
+ Hits           3912     3919       +7
  Misses         1579     1579
  Partials        237      237

Flag
Coverage Δ
Complexity Δ

#integration
?
?

#unittests
68.33% <100%> (+0.03%)
1431 <4> (+2)
:arrow_up:

Impacted Files
Coverage Δ
Complexity Δ

...toconfigure/pubsub/GcpPubSubAutoConfiguration.java
90.47% <100%> (+0.56%)
38 <4> (+2)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b48e8d0...0bdabdc. Read the comment docs.


add keyboard navigation to go up down the list of systems to jump the line to search

note: tab based navigation does currently work.


Fix invalid nodiscard attribute

nodiscard attribute with only 1 set of square brackets is not valid, neither is the all caps version.

@TylerMSFT

Can you review this PR?
IMPORTANT: When this content is ready to merge, you must add #sign-off in a comment or the approval may get overlooked.

#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#sign-off


Documentation is in JSDOC format

Need to swap documentation into a python style, and probably generate a readthedocs

I copied the doc from libmeshctrl, which would output markdown from jsdoc. Looking like I'm going to switch to sphinx with Google style strings for this one, I have a local branch with it started, just need to port the strings.


barrel-migrations is broken

Setup
Versions

Rust: nightly (rustc 1.29.0 (aa3ca1994 2018-09-11))
Diesel: 1.3.3 / master
Database: psql
Operating System linux

Feature Flags

diesel: barrel-migrations,postgres

Problem Description
Using the barrel-migrations-feature is broken in the current diesel / barrel version.
What are you trying to accomplish?
Use diesel with barrel as described in https://github.com/spacekookie/barrel/blob/master/guides/diesel-setup.md.
What is the expected output?
Working migrations.
What is the actual output?
[diesel 1.3.3, barrel <= 0.2.0]
$ diesel migration run
Running migration mod.rs
Received an empty query

[diesel master, barrel master]
$ diesel migration run
Running migration mod.rs
Executing migration script /path/to/project/migrations/2018-09-21-125612_init_db/mod.rs/up.sql
Failed with: Received an empty query

Are you seeing any additional errors?
[diesel 1.3.3, barrel <= 0.2.0]
The temporary created migration project build fails silently in the background:
$ cargo build
   Compiling tmp-generator v0.0.0 (file:///tmp/barrel.nkunnkN46LTx)
error: couldn't read "/path/to/project/migrations/2018-09-21-125612_init_db/mod.rs/mod.rs": Not a directory (os error 20)
 --> src/main.rs:8:1
  |
8 | include!("/path/to/project/migrations/2018-09-21-125612_init_db/mod.rs/mod.rs");
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to previous error

error: Could not compile `tmp-generator`.

To learn more, run the command again with --verbose.

Steps to reproduce
Follow https://github.com/spacekookie/barrel/blob/master/guides/diesel-setup.md.
Checklist

[x] I have already looked over the issue tracker for similar issues.
[ ] This issue can be reproduced on Rust's stable channel. (Your issue will be
closed if this is not the case)
You're kidding?! diesel_cli with barrel-migrations doesn't build on stable.

This seems to be a problem in combination with barrel, I'm going to investigate this further.

Barrel is a separate project, and is not maintained by Diesel. Please open an issue with that project instead.

There already is an issue. I doubt that it's actually the code in diesel that's causing the issue. But I'll have a look at it tomorrow.


restic diff single snapshot

I can diff two snapshots to find out what changes there were.  I would like to be able to get that information for the first snapshot, compared to an empty repo.  Then I thought this would also make sense for any single snapshot - giving you effectively the stats of what a restore of that snapshot would be.

I would like to be able to get that information for the first snapshot, compared to an empty repo.

You can use restic ls latest (or restic ls --long latest) to show all files of the latest snapshot (the first snapshot in your case).

Then I thought this would also make sense for any single snapshot - giving you effectively the stats of what a restore of that snapshot would be.

You can use restic stats SNAPSHOTID to show stats about a specific snapshot. The stats command can display different counting modes when using the --mode flag. Please use restic stats --help for more information.
PS: Please use the forum for questions about usage and the like.

ok, thanks for the info, and will use the forum


object_store: Use the official AWS SDK for s3 interactions

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Recently the official AWS SDK for Rust was made GA.
https://aws.amazon.com/blogs/developer/announcing-general-availability-of-the-aws-sdk-for-rust/
This ticket is created to discuss the potential to refactor the existing s3 implementation in favour of using the new SDK.
Describe the solution you'd like
remove the current HTTP(s) based implementation in favour of the official SDK
https://docs.rs/aws-sdk-s3/latest/aws_sdk_s3/
Describe alternatives you've considered

Additional context

cc: @tustvold @alamb

Regarding dependency footprint: https://crates.io/crates/aws-sdk-s3/1.4.0/dependencies
aws-sdk-s3 comes with 13 AWS-specific crates (direct dependencies only, haven't looked at indirect ones), 9 of which are used for the API generator generator smithy alone.

Also AWS is not the only implementation of the S3 API out there and I doubt they support features (like conditional operations) that other S3 implementations support.

@chitralverma  what would be the benefit of using the official SDK? Are there features it supports that are not supported by the custom one?
I can imagine potentially some of the more advanced access controls or something
Since the ObjectStore is a trait, perhaps we could create a new crate (could start outside of this repo) that uses the official SDK that could be used by anyone who needs the extra features

Closing as not planned, thank you all for the discussion

One of the limitations of the current object-store implementation is that its authentication support is more restrictive than the one provided by AWS, for example. https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-sourcing-external.html. How can this feature be added to the current implementation?

How can this feature be added to the current implementation?

Users can provide their own credential providers - https://docs.rs/object_store/latest/object_store/aws/struct.AmazonS3Builder.html#method.with_credentials
This could even be using the AWS SDK if they so wish

for users that are using those libraries via a Python wrapper, how should this work? I.e. object_store imported by polars used in Python

That is a question for the maintainers of the python wrapper

So it seems to me the summary of this ticket is:

There is a legitamate need for certain authentication / other features that are not implemented in the object_store's re-implementation of the S3
The "official" AWS s3 sdk provides these other features but at the expense of a significantly larger dependency set

Thus I propose (and I think this is what @tustvold is also proposing) for the first step is that someone implements the ObjectStore interface using the official AWS S3 SDK outside of the object_store crate. This would mean something like
struct OfficalAmazonS3 {
...
}

impl ObjectStore for OfficialAmazonS3 {
 // implementation based on aws sdk goes here
}

I won't have time to devote to this project (likely ever) but I think it would be straightforward for anyone with the need (the earlier versions of object_store were actually implemented this way).
Once that code is complete, I think we could then have a separate discussion about "is there willingness to maintain that implementation in the object_store crate"  (we would have to find some maintainer bandwidth to do so) to ease distribution
But the first step in all possible futures is for someone to implement OfficialAmazonS3.

There is a legitamate need for certain authentication / other features that are not implemented in the object_store's re-implementation of the S3

Just FYI, I'm working on a project focused on implementing all the request signing features provided by AWS. I believe it would be beneficial to have a separate signing crate instead of the entire AWS SDK.

One specific problem we have is how we inject this logic when the object store is wrapped by a crate which exposes python bindings, for example delta-rs, polars or DataFusion-python
Let's say this crate would already exists,  how can we integrate it with those frameworks above ?

Let's say this crate would already exists, how can we integrate it with those frameworks above ?

Reqsign will provide a Python binding, allowing the Python ecosystem to integrate with it directly.

Let's say this crate would already exists, how can we integrate it with those frameworks above ?

Either the same way datafusion-cli does by overriding the credential provider at construction time, or by the maintainers of said python wrapper adding first class support for credential providers. We already provide sufficient extension points for them to do this

@edmondop  and I spoke about this at the NYC DataFUsion meetup yesterday
I think the usecase is that

User uses object_store indirectly via polars
polars does not provide any way to modify / configure s3 connections at runtime

If they could control the source, they could use  the existing object_store CredentialProvider trait but they control neither the source or distribution
@edmondop mentioned something about some AWS SDK mechanism that calls out to an external program / process to get credentials. While less efficient this would allow someone to plug in whatever authentication mechanism they wanted without having to change the source code
Perhaps we can follow this model somehow and build a "get credentials from an external process / souce" into the object store to provide a way that did not require the aws sdk, nor a bug dependency tree, but could support arbitrary authentication 🤔

Thank you @alamb . My proposal is that we extend object_store to support an additional way of providing credentials, called credential_provider.
This process is also implemented as a part of the AWS SDK as described here and have the strong benefit of keeping the object_store crate simple, but support users in providing their custom authentication mechanism.
ObjectStore could be then extended to use a custom environment variable and use the credential provider instead. The documentation from AWS provides the type of schema required by the SDK, which we can follow
{
    "Version": 1,
    "AccessKeyId": "an AWS access key",
    "SecretAccessKey": "your AWS secret access key",
    "SessionToken": "the AWS session token for temporary credentials",
    "Expiration": "RFC3339 timestamp for when the credentials expire"
}

I am sure many have already implemented a custom credential provider, and the object store implementation would be minimal. It would allow people to use objecstore, and thefore datafusion and polars, with complex authentication mechanisms without polluting the objecstore crate
what do you tihnk @tustvold ? Is this solution acceptable? Can I open a new issue for it?

My only worry would be that the config system currently may be accepting "untrusted" user input, and this would then allow for a limited form of RCE.
Perhaps we could just provide a CredentialProvider that calls a process, and leave it to polars to hook it up based on a config?

Perhaps we could just provide a CredentialProvider that calls a process, and leave it to polars to hook it up based on a config? This would avoid surprises for people using object_store in a server context?

Yes, I agree it should be something explicitly enabled

Edit: it might also be worthwhile filing a polars issue about making CredentialProvider overridable, as this might be generally useful for them

I think most users of Polars use the python API (aka are not compiling rust programs) which means Rust level APIs (if I understand what you are proposing) is not likely to satisfy the usecase for their users

Edit edit: IIRC we provide a mechanism to use the Azure and GCP CLIs to source tokens, perhaps one option would be to add something similar for S3?

That certainly sounds like a good idea to me (follow the same model as the others)

@edmondop  -- can you pull the relevant discussion into a new (non closed) ticket ?

I think most users of Polars use the python API

I meant some way to expose it in the python API

Opened this https://github.com/apache/arrow-rs/issues/6422

FYI I have filed an upstream polars issue as this seems to be a fairly frequent source of these reports - https://github.com/pola-rs/polars/issues/18979

As an update here, polars now exposes the credential provider functionality allowing fetching credentials via alternative mechanisms


Update 125-fourier-series.md

Added my contribution to the Fourier Series challenge. You can click the keys to hear the waveform that's drawn.

This is awesome ❤ Thank you!


Switch to testify for testing

Description
We currently don't use any testing or mocking library to test functions.  This was my original goal, but now that there is quite a bit more code, tests have become ugly and are hard to read.  I would like to start using the testify library for mocking and assertions.  With this change, I would like to do a general overhaul of tests as well, so they look clean, layout what is being tested, and follow best testing practices.
Acceptance Criteria

[ ] Move mocking to testify mocking
[ ] Clearly define what each test is trying to accomplish
[ ] Make sure both happy and error paths are covered for each test
[ ] Implement table testing

May reevaluate this once #20 is completed.  I think there will still be a need for it, but it would be nice to get interface sizes down first.

Moving work for this ticket to #20


fixed link

I fixed the link in Router Architecture.md and Repeater Architecture.md

Sorry, I renamed it


Support for multiple log backends and formats

Since #16 we had a global definition of ClientLogger and ServerLogger in the log package and both of these were initialized in logging.go's init() function. In order to separate logging backends for client and server, we improved func NewLogger(), which is now capable of configuring more than one backend per logger. It now accepts three arguments:

module - a descriptive string (for instance "client" or "server"),
logFilePath - path to the file that logger will output its logs to in addition to stdout. If an empty string is given, file backend will not be used, e.g. logs will only be outputted to console.
loglevel - the initial loglevel of newly created logger (can be changed later with func (logger *Logger) SetLevel()).

Initialization of the logger used across client package was moved to init() in client.go. We kept this per-package logger for convenience, however the server-side logger is now hooked to a particular instance of the Server, as it seemed more appropriate. Logger initialization logic and calls in client.go, server.go, cli and test packages were updated accordingly. A new test was added that checks behavior in case invalid log level was passed to func NewLogger().
There are further changes, including:

Definition of three logging formats in a file log/formats.go. Note that FORMAT_SHORT is used by default for client logger (because it doesn't need as fine-grained logs as the server), FORMAT_LONG is used for the server's logger when outputting to console, and FORMAT_LONG_COLORLESS is used for the server's logger when outputting to file (because it doesn't need color information like console output). These updated formats output logs that are much more readable than before.
Definition of constants denoting log levels in log/levels.go.
Package function TurnOff() was discontinued in favor of func (logger *Logger) Disable().
The log level for server and client loggers during tests was modified from debug to notice.

One more thing: this PR also removes the problem of duplicated timestamps in logs. So, instead of this:
2017/09/06 12:38:52 [server] 12:38:52.562 Start ▶ NOTI 006 Emmy server listening for connections on port 7007

we now have something like this:
[server][Mon 25.Sep 2017,14:11:041] Start ▶ NOTI  Emmy server listening for connections on port 7007

Notice how much easier it is to read now :slightly_smiling_face:

@miha-plesko Last comments implemented, rebase done (successfully this time :wink:). Thanks a lot for a thorough review!

🎆 🍾 🎉


Can falco provide ARM execution pkg as default?

Motivation

Feature

our system use ARM and AMD64, falco do not support ARM as default, need to build from code which cost many work
Alternatives

Additional context

Hi @swxEmily! Would you expand on what you mean "by default"? If you are referring to the community releasing ARM artefacts, that's something that's about to happen from the next patch release: https://github.com/falcosecurity/falco/issues/2070

Hi @swxEmily! Would you expand on what you mean "by default"? If you are referring to the community releasing ARM artefacts, that's something that's about to happen from the next patch release: #2070

So when will 0.32.1 be released?

Follow that issue for the most up-to-date developments:
Targeting the week starting July 4th, 2022

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale

Falco now fully supports ARM64 and we regularly distribute builds for that architecture (e.g. https://download.falco.org/?prefix=packages/bin/aarch64/). Closing this.
/close

@jasondellaluce: Closing this issue.

In response to this:

Falco now fully supports ARM64 and we regularly distribute builds for that architecture (e.g. https://download.falco.org/?prefix=packages/bin/aarch64/). Closing this.
/close

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.


Deleted secret isn't recreated until refresh

Describe the solution you'd like
When a secret (created from an ExternalSecret) is deleted it isn't recreated until the refresh happens (if one will).
The status field on the ExternalSecret remains in Ready state despite one of the conditions being   # - The target secret exists (from this page).
Instead, it should be recreated immediately.
What is the added value?
Deleted secrets that should exist get recreated.
Give us examples of the outcome
Deleted secrets get recreated if they are owned by the operator.
Observations (Constraints, Context, etc):
This may only be possible with the creationPolicy: Owner due to needing the OwnerReference on the secret.
Using:

Azure Key Vault
K8s 1.20.5
0.3.3 of the helm chart

Moved this comment from PR to here since it's probably a better place to discuss.

This is tougher than it looks due to a catch 22.
While I was working on the above, I came to realise that we do not store the latest snapshot inside the externalsecret object. As a consequence, at the moment we do not know what's supposed to be inside the kind=secret.
Here are some thoughts:

We can easily deal with deleted secret, as long as es.Status.SyncedResourceVersion != "" , kind=secret should exist and we can trigger the global refresh which will recreate it with proper values.
At the moment we cannot detect drifting inside the kind=secret (i.e. external forces changing it).

If we say that any target secret name content's are under our responsibility, we can create a hash of the data (ie. marshalling in json and doing md5 on the text), and store that hash inside an annotation on the secret. If on the reconcile loop annotation differs from the recalculated hash, then force the refresh (expected behaviour for ref=owner?)
If it's expected that the target secret can receive additional keys from an external operation, then we need to compare secret map which we do not have. To fix this we would need to amend the CRD and add an additional field to the Status (lastSecretSynced) where we would store a base64 encoded json containing the secret. This approach is potentially concerning from the security perspective.
Your thoughts @moolen / @knelasevero ?

So, to sum it up (I am thinking out loud here).
In case of creationPolicy: Owner we can either:

Just trigger the whole resync and fetch secrets from cloud provider
Store the latest obtained secret data in the annotations (Or status, requires CRD change) on both ExternalSecret and kind=secret and perform full sync only in case they are different/missing
Store the latest obtained data inside ExternalSecret (base64 encoded?) and use that to avoid any premature calling to cloud provider (requires Crd Change)

In case of creationPolicy: Merge we will likely have more often calls to the cloudprovider, and we can:

perform full cycle on any changes to that secret (we do not know which fields are created by us)
have a hybrid solution where we store the hash in one annotation and list of fields created by us in another, and validate it with the hash written in externalsecret
Store the latest obtained data inside ExternalSecret (base64 encoded?) and use that to avoid any premature calling to cloud provide.

How precious are we in the current state of things about adding new fields to CRD?

adding new fields to CRD?

That's not a problem! We're consciously decided to stay in alpha to gather feedback and be free to add/remote fields to the v1alpha1 CRD.

creationPolicy: Owner

I think it would be best to perform a full cycle when the secret has been deleted: External state may have changed (e.g. templateFrom ConfigMap has changed). IMO we shouldn't store the plain text secret values in the status subresource.

creationPolicy: Merge

We should know which fields were created by us through the field ownership mechanism (secret.metadata.managedFields). I think we should do a full cycle here too: diff existing keys with the ones from the provider and apply a patch to that resource. If nothing has changed we don't need to patch the resource.

Fixed by https://github.com/external-secrets/external-secrets/pull/349
@moolen can we close this off?

Thanks!


⚠ okto-ageg (staging) has degraded performance

In 63d271c, okto-ageg (staging) (https://okto-ageg-staging.gitlab.io/) experienced degraded performance:

HTTP code: 500
Response time: 15321 ms

Resolved: okto-ageg (staging) performance has improved in 4713960.


mixture containing one point doesn't correctly sample for it, nor generate adequate statistics

I.
See the summary statistics for:

mixture(0)

II.
See the samples for:
sampleN(mixture(0), 10)

Still doesn't work correctly even with #1591, unfortunately.

I'd like to flag this as pretty important.
I previously thought that it's not a big deal, because who cares if it's exactly zero or 1e-16.
But then I tried to use mx to simulate multiple discrete branches, and got confused why my code wasn't working properly.

https://www.squiggle-language.com/playground?v=0.8.5#code=eNptjj0LwjAURf%2FKJVMLfpCMQl106dzROKT2FQNJWpqIgva%2Fm5qKCi6Bc3Pv4d1ZQ626mLDrGmIbdlKeyoajAJcugYggpHujj2hv2WFuLjC3jvl3Z7lFpWxvqKKwaofO7rUPMZXuE1vVZ9IB9%2BkBHmn7SKTbWYyiwPuqcCYHjvUa2mEgZXCKZ8dYe1y7i2lQE2yMDM%2Fy5CHj6Z9MJJmYZK%2BF%2BFlIVtq%2B817XhiSbPkbpcjY%2BARYzYbM%3D

Yea, that makes sense. I'd imagine that this specific issue isn't dramatically difficult? Maybe 3-20 hours, could be worth it soon.

Incidentally, the approach that squiggle.c takes here is, I think, more robust, so that these things don't happen, because it defines variables as samplers, which are conceptually clearer.
In fake pseudo-code, it is doing something like:
let a = () => 0
let b = () => sample_from_normal(1, 10)
let mixture = (samplers, weights) => {
   let normalized_cumsummed_weights = cumsum(weights) / sum(weights)
   let p = Math.random()
   let i = findFirstIndex(normalized_cumsummed_weights, x => p < x)
   return samplers[i]()
}
let resutlSampler = mixture([a,b], [1, 9])
let result = Array.from({length: 10000}, _ => resultSampler())
console.log(result)

You get more verbosity, but the type of objects which a and b are is a bit clearer, and there is no difference between a sampler which samples from discrete vs continuous distribution.
I don't know to what extent you will find this useful, but I thought I'd share.

Yea, I think that approach can be good in certain use cases. You can do similar things in Squiggle if you want, though it's not as emphasized.

Representing all distributions as generator functions that lazily return samples is a very tempting idea to me.
Benefits:

no need to decide on sample count in advance (important especially in the dependency chain when we add imports, and also in mixtures as discussed above)
I think it might unlock the capabilities similar to WebPPL and similar probabilistic languages, i.e. Bayesian inference (we'd have to implement some MCMC algorithm for that, of course; but with our current SampleSet implementation, non-trivial Bayesian inference is not possible at all)
no performance costs when the model is executed, which is important for imports

Risks/problems:

it's unclear if cdf, pdf, mean and other statistics functions would be compatible with that approach
also I'm not sure about the long-term performance implications; pulling one one at a time from a deeply nested chain of generator functions is slower because of non-locality, but we might pull N samples and then it's fine? anyway, not a problem in the short-term, we have many other performance bottlenecks
we'd have to be careful with correlation semantics, but I think that's doable and might even be more natural than with SampleSets (I'd have to think more about it to verify)

One more meta-level benefit is that if we do this, it'd be much clearer to me that Squiggle is necessary, and not just a trivial layer of syntax sugar on top of JS.

One less extreme way to try this, without changing the entire language and its semantics, is to add a 4th type of distributions: "GeneratorDist". I think it'd be mostly compatible with what we have right now: we could treat it as a lazy SampleSetDist at first, and pull sampleCount samples from it when it's first used. Then we could make it more lazy when it's possible.

Good point about imports. I think that generator functions are neat, but more complicated for users - so I was kind of hoping we wouldn't need them, but maybe we would for imports to scale.
I'm pretty nervous about doing another huge change now.

is to add a 4th type of distributions: "GeneratorDist". I think it'd be mostly compatible with what we have right now: we could treat it as a lazy SampleSetDist at first, and pull sampleCount samples from it when it's first used.

I would be interested/fine with doing something like this.

I think that generator functions are neat, but more complicated for users

Do you have examples where that might be the case?
My hope for this idea is that we could make a switch completely transparently from users, mostly. They don't need to know that their x = 2 to 5 doesn't generate a list of samples immediately.


[question]: How to integrate this with firebase?

I was able to setup the library and it is working fine with local notifications for Android and iOS.
How can i integrate this with Firebase so that it can work with android as well as with iOS?
 Basically, I want to make sure i push to single source for notification from backend. that single source handles the notification sending to mobile app.
I went through couple of questions and it confused me.

You can see this post: https://dev.to/jakubkoci/react-native-push-notifications-313i

@bhaskardabhi  Hey
were you able to do this?
I get notifications when app is in background or quit state but not when it is in foreground.


Multi dimestional array not implemented

I'm currently developing a Variant/BSON (to work with MongoDB) library, and I find really intersting to see how JKI would accomplish this.
I'm copying the drjdpowell Variant/JSON library, that seems to work well with any dimension size and array element type.
I tried to change this repo code to make Multi D support but using OpenG 'Array to Array of VData__ogtk.vi'  makes things difficult, as it reshapes to a 1D array.

I could make it work using the example of drjdpowell, but I'm not sure if this is the best way to do it.
Consider a VI 'Variant to Array of Variant.vi' that inputs 'Variant' and returns 'Array';
When the input is like Array it will return a Array normally, but in the case of input be Array (MultiD array) it would return Array. So when the reentrant Flatten to JSON.vi is called, the Variant passed would be an Array.
This way could even work for 1D or any D, but maybe the primitive flatten is optimized enough to make a each type case and sacrefice memory to processing, that might be important if someone may be using this for some real time acquisiton with 1D arrays.
The same idea could be applied with 2D arrays that are simple and very common to be used in this cases, like 2D array of numerics, but of course not for arrays of clusters.

Right now we don't support multidimensional arrays as they are not supported natively in JSON. Before we can support multidimensional arrays, we need to decide what is the expected behavior in flattening and unflattening such arrays.
In JSON you can create arrays of arrays that would be the closest item to multidimensional arrays in LabVIEW. For example the following would be a valid JSON object:
[ [ ], [1], [1, 2], [1, 2, 3, 4, 5, 6, 7], [[1,2],[1,2]]]
Such arrays can have rows and columns that vary in size from element to element. Furthermore the dimensionality can increase for some rows and columns. LabVIEW does not natively support this kind of data structures as multidimensional arrays.
What LabVIEW does support is arrays of clusters of arrays which would support variable sized array elements but using them raises a question how to flatten them. The natural flattening would be something like the following:
[ { [ 1 ] }, { [ 1, 2 ] } ]
But that may not be what the user expects or wants. Also clusters of arrays doesn't really solve the problem of variable dimensionality.
To include support for multidimensional arrays to the JSON library we need to come up with optimal and most natural expected behavior for both flattening and unflattening operations. Only after that can we focus solving the technical issues of how to exactly implement that. But I don't think the technical solution is a road block here.

I would expect that JSON Serialization input would accept "anything" and returns all as variants.
Example [ [ ], ["1"], [1, 2], [1, 2, 3, 4, 5, 6, 7], [["a","b"],[1,2]]]
It does not matter the input. If the type was expecting an integer the searlization would do it's best to convert whatever data is there to the type defined. So "1" would be converted to 1. The same if type expected was String, so the output would be converted to String, and in case it's a document, it would be a toString of the nested document.
If the format is not convertable, like type expected is number and field contains a document, then an error or warning would be throwed.

I have thoughts on this, too -- this is an important feature request.
Right now, I really need to be able to flatten and unflatten a 2D array of strings for Google Sheets access. For example:
{
  "range": "Sheet1!A1:E1",
  "majorDimension": "ROWS",
  "values": [
    [
      "1",
      "2",
      "3",
      "4",
      "5"
    ]
  ]
}

In the case of flattening a 2D array from LabVIEW to JSON, it's a no brainer since each LabVIEW rows/columns have to all be the same length in a multi-dimensional array.
In the case of unflattening it's trickier and the options are to either (A) truncate all LabVIEW rows to the shortest row/column, or (B) pad all rows/columns (in LabVIEW with default data elements) to the longest row/column.
Padding the LabVIEW array with default data seems to be the best option, since there is no loss of data. I can see cases where it would be nice to be able to unflatten each JSON row/column separately and I have other feature ideas on how to do that -- specifically, we need to be able to tag LabVIEW strings as #JSON similar to how EasyXML has an #xml tag.

Feature added to release 1.0.6


Create a custom prometheus metrics exporter

WHAT:
Create a prometheus metrics exporter for gateway API metrics to help with KSM nested status array limitations
Should be done for:

listeners status
httproute status

https://prometheus.io/docs/instrumenting/writing_exporters/

related issue with more info on why this is needed https://github.com/Kuadrant/gateway-api-state-metrics/issues/1

Repo with proof of concept & issues
https://github.com/Kuadrant/gateway-api-metrics-exporter/issues

Moving to v1.1 (post GA) after discussing with @jasonmadigan and @R-Lawton


BAU Release product page with fixed link

Any idea why this changes lots of resolved entries in the package-lock.json from HTTPS to HTTP?

Node rubbish, Jon's said its fine before. Would be ideal if that didn't happen but shouldn't be a problem

According to https://npm.community/t/some-packages-have-dist-tarball-as-http-and-not-https/285/47 it’s a registry bug but any actual downloads will always use HTTPS so it’s basically just cosmetic.

ah good to know, thanks 👍


Update 1.19.2 KillAura

Update KillAura to allow using item cooldown

did you test it?

did you test it?

Yes I did.

Thanks


BUG - Auto Tagged UTM Fields not Populating

Are you a current Fivetran customer?

Yes, @MarkMacArdle at Bought By Many
This is a replica issue witch directly mirrors the request opened by Mark on dbt_microsoft_ads_source

Hi @fivetran-joemarkiewicz this is working too.
One slight annoyance is that when I click on some of our ads I see utm_medium=paid-search in the url, rather than utm_medium=cpc. When I tried to search what it should be I only found this page saying it should be cpc so, unless that difference is ringing a bell with you as to what might cause it, probably best left as is. Maybe we just have an override for the particular ads I've been clicking on. I don't think we need utm_medium for joining to the attribution data so it's not of much importance to us anyway.

Hmm the page you linked is what I referenced for this as well. I am not aware of the paid-search utm_medium. Could it possibly be that some utm's leverage the auto tagging feature while others do not? Additionally, is there anything else in the url you mentioned that would identify it as not being a cpc utm-medium?

In Google's docs I did read that you could override the default utms so I guess we must be doing that. No sorry I didn't see a way identifying when utm_medium won't be the default.

I also would assume the overwritten utm_medium does not appear in the effective_final_url  field? I imagine not as the coalesce should have captured that if it was the case. I will continue to look a bit further into this to see if there is some other way to determine when a default auto-tag is overwritten.

@fivetran-joemarkiewicz now that I look some of the ads actually do have utms in the effective_final_url and these are getting parsed out correctly in the stg_google_ads__final_url_performance table. If I do a select distinct utm_medium,... on that table I get a few rows back. Nearly all are cpc but there some paid-search ones.
Apologies for not checking this sooner. In the Microsoft Ads none of them had utm fields so I presumed it'd be the same here.
So does seem to be working fine.

That's great to hear! Thanks for checking on that. With this, I believe we are good to go for review of the PRs. I will coordinate with our team to review and will keep you updated in the respective tickets once we are able to release new versions of the Microsoft Ads and Google Ads source packages!

@MarkMacArdle I just published a new release which includes the PR resulted from this issue. The new dbt_google_ads_source v0.3.1 should be live on the dbt hub at the top of the hour.
Feel free to reopen this ticket if you have any further questions on this topic, or you can open another issue if you have any other questions. Thanks!


Support for disabling/removing a validator in an inherited model

Initial Checks

[X] I have searched Google & GitHub for similar requests and couldn't find anything
[X] I have read and followed the docs and still think this feature is missing

Description
Given the following snippet that uses create_model (totally untested code - just for the sake of the example):
from typing import Optional
from pydantic import BaseModel, create_model

class MyBaseModel(BaseModel):
    a: str
    b: str
    c: Optional[str]

    @model_validator(mode="after")
    @classmethod
    def calculate_c(cls, values):
        values.c = a + b
        return values

new_model = create_model('MyNewModel', MyBaseModel)

I would like to remove (or disable) the validator called calculate_c from new_model. It currently seems that there's no easy way of achieving this.
Affected Components

[ ] Compatibility between releases
[ ] Data validation/parsing
[ ] Data serialization - .model_dump() and .model_dump_json()
[ ] JSON Schema
[ ] Dataclasses
[ ] Model Config
[ ] Field Types - adding or changing a particular data type
[ ] Function validation decorator
[ ] Generic Models
[X] Other Model behaviour - model_construct(), pickling, private attributes, ORM mode
[ ] Plugins and integration with other tools - mypy, FastAPI, python-devtools, Hypothesis, VS Code, PyCharm, etc.

As a temporary workaround, you can override with a no-op, but I get that this isn't a sustainable solution!

Thanks for your reply, @sydney-runkle
Can you point to some example of what you mention?

@tteguayco,
Sure thing:
from typing import Optional
from pydantic import BaseModel, create_model, model_validator

class MyBaseModel(BaseModel):
    a: str
    b: str
    c: Optional[str] = None

    @model_validator(mode="after")
    def calculate_c(self):
        self.c = self.a + self.b
        return self

NewModel = create_model('MyNewModel', __base__=MyBaseModel, __validators__={'calculate_c': model_validator(mode="after")(lambda x: x)})

print(repr(MyBaseModel(a="a", b="b")))
# MyBaseModel(a='a', b='b', c='ab')

print(repr(NewModel(a="a", b="b")))
# MyNewModel(a='a', b='b', c=None)

FWIW I think maybe a config setting could be used to disable / enable them

Thanks a lot, @sydney-runkle !


Polish up links to projects maintained by EdSchouten

Hi there,
The link to the Buildbarn project has become outdated. This pull request changes the page to point to the new location. While there, also document the existence of the rules for Elm.

Thank you for your contribution to the Bazel ecosystem!


Add 'nilness' to github actions jobs

Nilness checks for common programming errors like nil pointer dereference or degenerate nil pointer comparisons. It can find bugs, add a target for it.
https://pkg.go.dev/golang.org/x/tools/go/analysis/passes/nilness
Add it to the CI, and fix up the couple of legitimate failures it complains about in the codebase.
Technically the two identified errors are bugs, but they do not appear to be at all severe so I see no strong motivation to backport this PR to v1.9.

@christarazi actually it's part of my old git "fixes" formatting scripts:
https://github.com/joestringer/config/blob/1c050f3807e03cb8ba283c90b05416767f493576/.gitconfig#L8
https://github.com/joestringer/config/blob/1c050f3807e03cb8ba283c90b05416767f493576/.bashrc#L337-L366

Reassigned since it's probably more important to get the original authors to take a look both since they'll know that code and as a way to point out this pattern of bug to reduce recurrance of such issues.

test-me-please

GKE only issue, but seems unlikely that it's any way connected to the changes in this PR:
https://jenkins.cilium.io/job/Cilium-PR-K8s-GKE/3342/execution/node/115/log/
09:17:01  WARNING: cluster cilium-ci-13 is not running. The kubernetes API may not be available.
09:17:01  ERROR: (gcloud.container.clusters.get-credentials) cluster cilium-ci-13 is missing endpoint. Is it still PROVISIONING?

No need to backport to 1.9, all future patches should go through master and therefore have these checks applied at the time of submission.

I also received out-of-band confirmation from @jrajahalme that these patches do not need backport to v1.9 as the outer code already handles such error conditions.


Going back from viz page editor does not work for couple of clicks

When in the editor the back-on browser does not work

Thanks for reporting! This is a duplicate of

#495


About the training time.

Hi, I'm curious about how long does it take to train 32 Epochs on a V100？

May take about 24 hours, depending on your CPU, hard disk type and other hardware conditions


Page does not mention a mechanism to remove bot that is installed with ID

As part of the instructions on installing a bot, there is a method where you can use the Bot ID to initiate a 1-1 chat with it.
However, there is no mechanism to remove the Bot from the users chat once testing is done in this method. How do you remove such a bot after testing is complete?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 45ca0926-65fc-66a5-b881-5b99a589f2ec
Version Independent ID: b9e40da9-1029-f65f-18b5-f450a8ea87d2
Content: Test and debug your bot locally - Teams
Content Source: msteams-platform/bots/how-to/debug/locally-with-an-ide.md
Product: msteams
GitHub Login: @surbhigupta
Microsoft Alias: anclear

@codeepicure - We are looking into this I will get back to you soon.

@codeepicure - To remove your bot completely from Teams, go to your Bot Dashboard and edit the Microsoft Teams channel. Choose the Delete button at the bottom. This prevents users from discovering, adding, and interacting with your bot. This does not remove the bot from other user's Teams instances, however, it stops functioning for them as well.
Reference doc:-  https://docs.microsoft.com/en-us/microsoftteams/platform/bots/how-to/debug/locally-with-an-ide#remove-a-bot-from-a-team

@sayali-MSFT : This works well when the Bot was installed into your teams instance. My question is, how do you delete the Bot when you opened it via a direct link.

Once installed via the 'Open in Teams' link, for testing purposes, there is no way for me to remove it.

@codeepicure - There doesn't seem to be any way to remove it.
"Bots in personal context cannot be disabled or removed by users."

@sayali-MSFT : How do we request such a feature be available? Without this feature we are unable to test out bots locally and remove them at a later point.

@codeepicure - Could you please raise a user voice here:
https://microsoftteams.uservoice.com/forums/555103-public/filters/new

We are closing this issue for now, Please feel free to reopen it, if required.


Fix too eager API key presence check that blocked creating transcripts providers

Description
Fix too eager API key presence check that blocked creating transcripts providers
Risk

Deploy Plan

PMRR


Add playsinline to the players

On mobile they go full screen on play which defeats the point.
I almost wonder if we shouldn't go to custom full screen when playing on mobile automatically.

@heff do you want to keep this open for custom fullscreen?


CV in ORCA is not correct

ORCA-axis not good.pdf
It looks like the x-axis in the AT/distance map is a factor 10 too low.

coorected


[knife supermarket] set tarballs to uid/gid 0 to avoid large uid/gid errors on consumer machines

Description
The tar commands default to storing the user and group IDs--and the user/group names associated with them--of the directories and files being archived. This change updates the BSD and GNU forms for the tar command to set the owner and group to 0 and to not store the user/group name mapping in the resultant archive.
There is also has a benefit of avoiding bugs in tar readers that do not behave well when large uid/gid values are stored in tar's extended metadata headers. While this won't fix cookbooks that have already been published, it will minimize the bug's appearance in clients with older dependency resolvers as fewer latest cookbook versions will include large IDs.
Related Issue
To reiterate, this PR won't fix the issue for cookbooks already published. Any code that expands a cookbook—clients, Supermarkets, etc—must still handle tarballs with large IDs. These issues exhibit the downstream pain of the large uid/gid values being included in the cookbook tarballs.

https://github.com/chef/supermarket/issues/1806
https://github.com/chef/chef-dk/issues/1549
https://github.com/chef/chef-dk/issues/2018
https://github.com/chef/effortless/issues/54

Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
~New feature (non-breaking change which adds functionality)~
~Breaking change (fix or feature that would cause existing functionality to change)~
~Chore (non-breaking change that does not add functionality or fix an issue)~

Checklist:

[x] I have read the CONTRIBUTING document.
[ ] I have run the pre-merge tests locally and they pass.
[ ] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.
[x] All commits have been signed-off for the Developer Certificate of Origin.

Changed the title, since the actual breakage is much more important than the anonymization.  I was about to close this on the principle that the anonymization wasn't sufficient to warrant doing the work to fix it since it seems to be not as trivial at first glance.

yeah this is probably better left unfixed for now, its harder than it looks at first glance


[BEAM-5100][SQL]Test Arithmetic functions at the DSL level

Test math functions at the DSL level.
Remove BeamSqlMathFunctionsIntegrationTest.java as tests in it will be duplicates.

Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.


Speed up mget and other similar calls, by applying the pipeline parrelllel exection speedup under the hood.

As far as I can tell, the current implementation for commands like mget is to unroll them into a sequential series of get calls, to ensure the correctness, but that leads to very slow performance for calls for a large quantity of keys. See (https://github.com/Grokzen/redis-py-cluster/issues/375)
Would it be possible to apply the parallel execution speed ups that are used for pipelines though, so that the unrolled series of gets only need to block once on all of the calls at once, rather than sequentially waiting on many round trips to 1 node at a time?

@AlexBoyd I am closing this issue as this repo is no Archived and all future work and features should be redirected over to redis-py instead


Add a test on the search route for each possible error codes

I approved it because the tests are well-written, but could you look at why the error returned by the bad filter request is index_not_found?

This was expected actually BUT after a good night of sleep I realised I could also test the error we generate by hand if I were to create an index + provide a bad value.
Which I did and it highlighted a bug so, good catch 🎉

bors merge

bors merge


[docker compose] neuvector/allinone does not persist changes

Environment
Platform: docker compose
Docker compose Version(s): 2.20.2+ds1-0ubuntu1~22.04.1
neuvector/allinone version: 5.3.0
Describe the bug
Although the directory /var/neuvector is mounted with write permissions to the host, Neuvector does not store anything there.
This means that everything I change during the runtime of the container gets lost after a restart. This includes setting the admin password.
To Reproduce
Steps to reproduce the behavior:

Execute docker compose up -d
Wait a a few minutes until the UI is ready.
Login, change the password of the admin user.
Execute docker compose down
Execute docker compose up -d
The password of the admin user is reset to the default again.

Expected behavior
I expected persistence of all the changes I made to Neuvector and that they get stored in /var/neuvector. Instead the directory remains empty.
Screenshots
Here's the view of the directory /var/neuvector inside the container:
/ # whoami
root
/ # hostname
65efadbac180
/ # ls -la /var/neuvector
total 0
drwxrwxrwx    2 5006     5006             6 Jan 26 17:20 .
drwxr-xr-x    1 root     root            23 Feb  9 11:06 ..

And here's from the outside:
harbor@example:~/git/neuvector$ whoami
harbor
harbor@example:~/git/neuvector$ ls -la ./neuvector
total 0
drwxrwxrwx 2 harbor harbor  6 Jan 26 18:20 .
drwxrwxr-x 3 harbor harbor 86 Feb  8 18:32 ..

Additional context
docker-compose.yml:
name: neuvector

networks:
  neuvector:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: ${NEUVECTOR_SUBNET}
    driver_opts:
      com.docker.network.bridge.name: d-neuvector
  app_harbor:
    external: true

services:
  allinone:
    pid: host
    image: neuvector/allinone:${NEUVECTOR_VERSION:-latest}
    restart: always
    container_name: neuvector.allinone
    deploy:
      mode: global
    networks:
      neuvector:
        ipv4_address: ${NEUVECTOR_IP_ALLINONE}
    cap_add:
      - SYS_ADMIN
      - NET_ADMIN
      - SYS_PTRACE
      - IPC_LOCK
    security_opt:
      - apparmor=unconfined
      - seccomp=unconfined
      - label=disable
    environment:
      - CLUSTER_JOIN_ADDR=${NEUVECTOR_IP_ALLINONE}
      - NV_PLATFORM_INFO=platform=Docker
    ports:
      - 18300:18300
      - 18301:18301
      - 18400:18400
      - 18401:18401
      - 18301:18301/udp
      - 8443:8443
    volumes:
      - /lib/modules:/lib/modules:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /proc:/host/proc:ro
      - /sys/fs/cgroup:/host/cgroup:ro
      - ./neuvector:/var/neuvector
      # See https://open-docs.neuvector.com/configuration/console/replacecert
      - /home/harbor/certs/example.com.key:/etc/neuvector/certs/ssl-cert.key:ro
      - /home/harbor/certs/example.com.crt:/etc/neuvector/certs/ssl-cert.pem:ro

.env:
# Find the latest versions here: https://hub.docker.com/r/neuvector/allinone/tags
NEUVECTOR_VERSION=5.3.0
NEUVECTOR_SUBNET=<IP_ADDRESS>/24
NEUVECTOR_IP_ALLINONE=<IP_ADDRESS>
NEUVECTOR_IP_ADAPTER=<IP_ADDRESS>

When I startup Neuvector it shows this log line:
neuvector.allinone  | 2024-02-09 13:18:43,933 DEBG 'monitor' stdout output:
neuvector.allinone  | 2024-02-09T13:18:43.932|INFO|CTL|kv.(*configHelper).Restore:
neuvector.allinone  | 2024-02-09T13:18:43.932|INFO|CTL|kv.(*configHelper).Restore: Config persistence disabled

Has it anything to do with my issue? And how can I enable config persistence then?

Well, while digging through the Helm Charts I found the environment variable CTRL_PERSIST_CONFIG that seems to solve the issue.


v1.1.4 Release

Issue #, if available: N/A
Description of changes:
Update THIRD-PARTY-LICENSES file and update version for v1.1.4
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (a441dc5) 46.49% compared to head (1d14f8c) 46.49%.

Additional details and impacted files
@@           Coverage Diff            @@
##           develop     #112   +/-   ##
========================================
  Coverage    46.49%   46.49%
========================================
  Files          248      248
  Lines         8638     8638
  Branches       897      897
========================================
  Hits          4016     4016
  Misses        4313     4313
  Partials       309      309

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


kubevirt provider e2e create cluster test

This functional e2e test is very basic and is primarily meant as an entrypoint for us to begin fleshing out more advanced validation in the future. At the moment, we simply need a test case that we can wire up to Openshift CI to validate that a hosted cluster actually boots KubeVirt VMs successfully.
At the moment this test does the following.

Posts a HostedCluster using the kubevirt provider
Verifies the NodePool gets created as expected
Verifies the KubeVirt Machines come online (meaning that the VMs boot, that's it)
Verifies the KubeVirtCluster is ready

Unlike the AWS create cluster test, access to the guest cluster is not being exercised yet, and we aren't even verifying that the HostCluster reaches a completed state. This verification for the KubeVirt provider will come in time.

/test e2e-aws-all
to hit this test

/test e2e-aws-all
to hit this test

@sjenning We actually need this test to be skipped on AWS. The issue here is that KubeVirt requires nested virtualization, and the non-metal AWS EC2 instances used by OpenShift CI do not support nested virtualization.
@nunnatsa is in the process of preparing such a lane on Azure which does support nested virtualization, but this will take time.
Ideally, what I'd like to see is for this e2e test to get merged before a CI lane executes it as a way for @nunnatsa to have a simple e2e test to prove out the Azure lane works with kubevirt and nested virt. I'm sure we'll iterate on this test a few more times as that Azure lane is being shored up, but this at least gives us a foundation to work with.
Does that sound reasonable? These new test lanes are always kind of a pain due to the chicken and egg problem. We need a test exercise the new lane's abilities, but the test can't be proven until the lane exists. :/

So we need to exclude this test from e2e-aws-all then?
https://github.com/openshift/release/blob/master/ci-operator/config/openshift/hypershift/openshift-hypershift-main.yaml#L126
There will actually be no CI test that will execute this test this until the Azure work is done and... hypershift supports Azure which it currently does not?

There will actually be no CI test that will execute this test this until the Azure work is done and... hypershift supports Azure which it currently does not?

@sjenning Azure here is an arbitrary platform for the management cluster they choose to setup nested virt to accommodate Kubevirt. So then we can have a CI job which gives a management cluster with nested virt setup support (it'll happen to be in Azure but might be anywhere) so we can test hypershift platform=Kubevirt. This is not tied to hypershift supporting Azure. Make sense? cc @davidvossel

Does that sound reasonable? These new test lanes are always kind of a pain due to the chicken and egg problem. We need a test exercise the new lane's abilities, but the test can't be proven until the lane exists. :/

@davidvossel this sounds reasonable to me, if everyone else agrees I think we can merge after addressing comments and getting docs in place so we can enable @nunnatsa to follow up.

Also you'll need to ensure this Test* only run against your management cluster see docs/content/contribute/run-tests.md

I think I've addressed the feedback so far. Understand that this test is suboptimal. It's the highest bar I could come up with to validate the kubevirt platform given what exists right now. This will continue to improve in the future.

Also you'll need to ensure these TestKubeVirt* run only against your management cluster see docs/content/contribute/run-tests.md

yep, we'll filter by a KubeVirt regex in our CI lane.

So we need to exclude this test from e2e-aws-all then?

@sjenning I added usage of an environment variable KUBEVIRT_PLATFORM_ENABLED=true that indicates if the KubeVirt platform related tests can run. By default, unless that env var is set, the KubeVirt tests will be skipped. From what I can see, there's not a great way to tell go test to "run all tests except this subset" without using something like an env var.
This preserves the usage of the ci lanes that run on AWS infra.

/retest

So we need to exclude this test from e2e-aws-all then? https://github.com/openshift/release/blob/master/ci-operator/config/openshift/hypershift/openshift-hypershift-main.yaml#L126
There will actually be no CI test that will execute this test this until the Azure work is done and... hypershift supports Azure which it currently does not?

@sjenning that should be fine. We don't need hypershift to support azure. We just need "a" openshift cluster from azure (not hypershift) where we can deploy hypershift and kubevirt.

Hi folks, I want to help get this through, but the current factoring of the e2e code with regards to platform specific handling is not satisfactory to @alvaroaleman or myself. We have discussed how we'd rather handle it, and need to do some refactoring in here. The way this PR is currently factored seems to further expand and entrench a pattern I'm not sure we want. Can we please have a discussion about this before moving forward? I am working on a WIP to bring the code in line with what @alvaroaleman had briefly discussed.
I want to avoid wasting time working at cross purposes.

https://github.com/openshift/hypershift/pull/884 is where I'm going, but I don't want to make you rebase. I'm fine to let this through and then I can rebase.

/hold
I want to test my changes again here locally before this merges. Feel free to lgtm/approve in the mean time if someone feels comfortable and I'll remove the hold once I verify it's all still in order from my end.

I'm fine to let this through and then I can rebase.

Thanks, I followed similar patterns to aws and none platforms, so hopefully it's not much of an inconvenience.

/approve
/lgtm
No further feedback on my end for now, happy to get the skeleton in place for continued iteration

/hold cancel
Passes locally now. I had to make one minor change.
@ironcladlou For the test to pass, I had to make a change and add the capk scheme to the test client because i'm doing some introspection of the KubeVirtMachine and KubeVirtCluster objects.

/hold cancel
Passes locally now. I had to make one minor change so this will need another lgtm
@ironcladlou For the test to pass, I had to make a change and add the capk scheme to the test client because i'm doing some introspection of the KubeVirtMachine and KubeVirtCluster objects.

Awesome, thanks!
/lgtm

@davidvossel sorry I didn't mention this before, but can you please squash the commits?

yep, squashed

/lgtm

@nunnatsa  fyi, here's an example of how I ran this test locally from the source tree.
export KUBEVIRT_PLATFORM_ENABLED=true
make e2e
bin/test-e2e \
          -test.v \
          -test.timeout=0 \
          -test.run=TestKubeVirtCreateCluster \
          -e2e.node-pool-replicas=1 \
          -e2e.base-domain="fake" \
          -e2e.kubevirt-container-disk-image="$RHCOS_CD_IMAGE" \
          --e2e.pull-secret-file="$PULL_SECRET_PATH"


Add ContextProvider component to web-no-modal-sdk and web-modal-sdk

Now you can read the web3auth instance when using wagmi.

This PR helps a lot in understanding how to access the getUserInfo function with Wagmi, however we need to figure out a simpler way of doing it. Until then, we're keeping this as an open PR. Once figured out we will merge that approach rather.


Performance test (MPC and TEE)

In the last PATCG meeting, a discussion was raised about the use of MPC or TEE for aggregating the reports. I would like to take part in this group by comparing performances of both MPC and TEE in the attribution reporting case. It seems to be a consensus that MPC is better for privacy, but I would like to measure the execution time on a large dataset.
I have been through the IPA research prototype. MP-SPDZ seems powerful.  Do you think MP-SPDZ is suitable for performance test?
I would like to try different sorting algorithms. Do you know any other one than the one you are implementing?

Hi Victor, I think that this is a question better suited to the MP-SPDZ prototype in this repository.

There are many MPC sorting algorithms I'm familiar with.

We've been implementing the one from this paper: https://eprint.iacr.org/2019/695.pdf and the maliciously secure variant of this protocol that's soon to be published.
That paper references many other prior works that you can take a look at.
We understand that many people are using Quicksort (a secure variant of it) that seems to work well in practice.
I'm not sure exactly what sort algorithms are used for the default "sort" operations supported by frameworks like MP-SPDZ and Sharemind https://docs.sharemind.cyber.ee/2022.03/api/rmind-library


Integrate MS-AMP Support for FP8 as a seperate backend

Integrate MS-AMP to the Accelerator (round 2)
What does this add?
This PR introduces an additional backend for FP8 support through MS-AMP which has shown to decrease memory when using FP8 precision while maintaining accuracy
Who is it for?
Individuals training with FP8 (H100/4090's, etc)
Issues linked to
https://github.com/Azure/MS-AMP/issues/128
What parts of the API does this impact?
User-facing:
Two new arguments were added to the FP8RecipeKwargs:

backend (str): Whether a user should use MS-AMP or TE (transformerengine). Uses MS-AMP by default.
optimization_level (str), should be one of "O1" or "O2". "O3" is for DeepSpeed and we need to wait for them to update to v0.9.3 of deepspeed to match what Accelerate supports

General guideline to optimization levels:

O1: Weight gradients and all_reduce communications are done in fp8, reducing GPU
memory usage and communication bandwidth
O2: First-order optimizer states are in 8-bit, and second order states are in FP16.
Only available when using Adam or AdamW. This maintains accuracy and can potentially save the highest
memory.
03: Specifically for DeepSpeed, implements capabilities so weights and master weights of models
are stored in FP8. If fp8 is selected and deepspeed is enabled, will be used by default.
(Not available currently).

As a result, "O2" is the default. Here is an overview of each optimization level and what it does, taken from their docs:

Optimization Level
Computation(GEMM)
Comm
Weight
Master Weight
Weight Gradient
Optimizer States

FP16 AMP
FP16
FP32
FP32
N/A
FP32
FP32+FP32

Nvidia TE
FP8
FP32
FP32
N/A
FP32
FP32+FP32

MS-AMP O1
FP8
FP8
FP16
N/A
FP8
FP32+FP32

MS-AMP O2
FP8
FP8
FP16
N/A
FP8
FP8+FP16

MS-AMP O3
FP8
FP8
FP8
FP16
FP8
FP8+FP16

Basic Usage Example(s):
A user can either do:
accelerator = Accelerator(mixed_precision="fp8")

Or use the FP8RecipeKwargs:
# To use TransformerEngine instead
kwarg_handlers = [FP8RecipeKwargs(backend="TE")]

# To change the optimization level
kwarg_handlers = [FP8RecipeKwargs(optimization_level="O1")]

accelerator = Accelerator(
    mixed_precision="fp8",
    kwargs_handlers=kwarg_handlers,
)

Benchmarks
When running on bloomz-560m I saw a memory decrease of ~1/3.
More experiments need to be conducted on the behavior between TE x MS-AMP wrt performance. For instance, when running my sample script I use here for speed (here) running on the first 100 batches, I saw a stark contrast in the ending training loss between TE and MS-AMP:
BF16 (baseline): 2.4867
TE: 11.3125
MS-AMP: 2.89
I also found overall there wasn't much of a time save with MS-AMP, as it actually added time instead (BF16 was ~0.139s/batch, while MS-AMP was 0.169s/batch). I want to run some more tests to verify but these were some local results.
This performance difference isn't much in the case of BF16 vs MS-AMP, but it is starkly contrast when compared to TE. More work is needed to investigate why, so as a result I've opted to make MS-AMP just an entirely separate backend to use, rather than combine the two.
What went wrong in the last PR
While the training speed results looked very good, I quickly noticed issues with the losses that didn't make sense. Models just simply weren't training or converging, which was surprising. For now taking this more staged approach to the integration while we discover behaviors (both with TE and MS-AMP) through longer training runs.

TODO: write some doc guides on MS-AMP

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@BenjaminBossan re;

A cleaner solution would be to use a completely separate dataclass for MS-AMP, although that might clash with the accelerate philosophy of abstracting away such implementation details.

Eventually we will support a "mixed" backend that combines both, as MS-AMP has support for converting te.Linear layers to MS-AMP ones to get the best of both worlds. I'm not enabling this though until tests I run show that combining them does not degrade final accuracy, so for now they are disjoint and part of the same Plugin

@BenjaminBossan re; 1, we straight up don't allow it. In AcceleratorState we have:
            if mixed_precision == "fp8" and not is_fp8_available():
                raise ValueError("Using `fp8` precision requires `transformer_engine` to be installed.")

We should probably refactor this then as part of is_fp8_available. Thanks for the smell :)

@pacman100 I'd expect that likely to be the case, I'll check FLOPS mainly on the scaled up training to see what they can result in :)

cc @MKhalusova for docs and then we can merge 🤗

Thank you @muellerzr for working on the MS-AMP FP8 support! ✨ Overall looks good wrt integration and the memory savings of 33% (1/3) for 560M param model. However, It would be great to see more experiments at scale at intermediate model scales of ~10B model sizes as the paper claims:

Experiment results show that, during the training of GPT-175B model
on H100 GPU platform, our FP8 mixed-precision training framework not only achieved a
remarkable 42% reduction in real memory usage but also ran 64% faster than the widely
adopted BF16 framework (i.e., Megatron-LM), surpassing the speed of Nvidia Transformer Engine by 17%.

Here, it is odd that we see no savings in time. Maybe is it the case that with reduced memory, they fit larger batches leading to faster training?

Original paper shows that TE>MS-AMP>BF16 under same batch size setting (paper table 5). It is interesting to see MS-AMP slower than BF16.
Thank you are all for making this feature integrated to accelerate!

@skyshine102:

Original paper shows that TE>MS-AMP>BF16 under same batch size setting (paper table 5). It is interesting to see MS-AMP slower than BF16.

I have an inkling model size plays a huge factor in this. With TE I've pretty much always seen it to be slower unless our model size is > 3B

@skyshine102:

Original paper shows that TE>MS-AMP>BF16 under same batch size setting (paper table 5). It is interesting to see MS-AMP slower than BF16.

I have an inkling model size plays a huge factor in this. With TE I've pretty much always seen it to be slower unless our model size is > 3B

I see!

I have tried this work，It's  amazing. But  some questions:
1、TE and MSAMP didn't have  any difference？
2、BF16 is a little slower than the two，and  comsumed a little more GPU memory comparation to fp8
GPU ：L20
code：
compute_dtype = (
        torch.float16
        if training_args.fp16
        else (torch.bfloat16 if training_args.bf16 else torch.float32)
    )

    model = transformers.AutoModelForCausalLM.from_pretrained(
        model_args.model_name_or_path,
        cache_dir=training_args.cache_dir,
        device_map=device_map,
        trust_remote_code=True,
        torch_dtype=compute_dtype,

    )

    #kwargs = FP8RecipeKwargs(backend="te", fp8_format="HYBRID")
    kwargs = FP8RecipeKwargs(backend="msamp", opt_level="O1")

    accelerator = Accelerator(mixed_precision='fp8',  kwargs_handlers=[kwargs])
    #accelerator = Accelerator()
    #device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    device = accelerator.device
    model.to(device)

    tokenizer = transformers.AutoTokenizer.from_pretrained(
        model_args.model_name_or_path,
        cache_dir=training_args.cache_dir,
        model_max_length=training_args.model_max_length,
        padding_side="right",
        use_fast=False,
        trust_remote_code=True
    )
    tokenizer.pad_token = tokenizer.unk_token
    tokenizer.add_tokens(
        ['<eod>', '<sep>', '<pad>', '<mask>', '<predict>', '<FIM_SUFFIX>', '<FIM_PREFIX>', '<FIM_MIDDLE>',
         '<commit_before>',
         '<commit_msg>', '<commit_after>', '<jupyter_start>', '<jupyter_text>', '<jupyter_code>',
         '<jupyter_output>', '<empty_output>'], special_tokens=True)

    data_module = make_supervised_data_module(tokenizer=tokenizer, data_args=data_args)
    train_dataloader = DataLoader(data_module['train_dataset'], shuffle=True, batch_size=8,)  # 通过这里的dataloader，每个batch的seq_len可能不同
    eval_dataloader = DataLoader(data_module['eval_dataset'], batch_size=8, )
    optimizer = torch.optim.Adam(model.parameters(), lr=5e-5)

    num_epochs = 3
    num_training_steps = num_epochs * len(train_dataloader)  # num of batches * num of epochs
    lr_scheduler=torch.optim.lr_scheduler.StepLR(optimizer, num_training_steps, gamma=0.1, last_epoch=-1, verbose=False)

    model, optimizer, train_dataloader,eval_dataloader,lr_scheduler = accelerator.prepare(model, optimizer, train_dataloader,eval_dataloader,lr_scheduler)

    i=0
    for epoch in range(num_epochs):
        for batch in tqdm(train_dataloader):
            # 要在GPU上训练，需要把数据集都移动到GPU上：
            batch = {k: v.to(device) for k, v in batch.items()}
            loss = model(**batch).loss
            print("step: ",i,"loss: ",loss.item())
            #loss.backward()
            accelerator.backward(loss)
            optimizer.step()
            lr_scheduler.step()
            optimizer.zero_grad()
            if (i+1) % 500==0:
                accelerator.save_state("wpf_test_abcd")
            i=i+1


[c++] Add SOMAReader reset for stateful API changes

This PR adds the capability of "resetting" a SOMAReader while holding the underlying TileDB array open.

These changes work with the existing python API.
Additional minor C++ changes will be needed to match the stateful open/close API when implemented in python.

Codecov Report
Base: 63.88% // Head: 74.74% // Increases project coverage by +10.85% :tada:

Coverage data is based on head (32f5816) compared to base (d97e714).
Patch has no changes to coverable lines.

Additional details and impacted files
@@             Coverage Diff             @@
##             main     #624       +/-   ##
===========================================
+ Coverage   63.88%   74.74%   +10.85%
===========================================
  Files          61       26       -35
  Lines        3229     1853     -1376
===========================================
- Hits         2063     1385      -678
+ Misses       1166      468      -698

Flag
Coverage Δ

python
74.74% <ø> (ø)

r
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

R/SOMADenseNdArray.R

src/rinterface.cpp

src/rutilities.h

inst/tiledb/include/tiledb/config.h

R/TileDBObject.R

src/include/tiledbsoma/managed_query.h

inst/tiledb/include/tiledb/query_condition.h

R/TileDBArray.R

src/include/tiledbsoma/array_buffers.h

R/SOMADataFrame.R

... and 25 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Good point @eddelbuettel, let me know if you find any changes that would help adoption in R.
I'll commit an svg of the diagram too.

'Native code' called from R is usually asynchronous: you call a compiled routine, it sets up what it needs, does work, returns.  Intermediate are usually gone.   (Of course stateful ops happen to, standard example is a DB handle.)  So I would have to find a use case beyond what we already have where we init an 'iterating reader' and call 'next chunk' til done. It may fit best there. But @aaronwolen and I have not yet exposed that one...
All good stuff.


ES6 support question

Do ClearScript will support ES6 standard? any plan?

Hi @drearyrainDeng,
For JavaScript, ClearScript supports JScript and V8. JScript supports ES3 and select features from subsequent standards. V8 supports ES7 and beyond.
Thanks!

that is great,thanks

@ClearScriptLib above you mentioned that " V8 supports ES7 and beyond." when i try to run the following code.
engine.Evaluate(@"async function process1(data) {
return { xValue: 'asdasd' };
}");
var toReturn = engine.Script.process1(data);
I get the following error "no further information on this object could be discovered"
If i remove the async then it works fine, but having said that async is an ES7 feature


Implement user accounts and authentication

Please align the implementation of user accounts and authentication with other related tasks. Before we can authenticate with twitter, we need to be clear about how we implement authentication in loklak. What are your suggestions? This is kind of a basic feature that has a high priority.

General Notes

Sign in using Twitter has been done. Navigate to /auth/twitter and the auth flow begins.
After auth, the req.user object contains all the user data including token and token_secret.
User data is also sent to loklak_server after login. Keys and user profile data can be retrieved thereafter.
For implementing any kind of twitter related actions, I would recommend to use the twitter npm module. An example for posting a tweet to twitter has been already completed.

App Specific Notes (For twitter wall)
Using the apps property "apps" : {"wall" : {"type" : "horizontal"}}, I can store the wallOptions object with the account. (Milestone 2)

@aneeshd16 What is missing? How can the user sign in at loklak?

Navigate to /auth/twitter and the auth flow begins.
Do you mean? - http://loklak.net/auth/twitter

@mariobehling Yeah that's how the user logs in.

@sudheesh001 Did you try that link? Not working atm.

Yes it works on local, are the app keys and secrets put correctly on the application config as well as the twitter app ? I documented the details into the readme file.

@Mario I think currently due to #131
https://github.com/loklak/loklak_webclient/issues/131 and #133
https://github.com/loklak/loklak_webclient/issues/133
http://www.loklak.net is not working properly.
On Mon, Jun 8, 2015 at 6:11 PM, Sudheesh Singanamalla <
<EMAIL_ADDRESS>wrote:

Yes it works on local, are the app keys and secrets put correctly on the
application config as well as the twitter app ? I documented the details
into the readme file.
—
Reply to this email directly or view it on GitHub
https://github.com/loklak/loklak_webclient/issues/138#issuecomment-109979655
.

Look, everyone should deploy a test version on an online server, where we all can test. We already had a discussion about local vs. online yesterday. Time is limited and it is not my goals to figure out server setups. I would like to test stuff. Issues should only be closed when they clearly work online on loklak.net as well. Does this make sense to you?

@prasht63 have you set up the client on a server of yourself yet?

@prasht63 Could you help to fix this, please?

yes I have set it up http://prasht63loklak.eu-gb.mybluemix.net/ but its
not working properly. I am looking as soon as I have running one I will
post it on the group.
On Mon, Jun 8, 2015 at 6:18 PM, Mario Behling<EMAIL_ADDRESS>wrote:

@prasht63 https://github.com/prasht63 have you set up the client on a
server of yourself yet?
—
Reply to this email directly or view it on GitHub
https://github.com/loklak/loklak_webclient/issues/138#issuecomment-109981387
.

@prasht63 Where are the map parts that you made? How can I test and access it?

Hi
currently its not online.I am trying to put it online.If you have
loklak_webclient locally set up, you can go to localhost:3000/map to access
it.For the preview you can see the screen shots I am attaching here.[image:
Inline image 1]
On Mon, Jun 8, 2015 at 6:52 PM, Mario Behling<EMAIL_ADDRESS>wrote:

@prasht63 https://github.com/prasht63 Where are the map parts that you
made? How can I test and access it?
—
Reply to this email directly or view it on GitHub
https://github.com/loklak/loklak_webclient/issues/138#issuecomment-109993880
.

I can always set it up locally, but my interest is to see loklak running on a server. At the moment, it looks like none of the devs is able to set up a loklak server running without issues on the web. @/all Please keep it up! Hope to see some progress today.

This is already implemented and tested on my production server.
loklak.cloudapp.net:3001/auth/twitter/

@sudheesh001 I think you have a different understanding, what it means, that a feature is implemented. I would like to see a login button on the top right or a login page. After I login I get the features for logged in users. How can I login in the user interface? How is this feature integrated in the UI?

@mariobehling I think you're talking about #143 I seperated the UI UX issue of the login from the implementation of user authentication.

@sudheesh001 Thanks for clarifying. Please feel free to cross-reference such links in future. Cheers!

I don't have had the experience that I was able to log into twitter from the loklak_webclient interface since there is not even a log-in button. I expect that there is a button which I can click on, then authentify and finally see the loklak webinterface which shows that I am logged in.

@Orbiter The login button is present in the top right corner with the latest commit

Right now, no user account data is present in the loklak_server database which means that the /api/account.json servlet was never called. As long as we don't have account data of the user, this issue is not solved!

Will be solved after #171

#171 merged and checked, this is good to close.


Update circe-core, circe-generic, ... to 0.14.8

About this PR
📦 Updates

io.circe:circe-core
io.circe:circe-generic
io.circe:circe-parser

from 0.12.3 to 0.14.8
📜 GitHub Release Notes - Version Diff
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "io.circe" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "io.circe" }
}]

labels: library-update, early-semver-major, semver-spec-minor, version-scheme:early-semver, commit-count:1

Superseded by #330.


Add support samsung a51

your device (DEVICE_CODENAME, KERNEL_VERSION, etc.)
A51,4.14.113-18509473
what is not working
None of custom kernel
what exactly you would like to have / do.
Android Keyboard Gadget, what should I do?

If 4.1 is your Kernel version, instead of the custom kernel patch, you can try https://github.com/tejado/android-usb-gadget
Let me know if you have any issues with it.


Feature/semseg misclassification

Adds misclassification to semseg polygons

To fix typing issue in 3.8 and 3.9 https://stackoverflow.com/questions/76313592/import-langchain-error-typeerror-issubclass-arg-1-must-be-a-class


Formatting ticks to prevent overflowing.

This PR formats the tick value of the timeseries on the front page. This will prevent the need of padding changes in the future.
Type of PR

[x] Bugfix
[ ] New feature

Relevant Issues
Fixes #439
Checklist

[ ] Compiles and passes lint tests
[ ] Properly formatted
[ ] Tested on desktop
[ ] Tested on phone

Screenshots

Hey! Thanks for the PR! Unfortunately, this was patched during a PR merge that addressed logarithmic values.
Feel free to pick any available issues :)