Manually update dependabot's recommendations

Dependabot has a couple PRs open to update some of this project's dependencies but it looks like updating is going to take a bit of manual work, i.e. updating the symbols that changed / went away.

Would be happy to pick this up.

That would be great! Thanks!


[venti-stack] diff-subcharts

What this PR does / why we need it
[venti-stack] diff-subcharts
Which issue this PR fixes
(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

fixes #46

Special notes for your reviewer
Checklist

[ ] Chart Version bumped

@jmnote ➜ /workspaces/helm-charts/hack (46-venti-stack-diff-subcharts) $ ./diff_fluentbit.sh
diff --color 0.40.0/apps_v1_daemonset_xxx-fluent-bit.yaml 0.43.0/apps_v1_daemonset_xxx-fluent-bit.yaml
28c28
<         image: cr.fluentbit.io/fluent/fluent-bit:2.2.0
---
>         image: cr.fluentbit.io/fluent/fluent-bit:2.2.2

@jmnote ➜ /workspaces/helm-charts/hack (46-venti-stack-diff-subcharts) $ ./diff_prometheus.sh
diff --color 25.8.0/apps_v1_deployment_xxx-kube-state-metrics.yaml 25.11.1/apps_v1_deployment_xxx-kube-state-metrics.yaml
40a41
>           failureThreshold: 3
41a43
>             httpHeaders: null
43a46
>             scheme: HTTP
44a48,49
>           periodSeconds: 10
>           successThreshold: 1
50a56
>           failureThreshold: 3
51a58
>             httpHeaders: null
53a61
>             scheme: HTTP
54a63,64
>           periodSeconds: 10
>           successThreshold: 1
diff --color 25.8.0/apps_v1_deployment_xxx-prometheus-pushgateway.yaml 25.11.1/apps_v1_deployment_xxx-prometheus-pushgateway.yaml
30c30
<       - image: quay.io/prometheus/pushgateway:v1.6.2
---
>       - image: quay.io/prometheus/pushgateway:v1.7.0
diff --color 25.8.0/apps_v1_deployment_xxx-prometheus-server.yaml 25.11.1/apps_v1_deployment_xxx-prometheus-server.yaml
40c40
<         image: quay.io/prometheus-operator/prometheus-config-reloader:v0.67.0
---
>         image: quay.io/prometheus-operator/prometheus-config-reloader:v0.71.2
54c54
<         image: quay.io/prometheus/prometheus:v2.48.0
---
>         image: quay.io/prometheus/prometheus:v2.49.1
diff --color 25.8.0/v1_serviceaccount_xxx-kube-state-metrics.yaml 25.11.1/v1_serviceaccount_xxx-kube-state-metrics.yaml
2d1
< imagePullSecrets: null


[BUG]: Cannot insert into SQLite - No overload matches this call

What version of drizzle-orm are you using?
0.28.5
What version of drizzle-kit are you using?
0.19.13
Describe the Bug
Trying to insert a row into an SQLite table, but Typescript throws a compiler error.
Schema (note the primary key address is a TEXT column):
import { sqliteTable, text, integer } from 'drizzle-orm/sqlite-core';

export const accounts = sqliteTable('accounts', {
    address: text('address', { length: 44 }).primaryKey(), // human-readable address
    type: integer('type', { mode: 'number' }).notNull(), // account type
    balance: integer('balance', { mode: 'number' }).notNull(), // in luna
    firstSeen: integer('first_seen', { mode: 'timestamp' }).notNull(),
    lastSeen: integer('last_seen', { mode: 'timestamp' }).notNull(),
});

Insert code:
await db.insert(accounts).values({
    address: 'NQ07 0000 0000 0000 0000 0000 0000 0000 0000',
    type: 0,
    balance: 1000e5,
    firstSeen: Date.now(),
    lastSeen: Date.now(),
});

This produces the following error:
No overload matches this call.
  Overload 2 of 2, '(values: { address: string | SQL<unknown> | Placeholder<string, any>; type: number | SQL<unknown> | Placeholder<string, any>; balance: number | SQL<unknown> | Placeholder<...>; firstSeen: Date | ... 1 more ... | Placeholder<...>; lastSeen: Date | ... 1 more ... | Placeholder<...>; }[]): SQLiteInsert<...>', gave the following error.
    Argument of type '{ address: string; type: AccountType; balance: number; firstSeen: number; lastSeen: number; }' is not assignable to parameter of type '{ address: string | SQL<unknown> | Placeholder<string, any>; type: number | SQL<unknown> | Placeholder<string, any>; balance: number | SQL<unknown> | Placeholder<...>; firstSeen: Date | ... 1 more ... | Placeholder<...>; lastSeen: Date | ... 1 more ... | Placeholder<...>; }[]'.
      Object literal may only specify known properties, and 'address' does not exist in type '{ address: string | SQL<unknown> | Placeholder<string, any>; type: number | SQL<unknown> | Placeholder<string, any>; balance: number | SQL<unknown> | Placeholder<...>; firstSeen: Date | ... 1 more ... | Placeholder<...>; lastSeen: Date | ... 1 more ... | Placeholder<...>; }[]'.ts(2769)

When I change the values() call to insert an array, the error changes to complain about Overload 1 of 2 instead, but still errors with No overload matches this call.
Expected behavior
The usage matches overload 1 and does not produce a TS compiler error.
Environment & setup

TS version 5.2.2, but also happens with VSCode's own 5.1.6
Node 18.17.1
Fedora Linux 38

The error was that I used Date.now() which returns a number, instead of new Date() which returns a Date object, which is what is actually expected for the timestamps.
The error message saying address is not part of the expected object made no sense...

OMG, I SPENT 10 HOURS BEFORE I KNEW HOW TO EVEN GOOGLE IT...........
same thing.... Date was set as t.Date and it conflicted with the value i was inserting


Optimized the logic of focus

Summary
This change optimized the auto-focus feature for mobile devices through the react-device-detect library.
Test Plan
Simulated the behavior of auto-focus on mobile devices in Chrome Dev Tools, works as expected.
Issues
Closes #782

I think this is a PR worth pinging @EricPedley and @ap0nia on! We currently detect devices based on view width, but that's kind of a poor measuring stick...
Would this more properly standardize our isMobile (or whatever the helper function is called) across the app? I'm not super sure about the procedure for adding in new dependencies, so I'd love y'alls thoughts.
At first glance (haven't tested yet), it seems like it would be useful not just here, but essentially anywhere in the app we use view width as a proxy for device (like with the recent Zotistics graph hover)

Yeah bruh. I was planning on using width to distinguish between mobile and computers, but I did some research and it seems like that's very unstable across devices and platforms. And I figured out that react actually has this detect feature in their standard library. So I thought this might be a better approach lol

new library sounds good, we'll just do another PR to change it everywhere else to not use screen size.

Idk man don't we like having a wrapper?

We want a "react-specific" version to do this?
new UAParser().getDevice().type === 'mobile'

Oh yeah ur right

Okay so I digged into it a little bit. The ua-parser-js works in most cases as it returns a type of mobile or tablets which we should not auto-focus. But when it comes to certain tablets model like Ipad Air or Ipad Pro, the type returned by the library is actually undefined... So that we wouldn't be able to distinguish them.  I think when I test the isMobile from react library it works fine but it is indeed a little dumb to stick with a certain version. However, the ua-parser-js also have some constraints dealing with certain models. Any suggestions maybe guys? At least I think it works with most mobile types, but not for certain model like ipads

This is the full logic that they use: https://github.com/duskload/react-device-detect/blob/master/src/lib/selectors.js#L11
They check that it's either a mobile device, tablet, or ipad.
The full logic would look something like this.
const ua = new UAParser()
const device = ua.getDevice()
const isMobile = device.type === 'mobile' || device.type === 'tablet'

function isIpad() {
  if (typeof window === 'undefined') {
    return false
  }
return   (/iPad|iPhone|iPod/.test(window.navigator.platform) ||
      (window.navigator.platform === 'MacIntel' && window.navigator.maxTouchPoints > 1)) &&
    !window.MSStream
}

Okay just pushed the ua-parser way. However, idk I simulated the react library way but it didn't really workout lol. I still implemented the ua-parser since it detects most devices okay. For the ipad case it's annoying because right now it's OS becomes Mac too. The way I am currently checking is if it supports touch capabiity, if it is then we will identify it as mobile.

I just attended a fantastic web accessibility workshop by Alex and Eddy! Auto-focus is not accessibility-friendly, so I think we should strongly consider reverting the original auto-focus PR.

Can u elaborate on how it isn't accessible?

Can u elaborate on how it isn't accessible?

The issue mainly stems from the fact that autofocus creates a jarring effect for screen readers -- users end up in a location they didn't expect/navigate to

I agree with Kevin. From a developer perspective, I'd generally consider it a goal to make the app act in a predictable manner. This is basically already an impossible challenge because of React's consistently unpredictable rendering. Adding logic that arbitrarily abducts control from the user will invariable end up with a edge cases because of the inherently jarring behavior.

@EricPedley What do you think boss

ok how about we just add a setting to enable autofocus and make it disabled by default? IMO this feature is hella nice so I don't want to see it removed.


[config] Update readme

went through this to try to clean it up a little bit... not sure I made it much better but at least it is up to date

@bors-libra r=joshlind

:pushpin: Commit a629cd4 has been approved by joshlind

:hourglass: Testing commit a629cd4e7894a96b9628473a9d35b430db49e550 with merge 572c482ecdd7d8e3d947b79aca3a47fc007e6034...

:sunny: Test successful - checks-actions_land_blocking_test, checks-circle_commit_workflow
Approved by: joshlind
Pushing 572c482ecdd7d8e3d947b79aca3a47fc007e6034 to master...


Add access control to apps

https://user-images.githubusercontent.com/17034772/219683815-12213e08-1109-47d4-bfc7-c611613c5349.mp4

@josevalim updated! Here's how it looks:
https://user-images.githubusercontent.com/17034772/219811868-f1a90771-9f3c-4903-80c4-ce872869df24.mov
The loading screen is only applicable if the app has password, and we also skip it if they have Livebook-wide access.
I added a comment with description of the flow and integration tests.


Workload patching

I'm triyng to implement HPA provisioner and i'm facing the issue with Deployments spec.replicas set to 1. Documentation mentions this but doesn't explain why.
I'd like to unset it and the only supported option seems to be via patching in cli arguments
I've toyed with a similar tool and ended up implementing Json patches in provisioner definitions.
These returned along side with manifests and run at the end of the rendering pipeline.
This way provisioner functions similar to kustomize's components.
Makes provisioners more self contained and removes the need to monkey-patch with cli arguments or kustomize step later.
What do you think? Was this discussed already somewhere?

Hi @maxstepanov, just to make sure what do you mean by unset the spec.replicas field?
Also, what you want is generating/deploying a HorizontalPodAutoscaler resource, right?

When using HPA it is recommended to remove the spec.replicas field in Deployment. It's to avoid conflicts between manual scaling and HPA-controlled scaling. When kubectl applies the Deployment the replicas will be reset to spec.replicas value. Some pod might start to shutdown before HPA can react to adjust the replicaset to the desired value.
The default value for spec.replicas is 1 so i'm not sure why it's set explicitly by score-k8s.
That's what brought me to suggest the JSON patches in provisioners.
Deployment can be patched by the provisioiner itself then i don't need to patch the value from  command line and leak this aspect outside. I found this approach very useful in my implementations.
Like so:
- uri: template://default/hpa-deployment
  type: hpa-deployment
  manifests: |
    - apiVersion: autoscaling/v2
      kind: HorizontalPodAutoscaler
      metadata:
        name: {{ .SourceWorkload }}
      spec:
        scaleTargetRef:
          apiVersion: apps/v1
          kind: Deployment
          name: {{ .SourceWorkload }}
        minReplicas: {{ .Params.minReplicas }}
        maxReplicas: {{ .Params.maxReplicas }}
        metrics:
          - type: Resource
            resource:
              name: cpu
              target:
                type: Utilization
                averageUtilization: 80
  patches:
  - target:
      kind: Deployment
      name: "{{ .SourceWorkload }}"
    patch: |-
      - op: remove
        path: /spec/replicas

PS: Not sure if Workload Kind value is available to provisioners but would be nice to have.

Makes sense, thanks for the info.

That's what brought me to suggest the JSON patches in provisioners.

I totally agress with that, it would be very beneficial, to inject or patch Deployment's field. Example of: "I want to inject securityContext, etc. Note: I have filed in score-compose the same ask/approach: https://github.com/score-spec/score-compose/issues/142. CC: @astromechza

The default value for spec.replicas is 1 so i'm not sure why it's set explicitly by score-k8s.

That's a good point: https://github.com/score-spec/score-k8s/blob/main/internal/convert/workloads.go#L212, @astromechza, should we not set/generate this field/value by default?

Fix pr for replicas=1 is up!

I totally understand the need for modifying the converted output of score-k8s, and we have a few places where I've seen yq or kustomize used to modify the manifests prior to deployment.
I'm not clear really why a "resource" is the answer here though. It might need to be some other syntax and semantics that allows users to modify the workload output - similar to how we might allow score workloads to be converted to jobs/cronjobs or custom CRDs.

@astromechza There are many cases that require Workload patching along side the added manifests.
If patches defined somewhere else the implementation of the resource is incomplete in template provisioner.
I have a case where i need to create a Secret and mount it as file.
So I created the template provisioner that adds the Secret to output manifests. Next i need to patch the volumes into workload. I don't see any way to accomplish that in template provisioner definition. After all i'd like to have the code in one place.
I can run yq after score-k8s generates, detect if resource of this type is used. Read it and patch it with kustomize but this makes the implementation leak outside and spread it over several components.
At this point it's easier not to implement template resource at all and patch it with kustomize Component later.
That's what we have been doing so far.
If i can specify JSON patches and template together then provisioner implementation is self-contained and easy to understand. That's why kustomize added Components at the end and we use them heavily.
This approach would require adding WorkloadKind to provisioners Input so provisioner can support different kinds and error out if it's not supported.
Here is one example. The same this with serviceAccounts, PDBs etc...
- uri: template://default/mounted-secret
  type: mounted-secret
  manifests: |
    {{ if not (eq .WorkloadKind "Deployment") }}{{ fail "Kind not supported" }}{{ end }}
    - apiVersion: v1
      kind: Secret
      metadata:
        name: {{ .State.service }}
        annotations:
          k8s.score.dev/source-workload: {{ .SourceWorkload }}
          k8s.score.dev/resource-uid: {{ .Uid }}
          k8s.score.dev/resource-guid: {{ .Guid }}
        labels:
          app.kubernetes.io/managed-by: score-k8s
          app.kubernetes.io/name: {{ .State.service }}
          app.kubernetes.io/instance: {{ .State.service }}
      data:
        password: {{ .State.password | b64enc }}
  patches: |
    - op: add
      path: /spec/template/spec/volumes
      value:
        - name: "{{ .SourceWorkload }}"
          secret:
            secretName: "{{ .SourceWorkload }}"

The second route is not to use score-k8s templating facilities at all and implement custom CMD provisioners that leave the patches aside for the later stage of kustomize patching. This approach is cumbersome and devalues template provisioners.
I can work on this if there is interest.

I have a case where i need to create a Secret and mount it as file. So I created the template provisioner that adds the Secret to output manifests. Next i need to patch the volumes into workload. I don't see any way to accomplish that in template provisioner definition. After all i'd like to have the code in one place.

Ok cool, this is the use case.
We do already have a solution to this particular thing, although it's not documented particularly well. https://github.com/score-spec/score-k8s/blob/26e22116f2a7562d669256bc1d64e9013a6be346/internal/provisioners/default/zz-default.provisioners.yaml#L39C5-L39C5
The template provisioner returns an output key that looks like secret-thing: {{ encodeSecretRef "my-secret" "my-key" }} this is a way of encoding Secret Name, and key within the secret.
Then when you mount this as a file in the workload, you use container.x.files[n].content: ${resources.foo.secret-thing}} it will identify that this is a secret, and will mount it as a volume in the pod.
So your example above can look like this:
Provisioner:
- uri: template://default/mounted-secret
  type: mounted-secret
  outputs:
    reference: {{ encodeSecretRef .State.service "password" }}
  manifests: |
    {{ if not (eq .WorkloadKind "Deployment") }}{{ fail "Kind not supported" }}{{ end }}
    - apiVersion: v1
      kind: Secret
      metadata:
        name: {{ .State.service }}
        annotations:
          k8s.score.dev/source-workload: {{ .SourceWorkload }}
          k8s.score.dev/resource-uid: {{ .Uid }}
          k8s.score.dev/resource-guid: {{ .Guid }}
        labels:
          app.kubernetes.io/managed-by: score-k8s
          app.kubernetes.io/name: {{ .State.service }}
          app.kubernetes.io/instance: {{ .State.service }}
      data:
        password: {{ .State.password | b64enc }}

And your workload could look like
...
containers:
  example:
    ...
    files:
    - target: /mnt/password
      content: ${resources.secret.reference}
resources:
  secret:
    type: mounted-secret

That's what I was afraid of when choosing the Secret as example.
What about serviceaccounts? Pdbs? I don't want to expose this to the
developer at all. Add and patch is the regular case. I wish this tool can
meet me in the middle instead of forcing to implement CMD provisioners with
post processing hooks. Oh well...
On Wed, Sep 4, 2024, 20:06 Ben Meier @.***> wrote:

I have a case where i need to create a Secret and mount it as file. So I
created the template provisioner that adds the Secret to output manifests.
Next i need to patch the volumes into workload. I don't see any way to
accomplish that in template provisioner definition. After all i'd like to
have the code in one place.
Ok cool, this is the use case.
We do already have a solution to this particular thing, although it's not
documented particularly well.
https://github.com/score-spec/score-k8s/blob/26e22116f2a7562d669256bc1d64e9013a6be346/internal/provisioners/default/zz-default.provisioners.yaml#L39C5-L39C5
The template provisioner returns an output key that looks like secret-thing:
{{ encodeSecretRef "my-secret" "my-key" }} this is a way of encoding
Secret Name, and key within the secret.
Then when you mount this as a file in the workload, you use container.x.files[n].content:
${resources.foo.secret-thing}} it will identify that this is a secret,
and will mount it as a volume in the pod.
So your example above can look like this:
Provisioner:

uri: template://default/mounted-secret
type: mounted-secret
outputs:
reference: {{ encodeSecretRef .State.service "password" }}
manifests: |
{{ if not (eq .WorkloadKind "Deployment") }}{{ fail "Kind not supported" }}{{ end }}

apiVersion: v1
kind: Secret
metadata:
name: {{ .State.service }}
annotations:
k8s.score.dev/source-workload: {{ .SourceWorkload }}
k8s.score.dev/resource-uid: {{ .Uid }}
k8s.score.dev/resource-guid: {{ .Guid }}
labels:
app.kubernetes.io/managed-by: score-k8s
app.kubernetes.io/name: {{ .State.service }}
app.kubernetes.io/instance: {{ .State.service }}
data:
password: {{ .State.password | b64enc }}

And your workload could look like
...
containers:
example:
...
files:
- target: /mnt/password
content: ${resources.secret.reference}
resources:
secret:
type: mounted-secret
—
Reply to this email directly, view it on GitHub
https://github.com/score-spec/score-k8s/issues/25#issuecomment-2329579601,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAESDH7CZVTSNXLEVZWXD2DZU44ZPAVCNFSM6AAAAABNDUO6J2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRZGU3TSNRQGE
.
You are receiving this because you were mentioned.Message ID:
@.***>

Yeah I agree that it doesn't handle other kinds of links between resources and workloads. In general we should look for ways to have the workload consume a resource, rather than the resource submitting arbitrary patches to workloads.

As seen on #29.

@astromechza The same thing we do every day, add objects and patch references. If the reference in the workload like, service accounts, pod disruption budget, topology spread i must be able to patch them. In order to generate the patch i need to know the object Kind to provided the appropriate patch.
Our Developers, must be able to specify the service account, topology spread, pod disruption budget etc in score-k8s via resource!
score-k8s doesn't support this pattern! Only for secrets and volumes!
I'm trying to avoid forking this, that's why i'm begging to add the value. so i can re-implement templating in CMD provider that also leaves patches out-of-band(score is not aware of them).
then i can use wrap and dump pattern. Where i wrap the score-k8s execution in a script that picks up the patches left by the my CMD provisioners and apply them to manifests.yaml
This is such a common devops pattern I'm really surprised to get so much push back on this.

There's a suggestion that we should be able to access workload annotations or context which describes the output "shape" of the workload since without this the patch may be invalid or unnecessary.

I've been looking at other Score implementations and similar issues and conversations and I think I have a suggestion here.
Similar prior art:

Some score-compose users add a compose-mixins.yaml which gets applied after their main compose spec via docker compose -f compose.yaml -f compose-mixins.yaml. This mixin gets applied directly on top of the workload.

Humanitec has a resource type called a workload which can provide update outputs which are applied as json patches to the generated Humanitec "Deployment Set". See https://developer.humanitec.com/platform-orchestrator/reference/resource-types/#workload.

So I think we can probably do a similar thing here: Currently the provisioners have manifests as part of their execution output, these are new manifests to add to the output but do not get applied to the workload's manifest. We could add a patches output, which gets applied using rfc6902 to the workload output itself after being converted.
After converting the workload, we search all listed resources and apply any patch outputs in lexicographic order based on the resource name.
This mechanism could be re-used by most of our other Score implementations in order to apply meaningful mixins.
We currently pass the workload name through to the provisioner, but the provisioner will need to understand more about the structure of the output kubernetes spec in order to generate a meaningful patch. This could easily be done through the params or annotations on the resource for now while we investigate this further.
Example provisioners:
As an example, here's a workload which requests a k8s-service-account resource type. The application running inside the workload will expect this to provider the urls and k8s token needed to access a kubernetes API. Even if the workload was running on docker compose via score-compose or some other score implementation.
score-k8s implementation:
- uri: template://example-k8s-service-account
  type: k8s-service-account
  manifests:
  - <service account manifest here if it needs to be generated>
  patches:
  - op: add
    path: /spec/serviceAccountName
    value: my-service-account

score-compose implementation:
- uri: template://example-k8s-service-account
  type: k8s-service-account
  patches:
  - op: add
    path: /services/my-workload/environment/KUBERNETES_SERVICE_HOST
    value: localhost
  - op: add
    path: /services/my-workload/environment/KUBERNETES_SERVICE_PORT_HTTPS
    value: 6443
  - op: add
    path: /services/my-workload/volumes/...
	value: < a kubernetes token or something here>

Or something similar to that. This is also generic enough to work for other workload kinds in the future too.
Thoughts @mathieu-benoit ?

I'll see if I can put up a PR for this soon

Yes, I think patches is really good approach. I can add/inject securityContext, etc. or patch any field.
Now, I'm wondering with your example, in score file, we will need to add a resource dependencies k8s-service-account? If that is, It think that's not convenient. Because what about I patch any fields, for example I want to inject securityContext, how to inject that? Create a type workload-security-context? Not very user friendly... Could we have like in Humanitec a workload type, which is always resolved and that someone can implement/override?
Note: same note/remark for both score-k8s and score-compose.

@mathieu-benoit Or just create a type workload an add your custom provisioner for that.
resources:
  workload-patch:
    type: workload

I don't think we should add automatic workload resources that suprise users.


fix event color regression

Event Color wasn't updating properly, this will fix the issue.

Shouldn't typechecking have caught this bug?

The old function returned back an array of [string, number], where string was the Type and the number was the confidence pair.  The new one follows the naming more literally and returns the 'Type' string.  So taking the first index of a string technically means it is still a string just of length 1.  In the example email for the bug all of the types that were visible started with the same letter 'p' so all color mapping was mapped to the same color.  So it wasn't using the full name of the type for colors.

Shouldn't typechecking have caught this bug?

The old function returned back an array of [string, number], where string was the Type and the number was the confidence pair. The new one follows the naming more literally and returns the 'Type' string. So taking the first index of a string technically means it is still a string just of length 1. In the example email for the bug all of the types that were visible started with the same letter 'p' so all color mapping was mapped to the same color. So it wasn't using the full name of the type for colors.

I've been watching too many domain-driven design youtube videos, so my immediate thought is whether we can use the type system to enforce better constraints on these kinds of values. Something like TypeScript's type predicates or something like that... just something to think about.


Follow best practices for Git commit messages

@xuri Reminder of #230 because some recent commits still do not follow best practices...

3 years later this issue could be reopened... 😢

Thanks for your advice, I'll follow best practices in the future code commit.


More frequent on_event_position_update, subsampling of on_event_trade_update and on_event_orderbook_update

By analyzing the log information in swap,
updated frequencies of each method are:
on_ticker     4442
cancel_orders   4442
place_orders    4442
position update  1292
asset update    1932
kline update     5616
orderbook       99472
trade update    52824
Since async on_ticker calls
await self.cancel_orders()
await self.place_orders(),
the same number of on_ticker=cancel_orders=place_orders.
Since strategy depends on position for closing and entry price,
can one control to have more frequent update position?
In extreme market condition, orderbook(20 times of on_ticker) and trade update(10 times of on_ticker) causes overloading of system,
can one subsample orderbook and trade like 1/5 in extreme market condition?
Best regards,
Ingyu

Hi, @igkoh
Since strategy depends on position for closing and entry price,
can one control to have more frequent update position?
A: If you need to have more frequent update position,you can query the postion by the restful interface.
In extreme market condition, orderbook(20 times of on_ticker) and trade update(10 times of on_ticker) causes overloading of system,
can one subsample orderbook and trade like 1/5 in extreme market condition?
A: The orderbook and trade are updated once there is orderbook or trade event.I still have no better way to handle it.

@foonsun,
Following your comments, rest_api interface is queried as
position, error =  await self.trader.rest_api.get_position(self.symbol)
with the correct answer
{'status': 'ok', 'data': [{'symbol': 'BTC', 'contract_code': 'BTC-USD', 'volume': 2.0, 'available': 0.0, 'frozen': 2.0, 'cost_open': 12247.048309592923, 'cost_hold': 12247.048309592923, 'profit_unreal': -1.40774572778e-05, 'profit_rate': -0.00862036496786141, 'lever_rate': 10, 'position_margin': 0.001634454296571732, 'direction': 'buy', 'profit': -1.40774572778e-05, 'last_price': 12236.5}], 'ts':<PHONE_NUMBER>131}
question:  snce the return from query is synchronous return from rest_api, why I need await in front of trader.rest_api.get_position.
If the "await" is ommitted, error message
builtins.TypeError: 'coroutine' object is not iterable.
Is await required only for the syntax rule?

Why on_event_position_update and rest_api.get_position return DIFFERENT informations?  For example, in "on_event_position",
'D [2020-08-18 00:00:00,861] [-] [MyStrategy] position update: [platform: huobi_swap, account<EMAIL_ADDRESS>strategy: my_test_strategy, symbol: BTC-USD/SWAP, short_quantity: 0, short_avg_price: 0, long_quantity: 0, long_avg_price: 0, liquid_price: None, utime:<PHONE_NUMBER>229] '
"available" is not in return, while rest_api.get_position has "available" key-value.

@foonsun,
Following your kind comments, rest_api interface is queried as
position, error = await self.trader.rest_api.get_position(self.symbol)
with the correct answer
{'status': 'ok', 'data': [{'symbol': 'BTC', 'contract_code': 'BTC-USD', 'volume': 2.0, 'available': 0.0, 'frozen': 2.0, 'cost_open': 12247.048309592923, 'cost_hold': 12247.048309592923, 'profit_unreal': -1.40774572778e-05, 'profit_rate': -0.00862036496786141, 'lever_rate': 10, 'position_margin': 0.001634454296571732, 'direction': 'buy', 'profit': -1.40774572778e-05, 'last_price': 12236.5}], 'ts':<PHONE_NUMBER>131}
Question: since the return from query is synchronous return from rest_api, why I need "await" in front of trader.rest_api.get_position.
If the "await" is dropped, error message builtins.TypeError: 'coroutine' object is not iterable.
Is await required only for the syntax rule?
Best regards,
Ingyu

yes,await is required only for the syntax rule.

@foonsun, await is pythonic discipline for syntax.   Thanks

available


Revert "Emit Debug Variables for Errors Bound in Catch Clauses"

Reverts apple/swift#40372
This regressed the windows CI: https://ci-external.swift.org/job/swift-PR-windows/18932/consoleText

@swift-ci please smoke test

CC: @CodaFi

This is not the cause of the async regression AFAICT.

@CodaFi CI seems to indicate so, the buildbot seems to agree (note that the CI has returned green on Windows vs main).

Could we XFAIL this if I trade you a radar/SR pair? This test has been historically flaky and is usually a leading indicator of instability in async code gen.

@CodaFi if you throw in a forced merge, I'll bite.

https://bugs.swift.org/browse/SR-15552


reloadData() problem

Hi Kishikawa,
Thanks for a really excellent spreadsheet view, which I have incorporated into my app.
One problem. I change some settings and try to refresh the spreadsheetView with the usual:
spreadsheetView.reloadData()  or
spreadsheetView.setNeedsDisplay()
But this has no effect on
func spreadsheetView(_ spreadsheetView: SpreadsheetView, cellForItemAt indexPath: IndexPath) -> Cell?
Any suggestions?
Thanks
Roger

@rogpfreck Because the spreadsheetView (_: cellForItemAt:) method is called during the layout process, the needsReload flag is cleared at the end of the layout process.
Calling reloadData () inside spreadsheetView (_: cellForItemAt:) doesn't make sense, for example, doing the same thing with UITableView will also produce unexpected behavior. Could you please tell me why you want to do so, and show your code?

Hi Kishikawa,
Thanks for your reply.
I have/had no intention of calling reloadData inside the func spreadsheetView (_: cellForItemAt:) . As you rightly point out that would be daft.
Rather I wanted to call reloadData() outside of the func so that it would refresh the data that is being used by spreadsheetView. If spreadsheetView was a standard tableView this would be a trivial matter as reloadData() is used, as you know, precisely for that purpose.
My point is that treating spreadsheetView as though is was a tableView and trying to refresh it outside of the func does not work. Or at least as far as I can ascertain. Here's a trivial example.

The spreadsheetView is displayed with data.
User wants to change something so they go to a popover settings view and for example, select a different colour of text to be used in a cell.
ReloadData() is called when the user is finished.
In the spreadsheetView (_: cellForItemAt:) these changes can be manipulated easily, if it was invoked.

But spreadsheetView (_: cellForItemAt:) does not get called on reloadData()
I tried this with your fine Gantt example, by placing a button on a top toolbar on the view controller , and firing ReloadData() from there. No effect.
I trust this helps to clarify things.
Roger
On 16 May 2017, 01:32 +0100, Kishikawa Katsumi<EMAIL_ADDRESS>wrote:

@rogpfreck (https://github.com/rogpfreck) Because the spreadsheetView (: cellForItemAt:) method is called during the layout process, the needsReload flag is cleared at the end of the layout process.
Calling reloadData () inside spreadsheetView (: cellForItemAt:) doesn't make sense, for example, doing the same thing with UITableView will also produce unexpected behavior. Could you please tell me why you want to do so, and show your code?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub (https://github.com/kishikawakatsumi/SpreadsheetView/issues/21#issuecomment-301641776), or mute the thread (https://github.com/notifications/unsubscribe-auth/ANBL5ymucZt5yGH6gzruaA7agy5raqBoks5r6O6lgaJpZM4Nbpt8).

I am also seeing this issue, I am trying to reload the data after touching a header cell to sort data:
func spreadsheetView(_ spreadsheetView: SpreadsheetView, didSelectItemAt indexPath: IndexPath) {
        if let _ = spreadsheetView.cellForItem(at: indexPath) as? HeaderCell {
            self.data.sort { $0 < $1 }
            print(self.data)
            ssview.reloadData()
        }
    }

nothing happens.

I'm having the same issue, for the same use case as blainerothrock.

@rogpfreck @blainerothrock @jpopham I found a rendering issue after reloading. But reloading data seems to work correctly. Can anyone provide a reproducible sample project?

here you go. In this project, I am comparing SpreadsheetView and SwiftSpreadsheetView. The code for SpreadsheetView can be found in ViewController.swift and Cells.swift
Let me know if you have any questions.

Another Example.  Clicking on the "Change Value" button updates a UILabel, a UITableView and a UICollectionView - the latter two by calling there reloadData() method.  The reloadData() method of SpreadsheetView is also called, but the SpreadsheetView display doesn't update.

I'm also facing this issue.
I have found out that if the number of rows or columns wasn't changed It won't do the reload, hope this help.

@blainerothrock @jpopham Thanks for repro case. The cause of the issue is that reusing cell keeps old data after reloading 😥 It should be fixed by https://github.com/kishikawakatsumi/SpreadsheetView/pull/37, and I will release a new version soon.

Awesome, I tested on that branch and it is working! Thanks!

Yes that branch works for me too.  Thanks


Add missing 'six' requirement

Coverage remained the same at 85.23% when pulling 79ecce2bbf59ba4eb711dfa74ad9a63bee3c2125 on camomile-project:feature/fix_setup into 04f533cbe35bdfc9f5bf8b000185dad4ba16b0ff on redodo:develop.

Nice catch, thanks.


[EXPLORER] Add network-wide router liquidity display

Problem
We don't have a way to see all of the router liquidity available for a certain asset on a certain chain. We have to manually go to each router and see how much liquidity they provided.
Impact
Very helpful for speeding up debugging and for more technical users.
Proposed Solution
On the routers page in the explorer there should be a way to filter by asset and chain, and see a total amount for that liquidity provided by all the routers.
Acceptance Criteria
Self explanatory

duplicate of https://github.com/connext/monorepo/issues/3922


chore(init): Switch schema link to shorter one

... that is only used in generated Prisma schemas so we can fully identify how many people actually click the link in the schema generated by prisma init

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


HOL3 - .NET; steps 12 to 15 might be duplicated

I'd say that at some merge or at some issue fixing something got messed up since step 12 is super-similar to step 15, but the step 15 says "updated", even though it's unclear what the scope of the update is (update from what, with what etc.).
Also, the screenshots are just misleading, because the protocol is the same (HTTPS) but the port is completely different - may I suggest removing one of the screenshots, just for the sake of clarity?
Of course, following (by skipping) is always a solution, but every single attendee did however stop at these steps, read them through a few times and always raised their hand for assistance asking me to "translate" :)

Added bubbles to the screenshot noting that the local IIS Express port may be different.


kore-exec profile: why are there so many calls to prove?

This profile comes from #2757, on a 2-character input string. The proof takes ~800ms.
valid-jump-dests.json.

Note: this is a regular Haskell profile (not a kore-prof one). We need to see if this is still happening.


Allow access to total-usage method for client_credentials

Improvements

Added a new method to be allowed to access for requests made using client-credentials.

Also wanted find out, is it good if we add a new config element containing allowed methods for client_credentials? Instead of making changes to code based on the need?


Standalone App

Please make a standalone Android app that can allow you to login to any Commafeed account from commafeed.com or any self hosted server that is open source and can be downloaded on F-Droid.

I just use a web browser on my phone to access my self-hosted commafeed instance, and it works really well.

I just use a web browser on my phone to access my self-hosted commafeed instance, and it works really well.

Sorry I don't plan to write a native app. As already said, the website is mobile friendly though :)


Reproducible test_dataset crash on Windows / 1.10.2 from anaconda

https://github.com/aldanor/hdf5-rust/runs/924910831
Anyone who has a windows box is welcome to check this out ^
Seems to results in STATUS_HEAP_CORRUPTION in test_dataset.

@magnusuMET ^ btw: this will be a blocker for migrating everything fully to GHA and releasing a new version, I think.
There's another reproducible macOS test failure on 1.8.18 with homebrew build, but that one I'll take on myself since I have a mac box at home.

I can reproduce on a virtual machine running on linux. The error occurs in
test_read_write_varlen_struct -> test_read -> assert_eq (line 113)
The error is probably due to some undefined behaviour. Moving the ? operator to
let out_vec = ds.read_raw::<T>()?;

makes the assertation pass, but it again fails on the next assertation.

Hm, interesting, needs further investigation.
What we need to figure out, I guess - is it our fault or a known HDF5 bug? If it's the first one causing some UB (although it looks like some very reproducible UB), we have to fix it; if it's a known bug - just skip this test, or maybe try to work around it.

I was thinking also to schedule a weekly run of ~200-250 jobs, running a full test matrix over all known HDF5 versions from all sources on all platforms and all Rust versions - this may help uncover more crash bugs in the future.
(Will probably do it later after this crashbug is investigated and GHA PR is merged in)

@aldanor I have a question regarding memory managment of the string types. When reading raw values from a dataset, who has the responsibility for calling free on the allocated strings? If it is us, which I assume since the variable strings have a Drop implementation, are we sure we are using the same allocator to free as was used to allocate the strings in hdf5-c? STATUS_HEAP_CORRUPTION looks like an allocator error

See https://portal.hdfgroup.org/display/HDF5/H5_FREE_MEMORY,

It is especially important to use this function to free memory allocated by the library on Windows. The C standard library is implemented in dynamic link libraries (DLLs) known as the C run-time (CRT). Each version of Visual Studio comes with two CRT DLLs (debug and release) and allocating and freeing across DLL boundaries can cause resource leaks and subtle bugs due to heap corruption.
Only use this function to free memory allocated by the HDF5 Library. It will generally not be safe to use this function to free memory allocated by any other means.
Even when using this function, it is still best to ensure that all components of a C application are built with the same version of Visual Studio and build (debug or release) and thus linked against the same CRT.

All tests does indeed pass when leaking memory instead of calling libc::free.

Thanks for investigating!
Yes, it's our job to deallocate it and currently varlen strings in hdf5-types use libc::free because that crate does not depend on hdf5 crate and may be used without it (that was kind of the point, anyway...).
I remember switching to using H5free_memory() in a few places which, IIRC, is guaranteed to deallocate things in a matching fashion, but that's only within hdf crate... https://github.com/aldanor/hdf5-rust/commit/c2cda28b9ba535392870b2ea2311ec5f15520865

Should we add hdf5-sys as a dependency to hdf5-types then? The current situation is a security risk and needs to be fixed.

I don't see how we can access H5free_memory from within hdf5-types crates, maybe on Windows there's a way to just mismatch between MSVC runtime and the version used to build the library, or something like that?
Also, interesting, why does it happen to fail for that particular version only.

I guess it depends on which compiler (and settings) was used for building the library?

Should we add hdf5-sys as a dependency to hdf5-types then? The current situation is a security risk and needs to be fixed.

We can, but then you can no longer have downstream apps/crates that use hdf5-compatible memory layouts but don't actually depend on hdf5; that was my main point.

I guess it depends on which compiler (and settings) was used for building the library?

Yea, we need to figure if it was debug/release and which version; wonder if that's recoverable from the dll somehow? (Btw now I'm even more inclined to do scheduled "crater runs" over huge job matrices)

We can, but then you can no longer have downstream apps/crates that use hdf5-compatible memory layouts but don't actually depend on hdf5; that was my main point.

crates.io does not list any dependent crates, and the types must be somewhat tied in with hdf5 to allow the zero-copy approach. I suppose it is a necessary evil to rely on the hdf5 allocator.


Windows 10 20H2 - Explorer Task Killed

Describe the bug
Windows Explorer task killed when running tweaks on Windows 10 Pro 20H2.
To Reproduce
In the tweaks section I used the following options

Default Tweaks from the desktop section
Add Ultimate Performance
Change DNS to Google

Expected behavior
Run all tweaks without killing the Windows shell completely
Screenshots
N/A
Additional context
It was an easy enough fix to just run explorer from powershell but some users may not know to do so.


Review for usage of datetime.utcnow and datetime.utcfromtimestamp to prep for Python 3.12

Per this link datetime.utcnow and datetime.utcfromtimestamp will be deprecated in Python 3.12. I need to review the package to see if these are used in preparation for Python 3.12

added 3.12 and nothing errored out so this appears to be a non-issue


Convert arguments based on type hint or default value, not both

Currently if an argument has a type hint and a default value like arg: int = None, we first try conversion based on the type hint and if it fails we also try conversion based on the default value. For example, a keyword with the following argument specification could be called like
Keyword    42
Keyword    None

and both usages would work.
This doesn't make much sense, because the type hint not containing the default value type is actually a typing error. For example, the previous example should be arg: Union[int, None] = None (or arg: int|None = None when using Python 3.10 or newer). This logic is also bit more complicated to explain and also makes the conversion code a bit more complicated.
I believe we should change the conversion logic so that if an argument has a type hint, we do conversion solely based on it, regardless does the argument have a default value or not. If argument doesn't have a type hint, we should do conversion based on the possible default value, though.
In practice this change would mean that a keyword with an argument like arg: int = None could be called with an argument that can be converted to an integer like
Keyword    42

and using
Keyword    None

fails. This is obviously backwards incompatible, but, as already mentioned, the typing is wrong and should be fixed.

I don't expect there to be much code with totally invalid typing like arg: int = 'bad. Because it's not generally recommended to use mutable default values like arg: list = [], it is possible that code like arg: list = None exists, though. Even Python itself used to consider this usage ok until Python 3.11 and treated it the same way as arg: Optional[list] = None which in turn is just a shorthand for arg: Uniont[list, None] = None.

After thinking about this a bit more, and after changing the code accordingly and running all our acceptance tests, I don't think this is a good idea right now.
As I mentioned in the previous comment, Python itself handles the arg: list = None case differently depending on the version. With Python 3.10 and earlier, it is the same as arg: Union[list, None] = None, which means that our argument conversion code does None conversion already based on the type hint. With Python 3.11 it stays as arg: list = None, but because our conversion code looks also at the default value, None conversion is done as well. In other words, if we stop looking at the default value, the decision to do None conversion depends on the Python version. That would be confusing to users and also would make our tests version specific.
I still believe we should do conversion only based on the type hint if it exists, but I think it's better to wait until we have Python 3.11 as the minimum Python version. Then the conversion behavior won't change between versions and it ought to be more well known that the default value type needs to be specified explicitly in type hints.
I close this issue now, but I add a comment to the code about reconsidering this in the future. I'll also add a note to the User Guide that this behavior may change.


test executable_dll_and_plugin fails on Linux, GCC 8.1.0, -fsanitize=address

Description
Using commit b2611a1 (latest master), test executable_dll_and_plugin fails:
31/32 Test #31: executable_dll_and_plugin ...........***Failed    0.02 sec
Files "/home/jkt/work/cesnet/gerrit/CzechLight/cla-sysrepo/submodules/dependencies/doctest/examples/executable_dll_and_plugin/test_output/executable_dll_and_plugin.txt" to "/home/jkt/work/prog/_build/doctest/GCC_8_1_NETCONF-Debug/examples/executable_dll_and_plugin/CMakeFiles/temp_test_output.txt" are different.
==========================================================================
== CONTENTS OF /home/jkt/work/cesnet/gerrit/CzechLight/cla-sysrepo/submodules/dependencies/doctest/examples/executable_dll_and_plugin/test_output/executable_dll_and_plugin.txt
==========================================================================
[doctest] run with "--help" for options
I am a test from the dll!
I am a test from the implementation!
I am a test from the implementation_2!
I am a test from the executable!
===============================================================================
main.cpp(0):
TEST CASE:  executable

main.cpp(0): ERROR: test case THREW exception: char: 97

I am a test from the plugin!
===============================================================================
[doctest] test cases:      5 |      4 passed |      1 failed |      0 skipped
[doctest] assertions:      0 |      0 passed |      0 failed |
[doctest] Status: FAILURE!

==========================================================================
== CONTENTS OF /home/jkt/work/prog/_build/doctest/GCC_8_1_NETCONF-Debug/examples/executable_dll_and_plugin/CMakeFiles/temp_test_output.txt
==========================================================================
[doctest] run with "--help" for options
I am a test from the dll!
I am a test from the implementation!
I am a test from the implementation_2!
I am a test from the executable!
===============================================================================
main.cpp(0):
TEST CASE:  executable

main.cpp(0): ERROR: test case THREW exception: unknown exception

===============================================================================
[doctest] test cases:      4 |      3 passed |      1 failed |      0 skipped
[doctest] assertions:      0 |      0 passed |      0 failed |
[doctest] Status: FAILURE!

==========================================================================
== CONTENTS END
==========================================================================
CMake Error at /home/jkt/work/cesnet/gerrit/CzechLight/cla-sysrepo/submodules/dependencies/doctest/scripts/cmake/exec_test.cmake:57 (message):
  Running
  '/home/jkt/work/prog/_build/doctest/GCC_8_1_NETCONF-Debug/examples/executable_dll_and_plugin/executable_dll_and_plugin
  --no-version --dt-no-path-filenames=1 --dt-no-line-numbers=1
  --dt-no-exitcode=1 --dt-gnu-file-line=0' ended with code 'Output is
  different from reference file!'

It would be nice if the test suite run diff automatically, but anyway, here we go:
--- /home/jkt/work/cesnet/gerrit/CzechLight/cla-sysrepo/submodules/dependencies/doctest/examples/executable_dll_and_plugin/test_output/executable_dll_and_plugin.txt    2019-03-19 11:40:17.881073684 +0100
+++ /home/jkt/work/prog/_build/doctest/GCC_8_1_NETCONF-Debug/examples/executable_dll_and_plugin/CMakeFiles/temp_test_output.txt 2019-03-19 11:47:10.422304360 +0100
@@ -7,10 +7,9 @@
 main.cpp(0):
 TEST CASE:  executable

-main.cpp(0): ERROR: test case THREW exception: char: 97
+main.cpp(0): ERROR: test case THREW exception: unknown exception

-I am a test from the plugin!
 ===============================================================================
-[doctest] test cases:      5 |      4 passed |      1 failed |      0 skipped
+[doctest] test cases:      4 |      3 passed |      1 failed |      0 skipped
 [doctest] assertions:      0 |      0 passed |      0 failed |
 [doctest] Status: FAILURE!

Extra information

doctest version: b2611a1
Operating System: Gentoo Linux, amd64
Compiler+version: GCC 8.1.0
cmake invocation (via QtCreator IDE, sorry for a non-exact form of parameters)
Starting to parse CMake project, using: "-DCMAKE_BUILD_TYPE:STRING=Debug", "-DCMAKE_CXX_COMPILER:STRING=/usr/x86_64-pc-linux-gnu/gcc-bin/8.1.0/x86_64-pc-linux-gnu-g++", "-DCMAKE_CXX_FLAGS:STRING=-fsanitize=address", "-DCMAKE_C_COMPILER:STRING=/usr/x86_64-pc-linux-gnu/gcc-bin/8.1.0/x86_64-pc-linux-gnu-gcc-8.1.0", "-DCMAKE_C_FLAGS:STRING=-fsanitize=address", "-DCMAKE_EXE_LINKER_FLAGS:STRING=-fsanitize=address", "-DCMAKE_EXPORT_COMPILE_COMMANDS:INTERNAL=ON", "-DCMAKE_INSTALL_PREFIX:STRING=/opt/nc", "-DCMAKE_INSTALL_RPATH:INTERNAL=/opt/nc/lib64", "-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:INTERNAL=ON", "-DCMAKE_LINKER:STRING=/usr/x86_64-pc-linux-gnu/gcc-bin/8.1.0/x86_64-pc-linux-gnu-gcc-8.1.0", "-DCMAKE_MODULE_LINKER_FLAGS:STRING=-fsanitize=address", "-DCMAKE_PREFIX_PATH:STRING=/opt/nc", "-DCMAKE_SHARED_LINKER_FLAGS:STRING=-fsanitize=address", "-DENABLE_BUILD_TESTS:STRING=ON", "-DGEN_CPP_BINDINGS:BOOL=ON", "-DGEN_JAVA_BINDINGS:BOOL=OFF", "-DGEN_LANGUAGE_BINDINGS:BOOL=ON", "-DGEN_LUA_BINDINGS:BOOL=OFF", "-DGEN_PYTHON_BINDINGS:BOOL=OFF", "-DKEYSTORED_DEFER_SSH_KEY:BOOL=ON", "-DKEYSTORED_KEYS_DIR:INTERNAL=/opt/nc/etc/keystored/keys", "-DSYSREPOCTL_ROOT_PERMS:INTERNAL=-p 600", "-DTHREADS_HAVE_PTHREAD_ARG:BOOL=ON", "-DUSE_SR_MEM_MGMT:BOOL=OFF".

Custom env:
CC=gcc-8.1.0
CFLAGS=-ggdb -O2 -march=native -fsanitize=address
CXX=g++-8.1.0
CXXFLAGS=-ggdb -O2 -march=native -fsanitize=address
LDFLAGS=-fsanitize=address
LD_LIBRARY_PATH=/opt/nc/lib64
PKG_CONFIG_PATH=/opt/nc/lib64/pkgconfig:/opt/nc/lib/pkgconfig

I have not dug into this, but given that this is an ASAN build, it would be an error if the test links together TUs compiled or linked with and without -fsanitize=address. Maybe that's irrelevant. Either way, I'm quite sure that my build env is sane.

doctest runs all the tests on the travis CI with asan/ubsan/tsan and this passes there without trouble...
Can you test with Clang as well? The sanitizers are usually best integrated with clang.
I've hit many such toolchain issues in the past where a specific version of the compiler or something else is causing problems and I'm highly suspicious that this is the case here as well.

I am not hitting this with clang, neither v6, nor v7. However, the issue persists when I try with the latest GCC (8.3.0). I believe that sanitizers actually share the same backend these days (upstream says so, at least), and given that the latest compiler release from GCC fails the test suite, I think that this is relevant.

So the problem is present both with gcc 8.1 and 8.3? This is the GCC that is used on the CI: g++-8 (Ubuntu 8.1.0-5ubuntu1~14.04) 8.1.0 and there is no such issue with any of these compilers: https://travis-ci.org/onqtam/doctest (all tested in Debug/Release - but I'm using the gold linker for all sanitizer tests).
About the sanitizers - I've previously reported bugs to GCC on linux about their integration with the sanitizers - there were issues not present with clang, so it is very much a possibility that something isn't right for a particular patch of gcc. Maybe -march=native is the problem - I don't know - could you test without it? Does the issue persist without the sanitizers? I've had spectacular failures of normal exception use on OSX using an array of different gcc versions installed through homebrew - and there are no longer any CI builds on OSX with gcc - only clang from XCode. Toolchains can be amazingly broken.
Maybe something else on your platform is causing this since it is a project with multiple shared objects. Try with the gold linker? I use it because there were issues with unrecognized flags by the default linker on the CI like --push-state (or something like that) which when the sanitizers were enabled were passed to the linker.
doctest is using exceptions in a standards-conforming and simple way so I seriously doubt that there is an issue on the side of the framework. I'm keeping the issue open but for now I have no clue as to what the reason might be.

seems that this is the flag which fixes the issue: -static-libasan - see this comment: https://github.com/onqtam/doctest/pull/285/files#r333711432


Troubles with optimum

Run pipx install insanely-fast-whisper
Output:
pip seemed to fail to build package:
    optimum

Some possibly relevant errors from pip install:
    error: subprocess-exited-with-error
    FileNotFoundError: [Errno 2] No such file or directory: 'optimum/version.py'
    AssertionError: Error: Could not open 'optimum/version.py' due [Errno 2] No such file or directory: 'optimum/version.py'

Error installing insanely-fast-whisper.

Python 3.10.1
macОs Monterey 12.6 , M1

Hi! @VirtualRoyalty - Can you re-install with a Python version specified. Here's what you'd need to do:

pipx uninstall insanely-fast-whisper (if it is already installed)
pipx install insanely-fast-whisper --python3.11 (As mentioned here: https://github.com/Vaibhavs10/insanely-fast-whisper#-blazingly-fast-transcriptions-via-your-terminal-)

This should alleviate your issue.

I have the same problem. Running:

pipx install insanely-fast-whisper --python3.11
yields:
pipx: error: unrecognized arguments: --python3.11

The correct command to use is:
$ pipx install insanely-fast-whisper --python python3.11

Yeah I discovered that soon after, thanks.Sent from my iPhoneOn 12 Dec 2023, at 08:46, Alberto Lanaro @.***> wrote:
The correct command to use is:
$ pipx install insanely-fast-whisper --python python3.11
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

The command provided by @AlbertoLanaro worked for me.


Return first upstream and first downstream catchments?

This could be added using hy_features semantics looking at the up and dn hydrosequence in an sql query. It would fit into the base NLDI feature response document per #249


Alpine Linux: /bin/ps: unrecognized option: x

Alpine Linux' ps does not appear to have the x option:
[IPC Library: Pty Host] DEBUG ChildProcessMonitor: Fetching process tree failed Error: Command failed: /bin/ps -ax -o pid=,ppid=,pcpu=,pmem=,command=
/bin/ps: unrecognized option: x
BusyBox v1.34.1 (2021-11-23 00:57:35 UTC) multi-call binary.

Usage: ps [-o COL1,COL2=HEADER] [-T]

Show list of processes

        -o COL1,COL2=HEADER     Select columns for display
        -T                      Show threads

    at ChildProcess.exithandler (child_process.js:308:12)
    at ChildProcess.emit (events.js:315:20)
    at maybeClose (internal/child_process.js:1048:16)
    at Socket.<anonymous> (internal/child_process.js:439:11)
    at Socket.emit (events.js:315:20)
    at Pipe.<anonymous> (net.js:673:12)

Version: 1.64.0-insider
Commit:<PHONE_NUMBER>4279d4ecfe0575ca69d68211fb3798
Date: 2022-01-26T05:15:06.786Z
Electron: 13.5.2
Chromium: 91.0.4472.164
Node.js: 14.16.0
V8: 9.1.269.39-electron.0
OS: Darwin x64 20.6.0

@meganrogge FYI Rachel wrote this code so I guess we own it now https://github.com/microsoft/vscode/blob/6358042bbcd72bca2c3c2a667b74fd3a8a7172a2/src/vs/base/node/ps.ts#L232
@chrmarti completely new to Alpine is there any easy way to test this?

The options are very limited:
BusyBox v1.34.1 (2021-11-23 00:57:35 UTC) multi-call binary.

Usage: ps [-o COL1,COL2=HEADER] [-T]

Show list of processes

        -o COL1,COL2=HEADER     Select columns for display
        -T                      Show threads

Header names are different/limited:
ps: bad -o argument 'pcpu', supported arguments: user,group,comm,args,pid,ppid,pgid,etime,nice,rgroup,ruser,time,tty,vsz,sid,stat,rss

And the only way to hide the header name is to use -o for each:
ps -o pid="" -o ppid="" -o comm=""

Remote-Containers reads /proc to avoid the need for ps which isn't necessarily installed, e.g., the Debian Docker image does not have it (https://hub.docker.com/_/debian).

@chrmarti fyi I've found reading /proc throws security related errors sometimes, that's why this throw exists:
https://github.com/microsoft/vscode/blob/7f2580fa2ee290d1a87bf8ec0789f1bc2b7990f2/src/vs/platform/terminal/node/terminalProcess.ts#L545-L549
Could you take this one as I don't have an Alpine setup? We could try use Alpine's limited args or it might be better to detect that and fallback to ps.sh like when ps doesn't exist?
https://github.com/microsoft/vscode/blob/6358042bbcd72bca2c3c2a667b74fd3a8a7172a2/src/vs/base/node/ps.ts#L220-L228

The implementation I'm using ignores when it can't read a /proc file. (There is known bug with env values including newlines.) I see ps.sh is using awk, I have so far avoided that, but can't currently find a distro not including it.
The easiest way to get Alpine for testing is maybe as a container (I don't have a full install of Alpine).

Both the terminal's child process monitor and the process explorer work when connected to an Alpine container so I guess this is fixed? Maybe only in newer versions of Alpine?


ignore_list in method: generate_unwind_node_query

https://github.com/NCATS-Tangerine/kgx/blob/9ce3375832a1d1f85dcb620f7db88431013ddb6f/kgx/neo_transformer.py#L279-L283
Looks like a TODO: load it from a config file?

Not exactly. It was there for testing purposes.
Should have removed it when I was cleaning up that method.

Fixed via 7ffc7ab42e0aa865c05df8bedb7f1ca73945b683


question: TLS possible

i now use PubSubClient v2.8 for a ESP32-S3 Arduno IDE project
and want use TLS with HIVEMQ FREE ACCOUNT
possible?

Hi, you can refer the functions here


[Prototype] AuthNPolicy[Provider] / AuthorizationRequest[Evaluator]

Rough prototype of some of what we were discussing for Auth 2.1 changes to further discussion:

AuthenticationPolicy: gives us a named set of schemes, and the ability to resolve these on a per request basis (PolicyProvider)
AuthorizationRequest: is the new unit of work for AuthZ, it contains the authentication policy name, the authorization requirements, and the resource being authorized.  This is the complete set of data needed to process an authorization
IAuthorizationRequestEvaluator builds on top of both the new 2.0 PolicyEvaluator and the AuthenticationPolicyProvider, it has a single method:
PolicyResult AuthorizeAsync(AuthorizationRequest, HttpContext) and returns Success/Forbidden/Challenged

The new authorization metadata would just need to produce the AuthorizationRequest
I focused mostly on the AuthZ side of things first, exposing the authNPolicy concept for authN will come later
@davidfowl @blowdart @Tratcher @rynowak @Eilon @ajcvickers

@HaoK,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by .NET Foundation. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot

@HaoK can you outline which components use which APIs? Would the Authentication middleware be updated to use the AuthenticationPolicyEvaluator?

Yeah we should consider switching all of the public authentication entrypoints to expose the policy concept.
So

AuthenticationOptions would get a DefaultPolicy property which could be configured
The middleware from UseAuthentication() would switch to using IAuthenticationPolicyEvaluator.AuthenticateAsync(httpContext, policyName: null) which would fall back to the current context.AuthenticateAsync behavior if no policies are configured.
The authorization stack would switch to a new layer built on top of the old authZ layer (IPolicyEvaluator) + named authN policies (IAuthenticationPolicyEvaluator) = IAuthorizationRequestEvaluator)

I'm not sure the ability to name specific other non default authentication policies would be used outside of Authorization, but regardless it would allow dynamic per request resolution of the authentication policies via IAuthenticationPolicyProvider

And some kind of new attribute syntax like:
[Authorize(AuthenticationPolicy = "Cookies", AuthorizationPolicy = "Organization", Resource = Operation.Something)]

Don't forget the imperative calls as well, not just the attribute.

@Tratcher consumption is via IAuthenticationPolicyEvaluator, you call Authenticate/Challenge/Forbid with the policy name.  That's what AuthZ will use as well.  I don't really think SignIn/SignOut make sense unless they supports multiple schemes as it would just be an exact call to the underlying IAuthenticationService, we can keep it as just passthrough if we really want, but doesn't seem that useful

Specifically picking between cookies/bearer isn't all that different, you can do it on [Authorize] explicitly via AuthenticationPolicy, or you can do it dynamically by using a single "BlowdartHatesThis" policy where IAuthenticationPolicyProvider resolves it to cookies/bearer based on something that would never be request path...

To summarize the main benefits of this change: you get the ability to associate a set of named challenge/forbid/authenticate schemes (including a default).  And the ability to dynamically resolve these named schemes on a per request basis.  This will allow us to tidy things up more and finally define some of the undefined behaviors with multiple authentication schemes in Authorize

@blowdart imperative calls will basically just be getting the appropriate service and calling it, with the lowest level something like:
AuthorizeAsync(authenticationPolicy, authorizationRequirements, resource) which returns Success/Forbidden(failure when authentication succeeded)/Challenge(failure when authentication failed)

Specifically, can you show an example of defining two policies and a selector in Startup for the JWT vs Cookies scenario.

Sure, it would look something like:
services.AddAuthentication(o => {
    o.AddPolicy("Cookies", p => p.UseDefault("Cookies"))); // sets all the schemes to cookies
    o.AddPolicy("Bearer", p => p.UseDefault("Bearer"))); // sets all the schemes to bearer
});
services.AddScoped<IAuthenticationPolicyProvider, MyPolicyProvider>();

class MyPolicyProvider : IAuthenticationPolicyProvider {
    AuthenticationOptions _options;

     MyPolicyProvider(IOptions<AuthenticationOptions> options) => _options = options.Value;

    public Task<AuthenticationPolicy> GetAsync(string name, HttpContext context) {
        // If no policy is specified, pick one
        if (name == null) {
            if (ShouldUseBearer(context)) return _options.Policies["Bearer"];
            else return _options.Policies["Cookie"];
        }
        return _options.Policies[name];
    }
}

The layering looks a little odd. I was expecting a policy name selector more like: public Task<string> GetAsync(HttpContext context)
And then a services or auth builder extension that reduces that to a lambda:
`services.AddAuthentication(...).AddPolicySelector((context => ShouldUseBearer(context) ? "Bearer" : "Cookies")

Sure, we can always add something to make selecting only the default easier
AddDefaultPolicySelector(context =>  ShouldUseBearer(context) ? "Bearer" : "Cookies")
and there'd be a matching IDefaultAuthenticationPolicySelector

Example usage:
               services.Configure<AuthenticationPolicyOptions>(o =>
                {
                    o.DefaultPolicySelector = ctx => (string)ctx.Items["policy"];
                    o.AddPolicy("auth", p => p.DefaultScheme = "default");
                });

Are we happy with the current names (and planned homes) for this stuff?
New to Microsoft.AspNet.Core.Authentication[.Abstractions]:
AuthenticationPolicy - Authenticate/Challenge/Forbid schemes only (no sign in/sign out)
AuthenticationPolicyBuilder - Sugar to build policies
[I]AuthenticationPolicyEvaluator - used Authenticate/Challenge/Forbid via policy name (used instead of directly using IAuthenticationService)
[I]AuthenticationPolicyProvider - Per request resolution of name => AuthenticationPolicy
[I]DefaultAuthenticationPolicySelector - Per request resolution of default authentication policy
New to Microsoft.AspNet.Core.Authorization.Policy: (is this the right home?  we added it in 2.0 to link Authentication to Authorization):
AuthorizationRequest - Encapsulates the AuthZ request: authN policy name, resource, requirements
[I]AuthorizationRequestEvaluator - Evaluates the new AuthorizationRequest (replaces IAuthorizationPolicyEvaluator which itself replaces IAuthorizationService)

So I've started work in https://github.com/aspnet/Identity/pull/1442 trying to switch Identity's SignInManager over to using authentication policies.
Initial thoughts are:
There will be a "Identity" AuthenticationPolicy that is setup by default to:
Authenticate/Forbid/ChallengeScheme: "Identity.Application"
// There is no support for SignIn/SignOut right now, as they are kinda confusing
SignInScheme: We set this to "Identity.External" today but that is confusing since you'd expect SignIn to set the Application cookie, as we set this mostly for the external auth policies.
SignOutScheme: Identity needs to clear Application, External, and 2 two factor schemes
So I guess what we could do now is setup a new default "External" policy, which is what we use to set the SignInScheme to "Identity.External" in the templates for social logins, and I guess we'd just have to update the documentation since I don't think we can just make it all work with this, the external providers need to specify their authenticationPolicy somewhere, since there will be always be at least two authentication policies (Identity + External grrr)

Or maybe we should just add additional new dedicated setting for just the remote authentication SignIn default, which is separate from the DefaultSignInScheme, i.e.
services.AddAuthentication(o => {
    o.DefaultPolicy = "Identity.Application";
    o.DefaultRemoteSignInScheme = "Identity.External";

// Google Handler resolves the default SignInScheme by trying to use DefaultRemoteSignInScheme first, before falling back to the existing logic
}).AddGoogle();

No for DefaultRemoteSignInScheme, DefaultSignInScheme is primarily used for remotes anyways.

So if we don't add a new dedicated setting,I guess everything still hangs together for the template if we set the default AuthenticationPolicy to "Identity", then the remote auth handlers will resolve to the External cookie, and SignInManager never uses the default sign in scheme..  Its only users who call SignInAsync not specifying a scheme who will be surprised which hopefully won't be a huge number of people)

So tentative startup will look something like:
    services.AddIdentityCore<ApplicationUser>() // + UserManager
                .AddRoles<IdentityRole>() // + RoleManager
                .AddEntityFrameworkStores(); // + User/RoleStores
                .AddSignIn() // + SignInManager
                .AddDefaultCookies() // + Identity cookies, configures "Identity" auth policy
                .AddDefaultTokenProviders() // TFA stuff

    services.AddAuthentication(o.DefaultPolicy = IdentityConstants.DefaultPolicy)
            .AddGoogle()
            .AddJwtBearer() ...

Auth's relationship with funcs needs to go away. We need to use types like sane people 😄 . They are more DI friendly.

Went with virtual scheme instead: https://github.com/aspnet/Security/pull/1486


LUCENE-9188: Add jacoco code coverage support to gradle

jacoco plugin is activated by -Ptests.coverage=true
Each parallel jvm writes in append-mode to jacoco.db (we clear this out before test-runs to be safe). e.g. you'll see 4 sessions if you have tests.jvms=4
The jacocoTestReport task from the jacoco plugin generates reports,
currently with defaults (HTML only).

I'm reviving this issue as I think it's an easy win to keep an eye on this coverage. Maybe I can work with @mikemccand to add reports to luceneutil, or maybe ultimately we track coverage over time via jenkins plugins.
But we've found good bugs before just looking at untested code from these reports, so I think it is good to make a start.

Also, just casually browsing and randomly clicking the first file that looked interesting, I see huge chunks of untested code that surprised me. So I really do think the problem is just that nobody is looking at code coverage, anywhere.
For example it seems CMS's iothrottle logic is completely untested due to the fact that no merges exceed MIN_BIG_MERGE_MB? If you hover over if mergeMB < MIN_MERGE_MB statement, it explains that one of two branches are missed. so maybe this parameter should be exposed via setter at least so we can test it.
Just trying to make the argument for at least making steps to get more visibility on code coerage.

Looks fine to me, but haven't tested.
I would add a job to jenkins to run all tests with jacob and publish the report as javadocs or similar (depending on plugin choices).

@uschindler this uses defaults, which I think is HTML output only. We could attach reports as artifacts in jenkins as a start.
The plugin seems to support also CSV and XML output, and I think there are jenkins plugins that can use this structured output to generate things like graphs over time. For example, I think with python I used something to convert XML to cobertura format and used jenkins cobertura plugin to keep an eye on coverage and give fancy graphs.

Hi,
I tested with jenkins, works fine. For Jenkins it is enough to run tests with tests.coverage=true and configure the jenkins plugin to create the reports and save them inside jenkins. The task jacocoTestReport is obsolete for Jenkins.
I only had to tweak the settings a bit, because our build does not have default filenames (*.exec) or source location (src/main/java):

While running the jenkins Job, the replicator module produced a strange error with security manager:
> Task :lucene:replicator:test FAILED
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.jacoco.agent.rt.internal_f3994fa.core.runtime.InjectedClassRuntime$Lookup.privateLookupIn(InjectedClassRuntime.java:118)
	at org.jacoco.agent.rt.internal_f3994fa.core.runtime.InjectedClassRuntime.startup(InjectedClassRuntime.java:54)
	at org.jacoco.agent.rt.internal_f3994fa.PreMain.premain(PreMain.java:53)
	... 6 more
Caused by: java.security.AccessControlException: access denied ("java.lang.reflect.ReflectPermission" "suppressAccessChecks")
	at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
	at java.base/java.security.AccessController.checkPermission(AccessController.java:897)
	at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:322)
	at java.base/java.lang.invoke.MethodHandles.privateLookupIn(MethodHandles.java:189)
	... 13 more
FATAL ERROR in native method: processing of -javaagent failed

So the replicator reports zero coverage.
Here are the results of first successful run on your PR's branch:

Main build page: https://ci-builds.apache.org/job/Lucene/job/Lucene-Coverage-main/4/
Go to the coverage reports on left ("Testabdeckung" in German, I got crazy to find this): https://ci-builds.apache.org/job/Lucene/job/Lucene-Coverage-main/4/jacoco/

For easier setup on jenkins (maybe others may do this), let's rename the database file to jacoco.exec instead of jacoco.db. It looks like this is the standard.
P.S.: In future we should also start to remove the outdated "ant" folder layout and be more gradle/maven conformant.

I've changed the code slightly so that you can just say:
gradlew -p lucene/core test coverage

or even more briefly:
gradlew -p lucene/core coverage

This runs the tests and produces a report (printing its location to the console). Seems nicer than having to remember jacoco's task names + property?

Thanks Dawid, I changed the jenkins job to look for the other database file.

The security settings are still not working, aren't they?

My latest commit should fix the replicator module. It was a bug anyways, because this missing permission silently disabled MMapDirectory.

This runs the tests and produces a report (printing its location to the console). Seems nicer than having to remember jacoco's task names + property?

For developers, yes. For Jenkins: No. Jenkins only needs to collect the jacoco.exec files and generates report on its own. So the jenkins job just sets property and runs tests - nothing more!

Jenkins works now.
Replicator mdoule works, too; but it has 2 new test failures (looks like those 2 tests were never ran because MMAP was not working):

org.apache.lucene.replicator.nrt.SimpleServer.classMethod (no idea why this is a test!)
org.apache.lucene.replicator.nrt.TestNRTReplication.testCrashReplica

The replicator problem is not caused by MMAP not working. The missing permission made the whole TestNRTReplication.testCrashReplica test not work at all (the SimpleServer child process wasn't able to start at all because it needs sun.misc.Unsafe access to crush the JVM).
We have to fix the test or alternatively @Ignore it until fixed. It wasn't running correctly since years!

SimpleServer is invoked from another test. It shouldn't be included by default in test runs though (there is a test file name pattern?). I'm not sure why it's being called.

For developers, yes. For Jenkins: No. Jenkins only needs to collect the jacoco.exec files and generates report on its own. So the jenkins job just sets property and runs tests - nothing more!

I left the property there too, just in case, so it should still work if you invoke test with property enabled.

SimpleServer is invoked from another test. It shouldn't be included by default in test runs though (there is a test file name pattern?). I'm not sure why it's being called.

The other failing test starts it as a separate process. Because of the missing permission, the main test was never working and so the issue was not seen before.
I would disable this test and fix the setup in a separate issue. The problem was there since years.
Imho, simple server should not be marked as test.

I corrected the code - simple server actually utilizes test infrastructure and will fail if it's not a "test"... I think what I committed solves the problem with minimal effort?

It shouldn't be included by default in test runs though (there is a test file name pattern?)

Although this was not the problem here: We have no filename pattern! It looks like Gradle figures tests out automatically. But this did not work anymore as soon as you enable the Java Module system. Because of this in the MMapDirectory v2 I had to add the pattern manually. We should maybe always do this, because this was my first idea: "WTF? Again the missing pattern? What the hell?"
This should IMHO be applied on main branch in both solr and lucene: https://github.com/apache/lucene-solr/pull/2176/files#diff-5389a6d358feee1c8ed596fd91b01bcb373afac3570f463c66b825401afcb297R102-R103

I think we should at least run the test once with tests.nightly=true, as most tests in StresstestNRTReplication are only running nightly.

This all looks fine now: https://ci-builds.apache.org/job/Lucene/job/Lucene-Coverage-main/18/ -- I also ran the NRT replication tests with tests.nightly on my local machine, it all succeeded.
If nobody objects, I would like to merge this soon and reconfigure ASF jenkins to use the main branch.

I found a small problem with the delete task. @dweiss: can you check?

Do we need the delete task at all? From Gradle's documentation:

Tasks configured for running with the JaCoCo agent delete the destination file for the execution data when the task starts executing. This ensures that no stale coverage data is present in the execution data.

To me this does exactly what our delete task also does.

Tasks configured for running with the JaCoCo agent delete the destination file for the execution data when the task starts executing. This ensures that no stale coverage data is present in the execution data.

To me this does exactly what our delete task also does.

my guess is that this isn't always what gradle did :)
I put the delete task there because coverage was adding up.

Hi @rmuir,
I removed the extra task. In my tests, the coverage was not adding up. In all combinations (with "gradlew coverage", "gradlew -Dtests.coverage=true test",...) it started with empty database and number of sessions in report was number of JVMs used for tests.
I think it's ready now.

Config now looks like this:

Thanks @uschindler @dweiss for the assistance here!

This looks awesome!
Are our Jenkins builds publishing the Jacoco reports somewhere accessible?

Maybe I can work with @mikemccand to add reports to luceneutil, or maybe ultimately we track coverage over time via jenkins plugins.

+1
We could maybe also add a nightly chart of our total code coverage over time?

Are our Jenkins builds publishing the Jacoco reports somewhere accessible?

Sure: https://ci-builds.apache.org/job/Lucene/job/Lucene-Coverage-main/
I described this all in issue.

Jenkins "documentation": https://issues.apache.org/jira/browse/LUCENE-9188?focusedCommentId=17338040&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17338040

Thanks @uschindler!
Do we disable assertions when running the Jacoco coverage?  I see the assert statements are yellow e.g. here: https://ci-builds.apache.org/job/Lucene/job/Lucene-Coverage-main/lastCompletedBuild/jacoco/org.apache.lucene.index/IndexWriter/carryOverHardDeletes(ReadersAndUpdates, int, Bits, Bits, Bits, MergeState.DocMap, MergeState.DocMap)/

They are always yellow, because only one of the branches were running (successful assert). They would only be green, if assert also fails -- HAHA.
In Clover you were able to disable asserts. But Jacoco is so "simple" that an assert statement is a "standard" if/else branch.

Actually they show as 2 of 4 branches missed (hover over to see). Because when you write assert expr; it really looks something more like (just for demonstration):
if (ASSERTIONS_ENABLED) {
  if (expr) {
    throw new AssertionError
  }
}

So you have 4 branches: assertions=enabled/disabled, and expr=true/false

it really looks something more like (just for demonstration)

It is exactly that code. If you uncompile a class with assertions, you will see this.

There is a relevant jacoco issue: https://github.com/jacoco/jacoco/issues/324
It points to another issue (filtering feature), where you can see filtering asserts looks like an unimplemented TODO? So that seems to be the way forward if the yellow asserts are driving you crazy :)

Another tip: Never click on methods! Just go to "class" level and then scroll down (the problem is that the many method links make you automatically click on them). Jacoco also cannot handle lambdas correctly. If you view coverage of whole class, all looks fine (except asserts).

This looks awesome!
Are our Jenkins builds publishing the Jacoco reports somewhere accessible?

Maybe I can work with @mikemccand to add reports to luceneutil, or maybe ultimately we track coverage over time via jenkins plugins.

+1
We could maybe also add a nightly chart of our total code coverage over time?

jenkins already has a little "code coverage trend": https://ci-builds.apache.org/job/Lucene/job/Lucene-Coverage-main/jacoco/
but I don't know how far it looks back (seems like only a few builds?) Maybe it just needs tweaking. If it doesn't serve our purposes, I think there is also a Rest API to fetch the data in e.g. json: https://plugins.jenkins.io/code-coverage-api/

but I don't know how far it looks back (seems like only a few builds?) Maybe it just needs tweaking. If it doesn't serve our purposes, I think there is also a Rest API to fetch the data in e.g. json: https://plugins.jenkins.io/code-coverage-api/

ASF Jenkins does not seem to have the "uber" code coverage plugin with sophisticated API. It only has the "Publish Jacoco coverage" plugin (that I used). ASF Jenkins is using a Jenkins Cloudbees versions (LTS), which is stone-aged.
Policeman Jenkins is much more modern (weekly updates), so I may install a better plugin using that new API. I can play around a bit.

On ASF Jenkins I made the "Coverage" build to only run "daily". For testing purposes it was "hourly".

I also tried to dig and see if I could change the Lucene: Passing badge in the github README.md to instead show the coverage percentage just as an easily available tracker :) I couldn't figure this out after looking for a few minutes.

On ASF Jenkins I made the "Coverage" build to only run "daily". For testing purposes it was "hourly".

I also changed to keep last 30 days of builds, instead of 5 build items.

Thanks @uschindler . @mikemccand we could add a simple link to https://ci-builds.apache.org/job/Lucene/job/Lucene-Coverage-main/jacoco/ to luceneutil html, with Uwe's changes, it shows a graph of the last 30 days of coverage now.


Feat/allow managing categories

Expose the new setCategoryAllowed(categoryName, isAllowed) which does what it says on the tin. This will be used by the Product Pages to record any amendments to our users's cookies preferences.

Looks good to me, but I haven't got around to testing it locally yet.


ambient: test targetRefs instead of singular

We expect to push all users to use targetRefs in all cases. Both work,
but uses copy+paste from tests so its best to use best practices here.
Unit tests provide sufficient coverage that singular and plural are the
same.

@howardjohn: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-ambient_istio
79ed385597f50a964c048f5fb4f507ed03810746
link
true
/test integ-ambient

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


[release-1.11] Upgrade to latest dependencies

Update dependencies -pierDipi
/cc knative-extensions/kn-plugin-operator-approvers
/assign knative-extensions/kn-plugin-operator-approvers
Produced by: knative-extensions/knobots/actions/update-deps

/approve
/lgtm


How to disable log output from version 0.18

Maybe a stupid question, but we upgraded from version 0.17 to 0.18.2 and now get tons of log messages like this
14:33:04.318 [Worker-8: Building] INFO guru.nidi.graphviz.model.Serializer - INFO  node 'NODE_39' -> 'fixedsize' has its default value 'false'.
14:33:04.318 [Worker-8: Building] INFO guru.nidi.graphviz.model.Serializer - INFO  node 'NODE_40' -> 'fontcolor' has its default value 'black'.
14:33:04.318 [Worker-8: Building] INFO guru.nidi.graphviz.model.Serializer - INFO  node 'NODE_40' -> 'height' has its default value '0.5'.
14:33:04.318 [Worker-8: Building] INFO guru.nidi.graphviz.model.Serializer - INFO  node 'NODE_40' -> 'penwidth' has its default value '1.0'.
14:33:04.318 [Worker-8: Building] INFO guru.nidi.graphviz.model.Serializer - INFO  node 'NODE_42' -> 'fontcolor' has its default value 'black'.
14:33:04.318 [Worker-8: Building] INFO guru.nidi.graphviz.model.Serializer - INFO  node 'NODE_42' -> 'penwidth' has its default value '1.0'.
14:33:04.318 [Worker-8: Building] INFO guru.nidi.graphviz.model.Serializer - INFO  node 'NODE_42' -> 'fixedsize' has its default value 'false'.
14:33:04.318 [Worker-8: Building] INFO guru.nidi.graphviz.model.Serializer - INFO  node 'NODE_43' -> 'fontcolor' has its default value 'black'.
14:33:04.318 [Worker-8: Building] INFO guru.nidi.graphviz.model.Serializer - INFO  node 'NODE_43' -> 'penwidth' has its default value '1.0'.
14:33:04.318 [Worker-8: Building] INFO guru.nidi.graphviz.model.Serializer - INFO  node 'NODE_43' -> 'fixedsize' has its default value 'false'.
How can I turn of this messages?
Thanks, Max

No hint from nobody?

You can try the  no-operation (NOP) logger implementation.
Hope it helps.


Add MVP Workday Tools

Create MVP tools for Workday.
Initial Use Cases
The MVP tool set should allow agents to:

Request and manage time off requests
Perform people lookups

This is blocked until we have access to Workday. We're currently in the process of getting in contact with the Workday support.


Bot cannot access images added by clipboard

Hi there,
I am working on a bot which receives messages from Teams via the Botframework and sends them to another application. I'm using .NET Core.
When sending a normal file, I can access it with the Attachment.Content["downloadUrl"] of the activity which I receive from the Botframework.
But when I insert images from the clipboard, it is stored in another way, with the Activity I get two attachments with different urls (one for layout and one for the image resource, I suppose?), like the image below, but when trying to GET the image, I receive an 401 Unauthorized.
Is there a way to access these images pasted from the clipboard?
Or are these only visible/accessible from inside Microsoft Teams?
This way the Teams user thinks everything is alright, but the Bot/other application has no access to them.
Also, where do these clipboard-images be safed afterwards, if not in the SharePoint like "properly uploaded" files? Does the user ever has access to them again?
This applies to Stickers, too. Gifs can be sent without problem, because it's a giphy url.
I am using the OnMessageActivityAsync overload described in the Microsoft Docs
public class BotActivityHandler : Microsoft.Bot.Builder.Teams.TeamsActivityHandler
    {
        protected override async Task OnMessageActivityAsync(ITurnContext<IMessageActivity> turnContext, CancellationToken cancellationToken)
        {
            // it's either this url or the one parsed from the html, but both receive a 401 with a GET request like below or with Postman
            var imageUrl = turnContext.Activity.Attachments.Where(x => !x.ContentUrl.IsNullOrEmpty()).ToList().First().ContentUrl;

            var response = await myHttpClient.GetAsync(imageUrl);
            var img = await response.Content.ReadAsStreamAsync();
        }

The turnContext includes an Activity with two Attachments: One has Content like

\n
\n\n
\n\n\n
\n

And the other one a ContentUrl like

https://smba.trafficmanager.net/emea/v3/attachments/[...]/views/original

@joshamil ,
MS teams framework treat attachment object and clipboard image differently.

When user paste an image in MS teams chat, it does not treat it as an usual attachment. Rather it receives as an image type and stores it in MS Teams internal storage and sends the Image URL in Activity.Attachment[0].
you need to download the image content from that URL and upload to your blob storage.
MS Teams attachment URLs are secured by a JWT Token, so we need to pass the token from our bot to access it.

Here is a sample c# code to get the image form MS Teams URL and save it to solution file. You can add this logic in ActivityHandler method.
if (turnContext.Activity.Attachments != null && turnContext.Activity.Attachments.Count > 1
            && turnContext.Activity.Attachments[0].ContentType=="image/*")
        {
            var attachment = turnContext.Activity.Attachments[0];
            using (HttpClient httpClient = new HttpClient())
            {
                // MS Teams attachment URLs are secured by a JwtToken, so we need to pass the token from our bot.
                var token = await new MicrosoftAppCredentials("appId", "password").GetTokenAsync();
                httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
                var responseMessage = await httpClient.GetAsync(attachment.ContentUrl);
                var contentLenghtBytes = responseMessage.Content.Headers.ContentLength;
                // You could not use this response message to fetch the image for further processing.
                if (responseMessage.StatusCode == System.Net.HttpStatusCode.OK)
                {
                    Stream attachmentStream = await responseMessage.Content.ReadAsStreamAsync();
                    attachmentStream.Position = 0;
                    var image = System.Drawing.Image.FromStream(attachmentStream);
                    image.Save(@"ImageFromUser.png");
                    await turnContext.SendActivityAsync($"Attachment of {attachment.ContentType} type and size of {contentLenghtBytes} bytes received.");
                }
            }
        }

Thank you @Subhasish-msft for your great explanation and the code sample!
I completely forgot about the MicrosoftAppCredentials.
Now it works as intended :)