Pods remain idle and fail to join Kafka consumer group during scaling and rebalancing

In what area(s)?

/area runtime
What version of Dapr?
1.14.4
Expected Behavior
When using KEDA to automatically scale a Dapr app on a Kubernetes cluster to subscribe to Apache Kafka, all pods - both new and existing - should join the Kafka consumer group during scaling and rebalancing, and actively participate in consuming messages.
Actual Behavior
When scaling up the deployment, both new and existing pods sometimes fail to join or rejoin the Kafka consumer group during rebalancing. As a result, some pods (e.g., 2 out of 5) remain idle and do not process any messages. There are no error messages or warnings in the daprd sidecar logs to explain this behavior.
NAME                                   READY   STATUS    RESTARTS   AGE    IP              NODE  NOMINATED NODE   READINESS GATES
consumer-7669445bff-6gvgf              2/2     Running   0          54m    <IP_ADDRESS>    w01   <none>           <none>
consumer-7669445bff-7r7bg              2/2     Running   0          54m    <IP_ADDRESS>    w02   <none>           <none>
consumer-7669445bff-9rmsk              2/2     Running   0          54m    <IP_ADDRESS>    w01   <none>           <none>
consumer-7669445bff-nj8gg              2/2     Running   0          54m    <IP_ADDRESS>   w03   <none>           <none>
consumer-7669445bff-rhqkb              2/2     Running   0          54m    <IP_ADDRESS>   w03   <none>           <none>

~$ kubectl -n kafka exec -it kafka-0 -- ./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group consumer --describe
GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                 HOST            CLIENT-ID
consumer        tests1          1          810094          815770          5676            sarama-3d8c2edf-28c9-4f6f-86c7-574b7057b984 /<IP_ADDRESS>   sarama
consumer        tests1          0          15499017        15499017        0               sarama-3d8c2edf-28c9-4f6f-86c7-574b7057b984 /<IP_ADDRESS>   sarama
consumer        tests1          3          810583          816315          5732            sarama-3d8c2edf-28c9-4f6f-86c7-574b7057b984 /<IP_ADDRESS>   sarama
consumer        tests1          2          815650          815650          0               sarama-3d8c2edf-28c9-4f6f-86c7-574b7057b984 /<IP_ADDRESS>   sarama
consumer        tests1          7          814626          816542          1916            sarama-9050d6cf-e8d3-4d27-b6f0-387aad75f999 /<IP_ADDRESS>   sarama
consumer        tests1          9          815537          815537          0               sarama-9050d6cf-e8d3-4d27-b6f0-387aad75f999 /<IP_ADDRESS>   sarama
consumer        tests1          8          812852          814762          1910            sarama-9050d6cf-e8d3-4d27-b6f0-387aad75f999 /<IP_ADDRESS>   sarama
consumer        tests1          5          816473          816473          0               sarama-4bdbe62b-b48b-47f6-bf78-8e21212f3855 /<IP_ADDRESS>  sarama
consumer        tests1          4          815806          815807          1               sarama-4bdbe62b-b48b-47f6-bf78-8e21212f3855 /<IP_ADDRESS>  sarama
consumer        tests1          6          815878          815878          0               sarama-4bdbe62b-b48b-47f6-bf78-8e21212f3855 /<IP_ADDRESS>  sarama

Steps to Reproduce the Problem

Set up a Kubernetes cluster running Dapr, with Kafka 3.8 managed by Strimzi.
Deploy a Dapr-enabled Kafka subscriber, using the following configurations:

Dapr Kafka Component:
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: kafka-pubsub
  namespace: pipeline
spec:
  type: pubsub.kafka
  version: v1
  metadata:
  - name: brokers
    value: "kafka-bootstrap.kafka:9092"
  - name: authType
    value: "none"
  - name: initialOffset
    value: "oldest"
  - name: consumeRetryEnabled
    value: true
  - name: maxRetries
    value: "10"
  - name: consumeRetryInterval
    value: "100ms"

Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: consumer
  namespace: pipeline
  labels:
    app: consumer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: consumer
  template:
    metadata:
      labels:
        app: consumer
      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "consumer"
        dapr.io/app-port: "50051"
        dapr.io/app-protocol: "grpc"
        dapr.io/graceful-shutdown-seconds: "15"
    spec:
      containers:
      - name: consumer
        image: <image_path>
        imagePullPolicy: Always

Configure KEDA to scale the consumer deployment based on Kafka pub/sub lag.
KEDA ScaledObject:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: consumer
  namespace: pipeline
spec:
  scaleTargetRef:
    name: consumer
  pollingInterval: 15
  minReplicaCount: 1
  maxReplicaCount: 5
  cooldownPeriod: 300
  triggers:
  - type: kafka
    metadata:
      topic: tweets
      bootstrapServers: kafka-bootstrap.kafka:9092
      consumerGroup: consumer
      lagThreshold: "60"

Observe that during scaling and rebalancing, some pods (e.g., 2 out of 5) fail to join or rejoin the Kafka consumer group and remain idle without processing any messages. The daprd sidecar logs contain no errors related to this issue.

Release Note
RELEASE NOTE: FIX Issue with Kafka consumers remaining idle and not joining the consumer group on scaling and rebalancing.

@anneum What happens if you scale down your deployment to 0 and then back to 5? does it successfully rebalance across all the members?

@olitomlinson I have tried several times and the frustrating answer is: sometimes. Half of the attempts to scale from 0 to 5 worked, the other half only had 4 out of 5 pods in the consumer group. In the case where scaling from 0 to 5 did not work, I received the following log messages:
time="2024-10-29T11:59:35.111841369Z" level=info msg="application protocol: grpc. waiting on port 50051.  This will block until the app is listening on that port." app_id=consumer instance=consumer-7669445bff-fhcxn scope=dapr.runtime type=log ver=1.14.4
time="2024-10-29T11:59:37.826963052Z" level=info msg="application discovered on port 50051" app_id=consumer instance=consumer-7669445bff-fhcxn scope=dapr.runtime type=log ver=1.14.4
time="2024-10-29T11:59:37.827016221Z" level=info msg="actors: state store is not configured - this is okay for clients but services with hosted actors will fail to initialize!" app_id=consumer instance=consumer-7669445bff-fhcxn scope=dapr.runtime type=log ver=1.14.4
time="2024-10-29T11:59:37.82709866Z" level=info msg="Configuring actors placement provider 'placement'. Configuration: 'placement:dapr-placement-server.dapr-system.svc.cluster.local:50005'" app_id=consumer instance=consumer-7669445bff-fhcxn scope=dapr.runtime.actor type=log ver=1.14.4
time="2024-10-29T11:59:37.827146029Z" level=info msg="Configuring actor reminders provider 'default'. Configuration: ''" app_id=consumer instance=consumer-7669445bff-fhcxn scope=dapr.runtime.actor type=log ver=1.14.4
time="2024-10-29T11:59:37.827224098Z" level=info msg="Actor runtime started. Idle timeout: 1h0m0s" app_id=consumer instance=consumer-7669445bff-fhcxn scope=dapr.runtime.actor type=log ver=1.14.4
time="2024-10-29T11:59:37.827258328Z" level=info msg="Configuring workflow engine with actors backend" app_id=consumer instance=consumer-7669445bff-fhcxn scope=dapr.runtime type=log ver=1.14.4
time="2024-10-29T11:59:37.827273917Z" level=info msg="Registering component for dapr workflow engine..." app_id=consumer instance=consumer-7669445bff-fhcxn scope=dapr.runtime type=log ver=1.14.4
time="2024-10-29T11:59:37.827302607Z" level=info msg="Initializing Dapr workflow component" app_id=consumer component="dapr (workflow.dapr/v1)" instance=consumer-7669445bff-fhcxn scope=dapr.contrib type=log ver=1.14.4
time="2024-10-29T11:59:37.827338996Z" level=info msg="Workflow engine initialized." app_id=consumer instance=consumer-7669445bff-fhcxn scope=dapr.runtime type=log ver=1.14.4
time="2024-10-29T11:59:37.827532554Z" level=warning msg="Error processing operation DaprBuiltInInitializationRetries. Retrying in 528.2072ms…" app_id=consumer instance=consumer-7669445bff-fhcxn scope=dapr.runtime type=log ver=1.14.4
time="2024-10-29T11:59:37.827612213Z" level=info msg="Scheduler stream connected" app_id=consumer instance=consumer-7669445bff-fhcxn scope=dapr.runtime.scheduler type=log ver=1.14.4
time="2024-10-29T11:59:37.847052381Z" level=info msg="Placement tables updated, version: 0" app_id=consumer instance=consumer-7669445bff-fhcxn scope=dapr.runtime.actors.placement type=log ver=1.14.4
time="2024-10-29T11:59:40.069377186Z" level=error msg="error getting topic list from app: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp <IP_ADDRESS>:50051: connect: connection refused\"" app_id=consumer instance=consumer-7669445bff-fhcxn scope=dapr.runtime.processor.subscription type=log ver=1.14.4
time="2024-10-29T11:59:40.069441025Z" level=warning msg="failed to subscribe to topics: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp <IP_ADDRESS>:50051: connect: connection refused\" " app_id=consumer instance=consumer-7669445bff-fhcxn scope=dapr.runtime type=log ver=1.14.4
time="2024-10-29T11:59:40.069472254Z" level=info msg="dapr initialized. Status: Running. Init Elapsed 4975ms" app_id=consumer instance=consumer-7669445bff-fhcxn scope=dapr.runtime type=log ver=1.14.4

When I downscale from 5 to 1, the consumer group has no members. No log messages appear in the app or in the sidecar. Even if I increase the number of pods again, the "surviving" pod will still not become part of the consumer group. The last log messages from "surviving" daprd sidecar are those from the initial startup:
time="2024-10-29T12:16:25.108406325Z" level=info msg="Workflow engine initialized." app_id=consumer instance=consumer-7669445bff-xqcmx scope=dapr.runtime type=log ver=1.14.4
time="2024-10-29T12:16:25.108579521Z" level=warning msg="Error processing operation DaprBuiltInInitializationRetries. Retrying in 748.544521ms…" app_id=consumer instance=consumer-7669445bff-xqcmx scope=dapr.runtime type=log ver=1.14.4
time="2024-10-29T12:16:25.108664839Z" level=info msg="Scheduler stream connected" app_id=consumer instance=consumer-7669445bff-xqcmx scope=dapr.runtime.scheduler type=log ver=1.14.4
time="2024-10-29T12:16:25.125054735Z" level=info msg="Placement tables updated, version: 0" app_id=consumer instance=consumer-7669445bff-xqcmx scope=dapr.runtime.actors.placement type=log ver=1.14.4
time="2024-10-29T12:16:25.860068056Z" level=info msg="Recovered processing operation DaprBuiltInInitializationRetries after 2 attempts" app_id=consumer instance=consumer-7669445bff-xqcmx scope=dapr.runtime type=log ver=1.14.4
time="2024-10-29T12:16:25.860126424Z" level=info msg="app is subscribed to the following topics: [[test1]] through pubsub=kafka-pubsub" app_id=consumer instance=consumer-7669445bff-xqcmx scope=dapr.runtime.processor.subscription type=log ver=1.14.4
time="2024-10-29T12:16:25.860271651Z" level=info msg="dapr initialized. Status: Running. Init Elapsed 3290ms" app_id=consumer instance=consumer-7669445bff-xqcmx scope=dapr.runtime type=log ver=1.14.4

Thanks for the info!
The internal Kafka client that Dapr uses is over 6m old now and there has been a few of stability fixes in latest patches.
@yaron2 would it be possible to get a dependency bump, then OP could try an 'edge' build?

@olitomlinson Thank you.
During testing another strange behaviour has occurred. It seems that one partition is always ignored by Dapr. So overnight, with processing by only one dapr replica, some messages have accumulated in a partition that seems to be ignored. To rule out the possibility that this was due to my Kafka instance, I tested it with a consumer group consisting of a manually written consumer based on Kafka Python and two Dapr instances. Again, the result is that only n-1 partitions are processed by the Dapr instances.
I hope the dependency bump will help.
Overnight result with one Dapr instance:
GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                 HOST            CLIENT-ID
consumer        test1          5          829876          829876          0               sarama-0027d103-f909-4c23-b7cd-db75644e557c /<IP_ADDRESS>  sarama
consumer        test1          4          829094          829094          0               sarama-0027d103-f909-4c23-b7cd-db75644e557c /<IP_ADDRESS>  sarama
consumer        test1          7          830084          830084          0               sarama-0027d103-f909-4c23-b7cd-db75644e557c /<IP_ADDRESS>  sarama
consumer        test1          6          829396          829397          1               sarama-0027d103-f909-4c23-b7cd-db75644e557c /<IP_ADDRESS>  sarama
consumer        test1          9          828457          828457          0               sarama-0027d103-f909-4c23-b7cd-db75644e557c /<IP_ADDRESS>  sarama
consumer        test1          8          827927          827927          0               sarama-0027d103-f909-4c23-b7cd-db75644e557c /<IP_ADDRESS>  sarama
consumer        test1          1          817827          829246          11419           sarama-0027d103-f909-4c23-b7cd-db75644e557c /<IP_ADDRESS>  sarama
consumer        test1          0          15512348        15512348        0               sarama-0027d103-f909-4c23-b7cd-db75644e557c /<IP_ADDRESS>  sarama
consumer        test1          3          829514          829514          0               sarama-0027d103-f909-4c23-b7cd-db75644e557c /<IP_ADDRESS>  sarama
consumer        test1          2          828829          828829          0               sarama-0027d103-f909-4c23-b7cd-db75644e557c /<IP_ADDRESS>  sarama

Test result with two dapr instances and a manually written consumer:
GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                             HOST            CLIENT-ID
consumer        test1          1          829485          829485          0               kafka-python-2.0.2-3d652fb3-6bc5-4857-ace3-e860f9398c58 /<IP_ADDRESS>  kafka-python-2.0.2
consumer        test1          0          15512571        15512571        0               kafka-python-2.0.2-3d652fb3-6bc5-4857-ace3-e860f9398c58 /<IP_ADDRESS>  kafka-python-2.0.2
consumer        test1          3          829750          829750          0               kafka-python-2.0.2-3d652fb3-6bc5-4857-ace3-e860f9398c58 /<IP_ADDRESS>  kafka-python-2.0.2
consumer        test1          2          829073          829073          0               kafka-python-2.0.2-3d652fb3-6bc5-4857-ace3-e860f9398c58 /<IP_ADDRESS>  kafka-python-2.0.2
consumer        test1          5          830113          830113          0               sarama-f2a20fe2-c69a-484f-97dc-83a4d68c5fae             /<IP_ADDRESS>   sarama
consumer        test1          4          829317          829317          0               sarama-f2a20fe2-c69a-484f-97dc-83a4d68c5fae             /<IP_ADDRESS>   sarama
consumer        test1          6          829514          829634          120             sarama-f2a20fe2-c69a-484f-97dc-83a4d68c5fae             /<IP_ADDRESS>   sarama
consumer        test1          7          830311          830311          0               sarama-faea2adf-9165-49b4-a0f4-5b79b7d298f8             /<IP_ADDRESS>  sarama
consumer        test1          9          828569          828672          103             sarama-faea2adf-9165-49b4-a0f4-5b79b7d298f8             /<IP_ADDRESS>  sarama
consumer        test1          8          828170          828170          0               sarama-faea2adf-9165-49b4-a0f4-5b79b7d298f8             /<IP_ADDRESS>  sarama

I'll do a dependency bump

xref: https://github.com/dapr/components-contrib/pull/3587

Dependency updated. As a next step we'll update dapr/dapr so an edge build could be produced for testing


Update GitHub Actions

Code Coverage
Linting

We need to experiment with different lint types to see which one can co-exist with existing dependencies.
ES Lint requires separate peer dev dependencies which conflict with existing versions.


Compiler doesn't see required assignment in try..else structure.

The sample https://playground.ponylang.io/?gist=8b8975e6007b92a892779ac8f6a838ac doesn't compile, even though _tcp is being assigned in both branches.
Perhaps there are additional/hidden branches that I am unaware of.

Here is a more minimal example, exhibiting the same bug.
actor Main
  var _s: (String | None)

  new create(env: Env) =>
    try
      _s = "".usize()?.string()
    else
      _s = None
    end

As I am exploring the reason for this, there is more cases exhibiting the same behaviour.
while loops:
actor Main
  var _s: (String | None)

  new create(env: Env) =>
    var i: USize = 0
    while i < 5 do
       _s = i.string()
       i = i + 1
    else
      _s = ""
    end

The same is true for for loops as they desugar to while loops.
The problem with assignment in loops is that they might fail at runtime doing multiple assignments for let fields. I think we should forbid this.

There's no let in your example @mfelsche. You can't do multiple assignments to a let. That would be an error. I don't see anything wrong with the example that exists. I think that should be valid code. There's no way for that to error out.

I never wrote there is a let in the above example.
In the following example there is a let, but it errors as it thinks _s is undefined.
The example above should compile fine. I never wrote it shouldn't.
The one below should result in a different error, possibly doing multiple assignments, which i don't think is checked right now. This is what we should forbid.
actor Main
  let _s: (String | None)

  new create(env: Env) =>
    var i: USize = 0
    while i < 5 do
       _s = i.string()
       i = i + 1
    else
      _s = ""
    end

Sorry, I was wrong. We actually do check for assignments to let fields/variables in loops.

refer_while, refer_try, refer_repeat must be doing something different with  ast_consolidate_branches than refer_if as if works.

This change to refer_while fixes the issue despite some code that looks like it was intentionally trying to create the issue. Note that with this change, all the tests pass so, I think this would be fine for a PR. I'll open it later as it builds on another PR I did and I don't think they are connected at all, but I'm not positive of that and it is easier to redo this 1 method here once the other is merged and verify.
static bool refer_while(pass_opt_t* opt, ast_t* ast)
{
  pony_assert(ast_id(ast) == TK_WHILE);
  AST_GET_CHILDREN(ast, cond, body, else_clause);

  // All consumes have to be in scope when the loop body finishes.
  errorframe_t errorf = NULL;
  if(!ast_all_consumes_in_scope(body, body, &errorf))
  {
    errorframe_report(&errorf, opt->check.errors);
    return false;
  }

  size_t branch_count = 0;
  if(!ast_checkflag(else_clause, AST_FLAG_JUMPS_AWAY))
  {
    branch_count++;
    ast_inheritbranch(ast, body);

    ast_consolidate_branches(ast, branch_count);
  }

  // If all branches jump away with no value, then we do too.
  if(branch_count == 0 && ast_checkflag(body, AST_FLAG_JUMPS_AWAY))
    ast_setflag(ast, AST_FLAG_JUMPS_AWAY);

  // Push our symbol status to our parent scope.
  ast_inheritstatus(ast_parent(ast), ast);

  return true;
}

The refer_while code above is incorrect. I have a fix on a branch and it looks like tests are passing.

repeat is ok.
try has something weird going on where I am getting assertions for a symbol table being missing, so that requires more investigation.

So the issue with try appears to be that unlike if and while etc, if doesn't introduce a scope and without it, there's no symbol table. Not sure why that is.

Ok I figured out how to fix this and I think in the process I understand why try isn't a scope. I need to put more tests together and then I can open a PR.


Remove custom x-delivery-count header

Removing x-delivery-count as its no longer used in sdx

Codecov Report

Merging #23 into master will decrease coverage by 1.19%.
The diff coverage is 100%.

@@          Coverage Diff           @@
##           master   #23     +/-   ##
======================================
- Coverage    77.2%   76%   -1.2%
======================================
  Files           7     7
  Lines         443   421     -22
  Branches       14    14
======================================
- Hits          342   320     -22
  Misses         98    98
  Partials        3     3

Impacted Files
Coverage Δ

sdc/rabbit/consumers.py
47.89% <ø> (-2.61%)
:arrow_down:

sdc/rabbit/test/test_consumer.py
97.7% <100%> (-0.28%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8622bb9...a87bdd5. Read the comment docs.


translate /help/ops/missing-metrics

/lgtm

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: rootsongjc
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

content_zh/OWNERS [rootsongjc]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


False positive springframework

Jar that was found:
spring-core-3.2.13.RELEASE.jar	cpe:/a:springsource:spring_framework:3.2.13	org.springframework:spring-core:3.2.13.RELEASE
CVE that was found:
CVE-2014-1904  suppress
Severity: Medium
CVSS Score: 4.3
CWE: CWE-79 Improper Neutralization of Input During Web Page Generation ('Cross-site Scripting')
Cross-site scripting (XSS) vulnerability in web/servlet/tags/form/FormTag.java in Spring MVC in Spring Framework 3.0.0 before 3.2.8 and 4.0.0 before 4.0.2 allows remote attackers to inject arbitrary web script or HTML via the requested URI in a default action.
BUGTRAQ - 20140311 CVE-2014-1904 XSS when using Spring MVC
CONFIRM - http://docs.spring.io/spring/docs/3.2.8.RELEASE/changelog.txt
CONFIRM - http://www.gopivotal.com/security/cve-2014-1904
CONFIRM - https://github.com/spring-projects/spring-framework/commit/741b4b229ae032bd17175b46f98673ce0bd2d485
CONFIRM - https://jira.springsource.org/browse/SPR-11426
REDHAT - RHSA-2014:0400
SECUNIA - 57915
Vulnerable Software & Versions: (show less)
cpe:/a:springsource:spring_framework:3.0.0
cpe:/a:springsource:spring_framework:3.0.0:m1
cpe:/a:springsource:spring_framework:3.0.0:m2
cpe:/a:springsource:spring_framework:3.0.0:m3
cpe:/a:springsource:spring_framework:3.0.0:m4
cpe:/a:springsource:spring_framework:3.0.0:rc1
cpe:/a:springsource:spring_framework:3.0.0:rc2
cpe:/a:springsource:spring_framework:3.0.0:rc3
cpe:/a:springsource:spring_framework:3.0.1
cpe:/a:springsource:spring_framework:3.0.2
cpe:/a:springsource:spring_framework:3.0.3
cpe:/a:springsource:spring_framework:3.0.4
cpe:/a:springsource:spring_framework:3.0.5
cpe:/a:springsource:spring_framework:3.0.6
cpe:/a:springsource:spring_framework:3.0.7
cpe:/a:springsource:spring_framework:3.1.0
cpe:/a:springsource:spring_framework:3.1.1
cpe:/a:springsource:spring_framework:3.1.2
cpe:/a:springsource:spring_framework:3.1.3
cpe:/a:springsource:spring_framework:3.1.4
cpe:/a:springsource:spring_framework:3.2.0
cpe:/a:springsource:spring_framework:3.2.1
cpe:/a:springsource:spring_framework:3.2.2
cpe:/a:springsource:spring_framework:3.2.3
cpe:/a:springsource:spring_framework:3.2.4
cpe:/a:springsource:spring_framework:3.2.5
cpe:/a:springsource:spring_framework:3.2.6
cpe:/a:springsource:spring_framework:3.2.7 and all previous versions
cpe:/a:springsource:spring_framework:4.0.0:m1
cpe:/a:springsource:spring_framework:4.0.0:m2
cpe:/a:springsource:spring_framework:4.0.0:rc1
cpe:/a:springsource:spring_framework:4.0.1 and all previous versions

This is a known bug.  Version 3.2.13 is being flagged because the CVE contains the vulnerable software entry of:
cpe:/a:springsource:spring_framework:4.0.1 and all previous versions

This issue is being tracked as issue #180. This will be resolved in the next release.
Best Regards,
Jeremy


Cannot train an image classification model

System information

OS version/distro: Windows 10
.NET Version (eg., dotnet --info): .NET Core 3.1

Issue

What did you do? Visual Studio 'predict image' test dataset training and evalutation.
What happened? An error occurs while training vice the training cannot continue. ArgumentOutOfRangeException: Can't determine the number of source columns without valid data (Parameter 'Source')
What did you expect? The training to complete and able to test the model.

Source code / logs
Please paste or attach the code or logs or traces that would be helpful to diagnose the issue you are reporting.
be0d23b6-f85a-4e16-80f9-66eb241d55a6.txt
(i'm unsure if this is an issue related to ml.net or visual studio, or if this has been already reported).

sorry for hitting this issue, Thanks for sharing the log file! we will investigate it.

Before the training, I have prepared a dataset with 2 folders each containing different images with different dimensions (same file type), i have then fed the main folder into the file upload and hit train.

This error occurred before training seemed to begin, I can record all the steps I've encountered and logs before the error once I'm home.

Whole training process - https://speedy.is-a-qt.dev/media/1603128095588/XppxcZsRIU.mp4
Also if you could guide me towards the builder version, that'd be great.

@SpeedyCraftah,  thanks for sharing the recording! we will take look into it.

validated


Breakpoints are hidden by coverage indicator

Maybe there is way to provide a gutter decoration that is compatible with breakpoints?

Last I checked there wasnt a gutter decoration that played nice with breakpoints sadly ☹ . Another user had similar issues and I got him a fix in version 1.1.2. That fix should allow you to use line coverage and breakpoints https://github.com/ryanluker/vscode-coverage-gutters/issues/87 . Let me know if this fixes your issue.

@PatrickRoumanoff going to close this issue unless the gutter icon api changed recently (the workaround I mentioned should still work in the meantime). I will keep this issue in mind though as vscode releases more versions!


Auth - overriding LogicalId of UserPool don't change all reference to the ressource.

Before opening, please confirm:

[X] I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
[X] I have searched for duplicate or closed issues.
[X] I have read the guide for submitting bug reports.
[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

How did you install the Amplify CLI?
npm
If applicable, what version of Node.js are you using?
14.x
Amplify CLI Version
7.6.13
What operating system are you using?
mac
Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.
several month ago, the cloudformation stack was update manually.
the logical ID of the userPool was replace by "UserPool1" the original value was "UserPool"
Amplify Categories
auth
Amplify Commands
push
Describe the bug
Hi,
After upgrading from cli version 6 to 7, amplify cli update the auth stack.
During the migration, amplify replace the logical id of my userpool. (it replace UserPool1 by UserPool)
-> Cloudformation will replace my cognito user pool 😱
I try to override the logicalId of my userPool.
In the overrides.ts file, I can call overrideLogicalId function :
resources.userPool.overrideLogicalId('MyNewLogicalId')
The logical Id of the userPool is change in the cloudformation template but references to the old logical id aren't update.
ex: resources.userPoolClient.userPoolId refer to the old value
Is there a reason to keep reference to the old logical id ?
Is there an easy way to update the logicalId of the userPool ?
Expected behavior
overrideLogicalId should change the logical id of the ressource and update all references
Reproduction steps

create a new project.
add auth
override the auth and add : resources.userPool.overrideLogicalId('MyNewLogicalId')
push the project
push with failed in the cf  template, you will find reference to the old userpool logical id

GraphQL schema(s)
No response
Log output
No response
Additional information
No response

What is the use case for changing the user pool logical id?

Closing due to inactivity. @kouz75 if you are still experiencing this issue please reply back to this thread and we can re-open to investigate further 🙂


Javascript version

It would be interesting to see if a javascript version of this library could be created using SIMD.js.
https://developer.mozilla.org/nl/docs/Web/JavaScript/Reference/Global_Objects/SIMD
https://github.com/tc39/ecmascript_simd
https://github.com/jonbrennecke/node-simd

Yes, this looks possible now.

The Emscripten compilation of this lib also look promising, but I don't know what a dynamic call from javascript would look like. Emscripten generates a lot of constants and heap allocations.

@wshager
I am somewhat pessimistic regarding any approach that does not translate into SIMD instructions at the machine level.

Continuing from #10, the performance of the first example compiled to javascript with Emscripten performs well, but performance of other 2 tests is extremely poor. It seems SIMD.js is nowhere near being compatible with this project.

@wshager
Can you tell us which examples fare well and which ones fare badly?

When I compiled previously with my smmintrin shim the first two test would run, but simple_demo would hang, currently the second example didn't finish in any acceptable time, only the first compress_decompress example finishes in ff nightly.

What happens after this last commit https://github.com/lemire/simdcomp/commit/b7e1ecca96ebb5290c52e3738fb844e2436ffa74
?

@lemire yes, that script does finish, in about 8 seconds in FF nightly on my i7-3612QM (8GB).

Ok. Can you share the resulting output (just so we can document it).

@lemire sorry, I did, but I replied to the commit. Here goes:
http://lagua.nl/uploadfiles/example.html
http://lagua.nl/uploadfiles/example.uncompressed.html

example.js.txt
example.uncompressed.js.txt

@wshager
Thanks, but I am inviting you to publish the performance results you got.
My concern right now is whether SIMD instructions are used at all. On my laptop, the script completes, but very slowly and with speeds that indicate that SIMD instructions are not used.

_compress_decompress_demo: 10-50 ms
_varying_bit_width_demo: 0-10 ms
_simple_demo: 4600-7400 ms

@wshager  Do you have the screen output where it compares the memcpy speed with the decoding speed?
On my Firefox, things just freeze, so it is no use. On Google Chrome, I do not even get 1 million int. per second.

Ah sorry I misunderstood. Here it is:
== simple test
Code works!
== varying bit-width demo
encoded size: 242 (original size: 1024)
Code works!
== simple demo
gap = 1
compression ratio = 30.117647
decoding speed in million of integers per second 1.159420
memcpy speed in million of integers per second 320.000000
ignore me -589677036
All tests are in CPU cache. Avoid out-of-cache decoding in applications.
gap = 3
compression ratio = 15.515152
decoding speed in million of integers per second 1.154193
memcpy speed in million of integers per second 640.000000
ignore me -1852397360
All tests are in CPU cache. Avoid out-of-cache decoding in applications.
gap = 9
compression ratio = 7.876923
decoding speed in million of integers per second 1.176471
memcpy speed in million of integers per second 640.000000
ignore me -1604405666
All tests are in CPU cache. Avoid out-of-cache decoding in applications.
gap = 27
compression ratio = 6.320988
decoding speed in million of integers per second 1.118881
memcpy speed in million of integers per second 640.000000
ignore me -848559680
All tests are in CPU cache. Avoid out-of-cache decoding in applications.
gap = 81
compression ratio = 4.530973
decoding speed in million of integers per second 1.072925
memcpy speed in million of integers per second 213.333333
ignore me -893214422
All tests are in CPU cache. Avoid out-of-cache decoding in applications.
gap = 243
compression ratio = 3.968992
decoding speed in million of integers per second 1.076535
memcpy speed in million of integers per second 640.000000
ignore me -1238753396
All tests are in CPU cache. Avoid out-of-cache decoding in applications.

@lemire did you install nightly? https://nightly.mozilla.org

@wshager
These results are better than what I get with Google Chrome but still not close to what one would want. These results are thousands of times slower than C. That's not surprising given that it is JavaScript... but still...

@lemire agreed, too bad. Perhaps the generated js isn't optimal.

@wshager  Yes, manual inspection of the generated code might be telling.

@lemire the abstractions in SIMD.js may be clearer, but the downside is that there's no way to convert the code back to pure functions... unless I take on the quixotic task of rewriting SIMD.js to pure functions.

Currently the function mapping is as follows:
var SIMD_Int8x16_add=SIMD_Int8x16.add;
var SIMD_Int8x16_sub=SIMD_Int8x16.sub;
var SIMD_Int8x16_neg=SIMD_Int8x16.neg;
var SIMD_Int8x16_mul=SIMD_Int8x16.mul;
var SIMD_Int8x16_equal=SIMD_Int8x16.equal;
var SIMD_Int8x16_lessThan=SIMD_Int8x16.lessThan;
var SIMD_Int8x16_greaterThan=SIMD_Int8x16.greaterThan;
etc.
Anyway, the generated code is quite unreadable, because each transformation is assigned to a variable (which may also impede performance).

No matter. I think it is useful and important to experiment, even when the results are underwhelming.

I've created an interface from (client-side) javascript to the simdcomp library using Emscripten. What I fail to understand is how to decompress a Uint8Array.
var maxbits_length = Module.cwrap('maxbits_length', 'number', ['number','number']);
var simdpack_length = Module.cwrap('simdpack_length', 'number', ['number','number','number','number']);
var simdunpack_length = Module.cwrap('simdunpack_length', 'number', ['number','number','number','number']);

var n = 128;

var data = new Uint32Array(128);

for (var k = 0; k < n; k++){
    data[k] = k+3;
}
//var n = data.length;
// Get data byte size, allocate memory on Emscripten heap, and get pointer
var nDataBytes = n * data.BYTES_PER_ELEMENT;
var dataPtr = Module._malloc(nDataBytes);

// Copy data to Emscripten heap
var dataHeap = new Uint8Array(Module.HEAPU8.buffer, dataPtr, nDataBytes);
dataHeap.set( new Uint8Array(data.buffer) );

// Call the function by passing a number pointing to the byte location of
// the array of pointers on the Emscripten heap.  Emscripten knows what to do!
var b = maxbits_length(dataHeap.byteOffset, n);

var nOutBytes = nDataBytes + n / 128;
var outPtr = Module._malloc(nOutBytes);

simdpack_length(dataHeap.byteOffset, n ,outPtr,b);

var outHeap = new Uint8Array(Module.HEAPU8.buffer, outPtr, nOutBytes);

//var out = new Uint8Array(outHeap.buffer, outHeap.byteOffset, n + n);

var backPtr = Module._malloc(nDataBytes);

simdunpack_length(outHeap.byteOffset, n , backPtr, b);

var backHeap = new Uint8Array(Module.HEAPU8.buffer, backPtr, nDataBytes);

var back = new Uint32Array(backHeap.buffer, backHeap.byteOffset, n);

console.log(back);

// Free memory
Module._free(dataHeap.byteOffset);
Module._free(outHeap.byteOffset);
Module._free(backHeap.byteOffset);

@wshager  I am not very familiar with Emscripten.
Do you know how to achieve what you are trying to achieve in C? If you do then, presumably, it is simply a matter of translating the appropriate C code.

@lemire following the updated example code, the serialized compressed array now successfully decompresses. What I was wondering is that in client-side javascript, providing a license is somewhat complicated, as the code is compressed. How would you like me to ameliorate this?

@wshager
This C code falls under the BSD License. If you want to repackage it as JavaScript and redistribute it, you can use BSD as well.

@wshager
Though it is not a port of this library per se, the following JavaScript library might be of interest...
https://github.com/lemire/FastIntegerCompression.js

@lemire great! I'll check it out, it will no doubt be very useful until we have full SIMD support.

example.uncompressed.js.txt

3 years later and all of the examples return "speed in million of integers per second 0.184491" or less in Chrome on my system!
I added the following code inside the _simple_demo function at line 7536. Any idea why _simdpackwithoutmask returns undefined?
var data = [29,70,8,3,4,1,22,3,4,5,2,1,5,1,1,7,1,2,1,1,2,3,1,4,8,1,10,2,1,3,4,44,54,1,3,3,3,2,6,3,2,1,1,1,8,1,2,4,11,1,2,9,3,4,5,1,6,6,6,2,10,55,35,1,8,2,2,3,4,1,1,3,4,5,7,3,1,2,4,2,2,5,2,9,1,2,1,3,3,9,0,44,61,3,4,11,1,1,3,3,2,5,5,1,5,2,1,2,4,4,1,3,5,1,4,4,4,5,1,9,51,32,19,1,3,5,1,6,1,1,2,1,8,11,6,2,3,2,3,1,3,2,3,3,3,3,6,6,6,3];
console.log(_simdpackwithoutmask(data, new ArrayBuffer(8), _maxbits(data)));


Split train / test before train and use test in evaluate

At the moment train / test splits during train and as such we do the same split when we evaluate. Instead we should split before and pass the train data to train and test_data to evaluate.

Duplicate of #115


Pressing shift-tab inserts letter Z, instead of cycling completions

It turns out that this issue is a trivial one-line fix.  I'll push a fix soonish.


Option to transpile cjs to es5 with traceur

Now that nodejs v4 is out nodejs developers are combining es6 with commonjs modules.
The option cjs-to-es5 let traceur transpiles that code into es5.
This is useful for:

minification as uglify-js only supports es5
supporting other browsers than Chrome.

Thanks for posting. SystemJS will be moving towards https://github.com/systemjs/systemjs/issues/579.
Unfortunately a patch like this will fail when using transpilers. Also we need to make sure that build features always match SystemJS browser features, so can't create a difference here either.

@guybedford thanks for the review.
So what is the best way to proceed until https://github.com/systemjs/systemjs/issues/579 lands?

@guybedford "this will fail when using transpilers".
I am not sure I understand that and that is probably good news; maybe I am missing the obvious. When I run jspm bundle what should I also do to transpile commonjs-require with es6 into es5 code?

I mean that a lot of users use Babel for compilation instead of Traceur, while this approach only uses Traceur for the transpiling. For now I would suggest running this as a custom precompilation step, perhaps converting app to app-compiled or something like that.

Thanks for your patient explanation.


Horizontal line stays after you unsubscribed from pinned channel

If you have only one pinned channel and you unsubscribe from it,
 stays and dissappears after refreshing the page.

Hm, I can't seem to replicate this on master. I added two pinned streams, removed them both, and there was no stale
.

@brockwhittaker  it is very important to unsubscribe from them clicking down arrow and choose option "Unsubscribe"

Ah, I misinterpreted that. I see. @zulipbot claim.


Support for automatically publishing the release

The tool should be able to publish the release if it hasn't been published yet.
This should also support scheduling the release.

Is there an API for this? happy to have a crack at this one


converting to triton model

Hi, Thanks to this repository, I made the onnx model what I trained and I curious about the model is on cpu or gpu
Now I'm trying to convert espnet onnx model to triton server model
How can I do this? Let me some hint..
Thank you

Hi @Macsim2, sorry for the late reply. The converted model is on the CPU.
I am not familiar with Triton server models, so I cannot help you with that part of the conversion process.


Please release Pointer Generator implementation

Hi,
Pointer Generator is one of the most popular baseline in text summarisation studies. It also serves as the building block of several very competitive models. Therefore, if Pointer Summariser can be included into your model zoo, it will greatly encourage the development of Paddle-based summarisation systems:wink:
Many thanks!

Update: The implementation of Pointer Generator is under review now https://github.com/PaddlePaddle/PaddleNLP/pull/235 .

Thanks for your hard work! 辛苦了! ☕

Hi @Pzoom522, feel free to use the most current version here. We are still reconstructing the code to better align with the PaddleNLP framework but the current version is sufficient to reproduce the SOTA results in the source paper.
Let us know if you have any questions.
https://github.com/logan-siyao-peng/PaddleNLP/tree/60246220447f598c7a4f2ebed7a548fc83f8f0e7/examples/text_summarization/pointer_summarizer


Fix/augmentations swap entities

Checklist:

[ ] I've added Google style docstrings to my code.
[ ] I've used pydantic for typing when/where necessary.
[ ] I have linted my code
[ ] I have added tests to cover my changes.

This PR deals with solving issues which we faced in augmenting swap entities for clinical files


[AIRFLOW-3475] Move ImportError out of models.py

Make sure you have checked all steps below.
Jira

[x] My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-3475
In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.

Description

[x] Here are some details about my PR, including screenshots of any UI changes:

Move the ImportError class to a separate file as part of the big models.py refactor (/airflow/models/importerror.py).
Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters (not including Jira issue reference)
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[x] In case of new functionality, my PR adds documentation that describes how to use it.

When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
All the public functions and the classes in the PR contain docstrings that explain what it does

Code Quality

[x] Passes flake8

Codecov Report

Merging #4383 into master will increase coverage by <.01%.
The diff coverage is 83.33%.

@@            Coverage Diff             @@
##           master    #4383      +/-   ##
==========================================
+ Coverage   78.16%   78.17%   +<.01%
==========================================
  Files         204      205       +1
  Lines       16530    16534       +4
==========================================
+ Hits        12921    12925       +4
  Misses       3609     3609

Impacted Files
Coverage Δ

airflow/models/__init__.py
92.79% <ø> (-0.02%)
:arrow_down:

airflow/www/views.py
69.31% <100%> (ø)
:arrow_up:

airflow/models/importerror.py
100% <100%> (ø)

airflow/www_rbac/views.py
72.59% <100%> (ø)
:arrow_up:

airflow/jobs.py
77.41% <33.33%> (ø)
:arrow_up:

airflow/utils/dag_processing.py
59.74% <50%> (+0.07%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0cc078a...74ee999. Read the comment docs.

Good point. I'll rename to errors.py.
@mik-laj There's one other error-class in models.py -> InvalidFernetToken. I was thinking of grouping that into a fernet.py script since there's a few fernet-related functions and classes in models.py. So for now, errors.py will only contain ImportError.


Cache hasRunstime in chunk

Cache this.hasRuntime(), otherChunk.hasRuntime() in variable. Because in some cases these methods call twice.

What kind of change does this PR introduce?
Refactoring

Did you add tests for your changes?
No

Does this PR introduce a breaking change?
No

What needs to be documented once your changes are merged?
No

Thanks


feat: support anthropic function calling

todo:

handle partial streaming
add tests

take a look at https://github.com/jxnl/instructor/pull/485/files

@shreya-51 i added a test that fails

@shreya-51 i added a test that fails

yeah I noticed that this often fails with Lists because its not defined how the LLM should structure them - will add a fix

https://github.com/jxnl/instructor/issues/529


Resolve #1095 - Add flexibility to the start script

Signed-off-by: Ilya Shindyapin<EMAIL_ADDRESS>

Thanks for the PR!
Would #1148 satisfy your use case?
It seems a bit more generic.

We likely won't support this way specifically, but I'm open to making it work with BROWSER env variable we added. Please chime in in https://github.com/facebookincubator/create-react-app/issues/1533 if you're interested.

Seems like this will be fixed by https://github.com/facebookincubator/create-react-app/pull/1590.

Agreed


⚠ Bofrost.be has degraded performance

In 83fafdc, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 2354 ms

Resolved: Bofrost.be performance has improved in a6e23ae.


JetBrains: Automatic repository recognition is not working.

Cody Version
v5.5.7-nightly
IDE Information
IntelliJ IDEA 2023.3.3 (Community Edition)
Build #IC-233.14015.106, built on January 25, 2024
Runtime version: 17.0.9+7-b1087.11 aarch64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.
macOS 14.3
GC: G1 Young Generation, G1 Old Generation
Memory: 2048M
Cores: 8
Metal Rendering is ON
Registry:
ide.experimental.ui=true
Non-Bundled Plugins:
idea.plugin.protoeditor (233.13135.65)
DevKit (233.14015.152)
gherkin (233.11799.165)
cucumber-java (233.11799.196)
PythonCore (233.14015.106)
com.sourcegraph.jetbrains (5.5.7-nightly)
Kotlin: 233.14015.106-IJ
Describe the bug
Platform: MacOS
Steps to reproduce:

Have Cody plugin installed in your JB IntelliJ IDEA editor
Login to Cody with Enterprise user
Open project with enabled Git VCS. This repository must be publicly available on GitHub i.e. https://github.com/sourcegraph/codemod
Open to Cody tool window
Check the Context Selection

Observed behavior:
Repository is not automatically recognised.
Expected behavior
Automatic repository recognition should work and repo should be added automatically.
Additional context
Video link:
https://drive.google.com/file/d/1jNOjPuCUHpGlAj5G_7B57A_lu-Ju6I3O/view?usp=drive_link

@dominiccooney are you the best person to take a look at this?

@jay-fibi , you should perform a chat and see what context is fetched. Default context selection should be working. The test pass would be better if it verified that, too.
If there is an issue, it is to display the implicitly picked repo in the context list. This is tracked in #1322.
@taylorsperry I will work on this later. This is not a P0, it is a cosmetic issue with not displaying the implicitly picked repo. Before the repo autocomplete popup we had a similar problem—the context list always showed "local context" even when implicit detection failed. Now we never show it. So this is not a new issue exactly, just the surface appearance of it changed with the new repo picker.

Thank you, @dominiccooney ! I'm closing in favor of https://github.com/sourcegraph/jetbrains/issues/1322.

This was accidentally reopened, so re-closing, but just noting that I have changed the enterprise repo picker in 70e78539124b94b6964c29ab51c2ca953910d569 to always display the state synced with agent. This means we display the automatically selected repo (if any), you can't remove it, and there's a tooltip explaining what is going on.


Error while creating BIF file. Unable to set process priority.

My Jellyfin logs are filled with the following error for a lot of my content:
[2023-02-21 23:09:01.350 -05:00] [INF] [28] Nick.Plugin.Jellyscrub.Drawing.VideoProcessor: Creating trickplay files at 320 width, for "/mnt/manshare/Jellyfin/Movies/Wreck It Ralph/Wreck-It Ralph (2012) Bluray-1080px264 AAC.mp4" [ID: e6e77404-bc2f-fea6-7025-2d4d940c0211]
[2023-02-21 23:09:01.350 -05:00] [INF] [28] Nick.Plugin.Jellyscrub.Drawing.OldMediaEncoder: /usr/lib/jellyfin-ffmpeg/ffmpeg -loglevel error -init_hw_device vaapi=va:/dev/dri/renderD128 -filter_hw_device va -hwaccel vaapi -hwaccel_output_format vaapi -autorotate 0 -i file:"/mnt/manshare/Jellyfin/Movies/Wreck It Ralph/Wreck-It Ralph (2012) Bluray-1080px264 AAC.mp4" -autoscale 0 -an -sn -vf "fps=1/10,setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale_vaapi=w=320:h=134:format=nv12" -threads 0 -c:v mjpeg_vaapi -f image2 "/var/cache/jellyfin/temp/dbb20ce1c6894ada849a3cfc465ba9bd/img_%08d.jpg"
[2023-02-21 23:09:01.351 -05:00] [INF] [28] Nick.Plugin.Jellyscrub.Drawing.OldMediaEncoder: Setting generation process priority to AboveNormal
[2023-02-21 23:09:01.351 -05:00] [ERR] [28] Nick.Plugin.Jellyscrub.Drawing.OldMediaEncoder: Unable to set process priority: "Permission denied" (will not prevent BIF generation!)
[2023-02-21 23:09:01.546 -05:00] [INF] [13] Jellyfin.Plugin.PlaybackReporting.EventMonitorEntryPoint: Processing playback tracker : "TW96aWxsYS81LjAgKFdpbmRvd3MgTlQgMTAuMDsgV2luNjQ7IHg2NCkgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzEwOC4wLjAuMCBTYWZhcmkvNTM3LjM2IEVkZy8xMDguMC4xNDYyLjU0fDE2NzI3NTkyMzQ2NDc1-7f2c574c0ee14f479c8b41e8e1a795e1-392285dc052cdf93ea7fb5671d9fd329"
[2023-02-21 23:09:01.735 -05:00] [ERR] [28] Nick.Plugin.Jellyscrub.Drawing.VideoProcessor: Error while creating BIF file
System.InvalidOperationException: Cannot make BIF file from 0 images.
   at Nick.Plugin.Jellyscrub.Drawing.VideoProcessor.CreateBif(String path, Int32 width, Int32 interval, BaseItem item, MediaSourceInfo mediaSource, CancellationToken cancellationToken)
   at Nick.Plugin.Jellyscrub.Drawing.VideoProcessor.Run(BaseItem item, MediaSourceInfo mediaSource, Int32 width, Int32 interval, CancellationToken cancellationToken)

I have generated some .bif files for other movies or shows in the past but I am not sure why this is happening now. I am not sure what additional information you might need to look into this, so do let me know.

Well the "unable to set process priority" is fine. At least on linux, anything above normal seems requires elevated permissions and it will just default to normal priority on error.
"Cannot make BIF file from 0 images" is bad because it means ffmpeg exited without generating any images, but ffmpeg didn't output any errors so I'm not sure whats going on there. You say you have generated some in the past--was it working fine and just errored on this one movie? Was it working a couple weeks ago until you updated it? Did it stop working after you changed anything like hardware acceleration settings?

I used to have it working with HWA set to "None" without realizing but that was super stressing on the CPU so I changed it to Full. I am not sure if the error started happening after I switched to Full HWA or updating to <IP_ADDRESS>.  I will try downgrading the version and test again.
I am using an AMD RX 590 with VAAPI for HWA.

AMD does not support mjpeg encoding, so if you set it to "No Encode" it should start working. That's my bad since when I initially wrote the description about "on older hardware set to No Encode..." I didn't realize that all AMD hardware didn't support mjpeg encoding at all through vaapi.

Okay let me try that.

Whats weird is that you had no error from ffmpeg, as when I tried with an AMD graphics card I got an error about no suitable encoding profiles.

If I run the command mentioned in the logs straight through my terminal, I get the following:
[mjpeg_vaapi @ 0x56268664e8c0] No usable encoding entrypoint found for profile VAProfileJPEGBaseline (12).
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Not sure why it is not in the logs.
Changing the setting to "No Encode" seems to have fixed it. This is the log from the first bif file it created.
[2023-02-22 00:03:00.983 -05:00] [INF] [23] Nick.Plugin.Jellyscrub.Drawing.VideoProcessor: Finished creation of trickplay file "/mnt/manshare/Jellyfin/Movies/22 Jump Street (2014) [1080p]/trickplay/22 Jump Street (2014) Bluray-1080px264 AAC-320.bif"
[2023-02-22 00:03:01.004 -05:00] [INF] [13] Nick.Plugin.Jellyscrub.Drawing.VideoProcessor: Creating trickplay files at 320 width, for "/mnt/manshare/Jellyfin/Movies/Aftermath (2017) [1080p] [YTS.AG]/Aftermath (2017) Bluray-1080px264 AAC.mp4" [ID: 818f5dcd-7b95-4f4b-387f-f39ae7fcaf0d]
[2023-02-22 00:03:01.004 -05:00] [INF] [13] Nick.Plugin.Jellyscrub.Drawing.OldMediaEncoder: /usr/lib/jellyfin-ffmpeg/ffmpeg -loglevel error -init_hw_device vaapi=va:/dev/dri/renderD128 -filter_hw_device va -hwaccel vaapi -hwaccel_output_format vaapi -autorotate 0 -i file:"/mnt/manshare/Jellyfin/Movies/Aftermath (2017) [1080p] [YTS.AG]/Aftermath (2017) Bluray-1080px264 AAC.mp4" -autoscale 0 -an -sn -vf "fps=1/10,setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale_vaapi=w=320:h=132:format=nv12,hwmap,format=nv12" -threads 0 -c:v mjpeg -f image2 "/var/cache/jellyfin/temp/c49734f8e78d4d84ba4a4bfbec65271f/img_%08d.jpg"
[2023-02-22 00:03:01.005 -05:00] [INF] [13] Nick.Plugin.Jellyscrub.Drawing.OldMediaEncoder: Setting generation process priority to AboveNormal
[2023-02-22 00:03:01.005 -05:00] [ERR] [13] Nick.Plugin.Jellyscrub.Drawing.OldMediaEncoder: Unable to set process priority: "Permission denied" (will not prevent BIF generation!)

Glad that solved it for you. I'll have to look into why it's not logging errors though.


Frontend application flow update

The purpose of this PR is to establish the new application flow in the frontend. The designs implemented look like this:

The new reef appears in my profile list as expected but it looks like we need to add a few guards to avoid errors if we try loading the reef too early:


Implement PhraseQuery

The current positions encoding is very slow, but we should already be able to implement PhraseQuery on top of it.

Was done in #50 for no reason at all.


fix(go-client): scan will not recover when encounter error_invalid_state

What problem does this PR solve?
https://github.com/apache/incubator-pegasus/issues/903
If rpc's resp = error_invalid_state,  go client should support re-config, this feature has support most api but scan.
What is changed and how does it work?
In https://github.com/XiaoMi/pegasus-go-client/pull/86, we add it into nextBatch, but miss in startScanPartition, this pr add it.
Checklist
Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

Code changes

Has interface methods change

Related changes

Need to cherry-pick to the release branch
Need to update the documentation
Need to be included in the release note

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@b27f101). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #1106   +/-   ##
=========================================
  Coverage          ?   53.19%
=========================================
  Files             ?       27
  Lines             ?     2523
  Branches          ?        0
=========================================
  Hits              ?     1342
  Misses            ?     1135
  Partials          ?       46

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more


function "isFunction" is now defined in two files (graphics-commands and matrix-commands) and hence needs to be refactored.

isFunction = (functionToCheck) ->
getType = {}
functionToCheck and getType.toString.call(functionToCheck) is "[object Function]"

so this is actually in the bigcleanup pull request. can close once PR #255 is merged

fixed now #255 is merged


Customize the deployment

what file do i modify to get this working the file in dir example/kubernetes or the variable.tf in the top level directory, documentation was not clear. This will be a stock build without Tectonic features.

why am i getting this error:
Warning: output "etcd_sg_id": must use splat syntax to access aws_security_group.etcd attribute "id", because it has "count" set; use aws_security_group.etcd.*.id to obtain a list of the attributes across all instances
Warning: output "aws_api_external_dns_name": must use splat syntax to access aws_elb.api_external attribute "dns_name", because it has "count" set; use aws_elb.api_external.*.dns_name to obtain a list of the attributes across all instances
Warning: output "aws_elb_api_external_zone_id": must use splat syntax to access aws_elb.api_external attribute "zone_id", because it has "count" set; use aws_elb.api_external.*.zone_id to obtain a list of the attributes across all instances
Warning: output "aws_api_internal_dns_name": must use splat syntax to access aws_elb.api_internal attribute "dns_name", because it has "count" set; use aws_elb.api_internal.*.dns_name to obtain a list of the attributes across all instances
Warning: output "aws_elb_api_internal_zone_id": must use splat syntax to access aws_elb.api_internal attribute "zone_id", because it has "count" set; use aws_elb.api_internal.*.zone_id to obtain a list of the attributes across all instances
Warning: output "etcd_sg_id": must use splat syntax to access aws_security_group.etcd attribute "id", because it has "count" set; use aws_security_group.etcd.*.id to obtain a list of the attributes across all instances
Warning: output "aws_api_external_dns_name": must use splat syntax to access aws_elb.api_external attribute "dns_name", because it has "count" set; use aws_elb.api_external.*.dns_name to obtain a list of the attributes across all instances
Warning: output "aws_elb_api_external_zone_id": must use splat syntax to access aws_elb.api_external attribute "zone_id", because it has "count" set; use aws_elb.api_external.*.zone_id to obtain a list of the attributes across all instances
Warning: output "aws_api_internal_dns_name": must use splat syntax to access aws_elb.api_internal attribute "dns_name", because it has "count" set; use aws_elb.api_internal.*.dns_name to obtain a list of the attributes across all instances
Warning: output "aws_elb_api_internal_zone_id": must use splat syntax to access aws_elb.api_internal attribute "zone_id", because it has "count" set; use aws_elb.api_internal.*.zone_id to obtain a list of the attributes across all instances
Error: module.tectonic.template_dir.tectonic: 1 error(s) occurred:

module.tectonic.template_dir.tectonic: file: open : no such file or directory in:

${base64encode(file(var.license_path))}

is it bpoth config.tf and variable.tf that need to be modified?
i added my zone ID to variable.tf but getting this error
Error: Error refreshing state: 1 error(s) occurred:

module.dns.data.aws_route53_zone.tectonic: 1 error(s) occurred:

module.dns.data.aws_route53_zone.tectonic: data.aws_route53_zone.tectonic: no matching Route53Zone found


Alignment setting for indicator

Take vertical timelineTile for example, I need the indicator to alignment at the start of it's child. By default, indicator is center aligned. Allow users to decide indicator alignment to its child is very necessary.
This is a nice library. thanks.

Hey @Nerogee !
You can already do this with the indicatorXY property from IndicatorStyle.
The weather timeline aligns the indicator at 0.3 for example. As you can see here.

Let me know if this helped you.

@JHBitencourt yep, this helps me a lot. thanks.

Happy to help.
I'm closing this issue for now.


Add File APIs for setting timestamps (creation time, last write time, last access time)

We currently don't expose the APIs in System.IO.File for setting the timestamps in files:

SetCreationTime
SetCreationTimeUtc
SetLastAccessTime
SetLastAccessTimeUtc
SetLastWriteTime
SetLastWriteTimeUtc

hey, @augustoproiete I would like to work on Issue could you Assigned to me.

@DurveshKumarPal Thanks for the offer, but the work for this ticket is already done - just awaiting review & merge.
I assigned https://github.com/cake-build/cake/issues/2056 to you though as that one is still open and no work has been done yet


Allow running the test suite on a local Linux machine

Type of change

[x] Optimization

Description
The tests failed to run on a local Linux machine. To allow running it locally two main changes were required:

Run opensearch locally and instruct the tests to report to it
Run the tests with the locally built binary

In addition, instructions on running rootful Podman were added.

Needs some rebase after merging your previous pr @ygalblum

Needs some rebase after merging your previous pr @ygalblum

@rsevilla87 Done


CSS Parser: parse property at rule

https://developer.mozilla.org/en-US/docs/Web/CSS/@property
https://drafts.css-houdini.org/css-properties-values-api/#at-property-rule
@property <custom-property-name> {
  <declaration-list>
}

It should be straightforward to implement the rule, we can take counter-style as an example:
https://github.com/biomejs/biome/blob/main/crates/biome_css_parser/src/syntax/at_rule/counter_style.rs
In our grammar, it might look like:
https://github.com/biomejs/biome/blob/main/xtask/codegen/css.ungram
CssPropertyAtRule =
	'property'
	name: CssCustomProperty
	block: CssDeclarationListBlock

As an example to parse:
@property --my-color {
    inherits: false;
    initial-value: red;
}

@yossydev
Could you please make a comment here?
Doing so will allow me to assign the issue to you.

@denbezrukov
Sorry!
Once again, I would like to address this issue!

@denbezrukov
I am so sorry, I completely forgot that I was assigned to this issue.
If still possible I would like to work on this issue. I will get a PR out as soon as possible.

@ematipico I would love to do this one, I have started already. I used it as a way to get to know the project :)

Thank you @bjoroen , it's all yours


Add ability to schedule jobs

It'd be cool if I could schedule a job, by adding text to the subject, so that the job isn't run immediately.
E.g

Jenkins > Deploy @ 12pm
Jenkins > Deploy @ 12pm Tomorrow
Jenkins > Deploy @ 12.15pm 10/04/2016 EST

Use?:

compile 'com.joestelmach:natty:0.12'
compile 'org.ocpsoft.prettytime:prettytime:4.0.1.Final'

Closing, unless people vote to include it.


Need Checks for Multiplayer Games

could use either of the following:
https://partner.steamgames.com/doc/webapi/ISteamApps
https://api.steampowered.com/ISteamApps/GetAppList/v2/
https://store.steampowered.com/api/appdetails?appids=???&filters=categories

Closed by commit:
https://github.com/reiss-josh/gamesincommon/commit/fe9c65dc7939050684f778f626cc6c85b8303748


Make sulten reservation model support multiple tables

When #739 is resolved, reservations for multiple tables will simply be duplicated.
The better solution is to have a one-to-many relationship from reservation to tables so that the reservations are connected.
This is more complex and requires various changes in the sulten system.  This issue will be quite hard and requires a fair amount of work to solve, so I would wait to see if Lyche really needs this. It is really only important when reservations must be changed (they'd have to change each one individually).

This overlaps with #451


[Bug] Issue with SASS - either does not apply it or it breaks

Describe the bug
Good morning, I have an internal React library and I'm trying to setup storybook for it. I need to use SASS so I come up to this addons, but even following the documentation I always end up having the same error.
I've followed several possible solution through SO and github issues, but either storybook runs but styles are not applied, or I get the following issue:

ERROR in ./node_modules/css-loader/dist/cjs.js!./node_modules/sass-loader/dist/cjs.js??ruleSet[1].rules[13].use[2]!./node_modules/style-loader/dist/cjs.js!./node_modules/css-loader/dist/cjs.js!./node_modules/sass-loader/dist/cjs.js!./src/styles/main.scss
Module build failed (from ./node_modules/sass-loader/dist/cjs.js):
expected "{".
╷
2 │       import API from "!../../node_modules/style-loader/dist/runtime/injectStylesIntoStyleTag.js";
│                                                                                ^
╵
src/styles/main.scss 2:98  root stylesheet

can someone point me to the right direction?
Every help of suggestion is more than welcome!
Expected behavior
Storybook runs and sass/styles are applied
Dev dependencies
"@chromatic-com/storybook": "^1.9.0",
"@rollup/plugin-commonjs": "^24.0.1",
"@rollup/plugin-image": "^3.0.3",
"@rollup/plugin-json": "^6.0.0",
"@rollup/plugin-node-resolve": "^15.0.2",
"@storybook/addon-essentials": "^8.3.6",
"@storybook/addon-interactions": "^8.3.6",
"@storybook/addon-links": "^8.3.6",
"@storybook/addon-onboarding": "^8.3.6",
"@storybook/addon-styling-webpack": "^1.0.0",
"@storybook/addon-webpack5-compiler-swc": "^1.0.5",
"@storybook/blocks": "^8.3.6",
"@storybook/react": "^8.3.6",
"@storybook/react-webpack5": "^8.3.6",
"@storybook/test": "^8.3.6",
"@trivago/prettier-plugin-sort-imports": "^3.3.0",
"@types/node": "18.0.0",
"@types/react": "18.3.3",
"@types/react-dom": "18.3.0",
"axios": "1.6.8",
"babel-loader": "^8.2.5",
"css-loader": "^7.1.2",
"esbuild": "^0.11.18",
"eslint": "8.18.0",
"eslint-config-next": "^13.1.1",
"eslint-plugin-storybook": "^0.5.13",
"file-loader": "^6.2.0",
"npm-run-all": "^4.1.5",
"postcss": "^8.4.47",
"prettier": "^2.7.1",
"rollup": "^3.20.2",
"rollup-plugin-copy": "^3.4.0",
"rollup-plugin-delete": "^2.0.0",
"rollup-plugin-dts": "^5.3.0",
"rollup-plugin-peer-deps-external": "^2.2.4",
"rollup-plugin-postcss": "^4.0.2",
"rollup-plugin-typescript2": "^0.34.1",
"sass": "^1.80.4",
"sass-loader": "^16.0.2",
"storybook": "^8.3.6",
"style-loader": "^4.0.0",
"tslib": "^2.8.0",
"typescript": "4.7.4"
Additional context
This is a private library so I can't share the entire package, I'll try to add the file that I think are meaningful to this error, let me know if it's not clear and I'll try to give more context/info
.storybook/main.ts
import type { StorybookConfig } from '@storybook/react-webpack5';
import path from 'path';

const config: StorybookConfig = {
  stories: ['../src/**/*.mdx', '../src/**/*.stories.@(js|jsx|mjs|ts|tsx)'],
  addons: [
    '@storybook/addon-webpack5-compiler-swc',
    '@storybook/addon-onboarding',
    '@storybook/addon-links',
    '@storybook/addon-essentials',
    '@chromatic-com/storybook',
    '@storybook/addon-interactions',
    '@storybook/addon-styling-webpack',
    {
      name: '@storybook/addon-styling-webpack', // why does not this work?
      options: {
        rules: [
          // Replaces any existing Sass rules with given rules
          {
            test: /\.s[ac]ss$/i,
            use: [
              'style-loader',
              'css-loader',
              {
                loader: 'sass-loader',
                options: { implementation: require.resolve('sass') },
              },
            ],
            exclude: /node_modules/,
          },
        ],
      },
    },
  ],
  framework: {
    name: '@storybook/react-webpack5',
    options: {
      builder: {
        useSWC: true,
      },
    },
  },
  swc: () => ({
    jsc: {
      transform: {
        react: {
          runtime: 'automatic',
        },
      },
    },
  }),
  docs: {
    autodocs: 'tag',
  },
  webpackFinal: async (config) => {
    config.module?.rules?.push(
      {
        test: /\.scss$/,
        use: [
          'style-loader', // Injects CSS into the DOM
          'css-loader', // Resolves `@import` and `url()` like `import/require()`
          'sass-loader', // Loads and compiles Sass/SCSS files
        ],
      },
      {
        test: /\.(woff|woff2|eot|ttf|otf)$/,
        use: [
          {
            loader: 'file-loader',
            options: {
              name: '[name].[ext]',
              outputPath: 'fonts/',
              publicPath: 'fonts/',
              esModule: false,
            },
          },
        ],
      },
    );

    config.resolve = {
      ...config.resolve,
      alias: {
        ...config.resolve?.alias,
        '@': path.resolve(__dirname, '../src/'),
      },
    };

    return config;
  },
};

export default config;

.storybook/preview.ts
import '../src/styles/main.scss';
import type { Preview } from '@storybook/react';

const preview: Preview = {
  parameters: {
    controls: {
      matchers: {
        color: /(background|color)$/i,
        date: /Date$/i,
      },
    },
  },
};

export default preview;

main.scss
@import "./xyz/styles.scss";
@import "./xyz/printview.scss";

.react-slidedown.xyz-dropdown-slidedown {
    transition-duration: 1s;
    overflow-x: hidden;
}

styles.scss

// core Variables
@import 'core-components/variables';
@import 'core-components/mixins';
@import 'core-components/hco_fonts';

// XYZ Variables
@import 'components/xyz-fonticons';
@import 'core-components/xyz-mixins';

// Core
@import 'core-components/core';
@import 'core-components/globals';

// Common Components
@import 'components/aria_select';

_hco_fonts.scss
@font-face {
	font-family: 'HCo Gotham';
	src:url('#{$fa-font-path}/hco_fonts/woff2/Gotham-Light_Web.woff2') format('woff2'),
		url('#{$fa-font-path}/hco_fonts/woff/Gotham-Light_Web.woff') format('woff'),
		url('#{$fa-font-path}/Gotham-Light.ttf') format('truetype');
	font-weight: 300;
	font-style: normal;
}

@consuelo-sanna I have been struggling for a few days with apparently the same issue (Storybook 8, React framework, scss apparently not processed when only adding the webpack styling addon, and same exception as you when I defined a custom rule for sass and scss files with this addon).
Here is what I discovered.

Regarding your affirmation that "it breaks": if you run storybook with options '--debug' and '--debug-webpack' you will see the printed final webpack rules, and you will notice that the issue comes from the fact that the scss files are processed twice. This is actually implicitly told in the error message ERROR in ./node_modules/css-loader/dist/cjs.js!./node_modules/sass-loader/dist/cjs.js??ruleSet[1].rules[13].use[2]!./node_modules/style-loader/dist/cjs.js!./node_modules/css-loader/dist/cjs.js!./node_modules/sass-loader/dist/cjs.js!./src/styles/main.scss You can see in here that webpack tries to run sass-loader twice (and css-loader and style-loader in between, and again css-loader once more).
In your particular example, I see that you have rules for processing scss files both in the webpackFinal config section and in the addon options. This is why webpack tries to process them twice (from last rule to first rule), and fails on the second run of sass-loader on the already processed files.
Regarding your affirmation that "it does not apply", did you try having very simple SASS content in your main.scss file, for example with a simple SASS variable evaluation? When I did that, then I could see that the SASS variable was correctly interpreted and that the styling rule was applied! This helped me focus on what could actually cause the more complex content to not be properly evaluated.

As a side note for people who may come across a similar issue in the future without having explicitly defined redundant rules implying sass-loader on the same files: check the config rules added by other Storybook addons!
In my case, the conflict came not from my own rules but from the react preset addon, which already defines rules for processing styling (and my custom rules were added to the ones provided by the react preset addon!) I am not sure that the styling webpack addon is actually needed at all when using the preset react addon. I gave a try without this addon and only with the present react addon and it just worked when I had a scss file with following content:
body {
  color: $body-color !important;
}

(now, I only need to figure out which parts of my complex scss are causing the whole of it to be ignored...)


Game doesn't apply conditions for MOBs (?)

This manifests itself for example in Spiders not getting Monster Melee Poison condition.
It does work when spawning a new critter so it makes me think it's a MOB loading issue.

Nope, this is strictly a MOB issue - also present in non-TP Co8, and is fixed by replacing the MOB with the one from vanilla GoG ToEE.


Restore Java compilation failure rendering in Build Scans

Using the Problems API reports to render build failures made the build scans less readable. More specifically, we moved the compiler output from the exception message to BuildExceptionRenderer.
This PR restores the original error message for build scans while preserving the Problems API-based rendering on the CLI.

Fixes https://github.com/gradle/gradle/issues/31513
Reviewing cheatsheet
Before merging the PR, comments starting with

❌ ❓must be fixed
🤔 💅 should be fixed
💭 may be fixed
🎉 celebrate happy things

@bot-gradle test this

I've triggered the following builds:

Pull Request Feedback (Trigger) (Check)

Build Scan

@bot-gradle test RFM

I've triggered the following builds:

Pull Request Feedback (Trigger) (Check)

Build Scan

The following builds have failed:

Pull Request Feedback (Trigger) (Check)

Build Scan

@bot-gradle test this

The following builds have passed:

Pull Request Feedback (Trigger) (Check)

Build Scan

@bot-gradle test RFN

I've triggered the following builds:

Ready for Nightly (Trigger) (Check)

Build Scan

The following builds have passed:

Ready for Nightly (Trigger) (Check)

Build Scan

@bot-gradle merge


chore: allow enabling DLAS as it says in the docs

There are test failures in Spring Boot integration.


Change /leaderboard command to a weekly auto post

Credit to hutch, we think it'd be more fun to have the point totals be a mystery that gets revealed at the end of every week. No one will be able to see how many points anyone has until the bot posts the leaderboard on Friday or something like that.

To add to this, only display points that were bestowed in that last week. Total points don't change by that much to be noticeable week by week, so I think anyone can invoke /leaderboard if they want, and what will be the mystery is how many points each person got/lost that week.

I'd still like this to eventually be an auto post, but for the time being #32 will add the ability to see the leaderboard for just the past week, it just needs to be invoked manually.


fix: Update social media section in README

:tada: This PR is included in version 0.21.3 :tada:
The release is available on:

v0.21.3
GitHub release

Your semantic-release bot :package::rocket:


The example doesn't work in Firefox

I use FF62 and the example todo app gives me a blank page with "component(): Custom elements are not available, all components fallback to JS definition only" in the console.

In Firefox, Web Components are not enabled by default yet. To do so follow these instructions: https://www.designedbyaturtle.co.uk/2015/how-to-enable-web-components-in-firefox-shadow-dom/
They will be enabled by default in the next version (63)


Enhance troubleshooting: tunnels down, but IPSEC is up

Enhance the troubleshooting section to help people figure out conditions in which the tunnels are shown as down but IPSEC is up.  This may be due to BGP not being established across both tunnels.

Hi @ckamps , I actually encountered this issue where IPSEC is up but Tunnels are down. I see that in my strongSwan's BGP summary, the BGP's state is Active but Up/Down status is listed as never. May I know how to fix this?

In my case, I have to redo the Site-to-Site VPN connection and indicate an anything (<IP_ADDRESS>/0) for Remote IPv4 network CIDR and Local IPv4 network CIDR. My guess is because BGP traffic routes on the public IP attached to strongSwan, meaning over the VPC Internet Gateway, so I could not use my VPC CIDRs.


Execute with another engine e.g. basexclient

Hi @jakobjanot, thanks for this. Hopefully I can take a look at merging this next week.


implemented Google Tag Manager Datalayer api to push event

Hi @rdesai40!
Thanks for the PR, I'll review this week.


All  sequential steps are reported as nested for Allure

The problem
All  sequential steps are reported as nested for Allure
Environment

Xteps version 1.3
Xteps Allure version 1.0

Done


filiphr first submission

Really interesting and fun challenge @gunnarmorling.
I gave it a go, first implementation using Shenandoah GC (reduces GC pauses significantly) and uses parallel iteration.
My implementation is: [CalculateAverage_filiphr.java]
I used OpenJDK Runtime Environment Zulu21.30+15-CA (build 21.0.1+12-LTS) (21.0.1-zulu)
On Macbook Pro (8 core Intel 64 GB RAM) it runs in +/- 1m40s, the reference implementation runs in +/- 4m9s

Nice one, thanks a lot! Will evaluate this one tomorrow, first need to install that Zulu JDK.

Thanks. It doesn't have to be Zulu though. It needs a JDK that has Shenandoah GC, the ones provided by Oracle do not have it (Graal, Oracle JDK and Open JDK). Zulu, Temurin, RedHat, SAP have it (in case it is easier for you to use one of the others)

Hey @lobaorn, it would for sure be interesting to try out a different build from https://builds.shipilev.net/. Which one will you suggest me to try out?
I've been also working on using memory mapped files locally and it seems like the GC is not making any special differences now.

Thanks @lobaorn, I'll check Twitter and the Comments in #15 as well.
I've done some rework on this using memory mapped files thanks @bjhara and @royvanrijn for their submissions, I learned something new.
Changed to use BigDecimal parsing instead of Double.parseBoolean, that improves a bit the speed as well.
With the updates it now runs in ~35s on my machine

@filiphr, if any of the distros available from SDKMan can be used, could you add a line like this to your launch script?

@gunnarmorling, done. I've added sdk use java 21.0.1-graal. Using the Graal CE edition makes no difference for me.
If it isn't possible to use Oracle GraalVM (it is available on SDKMan) then it is fine to just run this with the default JDK

Nice, thanks for merging it. Such a fun experience. Will ty to speed it up a bit more :)


Make it so table view does not auto-open when a feature is selected

If a feature is selected, the table opens even if collapsed:

The table should stay collapsed unless explicitly opened.

Turns out it also pops open when you click and drag the track. May have to do with this:
https://github.com/GMOD/Apollo3/blob/89ccf49549f3bd640f6470e170da5048f3815988/packages/jbrowse-plugin-apollo/src/LinearApolloDisplay/components/LinearApolloDisplay.tsx#L173-L175


Epic bug: Please support my favorite package manager!

In order to avoid everyone creating one off bugs for the same package manager, this bug will act as a home to collect all such requests received so far. (Of course, once someone starts working on adding support for one of them, that could be a good time to create an individual bug).
Here is the current list of package managers we are considering (but not actively working on):

apt
chocolatey
flatpak
gofish
macports
rpm
snapcraft
spec
winget

NOTE: Some of these package managers may never get supported, as I'm not sure how many people actually want to use them, and supporting more releases results in more work on our end.
So even if you see your package manager on this list, it still may be worth leaving a comment below like "Yes, I want that one please! I use it particularly because..."
If you see a comment below from someone expressing support for a package manager that you also want, please give their comment a thumbs up so we can better roughly understand what people are using.

It would be nice to have this on Winget! Currently however, Winget doesn't accept zip files but rather installers so we'd have to either wait until they do support zip files (it's an open issue but there is no indication of this being implemented) or create an executable (exe or msi).

Thank you @russellbanks for your comment. I updated the initial list to include winget.
Hmmm, it might be possible to create an exe... but at that point I think I would be bundling a jre which might balloon the final size a bit, which may be unfortunate since to use Kobweb the user pretty has to have Java installed on their machine already.
Hopefully, though, winget decided to support .zip files. It is so convenient and would reduce so much friction!

I'm not the most well-versed in packaging Java applications (in terms of bundling the JRE, etc) but is it possible to create an executable without bundling the JRE? If this is not feasible, we could just bite the bullet and bundle the JRE with it, specifying that the recommended option is still the zip file?
Even if this is a temporary solution, I still think I would be advantageous to look into ways we can get an executable working in order to get this on Winget as it is the official package manager for Windows.
P.S. Just a correction to the updated initial list - the link for Winget is https://github.com/microsoft/winget-cli.

@russellbanks Thanks for letting me know about the wrong link. I fixed the original comment.
Even if we could build an .exe, just keep in mind I'd want a flow where this could all be done automatically every time I released a new version. With existing package managers, this is basically a one click operation for all of them so far. So we'd need to figure out how to automatically 1) unzip the file, 2) bundle it up into an exe, 3) and publish this to winget.
I personally won't have time to look into this for quite a while, but if someone knows how to do it, and/or has an example of this happening already, please add a comment -- that could help a lot!
P.S. Russell, you may want to check out the https://github.com/jreleaser/jreleaser project and file an issue with them to support winget. I'm relying on them pretty heavily for my other releases, and if winget is the official package manager for windows, they may be interested in getting some signal that people would want it supported.

Not much more I can comment on this other than I hope that either Winget supports zip files in the future or JReleaser adds support for Winget. There is an open issue on JReleaser already to ask for support for winget as a packager, but it doesn't seem to have any activity.

JReleaser now has an MSI installer which can be published to WinGet. I think we can now proceed.

Great, thanks for the update! With JReleaser leading the way, I will
definitely take a look.
On Fri, 6 May 2022 at 10:12, Begula @.***> wrote:

JReleaser now has an MSI installer which can be published to WinGet. I
think we can now proceed.
—
Reply to this email directly, view it on GitHub
https://github.com/varabyte/kobweb/issues/117#issuecomment-1119825835,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKNONAWHPPHGI6JK3EKE66TVIVHINANCNFSM5NJWQPUQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

As of 1.4, winget will have support for installing zip files. However, this is just a preview at the moment, with the stable version being 1.2, and the release candidate being 1.3. But, this means that Kobweb can be put on WinGet; it's just a matter of time.

FWIW you can use jpackage (on its own or via JReleaser) to create Windows installers (exe, msi) that may be used in combination with Winget.

(Sorry for the recent radio silence, folks. I just moved, which wiped out my schedule for a couple of weeks).
As I catch up on stuff, I do intend to look into Winget, although I have a few other priorities to get through first. I'm hoping to get to this by end of September ('22) at the very latest, so feel free to ping this bug if it's later with no progress.

Finally, I'm back to this bug and looking into adding winget support. Thanks again for everyone's patience.
@aalmiray I'm trying to follow the Gradle syntax for jpackage from here: https://jreleaser.org/guide/latest/reference/assemble/jpackage.html
but "jpackage" contents aren't autocompleting for me:

(I'm using latest, Jreleaser v1.5.0)
is there any demo code for using jreleaser + jpackage I can look at?
Also, according to @russellbanks , winget should support .zip files now (1.4 was released in Jan). Are there plans to add support for that into JReleaser, or will jpackageing be the way to go for a while?

@bitspittle
https://github.com/jreleaser/jreleaser/issues/185
https://jreleaser.org/guide/latest/reference/packagers/winget.html#_distribution_support

Right, that's what I'm looking at. winget doesn't support .zip file targets according to that distribution support list, so I assume at this point I still need to try running jpackage to create an .exe for now.

Zip support in Winget was added in 1.4 but the team has decided to wait until a critical mass of users has migrated to 1.4. See https://github.com/jreleaser/jreleaser/issues/185#issuecomment-1406908541
Once that happens we'll support releasing zips with Winget.
re: auto-completion. app is a free name which you can choose to be anythign you want. It identifies a named element inside a NamedContainer, thus you must use regular KotlinDSL such as
jreleaser {
  assemble {
    jpackage {
      create("kobweb-installer") {
        ...
      }
    }
  }
}

Ah perfect, it was a Kotlin script thing. Thanks for the quick reply!

Zip support in Winget was added in 1.4 but the team has decided to wait until a critical mass of users has migrated to 1.4. See jreleaser/jreleaser#185 (comment)

@aalmiray zip's were turned in on the pipelines a month or so ago:
https://github.com/microsoft/winget-pkgs/issues/87300

@russellbanks gotcha. Will have a look at zips on Winget in the next couple of weeks.

Filed a new ticket to keep track of zip support on Winget https://github.com/jreleaser/jreleaser/issues/1253

I think I'll hold off on winget until zip support is ready.
It looks like supporting building an MSI/EXE installer will be rather tricky, involving a bunch of structure just for winget that no other package manager has needed. For example, to run jlink / jpackage, I think I would need to set up a special CI workflow using Java 16, since everything else I've done works fine with JDK11 (or older).
And given the extra complexity around this, I have other priorities I should attend to first.

Understood. If it helps at a later stage, have a look a JReleaser's release workflow and configuration which take care of multiple targets during a release.

Thanks @russellbanks , I appreciate the offer!
However, I'd want to make sure that everything is handled in the build script so whenever I update the Kobweb binary, I can do everything with a single jreleaserPublish.
Probably best to figure everything out the correct way all at once (or otherwise I might release future versions of Kobweb and winget would fall behind)

Anything you do do to automate winget should take the previous manifest and update the values with that of your new artifacts, so a pre-existing manifest shouldn't stop you doing that in the future.

@aalmiray Yeah I was using one of your simple workflows as a guide, but seeing that I'd need  to set up a bunch of runtime JDKs just for winget seems like a lot of work.
Also, I can just jlink / jpackage a windows binary right? If I don't care about the other platforms?
And if I am building an installer for windows, do I need to make sure to run jreleaser on a Windows machine?

@russellbanks where does the winget manifest live? Currently, I have some local repositories (e.g. for homebrew) which jreleaser overwrites for me. Is winget different? (I was skimming the docs but haven't looked too closely yet)

Also, I can just jlink / jpackage a windows binary right? If I don't care about the other platforms?
Correct.

And if I am building an installer for windows, do I need to make sure to run jreleaser on a Windows machine?
Correct, again.

@russellbanks where does the winget manifest live? Currently, I have some local repositories (e.g. for homebrew) which jreleaser overwrites for me. Is winget different? (I was skimming the docs but haven't looked too closely yet)

microsoft/winget-pkgs
Pull requests are just made to winget-pkgs, usually using and created with a tool like Komac, YamlCreate or WinGetCreate.

@russellbanks Ah thanks, really helpful info! I don't think I'd mind you submitting an initial manifest for me, then, if it doesn't block me from updating or overwriting it later.
I was trying to search randomly through the manifests.... do you know of a manifest out there which uses the new zip type?
FYI latest Kobweb zip is here: https://github.com/varabyte/kobweb/releases/tag/cli-v0.9.11
However, it's not a standalone zip file. The user will need to have Java 11 (or newer) installed on their machine.
Does winget support the idea of specifying "Java 11" as a dependency, at which point it handles installing things automatically? Or would it be up to me to bundle Java 11 into the zip file?
@aalmiray Does this mean when I run jreleaser targeting winget, it won't immediately update anything but will submit a PR on my behalf?

Ah, winget doesn't support dependencies currently. It is planned as a "soon" feature.
See Dependencies.md on winget-cli.
For the time being, you could make a second zip that has a stripped JRE using JLink bundled with it? That's entirely up to you though.

@aalmiray Does this mean when I run jreleaser targeting winget, it won't immediately update anything but will submit a PR on my behalf?

Not by default, no. The original idea was to push to an intermediate repository (winget-kobweb for example) and trigger a GH workflow that pull create a branch and push. The PR would still have to be done manually unless the workflow were to use the gh command. This is how https://github.com/jreleaser/jreleaser-macports is handled.
However, I just added an option to specify a branchPush property on taps/repositories (https://github.com/jreleaser/jreleaser/issues/1129) which would give you the ability to directly create a branch on a target repository such as winget-pkgs, bypassing winget-kobweb. The actual PR would still have to be created manually at the moment.

Hmmm, I'm thinking I should probably should look into jlink at this point.
Give me a week or so, I have some other things I need to focus on first, but I'll ramp up on jlink / jpackage over the next few days.
Maybe I'll even give in and finally move publishing my artifacts via Jreleaser from a CI instead of my personal desktop, like an actual professional or something :stuck_out_tongue: Once that's migrated, then it wouldn't be too hard to add a workflow that runs on Windows for Winget.
(I just spent a week getting CI artifact support working for two of my other projects. It's a pain, to be honest, and I'm not looking forward to the game of CI trial and error yet again at the moment)
@russellbanks No createDistributable or anything like it in my tasks, sadly. Note that it seems pretty unlikely for Kobweb users to not have Java installed on their machine, since they're doing Kotlin development anyway. Having winget handle the dependencies for me would be the best though, I think.

(I just spent a week getting CI artifact support working for two of my other projects. It's a pain, to be honest, and I'm not looking forward to the game of CI trial and error yet again at the moment)

That's why JReleaser has a dry-run mode and offers plenty of options to run as much as possible on a local environment. Alas, when platform specific binaries or tools are required you must test on the given platform.

Hi there! It would be nice to support .rpm packages for Fedora and other RHEL-based Linux distros. Thanks!

Flatpak is also a pretty solid one. It's ships with almost every Linux distro by default, and Flathub allows publishing CLI apps (although they won't be shown in the apps store understandably).

@Skaldebane thanks! I updated the original comment.
Do you have any experience with either of those package managers? At one point, I checked out snap and it was a wasted day.

I've previously created an .rpm for a specific purpose, but nothing really proper.
As for Flatpak, I do have some experience with it (for making a GTK app), but not in the Java ecosystem.
However, I'll try to package Kobweb into one of them when I find some time!

FWIW JReleaser supports both packagers.
The Native Installer assembler relies on Jpackage to create an .rpm that bundles a Java Runtime along with the app.
The Spec packager OTOH builds an .rpm as expected by Fedora COPR’s build infrastructure.
Lastly, Flatpak may be used to package CLI and/or desktop apps

(Ah sorry @aalmiray meant to reply last week!)
Just wanted to say thanks for the last comment. If we can do everything with JReleaser, that's definitely the way to go.
@Skaldebane if you were going to try to experiment with this yourself, the jreleaser config / kobweb binary is actually in a different project. You can find it here.

Thanks for the reminder @Ayfri . I added it to my near-term todo list to revisit, to see if it will work seamlessly now with latest JReleaser.
I probably have a week or so more of Kobweb work to get through, and then I'll see if I can get winget support working after that. I'll report back here whether it goes well or if there are still obstacles.


Default to host endianness

It seems the Rust standard library makes dealing with the host's endianness a pain while making explicit endianness easier. I get the reason behind this, but some formats don't have an explicit endianness.
I'm working on a toy parser for Lua bytecode, which includes an endianness check, and luac apparently uses the host's endianness for its data format. I can assume little endian for now (that's my dev machine) and potentially have the endianness passed in (e.g. by testing the host's endianness), but it would be nice to just ignore explicit endianness for this particular format.
I would like i32 and friends to use the host's endianness if it's not explicitly passed in. This seems fair to me, and defaulting to little endian is a bit surprising IMO. Granted, little endian is the most common (x86), it's not universal, and big endian seems more common for formats anyway (network byte order).

I'd prefer the default to not depend on the host machine (no hidden host-dependent change in behavior). I'm not even sure if having a default is really needed. I thing it's better to require people to use le_u32 explicitly rather than relying on the default of u32. It requires people to actually consider endianness issues before writing their parser.
Regarding host endianness, I agree that it would be nice to have since there are use cases depending on it. What about adding Endianness::Host and host_u32 (etc.)?
Until it's supported, you could write your own host_* functions for your project (with detection of the host endianness).

byteorder has NativeEndian
:bike::house: It may be good for consistency across the ecosystem to use Native instead of Host. I initially went with "host" because it's shorter, but consistency may be more important.
You could write a macro to generate all the aliases based on the endianness (using #[cfg(target_endian = ...)])

i'd be open for a PR adding native endian combinators and parsers

the new number parsers with configurable endianness now have a native option: 1baaa7c


Update LinkedIn authorization url

See https://developer.linkedin.com/docs/oauth2
Looks better and guess that is the official one now?

Thanks 👍

Should probably also update the other urls in the examples btw.


Calculating a UI's size without rendering it

It would be cool if we were able to get a widgets size before actually rendering it. I know this is a fundamental limitation of immediate mode UIs, but it should be easily possible to do this, right? Basically we just make all of the calls except the ones that take any effect on the screen.
My situation is that I have a row of items with different sizes, and I want the entire row to have the same height. When I add the tallest item first, this works, because then the later elements know their height. But when the tallest element is not the first, then the first item will not know how tall it should be. So I would like to iterate through all elements before rendering them, to calculate their sizes and find the max height, then set use that height for all elements when rendering them.

The idea is basically to render to an offscreen buffer, just to get the final size of the Ui, without actually rendering anything. If there is some way to know if the Ui changed it would also be possible to cache something here, but I doubt that this will be possible without compromising developer experience.

Maybe this could be implemented in Painter by painting to something like a "null layer", which just means don't do any rendering?
https://github.com/emilk/egui/blob/784bac53f1156feff1b17afd3c982b2ccbb86543/egui/src/painter.rs#L12-L32

Perfect! This seems to suffice, at least for vertical and horizontal layouts:
fn measure_pattern(&self, ui: &mut Ui, pat: &PatternVis, graph: &GraphVis) -> Response {
    let old_clip_rect = ui.clip_rect();
    let old_cursor = ui.cursor();
    ui.set_clip_rect(Rect::NOTHING);
    let r = self.pattern(ui, pat, graph, None);
    ui.set_clip_rect(old_clip_rect);
    ui.set_cursor(old_cursor);
    r
}

I.e. before measuring we cache the Placer cursor and the Painter clip rect, then we set the clip rect empty, so nothing is drawn, then we do the draw call, take the response, and re-set the cursor and clip rect to the old values for the actual draw.
Now we can take the rect.height() from the Response and adjust the layout our elements:

I would like to make this a supported Api in Ui something like fn measure<R>(&mut self, add_children: Fn(&mut Ui) -> R) -> Response or maybe something like this returning a Rect.

Having something like this would be useful, but it can quickly lead to an exponential explosion of runtime. If N nested ui functions all call measure, then the innermost closure will be called 2^N times.
An alternative that I have been thinking about is to have an optional pre-pass, where we call all the apps Ui code once without any paining, and with a special "this is a prepass" flag in the Context. This will allow widgets to store sizes which can be recalled in the second and final pass. This requires a bit of design work though.

without any paining,

I definitely support this approach. :D

In the current version I don't see ui.set_cursor, was that removed?

It's there but it's a crate private function https://github.com/emilk/egui/blob/083d61fccdfc9d967a12be09b18d4546e2313580/crates/egui/src/placer.rs#L72
I made a fork and made it public to use it.


build: update angular dependencies to version 15.2.x

This will need a re-run once FW releases the stable versions.