Lens metrics not showing with custom kube-prometheus-stack

Describe the bug

Lens metrics not showing with custom kube-prometheus-stack

To Reproduce
Steps to reproduce the behavior:

Server settings - Extension (Lens Metrics) - Disable all three
helm Install kube-prometheus-stack (latest)
Apply following in helm chart config https://github.com/lensapp/lens/blob/master/troubleshooting/custom-prometheus.md
Lens UI not showing any metrics...

Expected behavior
Metrics to show up
Screenshots

If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):

Lens: 2022.11.81921-latest
Extension API: 6.1.17
Electron: 19.1.4
Chrome: 102.0.5005.167
Node: 16.14.2
© 2022 Mirantis, Inc.
OS: OSX
Installation method (e.g. snap or AppImage in Linux): dmg for M1

Logs:
When you run the application executable from command line you will see some logging output. Please paste them here:
Your logs go here...

Kubeconfig:
Quite often the problems are caused by malformed kubeconfig which the application tries to load. Please share your kubeconfig, remember to remove any secret and sensitive information.
your kubeconfig here

Additional context
Add any other context about the problem here.

Did it work before?

Did it work before?

It worked with default Lens metrics before but in the latest version, both default ones (lens metrics) and custom ones (kube-prometheus-stack) do not work...
Currently, we are only interested in the custom (kube-prometheus-stack) tho!

What metrics provider is configured for that cluster?

You mean like metric server & node exporter?
I am using k8s.gcr.io/metrics-server/metrics-server:v0.6.1 and quay.io/prometheus/node-exporter:v1.3.1 (prometheus-node-exporters from kube-prometheus-stack)

No I mean what is the setting within Lens under cluster preferences > Metrics for "Promtheus provider"

No I mean what is the setting within Lens under cluster preferences > Metrics for "Promtheus provider"

Oh, I chose Prometheus Operator and used headless service of prometheus as below.
(Also tried kube-prometheus-stack-prometheus)

If you installed prometheus from Helm you should probably use the "Prometheus Helm" provider. But until #6576 is merged and released it will be hard for us to diagnose why it isn't showing up.

If you installed prometheus from Helm you should probably use the "Prometheus Helm" provider. But until #6576 is merged and released it will be hard for us to diagnose why it isn't showing up.

I will try that and let you know! Really looking forward to seeing the new merge!

If you installed prometheus from Helm you should probably use the "Prometheus Helm" provider. But until #6576 is merged and released it will be hard for us to diagnose why it isn't showing up.

Switched to Helm, but no changes.

@Nokel81 I have the following error stack and the cause seems to be a socket hang up error
[0] warn:    ┏ [METRICS-ROUTE]: failed to get metrics for clusterId=bf5a8d06fa892a342094edf8835c4b42: Metrics not available +6ms
[0] warn:    ┃ [ 1] Error: Metrics not available
[0] warn:    ┃ [ 2]     at loadMetricHelper (/Users/XXXX/Downloads/lens-6.2.0/static/build/main.js:45121:31)
[0] warn:    ┃ [ 3]     at process.processTicksAndRejections (node:internal/process/task_queues:96:5)
[0] warn:    ┃ [ 4]     at async Promise.all (index 0)
[0] warn:    ┃ [ 5]     at async /Users/XXXX/Downloads/lens-6.2.0/static/build/main.js:45162:32
[0] warn:    ┃ [ 6]     at async Object.route (/Users/XXXX/Downloads/lens-6.2.0/static/build/main.js:43974:32)
[0] warn:    ┃ [ 7]     at async Router.route (/Users/XXXX/Downloads/lens-6.2.0/static/build/main.js:44247:9)
[0] warn:    ┃ [ 8]     at async LensProxy.handleRequest (/Users/XXXX/Downloads/lens-6.2.0/static/build/main.js:41912:9)
[0] warn:    ┃ [ 9] Cause: RequestError: Error: socket hang up
[0] warn:    ┃ [10]     RequestError: Error: socket hang up
[0] warn:    ┃ [11]         at new RequestError (/Users/XXXX/Downloads/lens-6.2.0/node_modules/request-promise-core/lib/errors.js:14:15)
[0] warn:    ┃ [12]         at plumbing.callback (/Users/XXXX/Downloads/lens-6.2.0/node_modules/request-promise-core/lib/plumbing.js:87:29)
[0] warn:    ┃ [13]         at Request.RP$callback [as _callback] (/Users/XXXX/Downloads/lens-6.2.0/node_modules/request-promise-core/lib/plumbing.js:46:31)
[0] warn:    ┃ [14]         at self.callback (/Users/XXXX/Downloads/lens-6.2.0/node_modules/request/request.js:185:22)
[0] warn:    ┃ [15]         at Request.emit (node:events:526:28)
[0] warn:    ┃ [16]         at Request.emit (node:domain:475:12)
[0] warn:    ┃ [17]         at Request.onRequestError (/Users/XXXX/Downloads/lens-6.2.0/node_modules/request/request.js:877:8)
[0] warn:    ┃ [18]         at ClientRequest.emit (node:events:526:28)
[0] warn:    ┃ [19]         at ClientRequest.emit (node:domain:475:12)
[0] warn:    ┃ [20]         at Socket.socketOnEnd (node:_http_client:466:9) {
[0] warn:    ┃ [21]       cause: Error: socket hang up
[0] warn:    ┃ [22]           at connResetException (node:internal/errors:691:14)
[0] warn:    ┃ [23]           at Socket.socketOnEnd (node:_http_client:466:23)
[0] warn:    ┃ [24]           at Socket.emit (node:events:538:35)
[0] warn:    ┃ [25]           at Socket.emit (node:domain:475:12)
[0] warn:    ┃ [26]           at endReadableNT (node:internal/streams/readable:1345:12)
[0] warn:    ┃ [27]           at process.processTicksAndRejections (node:internal/process/task_queues:83:21) {
[0] warn:    ┃ [28]         code: 'ECONNRESET'
[0] warn:    ┃ [29]       },
[0] warn:    ┃ [30]       error: Error: socket hang up
[0] warn:    ┃ [31]           at connResetException (node:internal/errors:691:14)
[0] warn:    ┃ [32]           at Socket.socketOnEnd (node:_http_client:466:23)
[0] warn:    ┃ [33]           at Socket.emit (node:events:538:35)
[0] warn:    ┃ [34]           at Socket.emit (node:domain:475:12)
[0] warn:    ┃ [35]           at endReadableNT (node:internal/streams/readable:1345:12)
[0] warn:    ┃ [36]           at process.processTicksAndRejections (node:internal/process/task_queues:83:21) {
[0] warn:    ┃ [37]         code: 'ECONNRESET'
[0] warn:    ┃ [38]       },
[0] warn:    ┃ [39]       options: {
[0] warn:    ┃ [40]         timeout: 0,
[0] warn:    ┃ [41]         resolveWithFullResponse: false,
[0] warn:    ┃ [42]         json: true,
[0] warn:    ┃ [43]         method: 'POST',
[0] warn:    ┃ [44]         form: {
[0] warn:    ┃ [45]           query: 'sum(node_memory_MemTotal_bytes * on (pod,namespace) group_left(node) kube_pod_info{node=~"lambda-server|lambda-server-2|lambda-server-cpu-1"} - (node_memory_MemFree_bytes * on (pod,namespace) group_left(node) kube_pod_info{node=~"lambda-server|lambda-server-2|lambda-server-cpu-1"} + node_memory_Buffers_bytes * on (pod,namespace) group_left(node) kube_pod_info{node=~"lambda-server|lambda-server-2|lambda-server-cpu-1"} + node_memory_Cached_bytes * on (pod,namespace) group_left(node) kube_pod_info{node=~"lambda-server|lambda-server-2|lambda-server-cpu-1"}))',
[0] warn:    ┃ [46]           start: '1668735048',
[0] warn:    ┃ [47]           end: '1668738648',
[0] warn:    ┃ [48]           step: '60',
[0] warn:    ┃ [49]           kubernetes_namespace: ''
[0] warn:    ┃ [50]         },
[0] warn:    ┃ [51]         headers: { Host: 'bf5a8d06fa892a342094edf8835c4b42.localhost:53298' },
[0] warn:    ┃ [52]         uri: 'http://localhost:53298/api-kube/api/v1/namespaces/prometheus/services/prometheus-operated:9090/proxy/api/v1/query_range',
[0] warn:    ┃ [53]         callback: [Function: RP$callback],
[0] warn:    ┃ [54]         transform: undefined,
[0] warn:    ┃ [55]         simple: true,
[0] warn:    ┃ [56]         transform2xxOnly: false
[0] warn:    ┃ [57]       },
warn:    ┃ [58]       response: undefined
[0] warn:    ┃ [59]     }
[0] warn:    ┃ [60] Cause: Error: socket hang up
[0] warn:    ┃ [61]     Error: socket hang up
[0] warn:    ┃ [62]         at connResetException (node:internal/errors:691:14)
[0] warn:    ┃ [63]         at Socket.socketOnEnd (node:_http_client:466:23)
[0] warn:    ┃ [64]         at Socket.emit (node:events:538:35)
[0] warn:    ┃ [65]         at Socket.emit (node:domain:475:12)
[0] warn:    ┃ [66]         at endReadableNT (node:internal/streams/readable:1345:12)
[0] warn:    ┃ [67]         at process.processTicksAndRejections (node:internal/process/task_queues:83:21) {
[0] warn:    ┃ [68]       code: 'ECONNRESET'
[0] warn:    ┗ [69]     }

@Nokel81 Would you have a look at the error stack when you are free!

I'm having same error, but with only one cluster that deployed to GKE, same error message Error: socket hang up, prometheus installed via operator.

same here, - not working Lens metrics with Lens prometheus, neither with latest kube-prometheus-stack, - @imageschool @vsychov how do you people got these errors, where did you get'em from? where do I click.. )
after TONS of googling I assumes that Lens and Prometheus projects both updated so quickly Lens just cannot keep up parsing all those Prometheus logging format changes somehow. So downgrading to old Prometheuses often fixes that, (probably accompanied by downgrading Lens also, but not necessarily - they try their best to store/keep all the previos Prometheuses'es fixes I guess).
By googling I sees sometimes it helps to mess with "Metrics" settings a lot: auto-detect, helm, lens, operator, set service address manually (auto-detection is seldom) etc. but greatly depending on the Prometheus and Lens versions used (and Cluster configuration also).
community would greatly benefit if someone please pleeease describe the exact step-by-step applying of this here https://github.com/lensapp/lens/blob/master/troubleshooting/custom-prometheus.md curious fixes.
they are way confusing as for me, where to click.. how to where wha?.. ))

@anonimnor to get the same error stack, I built it (following https://docs.k8slens.dev/contributing/development/) and run it on my own.
Or you can also run your Lens with the following command (on Mac) DEBUG_PROXY=true /Applications/Lens.app/Contents/MacOS/Lens

@anonimnor To apply the custom Prometheus, you have to include the following in the Helm Chart.
kubelet:
  serviceMonitor:
    metricRelabelings:
    - action: replace
      sourceLabels:
      - node
      targetLabel: instance

prometheus-node-exporter:
  prometheus:
    monitor:
      metricRelabelings:
      - action: replace
        regex: (.*)
        replacement: $1
        sourceLabels:
        - __meta_kubernetes_pod_node_name
        targetLabel: kubernetes_node

Lens - Cluster Setting - Metrics, include the following.

@vsychov @anonimnor If it still doesn't work after applying the above, try the below.
I use the latest Lens IDE & kube-prometheus-stack v41.9.1
The first thing you can try for the kube-prometheus-stack Helm Chart is enabling the host network and see if it works.
prometheus:
  prometheusSpec:
     hostNetwork: true # this

prometheus-node-exporter:
 hostNetwork: true # and this

After setting them to true I was able to see all the metrics in worker nodes (on-prem) but still could not see the master node metrics (master node on AWS EC2).
Investigating further, we ended up finding a network plugin (flannel) not correctly inserting the VXLAN interface (flannel.1) into the master node.
After restarting the canal on our master node then I put the host network back to false and everything was working correctly!

It might also be a permissions issue with cloud provider clusters. For example with GKE you need container.services.proxy permissions for lens to display metrics

For us the issue was that the Kubernetes API Server was not allowed to reach prometheus.
Since lens is relying on the proxy-functionality of the service API, the API server must be allowed to reach the prometheus pod on port 9090 (or whatever port prometheus is running on inside the pod)
To debug stuff I was using this (adjust namespace/service on copy):
kubectl get --raw /api/v1/namespaces/monitoring-system/services/prometheus-server:80/proxy/

This was giving me this error after 60s or so:
Error from server (ServiceUnavailable): error trying to reach service: dial tcp <IP_ADDRESS>:9090: connect: connection timed out

I am using the EKS terraform module. So I added this inside the module block:
  node_security_group_additional_rules = {
    prometheus_service_from_api_server = {
      description                   = "With this rule, the API server would be allowed to reach prometheus port. With this metrics work in lens"
      protocol                      = "TCP"
      from_port                     = 9090
      to_port                       = 9090
      type                          = "ingress"
      source_cluster_security_group = true
    }
  }

But of curse how exactly this rule is added depends on the setup...
With this extra-rule most metrics work (I could not get the Metrics on the cluster Tab working yet through)
Felix

For future devs, as of April 22nd 2024 this is finally was fixed it for me 🐐🙏

My issue resolved after I selected prometheus operator and address operations/prometheus-operated:9090


HHH-17050 Introduce JPA Criteria API support for timestampadd and timestampdiff

https://hibernate.atlassian.net/browse/HHH-17050

I'm against this. These functions are not on the official list of portable functions, because HQL has subtraction and addition operators for datetimes.
I just added support for these operations to the criteria API, so we should point users to those instead:

Well, subtraction and addition between datetimes have different semantic than what timestampadd and timestampdiff provide, so they are not really direct alternatives.

These functions are not on the official list of portable functions

They were not listed in the documentation, yes, but AFAIU they work in HQL in most (all?) supported dialects either by direct function call or emulation. They looked portable to me, that's why I also added them to the UG within this PR.

I guess one could get the same effect by combining the new Duration APIs you just implemented, didn't notice that because I was already working on this while they got merged.
@beikov actually suggested I implement timestampadd/diff APIs, I guess he could provide some additional feedback on this.

Well, subtraction and addition between datetimes have different semantic than what timestampadd and timestampdiff provide, so they are not really direct alternatives.

Not sure what you mean. (x - y) by day is the same as timestampdiff(day,x,y) and x + 1 day is the same as timestampadd(day, 1, x), and translate down to essentially the same SQL.

They were not listed in the documentation,

The canonical list is the java doc of Dialect.initializeFunctionRegistry()

AFAIU they work in HQL in most (all?) supported dialects either by direct function call or emulation.

They work because I wrote code to emulate them on all dbs as part of the underlying implementation of the date/time arithmetic stuff. But they were always meant for internal use.

So I guess HHH-17050 can be considered a duplicate of HHH-17109 then?

Well it's not really a duplicate, as the user was asking for way to call the native datediff  function through Criteria API. I guess we could say it's a viable alternative, using standardized Hibernate duration functions.

I guess we could say it's a viable alternative, using standardized Hibernate duration functions.

Right, point the users to the new API then. So I think we can close this PR and HHH-17050


[CTL] Improve font fallback order selection.

Fix loosing fallback fonts in some cases, and decrease overall number of is_language_supported/is_script_supported checks.

Thanks!


Decide whether OpenLane commands alter state or not

Prompt
Arguably arbitrarily, some procedures take an input and an output, and some work in-place. Some leave modifying the global state to their caller, some take it upon themselves to do so. There's no rhyme or reason to it and it's never been codified.
Proposal
We should:

A. Decide whether the ideal for this kind of project is stateful commands or pure commands
B. Decide where the line is drawn

It took a rewrite: https://openlane.readthedocs.io/en/latest/flow_overview.html


OCM-9944 | test: fix id:74761,65798

OCM-9944 [CI] Debug and fix : 74761, 65798
$ ginkgo --focus "(74761|65798)" tests/e2e/
Ginkgo detected a version mismatch between the Ginkgo CLI and the version of Ginkgo imported by your packages:
Ginkgo CLI Version:
2.11.0
Mismatched package versions found:
2.17.1 used by e2e
Ginkgo will continue to attempt to run but you may see errors (including flag
parsing errors) and should either update your go.mod or your version of the
Ginkgo CLI to match.
To install the matching version of the CLI run
go install github.com/onsi/ginkgo/v2/ginkgo
from a path that contains a go.mod file.  Alternatively you can use
go run github.com/onsi/ginkgo/v2/ginkgo
from a path that contains a go.mod file to invoke the matching version of the
Ginkgo CLI.
If you are attempting to test multiple packages that each have a different
version of the Ginkgo library with a single Ginkgo CLI that is currently
unsupported.
Running Suite: ROSA CLI e2e tests suite - /home/akanni/OCP-Repository/rosa/tests/e2e
Random Seed:<PHONE_NUMBER>
Will run 2 of 158 specs
SSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
Ran 2 of 158 Specs in 26.461 seconds
SUCCESS! -- 2 Passed | 0 Failed | 0 Pending | 156 Skipped
PASS
Ginkgo ran 1 suite in 27.859006131s
Test Suite Passed

/hold

/test ?

/test e2e-presubmits-pr-rosa-hcp-advanced e2e-presubmits-pr-rosa-sts-advanced

@xueli181114 Could you have a look to failing jobs ?

/retest

/test e2e-presubmits-pr-rosa-hcp-advanced

/retest

/unhold


attributes defined in early step lost in a foreach flow

Hi,
I have a flow (link to the gist) that looks like the following:

Click to expand
from metaflow import FlowSpec, step, Parameter

def script_path(filename):
    """
    A convenience function to get the absolute path to a file in this
    tutorial's directory. This allows the tutorial to be launched from any
    directory.

    """
    import os

    filepath = os.path.join(os.path.dirname(__file__))
    return os.path.join(filepath, filename)

class TestForeachFlow(FlowSpec):
    """
    A flow to test foreach flow.

    The flow performs the following steps:
    1) Setup parameters.
    2) Fork to foreach step.
    3) Alter the value of input c.
    4) Joins all the altered c as a list.
    5) End.

    """
    # initial attributes
    init_a = Parameter('init_a',
                         help = 'the initial attribute a',
                         default = 100)

    @step
    def start(self):
        """
        Setup parameters.

        """

        self.param = dict(foo = 2)

        self.next(self.fork)

    @step
    def fork(self):
        """
        Fork to foreach step.

        """

        self.c_list = [1,2,3,4]

        self.next(self.alter_c, foreach='c_list')

    @step
    def alter_c(self):
        """
        This step alters the value of input c.

        """

        self.c = self.input
        self.altered_c = self.c + 1000

        self.next(self.join)

    @step
    def join(self, inputs):
        """
        This step joins all the altered c as a list
        """
        self.altered_c_list = [input.altered_c for input in inputs]
        self.foo = self.param.get('foo')
        self.result = [i+ self.foo for i in self.altered_c_list]

        self.next(self.end)

    @step
    def end(self):
        """
        end.

        """
        pass

if __name__ == '__main__':
    TestForeachFlow()

But when I ran it it failed at the join step because the attribute self.param is lost. The full trace is show below:

Click to expand
Metaflow 2.0.1 executing TestForeachFlow for user:ji.xu
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
2020-05-22 11:05:25.324 Workflow starting (run-id<PHONE_NUMBER>321423):
2020-05-22 11:05:25.327 [1590170725321423/start/1 (pid 54171)] Task is starting.
2020-05-22 11:05:25.690 [1590170725321423/start/1 (pid 54171)] Task finished successfully.
2020-05-22 11:05:25.715 [1590170725321423/fork/2 (pid 54255)] Task is starting.
2020-05-22 11:05:26.050 [1590170725321423/fork/2 (pid 54255)] Foreach yields 4 child steps.
2020-05-22 11:05:26.050 [1590170725321423/fork/2 (pid 54255)] Task finished successfully.
2020-05-22 11:05:26.057 [1590170725321423/alter_c/3 (pid 54296)] Task is starting.
2020-05-22 11:05:26.080 [1590170725321423/alter_c/4 (pid 54297)] Task is starting.
2020-05-22 11:05:26.086 [1590170725321423/alter_c/5 (pid 54299)] Task is starting.
2020-05-22 11:05:26.092 [1590170725321423/alter_c/6 (pid 54304)] Task is starting.
2020-05-22 11:05:26.417 [1590170725321423/alter_c/3 (pid 54296)] Task finished successfully.
2020-05-22 11:05:26.424 [1590170725321423/alter_c/5 (pid 54299)] Task finished successfully.
2020-05-22 11:05:26.431 [1590170725321423/alter_c/4 (pid 54297)] Task finished successfully.
2020-05-22 11:05:26.438 [1590170725321423/alter_c/6 (pid 54304)] Task finished successfully.
2020-05-22 11:05:26.443 [1590170725321423/join/7 (pid 54357)] Task is starting.
2020-05-22 11:05:26.751 [1590170725321423/join/7 (pid 54357)] <flow TestForeachFlow step join> failed:
2020-05-22 11:05:26.753 [1590170725321423/join/7 (pid 54357)] Internal error
2020-05-22 11:05:26.753 [1590170725321423/join/7 (pid 54357)] Traceback (most recent call last):
2020-05-22 11:05:26.753 [1590170725321423/join/7 (pid 54357)] File "/opt/miniconda/lib/python3.7/site-packages/metaflow/cli.py", line 853, in main
2020-05-22 11:05:26.753 [1590170725321423/join/7 (pid 54357)] start(auto_envvar_prefix='METAFLOW', obj=state)
2020-05-22 11:05:26.753 [1590170725321423/join/7 (pid 54357)] File "/opt/miniconda/lib/python3.7/site-packages/click/core.py", line 764, in __call__
2020-05-22 11:05:26.753 [1590170725321423/join/7 (pid 54357)] return self.main(args, kwargs)
2020-05-22 11:05:26.753 [1590170725321423/join/7 (pid 54357)] File "/opt/miniconda/lib/python3.7/site-packages/click/core.py", line 717, in main
2020-05-22 11:05:26.754 [1590170725321423/join/7 (pid 54357)] rv = self.invoke(ctx)
2020-05-22 11:05:26.754 [1590170725321423/join/7 (pid 54357)] File "/opt/miniconda/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
2020-05-22 11:05:26.754 [1590170725321423/join/7 (pid 54357)] return _process_result(sub_ctx.command.invoke(sub_ctx))
2020-05-22 11:05:26.754 [1590170725321423/join/7 (pid 54357)] File "/opt/miniconda/lib/python3.7/site-packages/click/core.py", line 956, in invoke
2020-05-22 11:05:26.754 [1590170725321423/join/7 (pid 54357)] return ctx.invoke(self.callback, ctx.params)
2020-05-22 11:05:26.754 [1590170725321423/join/7 (pid 54357)] File "/opt/miniconda/lib/python3.7/site-packages/click/core.py", line 555, in invoke
2020-05-22 11:05:26.754 [1590170725321423/join/7 (pid 54357)] return callback(args, kwargs)
2020-05-22 11:05:26.754 [1590170725321423/join/7 (pid 54357)] File "/opt/miniconda/lib/python3.7/site-packages/click/decorators.py", line 27, in new_func
2020-05-22 11:05:26.754 [1590170725321423/join/7 (pid 54357)] return f(get_current_context().obj, args, kwargs)
2020-05-22 11:05:26.801 [1590170725321423/join/7 (pid 54357)] File "/opt/miniconda/lib/python3.7/site-packages/metaflow/cli.py", line 430, in step
2020-05-22 11:05:26.801 [1590170725321423/join/7 (pid 54357)] max_user_code_retries)
2020-05-22 11:05:26.801 [1590170725321423/join/7 (pid 54357)] File "/opt/miniconda/lib/python3.7/site-packages/metaflow/task.py", line 373, in run_step
2020-05-22 11:05:26.801 [1590170725321423/join/7 (pid 54357)] self._exec_step_function(step_func, input_obj)
2020-05-22 11:05:26.801 [1590170725321423/join/7 (pid 54357)] File "/opt/miniconda/lib/python3.7/site-packages/metaflow/task.py", line 49, in _exec_step_function
2020-05-22 11:05:26.801 [1590170725321423/join/7 (pid 54357)] step_function(input_obj)
2020-05-22 11:05:26.801 [1590170725321423/join/7 (pid 54357)] File "99-flow-testing/test-foreach-flow.py", line 73, in join
2020-05-22 11:05:26.802 [1590170725321423/join/7 (pid 54357)] self.foo = self.param.get('foo')
2020-05-22 11:05:26.802 [1590170725321423/join/7 (pid 54357)] File "/opt/miniconda/lib/python3.7/site-packages/metaflow/flowspec.py", line 128, in __getattr__
2020-05-22 11:05:26.802 [1590170725321423/join/7 (pid 54357)] (self.name, name))
2020-05-22 11:05:26.802 [1590170725321423/join/7 (pid 54357)] AttributeError: Flow TestForeachFlow has no attribute 'param'
2020-05-22 11:05:26.802 [1590170725321423/join/7 (pid 54357)]
2020-05-22 11:05:26.804 [1590170725321423/join/7 (pid 54357)] Task failed.
2020-05-22 11:05:26.805 Workflow failed.
2020-05-22 11:05:26.805 Terminating 0 active tasks...
2020-05-22 11:05:26.805 Flushing logs...
    Step failure:
    Step join (task-id 7) failed.

May I know if this behaviour is expected or is it a bug? I can confirm that if I don't have the foreach step and have a linear flow instead, the flow can be ran successfully. I am using metaflow 2.0.1 in Python 3.7.5

@xujiboy This is the expected behavior because you will now have multiple copies of the same param. You can use merge_artifacts as documented in this section - data flow through the graph.

Thank you very much @savingoyal .

Can someone talk about the design decision here? I mean unambiguous variables can be persisted.


Add timeout to requests

No need to wait more than five seconds on a request.

https://github.com/pcarn/vaccine-watch/commit/2e2b3e89a45dd30034a75c28d7be411dd717fd8b


Fixed #590

Description
BulkGetSecret method now returns a map[string]map[string]string
Issue reference
We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.
Please reference the issue this PR will close: #590
Checklist
Please make sure you've completed the relevant tasks for this PR, out of the following list:

[x] Code compiles correctly
[ ] Created/updated tests
[x] Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#1089

hey @rynowak can you please review this? it's about the unflattening of Bulk Get for secrets.

hey @rynowak can you please review this? it's about the unflattening of Bulk Get for secrets.

@yaron2 will need someone with write access to approve, I'm not an approver for this repo 😆

@yaron2 will need someone with write access to approve, I'm not an approver for this repo 😆


Blaze style additionalProperties for Objects?

In my effort to convert my Blaze rules into Bolt rules I'm trying to figure out exactly how a Bolt Object subclass of my own can support my specified properties, as well as additional ones I do not specify. In Blaze this was done by setting additionalProperties=true. Is there an equivalent in Bolt? or what is the strategy if I want an open object with only some of the properties defined/checked?

Or at least allow the rule for $other to be specified.. maybe add an other() { } to a type? or allow 'path /somepath/{other} { validate() { true }}' even when a type is used at /somepath (currently not supported, gets ignored)

Using Bolt types, a node in a Firebase datastore is either a collection (no specifically named property keys), or an Object (doesn't now allow additional un-named properties).
The escape hatch is to use path statements instead of types.  So if you wanted:
// This example is NOT supported in Bolt
type User {
  name: String;
  age: Number;
  $other: String;
}

path /users/{uid} is User;

you could do this:
path /users/{uid} {
   /name is String;
  /age is Number;
  /{other} is String;
}

Another way to approach this is to put all your additional properties in a sub-collection:
type User {
  name: String;
  age: Number;
  other: String[];
}

path /users/{uid} is User;

This changes the structure of your data - but isolates your unstructured data into a single collection.

Would you please consider a solution that supports open types? Throwing away the type system doesn't seem like a reasonable solution to this problem.
Seems reasonable for types to have an option to specify how other, non-specified properties, should be handled. This would give everyone the benefits of the Type system in Bolt, as well as the capabilities we are accustomed to, without making us bend over backwards to work around this deficiency and resort to non-reusable path statements everywhere.
As an example. If the customer pain is not obvious enough yet. Lets say my User type has 15 properties and allows for open ended properties (currently specified using Blaze just fine). Now assume that User type is appropriate at a half-dozen paths in the database. I either have to change how my data is defined because of Bolt (which seems like an anti-goal), or I have to abandon the re-usable type notation and specify the full User spec at every path in my rule system, which also seems like an anti-goal to having the Type system in the first place.
Is forcing types to be closed a really important design feature for Bolt? What does it get us that I'm missing?

I think an open type system is probably a bad idea in most cases - any such system is very difficult to update over time unless you're careful to restrict the class of names that can be used for the "extensible" properties in your types (if you add a named property to an existing type - you may already have old data in the extensible portion that will now overlap unexpectedly).  Using a sub-collection for extensible proprties would be preferable, IMO.
So, we have a goal to dissuade people from using this as a common design pattern.  That said, I take your point that if the developer REALLY WANTS to do this - we should not actively get in the way to make things extra hard.

It might have less to do with a developer really wanting to do something, and more that data and schemas already exist that do this. So maybe Bolt needs a warning that it is not intended as a general-purpose rule syntax for Firebase. There are schemas that are either not-possible, or not-practical to define in Bolt because of some of the design decisions. Bolt then has a highly opinionated way of defining schemas and if your data does not conform, it will be difficult/impossible to use Bolt.
I went into this assuming this was a replacement for Blaze so we can move to a more attractive format instead of YAML. With my current app and data, Bolt won't work. I'll need to change my schema or wait for Bolt to be more flexible before I can use the nicer notation. It is unfortunate, Bolt has a lot of advantages over Blaze.

As a work-around to our limitation on types, you mentioned the path-only design is not tenable since you may have the same type represented in across different locations in the database.
Do you think it would work to use functions instead of types to capture the re-usable portion of your type definition?
isUser(ref) { ref.name.isString() && ref.age.isNumber() }

type UserLike {
  validate() { isUser(this) }
  $other: String;
}

path /x is UserLike;
path /y is UserLike;
path /z is UserLike {
  /karma is Number;
}

which generates:
{
  "rules": {
    "x": {
      ".validate": "newData.hasChildren() && (newData.child('name').isString() && newData.child('age').isNumber())",
      "$other": {
        ".validate": "newData.isString()"
      }
    },
    "y": {
      ".validate": "newData.hasChildren() && (newData.child('name').isString() && newData.child('age').isNumber())",
      "$other": {
        ".validate": "newData.isString()"
      }
    },
    "z": {
      "karma": {
        ".validate": "newData.isNumber()"
      },
      ".validate": "newData.hasChildren() && (newData.child('name').isString() && newData.child('age').isNumber())",
      "$other": {
        ".validate": "newData.isString()"
      }
    }
  }
}

First of all, thanks for the quick responses!
Are suggesting this as a temporary workaround? or as a fundamental adjustment given the design of Bolt? To be candid, I consider both of these workarounds as anti-patterns and if I have to I would rather go through the painful process of changing my schema to fit Bolt than to use these. It would seem to me that part of the goal of Bolt is to be clear and straightforward so it is easy to interpret rules. Given that goal, I would expect that the Type System (the center of that communication) to be a servant of the developer and assist us in communicating clear intent to the Firebase System, as well as the other developers on the team. Communicating what to do with $other properties found on an object seems to me to be a reasonable expectation of this system.
I appreciate your effort in proposing two workarounds to the current version of Bolt. However, I want to be clear so I understand if they are permanent solutions to the issue or not, because I see some serious flaws. They make it difficult and counterintuitive to understand the rules being introduced. Aren't we still early enough with Bolt that we should be evaluating it and verifying it meets the needs of the developers?
From my perspective, the conversation should be about how and when Bolt can be improved to accommodate reasonable use-cases. Or a discussion about the reasonableness of this use-case if that is in question.

Yes, I was mainly suggesting as a temporary work around; but also wanted your feedback (as you've given) on whether they are good enough for you to move forward - or if you really need an update to types sooner rather than later.
I have been viewing Bolt as a test-bed for getting feedback on the design possibilities.  For example, the type system (as opposed to path statements) could be viewed as something that belongs in a separate language as a companion to the access-control rules of the path-statements.
They already have some overlap - so it's occurred to me that the type language might be considered "optional" from a completeness point of view; but I see from your feedback that we don't have the same powerful yet simple constructs w/o types that we do with them.
Thanks for hashing it out with me!


Instantaneous generation, load, ... not available

Thank you for the last update. Unfortunately, since then all "instantaneous" entities show only 0.
Thank you for a short update

That's unexpected - all other measurements are ok?
If you restart home assistant does it fix the issue?  If so that might suggest it's a problem with the API library re-authenticating to Alpha ESS cloud.

Yes - all other measurements are fine. restart does not fix the issue. I will try to re-authenticate! Thank you!!

Re-authentication has not solved the problem - strange

A new version of the API library was published this morning with some alternative code for handling authentication refreshes, I'd be interested to know if the update fixes your issue.

Installed, solved the other problem but not my issue. I saw that, also on AlphaESS online Dashboard, Real-Time Power Graph is not available - maybe this is the reason
Thank you for your help!!

Aha,
Ok if you are not seeing the real time statistics from the official AlphaESS dashboard then that’s a different issue.  My code basically grabs the data that the official dashboard uses so if there’s nothing there to pull I won’t be able to do anything either.
Charles Gillanders | Personal Account
<EMAIL_ADDRESS>Please consider the environment before printing this e-mail
From: pawlikch @.>
Sent: Monday, October 17, 2022 12:30 PM
To: CharlesGillanders/homeassistant-alphaESS @.>
Cc: Charles Gillanders @.>; Comment @.>
Subject: Re: [CharlesGillanders/homeassistant-alphaESS] Instantaneous generation, load, ... not available (Issue #36)
Installed, solved the other problem but not my issue. I saw that, also on AlphaESS online Dashboard, Real-Time Power Graph is not available - maybe this is the reason
Thank you for your help!!
—
Reply to this email directly, view it on GitHubhttps://github.com/CharlesGillanders/homeassistant-alphaESS/issues/36#issuecomment-1280712917, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AMWX2MGNGFVFVYRCENZGLX3WDU2DBANCNFSM6AAAAAARDCOQSE.
You are receiving this because you commented.Message ID<EMAIL_ADDRESS>


Drop buggy TableViewLastColumnResizingFixer class

In Qt 5 the last column resizing with dragging its left edge works out-of-the-box.
The current TableViewLastColumnResizingFixer implementation could put the last column content out of the view port and confuse a user:

Historical context:

https://github.com/bitcoin/bitcoin/pull/2862
https://github.com/bitcoin/bitcoin/pull/3626
https://github.com/bitcoin/bitcoin/pull/3738
https://github.com/bitcoin/bitcoin/pull/3920

#205 is a nice addition.

@promag @jonatack Do you mind reviewing this PR?


Reading an Arrow file with no message batches after the schema seems to produce a partly initialized Table?

julia> t1 = load_t1();
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:294 checking for next arrow message: pos = 1
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:317 parsing message: pos = 217, msglen = 208, bodyLength = 0
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:217 parsing schema message
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:224 parsed column from schema: field = Arrow.Flatbuf.Field(name = "c1", nullable = false, type = Arrow.Flatbuf.Date(unit = Arrow.Flatbuf.DateUnitModule.DAY,), dictionary = nothing, children = Arrow.Flatbuf.Field[], custom_metadata = Arrow.Flatbuf.KeyValue[])
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:224 parsed column from schema: field = Arrow.Flatbuf.Field(name = "c2", nullable = false, type = Arrow.Flatbuf.Utf8(), dictionary = nothing, children = Arrow.Flatbuf.Field[], custom_metadata = Arrow.Flatbuf.KeyValue[])
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:294 checking for next arrow message: pos = 217
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:310 message has 0 length; terminating message parsing

julia> t2 = load_t2();
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:294 checking for next arrow message: pos = 1
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:317 parsing message: pos = 217, msglen = 208, bodyLength = 0
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:217 parsing schema message
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:224 parsed column from schema: field = Arrow.Flatbuf.Field(name = "c1", nullable = false, type = Arrow.Flatbuf.Date(unit = Arrow.Flatbuf.DateUnitModule.DAY,), dictionary = nothing, children = Arrow.Flatbuf.Field[], custom_metadata = Arrow.Flatbuf.KeyValue[])
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:224 parsed column from schema: field = Arrow.Flatbuf.Field(name = "c2", nullable = false, type = Arrow.Flatbuf.Utf8(), dictionary = nothing, children = Arrow.Flatbuf.Field[], custom_metadata = Arrow.Flatbuf.KeyValue[])
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:294 checking for next arrow message: pos = 217
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:317 parsing message: pos = 425, msglen = 200, bodyLength = 136
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:252 parsing record batch message: compression = nothing
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:294 checking for next arrow message: pos = 561
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:310 message has 0 length; terminating message parsing
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:335 building top-level column: field = Arrow.Flatbuf.Field(name = "c1", nullable = false, type = Arrow.Flatbuf.Date(unit = Arrow.Flatbuf.DateUnitModule.DAY,), dictionary = nothing, children = Arrow.Flatbuf.Field[], custom_metadata = Arrow.Flatbuf.KeyValue[]), columnidx = 1, nodeidx = 1, bufferidx = 1
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:520 building array: L = Arrow.Flatbuf.Date
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:527 storage type for primitive: T = Arrow.Date{Arrow.Flatbuf.DateUnitModule.DAY, Int32}
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:531 final julia type for primitive: T = Date
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:337 built top-level column: A = Arrow.Primitive{Date, Vector{Arrow.Date{Arrow.Flatbuf.DateUnitModule.DAY, Int32}}}, columnidx = 1, nodeidx = 2, bufferidx = 3
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:335 building top-level column: field = Arrow.Flatbuf.Field(name = "c2", nullable = false, type = Arrow.Flatbuf.Utf8(), dictionary = nothing, children = Arrow.Flatbuf.Field[], custom_metadata = Arrow.Flatbuf.KeyValue[]), columnidx = 2, nodeidx = 2, bufferidx = 3
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:408 building array: L = Arrow.Flatbuf.Utf8()
DEBUG: ~/.julia/packages/Arrow/Re9EM/src/table.jl:337 built top-level column: A = Arrow.List{String, Int32, Vector{UInt8}}, columnidx = 2, nodeidx = 3, bufferidx = 6

julia> t1
Arrow.Table: Error showing value of type Arrow.Table:
ERROR: KeyError: key :c1 not found
Stacktrace:
  [1] getindex
    @ ./dict.jl:482 [inlined]
  [2] getcolumn
    @ ~/.julia/packages/Arrow/Re9EM/src/table.jl:175 [inlined]
  [3] #1
    @ ./none:0 [inlined]
  [4] iterate
    @ ./generator.jl:47 [inlined]
  [5] collect(itr::Base.Generator{Vector{Symbol}, Tables.var"#1#2"{Arrow.Table}})
    @ Base ./array.jl:678
  [6] _totuple
    @ ./tuple.jl:331 [inlined]
  [7] Tuple
    @ ./tuple.jl:303 [inlined]
  [8] NamedTuple(r::Arrow.Table)
    @ Tables ~/.julia/packages/Tables/8FVkV/src/Tables.jl:190
  [9] show(io::IOContext{Base.TTY}, x::Arrow.Table)
    @ Tables ~/.julia/packages/Tables/8FVkV/src/Tables.jl:196
 [10] show(io::IOContext{Base.TTY}, #unused#::MIME{Symbol("text/plain")}, x::Arrow.Table)
    @ Base.Multimedia ./multimedia.jl:47

julia> t2
Arrow.Table: (c1 = [...], c2 = [...])

I've tried to put together the above demonstration, though I can't share the exact files unfortunately. The first example, t1, has no rows / message batches, but otherwise has the same schema as t2. t1 errors when trying to access any column (or show it, etc), but t2 is fine.
Arrow.schema(t1) looks healthy, and identical to Arrow.schema(t2).
Both files read fine with pyarrow.
If you're able to make sense of it, it would be very appreciated. Thanks!
Versions:
julia 1.6.0
Arrow v1.2.4

A reproduce is actually quite easy. I think this is close to the Arrow message I deal with - schema/metadata, but zero record batches:
import Arrow
import PyCall
pyarrow = PyCall.pyimport("pyarrow")

# write in julia
io = IOBuffer()
t = (;a=Int[])
Arrow.write(seekstart(io), t)
jl_buf = read(seekstart(io))
length(jl_buf) # 256

# write in python
py_data = [pyarrow.array([], type=pyarrow.int64())]
batch = pyarrow.record_batch(py_data, names=["a"])
sink = pyarrow.BufferOutputStream()
writer = pyarrow.ipc.new_stream(sink, batch.schema)
writer.close()
py_buf = collect(reinterpret(UInt8, sink.getvalue())) # comes as Vector{Int8} for some reason
length(py_buf) # 136

# Reading back in python - all 3 of the following work:
pyarrow.ipc.open_stream(py_buf).read_pandas()
pyarrow.ipc.open_stream(jl_buf).read_pandas()
pyarrow.ipc.open_stream(jl_buf[1:128]).read_pandas() # Turns out you only need first 128 bytes (metadata is repeated)

# Reading back in julia: can't read python
Arrow.Table(jl_buf) # reads fine
Arrow.Table(py_buf) # errors

# Arrow.Table: ERROR: KeyError: key :a not found
# Stacktrace:
#   [1] getindex
#     @ ./dict.jl:482 [inlined]
#   [2] getcolumn
#     @ ~/.julia/packages/Arrow/Re9EM/src/table.jl:175 [inlined]
#   [3] #1
#     @ ./none:0 [inlined]
#   [4] iterate
#     @ ./generator.jl:47 [inlined]
#   [5] collect(itr::Base.Generator{Vector{Symbol}, Tables.var"#1#2"{Arrow.Table}})
#     @ Base ./array.jl:678
#   [6] _totuple
#     @ ./tuple.jl:331 [inlined]
#   [7] Tuple
#     @ ./tuple.jl:303 [inlined]
#   [8] NamedTuple(r::Arrow.Table)
#     @ Tables ~/.julia/packages/Tables/8FVkV/src/Tables.jl:190
#   [9] show(io::IOContext{Base.TTY}, x::Arrow.Table)

julia> pyarrow.__version__
"1.0.1"

Digging in now (sorry for the slow reply); it looks like in Arrow.jl, we're writing an empty record batch whereas pyarrow doesn't.

Ok, fix is up: https://github.com/JuliaData/Arrow.jl/pull/175

Thanks! This fixes the example I gave above, but seems to choke when the field supplies custom_metadata. Here's an example:
import Arrow
import PyCall
pyarrow = PyCall.pyimport("pyarrow")

py_data = [pyarrow.array([], type=pyarrow.date32())]
my_schema = pyarrow.schema([
    pyarrow.field("a", pyarrow.date32(), false, metadata=Dict("asd" => "asd")),
])
batch = pyarrow.record_batch(py_data, schema=my_schema)
sink = pyarrow.BufferOutputStream()
writer = pyarrow.ipc.new_stream(sink, batch.schema)
writer.close()
py_buf = collect(reinterpret(UInt8, sink.getvalue())) # comes as Vector{Int8} for some reason
length(py_buf) # 136

Arrow.Table(py_buf)

ERROR: LoadError: MethodError: no method matching juliaeltype(::Arrow.Flatbuf.Field, ::Arrow.FlatBuffers.Array{Arrow.Flatbuf.KeyValue, UInt32, Arrow.Flatbuf.Field}, ::Bool)
Closest candidates are:
  juliaeltype(::Arrow.Flatbuf.Field, ::Union{Arrow.Flatbuf.Binary, Arrow.Flatbuf.LargeBinary}, ::Any) at ~/.julia/packages/Arrow/u4Ye6/src/eltypes.jl:136
  juliaeltype(::Arrow.Flatbuf.Field, ::Union{Arrow.Flatbuf.LargeUtf8, Arrow.Flatbuf.Utf8}, ::Any) at ~/.julia/packages/Arrow/u4Ye6/src/eltypes.jl:131
  juliaeltype(::Arrow.Flatbuf.Field, ::Union{Arrow.Flatbuf.LargeList, Arrow.Flatbuf.List}, ::Any) at ~/.julia/packages/Arrow/u4Ye6/src/eltypes.jl:380
  ...
Stacktrace:
 [1] Arrow.Table(bytes::Vector{UInt8}, off::Int64, tlen::Nothing; convert::Bool)
   @ Arrow ~/.julia/packages/Arrow/u4Ye6/src/table.jl:278
 [2] Table (repeats 2 times)
   @ ~/.julia/packages/Arrow/u4Ye6/src/table.jl:191 [inlined]

Ah, thanks for checking that out. There was something in my brain that was telling me that just passing f.custom_metadata wasn't quite right, but I didn't take enough time to remember that we need to convert arrow metadata from a Vector{Pair{String, String}} to a Dict{String, String}. I just pushed a commit that should fix it.


Fix incorrect insets for sections

🐛 I think the code mostly explains it, but it was using top/leading to add to the "end" of the size when calculating frames for decorations.
This is definitely an easy one to miss! In most of the code samples (from Apple, etc), the insets are the same on all edges, but I have to use different insets for top/bottom (and leading/trailing) for the layout I'm building, and this patch lays it out correctly 🙌

Thank you for your contribution!


Feature Request: Allow config of the "add archive" action from the web UI

Type

[ ] General question or discussion
[ ] Propose a brand new feature
[X] Request modification of existing behavior or design

What is the problem that your feature request solves
In some cases an end user may want to just grab the WARC or just the PDF to be archived from a given site. Similar to the CLI, it would be handy if you could override the current config values on the page where you add a URL for archival.
Describe the ideal specific solution you'd want, and whether it fits into any broader scope of changes
Ideally this would just be a small dropdown/hidden menu to "advanced options" or something similar with checkboxes for each archival type or other config value that are then passed to the backend to override the default config values.
What hacks or alternative solutions have you tried to solve the problem?
Nothing so far as it seems like a pretty quick change. I'll look into working on the change myself.
How badly do you want this new feature?

[ ] It's an urgent deal-breaker, I can't live without it
[X] It's important to add it in the near-mid term future
[ ] It would be nice to have eventually

[X] I'm willing to contribute dev time / money to fix this issue
[X] I like ArchiveBox so far / would recommend it to a friend
[ ] I've had a lot of difficulty getting ArchiveBox set up

Thanks again for this @BlipRanger! I added a credit to you in the release notes of v0.5.0 https://github.com/ArchiveBox/ArchiveBox/pull/552#issue-528926837
Don't hesitate to open a ticket for any other suggestions in the future!


docs: add section about custom REST handlers

Adding section about custom REST handlers after question here: https://splunk-usergroups.slack.com/archives/C03SG3ZL4S1/p1662455838235229

Codecov Report

Merging #517 (77749c2) into main (be3fb76) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #517   +/-   ##
=======================================
  Coverage   79.66%   79.66%
=======================================
  Files          33       33
  Lines        1770     1770
=======================================
  Hits         1410     1410
  Misses        360      360

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:tada: This PR is included in version 5.14.0 :tada:
The release is available on:

v5.14.0
GitHub release

Your semantic-release bot :package::rocket:


Update enumeration_units.py

Error messages on Place retrieval were vague / incorrect, wanted to clarify

thanks!  I'll merge these PRs shortly.


ngnix sets Content-type even though \Slim\App finalize removes it

During testing, I found that I was getting a "Content-Type: text/html; charset=UTF-8" header being sent when I was creating a 204 No Content page (in Slim 3 RC 2 within \Slim\App::finalize and \Slim\App::isEmptyResponse it removes the Content-length and Content-type headers if the response code is 204, 205 or 304).
Obviously, this isn't a fault in Slim "per-se" but an unexpected "feature" of nginx/1.8.0 running under fastcgi. I have managed to remove the header myself using an empty Content-type header setting like:
header('Content-type:',true);

would it be an idea of Slim to "sniff" for nginx/fastcgi and do the same if detected (I haven't been able to test this "fix" on other web servers to see if they will just strip out the empty Content-type header or not).

are you sure that you are not writing to the browser?
If you say do a print "" or echo "" it will over write the content type header.

Yep, I'm, sure. I've just done new test case which consists of a single file:
<?php
// file /var/www/test/index.php
require __DIR__.'/vendor/autoload.php';

$app=new \Slim\App();
$app->get('/204test',function ($request,$response,$arg) {
    $response = $response->withStatus(204, 'No Content');
    return $response;
});
$app->run();

with my nginx configuration being:
server {
    listen 88;
    server_name "";
    root   /var/www/test/;

    location / {
        try_files $uri $uri/ /index.php$is_args$args;
    }

    include php.conf;

}

and /etc/nginx/php.config being:
location ~ \.php {
        include                  fastcgi_params;
        fastcgi_keep_conn on;
        fastcgi_index            index.php;
        fastcgi_split_path_info  ^(.+\.php)(/.+)$;
        fastcgi_param PATH_INFO $fastcgi_path_info;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_intercept_errors on;
        fastcgi_pass             unix:/var/run/php-fpm.sock;
    }

and a test curl transaction is:
vagrant@host:/vagrant/project$ curl --raw http://<IP_ADDRESS>:88/204test -v
* Hostname was NOT found in DNS cache
*   Trying <IP_ADDRESS>...
* Connected to <IP_ADDRESS> (<IP_ADDRESS>) port 88 (#0)
> GET /204test HTTP/1.1
> User-Agent: curl/7.38.0
> Host: <IP_ADDRESS>:88
> Accept: */*
>
< HTTP/1.1 204 No Content
* Server nginx/1.8.0 is not blacklisted
< Server: nginx/1.8.0
< Date: Tue, 24 Nov 2015 16:47:22 GMT
< Content-Type: text/html; charset=UTF-8
< Connection: keep-alive
< X-Powered-By: PHP/7.0.0-dev
<
* Connection #0 to host <IP_ADDRESS> left intact

I believe nginx might be adding the content-type header because of this little nugget...
The 204 response MUST NOT include a message-body, and thus is always terminated by the first empty line after the header fields.

Your browser would lose the extra headers after content-type if it were broken.
You can test this by manually adding a Content-Type header to your response. NGinx should let your's go through.

I've amended the test script to add a
$response=$response->withHeader('Content-type','text/plain');

but if the status code is 204, Slim (in App->finalize) still strips it out: causing ngnix to add the text/html Content-type back in (I've even tried add body content to see if it makes a difference: it doesn't) [changing the status code to 200 does give my text/plain content type and the body].
Changing App->finalize to:
 if ($this->isEmptyResponse($response)) {
            return $response->withoutHeader('Content-Type')->withoutHeader('Content-Length')->withHeader('Content-Type','');
}

did work (i.e. no Content-Type header was actually sent by nginx).
(as an aside, /etc/nginx/nginx.conf has application/octet-stream as the default_type )

I'm confused. The spec for 204 states:

The 204 response MUST NOT include a message-body, and thus is always terminated by the first empty line after the header fields.

Surely that means that there cannot be a Content-Type or a Content-Length header?

That is my understanding as well.

@akrabat That's how I read it as well. I believe nginx is trying to be "helpful" and if there is absolutely no Content-type header set, it is adding one - however, if a Content-type header is set, but empty, "it trust you know what you are doing" but won't send the blank content-type (but also will not add its own content-type)

Right, so really we should look at seeing if we can let you manipulate the response after finalize() has run.

Just doing some tests on an Apache server and the inbuilt PHP server with a very very basic script:

No Content-type setting at all: Apache, PHP and nginx all add Content-Type: text/html
With Content-Type setting: All provide the coded Content-Type
With Content-Type:'' (empty string): Apache, PHP Web server and Nginx do NOT provide a Content-Type heading.

It's looking more like, to achieve what Slim is trying to do by unsetting the Content-Type header, it is better off setting a blank content header (similar to "header('Content-Type:');") which will stop the content-type line being sent on all three tested servers. Counter intuitive, but works on two of the major web servers and the inbuilt one. Is there a platform where this "hack" doesn't work?

This is happening because of the php.ini default_mimetype setting:

By default, PHP will output a character encoding using the Content-Type header. To disable sending of the charset, simply set it to be empty.

Fixed by #1629.


Lagging behavior when clicking a point

As in #7, clicking a point will "freeze" the view for a few seconds, which is not friendly.

Solved with Object.freeze() and shadowRef


pyclass: allow #[pyo3(get, set, name = "foo")]

For #1601
This implements #[pyo3(get, set, name = "...")] to set the python name of the exposed property.
This enables #[pyo3(get, set)] for tuple #[pyclass], by making name required in that case. Closes #1593

👍 I'll try to add some extra coverage before merging!

So it looks like I've got coverage for the lines which codecov claims aren't covered, but it's in UI tests where the coverage isn't currently measured. I'd love to fix that some day, but for now I'll just merge this.


SVProgressHUD : Sometimes does not show up during Back Navigation.

Problem:
Sometimes during Back Navigation of screens in the controller hierarchy SVProgressHUD does not show up. But Once the controller popped from Navigation stack and deallocated, next time it starts to show up again.

Closing this issue since this was a code error where SVProgressHUD's containerView not reset. This should be done since we are using SVProgressHUD as singleton.


Local organization/grouping/labeling of repos

Closes #5781. The final PR for repo overview improvements. (to be merged after #5790)
A quick video overview :)
https://user-images.githubusercontent.com/15635498/198895429-6c6ec741-c7ce-4af1-a412-34fbc0e6ee2e.mp4

Main feature:

possibility to add labels to repos locally, they will display in repo overview
filtering by label is possible and easily accessible from the top tool bar. filtering is technically integrated in the common filtering logic
label list respects favorites show/hide mode; also it respects "no labels" mode and hides the column respectively
labels can be colored. This is configured in AG settings per user (hmmm ... maybe it should be global?)

colors can be chosen from predefined (~20) styles
or custom #fg_rgb/bg_rgb format

docs are updated as well - https://github.com/abapGit/docs.abapgit.org/pull/53

Technically:

labels attr in local_settings structure
parsing/validation utils for labels and colors are in zcl_abapgit_repo_labels (UT covered)
labels are rendered by chunk lib render_label_list - rather universably and can be re-used. Slight visualization difference if repo list is rendered within a table
in CSS:

color classes start from rl-
repo label list classes: repo-labels and repo-label-catalog
help hint (below) - form-field-help-tooltip

side features:

html->a/add_a accepts iv_query - to specify ?query in actions
a new feature render_help_hint in chunk lib to render hoverable help hint (supposedly to be used in forms for more detailed help - see example in Personal settings for label colors in the screen above)
lcl_table_scheme helper in repo overview - a candidate for a separate class, potentially a part of table rendering component

Avoid "Tag", indeed. How about "Category"? zcl_abapgit_repo_categories, package $abapgit_repo (or "Group")

Or maybe "label".

Thanks guys ! I was thinking about "group" as well, but label looks like the match ! I'll rename it to label. Maybe move the util class to utils package too.

No styling yet, but already fun to play with :)

Hmmm :)

just thinking about the table layout. maybe "Labels" becomes a checkmark like "Details". Also if selecting "Labels" or "Details" could hide the URL and maybe branch as well to make room. The labels could then be put next to each other - comma separated.

They will, I didn't do any styling yet as I mentioned. It will be a typical bordered text items. Probably.
Hiding other fields - probably not. All the current columns are important. But the labels (the column) are hidden if there are no of them.

OK, I guess it's ready for review. The first post also updated with screenshots, user and tech details and a video :)

@mbtools can you provide details why to avoid "tag"? The SAP Community uses Tags. In SAP BTP Labels are used as a key / value combination:

@gregorwolf tag makes sense in general but in git clients tags have a different meaning. So sticking to labels for categorizing things like in GitHub or Gitlab looks good to me.

Overall, this is an awesome enhancement. Thank you!

Overview

Can we move the label column to the right? Feels much better there and less awkward if they appear after adding them.

The screen is not too crowded but I can image a "Labels on/off" switch like for "Details" could make sense esp. if you use many or longer labels.

Colors

Initially, all labels are white. Some default colors would be nice to have.

The predefined colors are not that intuitive and don't match the predefined CSS names.

Do we really need labels that match the fg or bg colors? Why not use the standard CSS colors https://www.w3schools.com/cssref/css_colors.php? Then one color name would be sufficient instead of fg/bg (based on the colors of that particular page).

Settings

With just a few labels, the input box becomes too small. Let's try textarea and just call it "Label Colors".

Very cool!

Thanks for the detailed comments.

"Can we move the label column to the right" and "Labels on/off"

I think it is a matter of preference. E.g for me the most important columns are: on/offline, repo name, branch and now labels. They should be on the left, others can be hidden.
As for labels - they are a grouping factor, thus imho they should be close to the name.
I don't think moving column to the right and hiding option would be "good for all" solution. "Good for all" solution is a mechanism to completely customize order and visible fields. This would solve all of that with one shot. But not in this PR.

"Initially, all labels are white."

This is intentional. I wanted to keep them neutral and non-distractive until the user intentionally want to change that.

Color names.

Not sure I understood the idea 100%. To admit, I also feel that there is some room for improvement in the color naming. But ... how to express it properly ... I spent quite some iterations on coloring to get to the current system so if it has to be changed it should be more systematic and more well-thought than now and has to take all previous "rakes" into account. Namely:

initially I though to implement just FG rgb color.
but then I though also to include html colors (the ones you mentioned)
but then it appeared ugly, because it changes just fg and labels were not that distinguishable
and I came to fg/bg pairs
and so I got rid of html color because:

A) they were not designed to match together (fg/bg)
B) they don't help unless you're a CSS expert remembering them
C) coloring, especially fg-bg relation goes well with HSL color concept, which has nothing to do with predefined html colors

also I got to the understanding that assigning color pairs, especially in abap, is messy, has poor readability and usability
in fact, now I'm inclined to remove the fg/bg rgb pairs at all ... they are so messy ... so after a couple of iterations and practical experiment I got to understanding that my initial idea was bad. (I left it to have this possibility but I personally will use styles below)
so, instead fg/bg pairs, I decided to give user a small, but easy and pretty predefined choice with CSS styles
... which also have border style, which looks nicer (so we have already 3 color components)
... which is rather easily extendable by later PRs (rl- css styles)
to compare, most of services give a predefined, rather small, yet well balanced choice (github, gmail ...)
designing the style set (partially inspired by existing color set in gmail theme) I got to the conclusion that white and dark backgrounds naturally goes well with several colors. Thus dark-w/y/r/b were added and a couple of other tints.
so the naming does not represent a single color ...
finally, and importantly. Does anyone really cares about the color names ? The user just wants to choose a color. The fact is that current approach (with entering label-color pairs in text) is not too convenient and dictated more by technical constrains. The best solution would be a palette picker, with no names, but just a color. This is possible, but needs a lot of JS or multistep, potentially inconvenient, step-by-step abap procedure. Which is ... not in this PR.

Sorry for long text. Just wanted to explain that current approach is a product of rather long way.

Textarea in setting

Yes, good idea. Done :)

(hmm, but now it has ugly overflow ... maybe I should render comma separator with a space after it ... but next PR)

It's possible that we are overthinking the colors a bit 😉
GH labels are defined by one, randomized #rrggbb. Simple, works perfectly well.

Could be, could be ... though I wouldn't harry with "simple". All colors are of limited saturation range, auto bright/dark foreground, text readability seems to be also considered ... I wonder how much engineering and attempts were behind ...

Also, there is no limit for perfection and further improvements ... ;P

🥳


Feature Request: "Paid /w Demo" category?

There's a demo category and a paid category, perhaps there can be a paid + demo category so an app shows up in both? We offer the demo and paid app on the same itch.io listing.

Thanks for responding. I think the end result I'd like is a "Paid w/ Demo" label in the listings, not necessarily that the app shows up as free.


JSON.isEmpty returns true by default.

Not sure if its a bug or not, but the comment and actual results are different

comments says that by default it returns false, but it actually returns true. This behavior also differs from the past versions.

Missed comment, we do return true at this situation.

https://github.com/SwiftyJSON/SwiftyJSON/commit/5f5d00dbdcd256600a250765d9f674964080ceee

Hey, Its me again. Somehow I bumped again into this. What is the reasoning behind this behaviour ? If an object is a String or some other type - it can still be non empty. But now if an object is non-dict and non-array, it automatically counts as empty even tho it is not.

I created a PR that fixes it. Hope it makes sense. It did fix my problem.
https://github.com/SwiftyJSON/SwiftyJSON/pull/464

@lingoer By any chance someone could explain to me this behaviour ?

It feel to me this method is really confusing to people, and also myself.
And it's not add enough value for us to clarify and document this method.
Also every team might have different definition of empty, so maybe this should not be part of the SwiftyJSON, but rather a simple extension on the actual product code.


Simplify Windows Start Meny entry

The current Start Menu entry is unnecessarily in a folder. This seems a bit unclean and frustrates me. I would recommend removing the folder, simply having the .lnk file.
Current:

Suggestion:

Minor cleanup

Could my PR be considered or could there be made another solution? The folder is not used for anything else. While Windows 11 automatically just shows the icon, Windows 10 does not. And it could generally be a good practice to keep the file structure as clean as possible.


Remove aggregator's 10ms sleeping and add breakdown of aggregator for performance improvment

This PR is to

remove aggregator's 10ms sleeping for performance improvement
add breakdown of aggregator for performance improvement analysis

The codes have been tested successfully in AWS integration test environment (500K nodes / 20 schedulers).

please close this PR. once we moved to the LW approach, this is no longer needed.


Support PATCH http operation

A patch-operation should be implemented to support documentation of patch-operations.
Maybe something like this?
@ContextDsl
inline fun <reified LOCATION : Any, reified ENTITY : Any> Route.patch(
    metadata: Metadata,
    noinline body: suspend PipelineContext<Unit, ApplicationCall>.(LOCATION, ENTITY) -> Unit
): Route {
    application.swaggerUi.apply {
        metadata.apply<LOCATION, ENTITY>(HttpMethod.Patch)
    }

    return patch<LOCATION> {
        body(this, it, call.receive())
    }
}

PRs are welcome!

Ok, will attempt one later today.

@JLLeitschuh , PR ready for review


chore: add Node.js 10

@DanielRuf Thank you for this PR.
Since this will launch two simultaneous builds on Travis every commit we need to check Sauce Labs behavior.
Maybe we can go with 10 only.
@marvinhagemeister What's your view on this?

Since this will launch two simultaneous builds on Travis every commit we need to check Sauce Labs behavior.
Maybe we can go with 10 only.

We can also use the build matrix feature with different env settings and run Saubelabs just on one of both.

Coverage remained the same at 100.0% when pulling 682ab53af1044382a9efe130d539b148d0e577a1 on DanielRuf:chore/add-nodejs-10 into 7f0ee264d866ef09af2e737b877ba48db2288021 on developit:master.