[BUG] Egeria Metadata Server crashes silently while processing massive amount of elements

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
While processing 500+ elements (KafkaTopics from StrimziMonitorIntegrationConnector) the Egeria Metadata Server crashes silently. Last few lines of log:
Tue Aug 16 12:02:43 GMT 2022 mds1 Event OMRS-AUDIT-8009 The Open Metadata Repository Services (OMRS) has sent event of type Instance Event to the cohort topic egeria.omag.openmetadata.repositoryservices.cohort.mds.OMRSTopic.instances
Tue Aug 16 12:02:43 GMT 2022 mds1 Event OMAS-SECURITY-MANAGER-0013 The Security Manager Open Metadata Access Service (OMAS) has sent event of type: New Element
Tue Aug 16 12:02:46 GMT 2022 mds1 Event OMRS-AUDIT-8009 The Open Metadata Repository Services (OMRS) has sent event of type Instance Event to the cohort topic mds1.openmetadata.repositoryservices.enterprise.mds1.OMRSTopic
Tue Aug 16 12:02:46 GMT 2022 mds1 Event OMRS-AUDIT-8009 The Open Metadata Repository Services (OMRS) has sent event of type Instance Event to the cohort topic egeria.omag.openmetadata.repositoryservices.cohort.mds.OMRSTopic.instances
Tue Aug 16 12:02:46 GMT 2022 mds1 Event OMAS-GOVERNANCE-ENGINE-0019 The Governance Engine Open Metadata Access Service (OMAS) has sent event of type Metadata change event
Tue Aug 16 12:02:46 GMT 2022 mds1 Information OMAS-GOVERNANCE-ENGINE-0017 The Governance Engine Open Metadata Access Service (OMAS) sent a metadata change event to listening Open Watchdog Governance Action Services: New Relationship
Tue Aug 16 12:02:46 GMT 2022 mds1 Event OMAS-DATA-MANAGER-0013 The Data Manager Open Metadata Access Service (OMAS) has sent event of type: New Element
Tue Aug 16 12:02:46 GMT 2022 mds1 Event OMAS-ASSET-MANAGER-0013 The Asset Manager Open Metadata Access Service (OMAS) has sent event of type: New Element
Tue Aug 16 12:02:48 GMT 2022 mds1 Event OMRS-AUDIT-8009 The Open Metadata Repository Services (OMRS) has sent event of type Instance Event to the cohort topic mds1.openmetadata.repositoryservices.enterprise.mds1.OMRSTopic
Tue Aug 16 12:02:48 GMT 2022 mds1 Event OMAS-STEWARDSHIP-ACTION-0012 The Stewardship Action Open Metadata Access Service (OMAS) has sent event of type: Element Declassified
Tue Aug 16 12:02:48 GMT 2022 mds1 Event OMRS-AUDIT-8009 The Open Metadata Repository Services (OMRS) has sent event of type Instance Event to the cohort topic egeria.omag.openmetadata.repositoryservices.cohort.mds.OMRSTopic.instances
Tue Aug 16 12:02:48 GMT 2022 mds1 Event OMAS-IT-INFRASTRUCTURE-0013 The IT Infrastructure Open Metadata Access Service (OMAS) has sent event of type: Element Declassified
Tue Aug 16 12:02:48 GMT 2022 mds1 Event OMAS-ASSET-CONSUMER-0012 The Asset Consumer Open Metadata Access Service (OMAS) has sent event of type: Element Declassified
Tue Aug 16 12:02:48 GMT 2022 mds1 Event OMAS-GOVERNANCE-ENGINE-0019 The Governance Engine Open Metadata Access Service (OMAS) has sent event of type Metadata change event
Tue Aug 16 12:02:48 GMT 2022 mds1 Information OMAS-GOVERNANCE-ENGINE-0017 The Governance Engine Open Metadata Access Service (OMAS) sent a metadata change event to listening Open Watchdog Governance Action Services: New Classification
Tue Aug 16 12:02:48 GMT 2022 mds1 Event OMAS-DIGITAL-ARCHITECTURE-0012 The Digital Architecture Open Metadata Access Service (OMAS) has sent event of type: Element Declassified
Tue Aug 16 12:02:48 GMT 2022 mds1 Event OMAS-SECURITY-MANAGER-0013 The Security Manager Open Metadata Access Service (OMAS) has sent event of type: Element Declassified
Tue Aug 16 12:02:49 GMT 2022 mds1 Event OMAS-ASSET-OWNER-0012 The Asset Owner Open Metadata Access Service (OMAS) has sent event of type: Element Declassified
Tue Aug 16 12:02:49 GMT 2022 mds1 Event OMAS-DATA-MANAGER-0013 The Data Manager Open Metadata Access Service (OMAS) has sent event of type: Element Classified
Tue Aug 16 12:02:49 GMT 2022 mds1 Event OMAS-ASSET-MANAGER-0013 The Asset Manager Open Metadata Access Service (OMAS) has sent event of type: Element Declassified
Tue Aug 16 12:02:49 GMT 2022 mds1 Event OMAS-STEWARDSHIP-ACTION-0012 The Stewardship Action Open Metadata Access Service (OMAS) has sent event of type: New Element
Tue Aug 16 12:02:49 GMT 2022 mds1 Event OMAS-IT-INFRASTRUCTURE-0013 The IT Infrastructure Open Metadata Access Service (OMAS) has sent event of type: New Element
Tue Aug 16 12:02:49 GMT 2022 mds1 Event OMAS-ASSET-CONSUMER-0012 The Asset Consumer Open Metadata Access Service (OMAS) has sent event of type: New Element
Tue Aug 16 12:02:49 GMT 2022 mds1 Event OMAS-DIGITAL-ARCHITECTURE-0012 The Digital Architecture Open Metadata Access Service (OMAS) has sent event of type: New Element
Tue Aug 16 12:02:49 GMT 2022 mds1 Event OMAS-ASSET-OWNER-0012 The Asset Owner Open Metadata Access Service (OMAS) has sent event of type: New Element
Tue Aug 16 12:02:49 GMT 2022 mds1 Event OMAS-DATA-MANAGER-0013 The Data Manager Open Metadata Access Service (OMAS) has sent event of type: New Element
Tue Aug 16 12:02:49 GMT 2022 mds1 Event OMAS-ASSET-MANAGER-0013 The Asset Manager Open Metadata Access Service (OMAS) has sent event of type: New Element
Tue Aug 16 12:02:50 GMT 2022 mds1 Event OMRS-AUDIT-8009 The Open Metadata Repository Services (OMRS) has sent event of type Instance Event to the cohort topic mds1.openmetadata.repositoryservices.enterprise.mds1.OMRSTopic
Tue Aug 16 12:02:50 GMT 2022 mds1 Event OMAS-GOVERNANCE-ENGINE-0019 The Governance Engine Open Metadata Access Service (OMAS) has sent event of type Metadata change event
Tue Aug 16 12:02:50 GMT 2022 mds1 Information OMAS-GOVERNANCE-ENGINE-0017 The Governance Engine Open Metadata Access Service (OMAS) sent a metadata change event to listening Open Watchdog Governance Action Services: New Element
Tue Aug 16 12:02:50 GMT 2022 mds1 Event OMRS-AUDIT-8009 The Open Metadata Repository Services (OMRS) has sent event of type Instance Event to the cohort topic egeria.omag.openmetadata.repositoryservices.cohort.mds.OMRSTopic.instances
Tue Aug 16 12:02:50 GMT 2022 mds1 Event OMAS-SECURITY-MANAGER-0013 The Security Manager Open Metadata Access Service (OMAS) has sent event of type: New Element
Tue Aug 16 12:02:53 GMT 2022 mds1 Event OMRS-AUDIT-8009 The Open Metadata Repository Services (OMRS) has sent event of type Instance Event to the cohort topic mds1.openmetadata.repositoryservices.enterprise.mds1.OMRSTopic
Tue Aug 16 12:02:53 GMT 2022 mds1 Event OMRS-AUDIT-8009 The Open Metadata Repository Services (OMRS) has sent event of type Instance Event to the cohort topic egeria.omag.openmetadata.repositoryservices.cohort.mds.OMRSTopic.instances
Tue Aug 16 12:02:53 GMT 2022 mds1 Event OMAS-GOVERNANCE-ENGINE-0019 The Governance Engine Open Metadata Access Service (OMAS) has sent event of type Metadata change event
Tue Aug 16 12:02:53 GMT 2022 mds1 Information OMAS-GOVERNANCE-ENGINE-0017 The Governance Engine Open Metadata Access Service (OMAS) sent a metadata change event to listening Open Watchdog Governance Action Services: New Relationship
Tue Aug 16 12:02:53 GMT 2022 mds1 Event OMAS-DATA-MANAGER-0013 The Data Manager Open Metadata Access Service (OMAS) has sent event of type: New Element
Tue Aug 16 12:02:53 GMT 2022 mds1 Event OMAS-ASSET-MANAGER-0013 The Asset Manager Open Metadata Access Service (OMAS) has sent event of type: New Element
Tue Aug 16 12:02:53 GMT 2022 mds1 Event OMAS-STEWARDSHIP-ACTION-0012 The Stewardship Action Open Metadata Access Service (OMAS) has sent event of type: Element Declassified
Tue Aug 16 12:02:53 GMT 2022 mds1 Event OMAS-IT-INFRASTRUCTURE-0013 The IT Infrastructure Open Metadata Access Service (OMAS) has sent event of type: Element Declassified
Tue Aug 16 12:02:53 GMT 2022 mds1 Event OMAS-ASSET-CONSUMER-0012 The Asset Consumer Open Metadata Access Service (OMAS) has sent event of type: Element Declassified
Tue Aug 16 12:02:53 GMT 2022 mds1 Event OMAS-DIGITAL-ARCHITECTURE-0012 The Digital Architecture Open Metadata Access Service (OMAS) has sent event of type: Element Declassified
Tue Aug 16 12:02:53 GMT 2022 mds1 Event OMAS-GOVERNANCE-ENGINE-0019 The Governance Engine Open Metadata Access Service (OMAS) has sent event of type Metadata change event
Tue Aug 16 12:02:53 GMT 2022 mds1 Information OMAS-GOVERNANCE-ENGINE-0017 The Governance Engine Open Metadata Access Service (OMAS) sent a metadata change event to listening Open Watchdog Governance Action Services: New Classification
Tue Aug 16 12:02:53 GMT 2022 mds1 Event OMAS-ASSET-OWNER-0012 The Asset Owner Open Metadata Access Service (OMAS) has sent event of type: Element Declassified
Tue Aug 16 12:02:53 GMT 2022 mds1 Event OMAS-SECURITY-MANAGER-0013 The Security Manager Open Metadata Access Service (OMAS) has sent event of type: Element Declassified
Tue Aug 16 12:02:53 GMT 2022 mds1 Event OMAS-DATA-MANAGER-0013 The Data Manager Open Metadata Access Service (OMAS) has sent event of type: Element Classified
Tue Aug 16 12:02:53 GMT 2022 mds1 Event OMAS-ASSET-MANAGER-0013 The Asset Manager Open Metadata Access Service (OMAS) has sent event of type: Element Declassified
Tue Aug 16 12:02:55 GMT 2022 mds1 Event OMRS-AUDIT-8009 The Open Metadata Repository Services (OMRS) has sent event of type Instance Event to the cohort topic mds1.openmetadata.repositoryservices.enterprise.mds1.OMRSTopic
Tue Aug 16 12:02:55 GMT 2022 mds1 Event OMRS-AUDIT-8009 The Open Metadata Repository Services (OMRS) has sent event of type Instance Event to the cohort topic egeria.omag.openmetadata.repositoryservices.cohort.mds.OMRSTopic.instances
Tue Aug 16 12:02:55 GMT 2022 mds1 Event OMAS-IT-INFRASTRUCTURE-0013 The IT Infrastructure Open Metadata Access Service (OMAS) has sent event of type: New Element
Tue Aug 16 12:02:55 GMT 2022 mds1 Event OMAS-STEWARDSHIP-ACTION-0012 The Stewardship Action Open Metadata Access Service (OMAS) has sent event of type: New Element
Tue Aug 16 12:02:55 GMT 2022 mds1 Event OMAS-DIGITAL-ARCHITECTURE-0012 The Digital Architecture Open Metadata Access Service (OMAS) has sent event of type: New Element
Tue Aug 16 12:02:55 GMT 2022 mds1 Event OMAS-ASSET-CONSUMER-0012 The Asset Consumer Open Metadata Access Service (OMAS) has sent event of type: New Element
Tue Aug 16 12:02:55 GMT 2022 mds1 Event OMAS-ASSET-OWNER-0012 The Asset Owner Open Metadata Access Service (OMAS) has sent event of type: New Element
Tue Aug 16 12:02:55 GMT 2022 mds1 Event OMAS-DATA-MANAGER-0013 The Data Manager Open Metadata Access Service (OMAS) has sent event of type: New Element
Tue Aug 16 12:02:55 GMT 2022 mds1 Event OMAS-ASSET-MANAGER-0013 The Asset Manager Open Metadata Access Service (OMAS) has sent event of type: New Element
Tue Aug 16 12:02:57 GMT 2022 mds1 Event OMAS-GOVERNANCE-ENGINE-0019 The Governance Engine Open Metadata Access Service (OMAS) has sent event of type Metadata change event
Tue Aug 16 12:02:57 GMT 2022 mds1 Information OMAS-GOVERNANCE-ENGINE-0017 The Governance Engine Open Metadata Access Service (OMAS) sent a metadata change event to listening Open Watchdog Governance Action Services: New Element
Tue Aug 16 12:02:57 GMT 2022 mds1 Event OMAS-SECURITY-MANAGER-0013 The Security Manager Open Metadata Access Service (OMAS) has sent event of type: New Element
Tue Aug 16 12:02:58 GMT 2022 mds1 Event OMRS-AUDIT-8009 The Open Metadata Repository Services (OMRS) has sent event of type Instance Event to the cohort topic mds1.openmetadata.repositoryservices.enterprise.mds1.OMRSTopic
Tue Aug 16 12:02:58 GMT 2022 mds1 Event OMRS-AUDIT-8009 The Open Metadata Repository Services (OMRS) has sent event of type Instance Event to the cohort topic egeria.omag.openmetadata.repositoryservices.cohort.mds.OMRSTopic.instances

Expected Behavior
Should have processed every KafkaTopic element.
Steps To Reproduce

Install metadata server and view server
Configure StrimziMonitorIntegrationConnector so that it connects to a Strimzi installation with 500+ topics
Start StrimziMonitorIntegrationConnector
Wait about 40 minutes

Environment
- Egeria:3.10
- OS: Kubernetes (Openshift 4.9)
- Java: 11.x

Any Further Information?
No response

Also, how was the behaviour after the crash? Did the system recover? Was any data lost? Did the connector resume retrieving topics?

For example, I had an issue with my Jupyter container (a bug) which led to the container restarting and I'd see something like:
Containers:
  jupyter:
    Container ID:   cri-o://3e6a2f6d264f9e8c6f052f23e49cb2da6a1cf35c869104c0686da689ffd2604f
    Image:          quay.io/odpi/jupyter:lab-3.4.5
    Image ID:       quay.io/odpi/jupyter@sha256:34f7ddbdebf759e4121e25c1ebe06929a118346ecd8a3ea06dbd17ca75fb6c42
    Port:           8888/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Mon, 15 Aug 2022 17:47:49 +0100
    Last State:     Terminated
      Reason:       Error
      Exit Code:    128
      Started:      Mon, 15 Aug 2022 17:47:46 +0100
      Finished:     Mon, 15 Aug 2022 17:47:48 +0100
    Ready:          True
    Restart Count:  1

I'm not sure how useful it would be, but the exit code may help us confirm running out of heap (for example) could be the cause.
What about the pod itself, rather than kubectl logs how about kubectl logs --previous - is there any extra information there?

Just checking the log again - I presume this is the metadata server -- not where you are running the integration connector?
What repository connector are you using? I'm guessing it could be XTDB?
If we can't get any info from the suggestions above quickly, I wonder if it's worth testing also with local graph or in-memory - that wouldn't completely rule a core issue out (as this could relate to timing, concurrent requests etc) but may give some extra insight.
if it is xtdb, what's the xtdb configuration like?

We have limited licenses for 'yourkit' java profiler (there are other tools available including using intellij+async profiler for example), which can help in pinning down memory issues, may be an option if the other approaches don't get us answers.

We use a local Janusgraph Repository. On the first restart after the crash, the server is unable to connect to the JanusGraph. This looks like a problem with JanusGraph. The oc describe does not show any problems. The previous log file is the one I attached to this issue. I will try to reproduce the error and check the events of the Stateful Set and the Pod.
Would you recommend using XTDB instead of JanusGraph?

@juergenhemelt I think generally we would suggest xtdb - it looks like a more scalable, reliable approach, and some performance analysis has been done on it.
That being said there's no evidence one way or the other yet to suggest your issue is specific to graph, we don't know. So whilst you could try it, equally it may be good to understand why it broken even with graph.
Were you using the default JanusGraph setup (uses berkeleydb)?
I think capturing the data would help, interesting to see if we can figure out if, for example, the jvm failed due to heap exhaustion. Or perhaps openshift killed it due to resource - perhaps you could also check the openshift dashboard to see what it's reporting?
I'd be interesting in trying to reproduce, though I may struggle to do this until later next week. I presume you just pointed your topic integrator at a strimzi deployment hosting lots of topics? Do you have any assets to share that may speed me trying to reproduce? (server config, chart config etc)?

This is an interesting kind of issue - how do we debug apparently the jvm going away. If the cause isn't external and anything shows up in k8s log, I wonder if we need to do something like:

set jvm flag to capture heapdump
modify the helm charts so that not only is the flag set, but the output either goes to, or is copied to, a persistent volume
we can then retrospectively analyze the heap dump

^ I'll raise an issue on this
OR maybe we can somehow modify the startup script of the container so that it does not exit (as you'd normally want) on this error condition, so that we could then interactively connect and analyze
But it still might be easier to just connect to the JVM using an agent, which should give us similar information (ie using YourKit profiler or similar)
For your issue another alternative might be to try and run just the mds server locally

I have just switched to XTDB in-memory and works. The good news is as the performance is tremendously better I can test faster than before. The bad news is that I still have the same problem. Will keep on analyzing.

I have just switched to XTDB in-memory and works. The good news is as the performance is tremendously better I can test faster than before. The bad news is that I still have the same problem. Will keep on analyzing.

Good to hear the performance is better.
I'm also completing/fixing/merging https://github.com/odpi/egeria-charts/pull/127 which is for our lab demo (notebook) chart to allow spec of additional connectors. I'll look at same in base. This would help me/us setting up a strimzi connector environment, and something I've intended to do for a while.
Please do share any data you capture especially from 'describe' against the k8s objects

In deed the oc describeof the pod gives a lastState:
    State:          Running
      Started:      Mon, 22 Aug 2022 10:49:56 +0200
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Mon, 22 Aug 2022 10:42:06 +0200
      Finished:     Mon, 22 Aug 2022 10:49:55 +0200

Obviously an out of memory issue.

The ressource limits in Kubernetes were at 1GB memory. I increased it and now it works. For the 547 topics it requires 3.4GB memory. I think that's expected behaviour as I am still on in-memory XTDB.

Did you have explicit 'resources' entries in your modified charts? Is that where you were setting the 1GB limit.
The standard charts don't specify a limit by default
When the pod restarted, did it continue processing ok?. In a k8s environment we need to expect pods to fail - for example nodes may fail or be rebooted, and the pod rescheduled elsewhere. It's important processing then continues normally - including resuming where it left off.
Are there any aspects of that which is problematic with the testing you have done?
What happens if you go far beyond 500 topics?
I am also thinking if our jam limits are set such that a failure could occur before aggressive garbage collection has kicked in. I can't see any explicit setting of heap in the container, though we do set MaxMetaSpaceSize to 1G -- it's extremely unlike we could get close to that, but may be worth checking./removing. By default heap is 1/2 physical ram, which should be the allocated size of the container
I think this aspect is worth pursuing - we'd mostly expect java heap to be exceeded (for example if we had an effective memory leak) with an exception before the allocated memory on the container ran out ? Need to look if we following best practice and/or how to do better.

Yes, I have explicitly set resource limits. This is a policy in my organization. It was set to 1GB. I now switched from XTDB in-memory to RocksDB persistence. That brings the used memory down again to less than 1GB. So the in-memory XTDB was the one which took all the memory. It could be good to check that XTDB is not using more than the available memory and log an exception then. But I can imagine that this  is not easy to implement.
For me this issue is fixed and you could close it.

If we get killed by k8s for exceeding limits there's nothing we can do in the java application -- only ensure recovery through the k8s deployment. If we hit a memory exception (ie out of heap) in the jvm there's not much we can do (especially anything that allocates memory). A output message, Maybe (and agree).
I'll keep the issue open to look into this behaviour. there may be something about the container / image we use/ settings that are allowing the jvm to continue unhindered yet where physical is restricted. could we do better. I don't know, but I see it as an important question for production.

I am seeing similar (if not the same) behavior when running the Lab Helm charts with the notebook "working-with-standard-models". In particular, if I run with the in-memory repository, the pod silently fails.  I am using Rancher-Desktop with an allocation of 14gb and 4 cores. When I switch to the graphDB repository, it usually works but occasionally dies (happened once or twice). I suspect that the JVM may need more resource?

I'm fairly sure the orchestration environment is killing our jvm - hence why we see no last runtime exception from the jvm.
I was wondering what memory the jvm is seeing
In java 8 timeframe this was an issue. See https://docs.openshift.com/container-platform/3.11/dev_guide/application_memory_sizing.html for example, where it was essential to set flags like
-XX:+UnlockExperimentalVMOptions \
  -XX:+UseCGroupMemoryLimitForHeap \

Moving forward in jvm options, we now have better options. There's now a flag UseContainerSupport, though this is enabled by default. Also we can then control with InitialRAMPercentage, MinRAMPercentage, and MaxRAMPercentage. These all work off the memory available to the container
But we also had a change in the kernel from cgroup1 to cgroups2, and it was only Java 15 that added support for cgroups2. I expect Openshift & other modern distros are using kernels with that.
So we're then back to the pre-container days of incorrect memory calculatinos.
From Java 15 onwards this seems corrected.
So.. now that we are with v4 of egeria, and using 17, it maybe these issues have been addressed by default.
@dwolfson were you running with 3.15 or 4? I wonder if both fail in the same way

Useful article - https://developers.redhat.com/articles/2022/04/19/java-17-whats-new-openjdks-container-awareness

Checking some real environments
Egeria 3.15, in openshift, 16GB node, default resource limits
sh-4.4$ java -XshowSettings:system -version
Operating System Metrics:
    Provider: cgroupv1
    Effective CPU Count: 4
    CPU Period: 100000us
    CPU Quota: -1
    CPU Shares: 2us
    List of Processors, 4 total:
    0 1 2 3
    List of Effective Processors, 4 total:
    0 1 2 3
    List of Memory Nodes, 2 total:
    0 1
    List of Available Memory Nodes, 2 total:
    0 1
    Memory Limit: 13.08G
    Memory Soft Limit: Unlimited
    Memory & Swap Limit: Unlimited
    Maximum Processes Limit: 44753

openjdk version "11.0.18" 2023-01-17 LTS
OpenJDK Runtime Environment (Red_Hat-<IP_ADDRESS>.10-2.el8_7) (build 11.0.18+10-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-<IP_ADDRESS>.10-2.el8_7) (build 11.0.18+10-LTS, mixed mode, sharing)

This may mean the problem is simpler - we're not setting a container size limit at all -- so it really can be as big as available on a node -- the subtlety above isn't relevant. And each of our processes will see the same, so the potential is for a massive over allocation
So we need to ensure limits are applied - and configurable. This isn't an egeria or even container consideration, rather it's at the deployment level ie in the charts (or operator). We must allow this to be config. Just as other apps do, or when a user is creating a deployment directly in a yaml, they would add
This consideration is part of the sizing activity needed in any deployment.. what to exactly set those values too
Platforms like openshift may also have the ability to dynamically override resource specifications - for example see https://docs.openshift.com/container-platform/4.11/nodes/clusters/nodes-cluster-overcommit.html , though we note that the default is not to overcommit

In summary:

currently pods will potentially use as much RAM as they can (especially in simpler k8s environments)
we may fail before even trying to do garbage collection
The k8s environment may offer controls to manage this (ie openshift) by modifying the pod definitions as they are being created
our helm charts need to provide a 'sensible' memory limit, at least for demos, to aim for more predictable behaviour (we do already for cts)
The limits need to be configurable
Docs should be updated with an explanation of our defaults, overriding & simple approach, ideally with links to standard k8s docs in this area
Note that in production environments this is a specialist skill


Multiple CompletionSuggesters?

elastic.v3 (for Elasticsearch 2.x)
I have a suggester set up and working for a single field:
const suggestorName = "typeahead"
var suggestor = elastic.NewCompletionSuggester(suggestorName)
field := "source_ip_suggest"
text := "10"
res, err := Client.Suggest(indices).Suggester(suggestor.Field(field).Text(text)).Do()

I can't figure out a way to use multiple suggesters - is that supported? I have two fields, source_ip and dest_ip and I would like to get a combined list of completions across the fields.
I realize I can do this at index time by adding both source and dest IPs to the same completion field, but then I would be storing the same data twice.
I can do so using curl and the elasticsearch http api like so:
curl -XPOST "http://localhost:9200/<index>/_search" -d'
{
   "size": 0,
   "suggest": {
      "text": "10",
      "dip-suggest": {
         "completion": {
            "field": "dest_ip_suggest"
         }
      },
      "sip-suggest": {
         "completion": {
            "field": "source_ip_suggest"
         }
      }
   }
}'

Oh, a long time since I touched suggester code ;-)
This test illustrates that you can add multiple suggesters. Is that what you're looking for?
HTH

Yes, that works! I had not realized from the documentation that you could add multiple suggesters.


Proposal: Improving text rendering

The community has said it - Pixi's text rendering performance needs to improve. This issue is devoted to how we can improve and what I think will be the best way to deliver:

Signed-distance fields: This technique renders the text into a texture (using Canvas 2D API) and applies a distance transform. In simple terms, the distance transform will set the value of each pixel in the output texture to the distance of that pixel to the nearest outline of the text in the input texture. pixi-sdf-text is an example: https://github.com/PixelsCommander/pixi-sdf-text

VTMs: Vector texture maps encode curve discontinuities at pixel level in textures.

Exact bezier curve rendering: This renders the font "as is" by tessellating everything except the curves. The curves are rendered using a special fragment shader (no sampling of the curve is done, it is rendered exactly on the GPU with antialiasing): https://www.microsoft.com/en-us/research/wp-content/uploads/2005/01/p1000-loop.pdf. I created a demo for quadratic bezier curve: https://codepen.io/sukantpal/pen/GRJawBg?editors=0010

@bigtimebuddy and I discussed these methods and we thought the third approach is the best because:

SDFs require pre-generated atlases. Generating multi-channel SDFs is very complicated (if we want to do it at runtime).

VTMs are way too complicated for just fonts.

Exact bezier curves just requires you to render the font as a path like everything else in Graphics.

It is should not be in core.
It is heavy package and MUST BE implemented outside main package, core team should not worries about implementation of it.
Pixi SDF is right example.

@eXponenta PIXI.Text implementation will be replaced by a higher performance implementation. This has already been decided.

I'd like to give some more context. I have asked @SukantPal to research alternative Text/BitmapText approaches that have different trade-offs. Particularly, if we can find something that performs better than Text and looks great.

There are a few pieces of criteria that I'd use judge a good Text display object:

Good Runtime Performance - Rendering text and changing text is low-cost
Low memory  - RAM usage is low
Small Dependencies - dependency footprint file-size is small
Great Fidelity - works well with different resolutions
Flexible styling - support for strokes, drop shadows, gradients
Supports Large Glyph Languages - E.g., Chinese, Japanese, Korean
Backward compatible - Works with with Context2D

Perf
Memory
Dependencies
Fidelity
Styling
CJK
Context2d

Text
👎
👎
👍
👍
👍
👍
👍

BitmapText
👍
👎
👍
👎
👎
👎
👍

Ok.
Describes fragment interpolation:

Needs extra shader and extra data - thats will prevent batching.
Needs rebuild geometry when change.
Require a shape table for all glyphs, that require load TTF to memory totally, because ot can't support streaming, and pase it. Looks like as SWF.
Language specifics rules are moving to lib team from browser team, that implement render rules: ltr/rtl, hindy/arabic glyph concatenation rules, etc, same as move 10% browser to lib. Unity wasn't implement it right still.
Heavy shader with extra arguments for supporting styles. That increases render cost, because we should render it in realtime.
And of course we should be cache it as texture.

@eXponenta

Current Text implementation caches each glyph multiple times (if a letter is used multiple times across multiple PIXI.Texts). Having one geometry per glyph is better. Also, you don't need to cache a different font sizes.

Geometry doesn't need to be rebuilt per say - if the letters in the text change, you need to fetch again from the glyph geometry table. If the transform changes, you just need to change one uniform.

We can use a library for font parsing. The conversion to geometry is trivial.

I don't know what glyph concatenation rules are. Could you explain?

Left-to-right vs right-to-left - if you want to reverse the letters, you can. If you want mirror image, then set scale.x=-1 internally. Am I missing something?

Text styling won't add more arguments to the shader. It will change the geometry. Now, storing the glyph as a geometry takes less memory than caching it in a texture.

U very naive:

right-to-left and left-to-rightand their mix is very complex problem.
Opened issues in PDF rendering lib:
https://github.com/asciidoctor/asciidoctor-pdf/issues/175
Hindi problem (and similar language)
https://docs.microsoft.com/en-us/typography/opentype/spec/gsub

@eXponenta

Mixed layout (ltr and rtl combined) is less of a priority than high performance. You can always fallback to current algorithm. Similar for Devanagari. Special features like gradient stops will also need a fallback.

Batching can work here because it will be easier without any textures. Can create a plugin like CanvasGraphicsRenderer.

I also suggest you read this https://en.wikipedia.org/wiki/Ad_hominem

Ok. U win. Do it. But outside core, then we will looks it and decide what we should do then.
But without heavy dependencies.

@eXponenta please stop being so hostile. We are exploring approaches nothing is solid yet. Please be constructive and do not use insults.

@eXponenta Are you sure we support right-to-left text as of now?
Another approach we are thinking is to render each glyph using canvas 2D (same as of now) separately and cache them into an atlas. The atlas can be reused globally for all PIXI.Text instances. Each glyph + font-size + font-style combination will be cached separately.
What do you think about accepting that into core?

Hey peeps, this is really exciting to hear these new approaches on the table. @eXponenta, echoing @bigtimebuddy, let's keep it constructive please. You clearly have some good insight into the new proposed approach, but lets aim to see if we can mitigate the issues raised and keep it friendly :)
On text, heres my 2 cents (pence?)..
Personally I think the most valuable approach is to gain the speed and performance of bitmap text but without the need to actually create a bitmap font beforehand. Dynamic bitmap texture building if you will!
Dynamic bitmap fonts
pros

Tools for creating bitmap fonts an are few and far between! Would love to remove that barrier.
we could create various sizes of text size textures.
Works with Canvas
Performance as each glyph would can be batched
lots off cool effects!
API would only need slight adjustment:

const textStyle = new TextStyle({
    font:'comic sans',
    fill:'bright green'
});

const bitmapFont = new BitmapFont(textStyle);

cons

all the other cons of bitmap current bitmap fonts :P
would dynamically updating the textures atlas cause slow down?

Although this would stabilise after a while, and could maybe be cached between sessions?
Current Text also has to upload a texture each time it changes frame, this approach would do it less?

ideas

Could cache texture generated in a localstorage? for subsequent visits?
Only add characters that are used, so textures is as big as it needs to be. Update as we go
bake in filters to this texture for extra cool effects (outline, shadow etc)
we can expore reducing memory of bitmap font by not making each glyph a sprite?

SDF
Can the SDF be an extension of the above approach? How complicated would it be to dynamically generate that texture? Would it be super janky? Are we talking increasing the code base by a large amount?
Most people have no idea what that is so it should ideally be hidden. If we need to externally generate the SDF fields or code required to generate is too large or complex then this would be better at home as a plugin for sure:
const textStyle = new TextStyle({
    font:'comic sans',
    fill:'bright green',
    sdf:true, <----- how sweet would this be, maybe rename to more user friendly like 'cleanEdges'
});

const sdfFont = new BitmapFont(textStyle);

Exact bezier curve rendering
This is an interesting approach for sure! I had looked into this in the past and decided that the shader switches introduced into the rendering loop and the extra tessellation required would have a large overhead. @SukantPal , you are closer to this idea than I am right now so would like to get your take on my concerns!
Having said that, maybe it might be worth building a prototype that we can test and bash about a bit to validate our thinking?
Exciting times for text :D

Can the SDF be an extension of the above approach? How complicated would it be to dynamically generate that texture? Would it be super janky? Are we talking increasing the code base by a large amount?
SDF generation is costly , and there are a packing time (when we  med pack different fonts in atlass), but it can be cached on client (as all other variant).
There is a canvas (hehe) implementation of it:
https://github.com/mapbox/tiny-sdf
But SDF compile is not fully correct   (glyph change when increase/decrease font size)
Looks like as usiful, but for MSDF we should execute it 3 times (per channnel).

oh wow, thats a tiny bit of code! Looks ripe for a shader to be doing too!

@mat The tiny-sdf package generates a single channel distance field. It doesn’t produce sharp corners when you cache the text at smaller font sizes. That can be mitigated by caching at the same font size being rendered.
Single channel SDF generation shouldn’t be too bad. The multi channel SDF is superior except that it is way too complicated. I couldn’t understand how Chumskly was encoding corners.
Now, everything we are considering won’t support rtl and Ltr layout mixed (as @eXponenta pointed out correctly). But I don’t think we are officially supporting it right now either.
The exact approach has two benefits - small cache size (you are caching vertices not all the pixels), antialiasing even when the setting “antialiasing” is turned off (text is expected to be antialiased always I think. Current implementation does that by using canvas 2D).
As for the demo, did you look at my quadratic Bézier curve demo?  https://codepen.io/sukantpal/pen/GRJawBg?editors=0010
——
Another thing I and @bigtimebuddy discussed as the most simple solution was to keep using canvas 2D and cache the glyphs into a texture. The glyphs can be rendered as a quad. The cache atlas can be reused between texts. Of course, this means large text is being cached at same size and hence large memory usage when a lot of text is being rendered but only a portion of it (like current page in PDF) is visible.

Nice,
Yes, no worries about rtl or ltr that will always be available through current method and is a problem all custom text rendering techniques will need solving at a later date.
These decisions. should ideally be focus on memory consumption and runtime performance.
The Bezier demo is cool, but I do not feel it is enough for understand the true complexities that may be hiding if we render a full sentence.
The stuff I wrote about dynamic bitmapfonts above is pretty much exactly what you mentioned about glyph caching. I think that is definitely a useful route to go down!
In summery:
The Bezier curve solutions: This could be good route, but I feel this requires further R&D to figure how it fits into a real production use cases.
Dynamic Bitmapfonts from normal fonts: This we should definitely explore as we know this will give good perf whilst also keeping API simple. My Favourite :D
Single channel SDF Appreciate its not going to be as good multichannel SDF, but the demo looked pretty good to me! If text can scale nicely with and we can have one texture per font, then I think this route could worth investigating too!

@GoodBoyDigital I think the demo does have pretty good text. But there are visible differences if you try to look for details:

The text on the top is not sharp - the edges have wiggles. Each font+(~12-15px differences in font-size) combinations might need to be cached separately.
I totally fine with going for SDF if the those wiggles are unimportant or if we can go with caching at multiple font sizes.

Hey all.
I'm a super laymen on all things Pixi, but am a user of GDevelop, and I've been digging into this for a while for GDevelop's implementation of PixiJS and text rendering. It's actually impactful for a game I'm working on that has a lot of text. (You can see my research/examples in the issue I've posted in GDevelop: https://github.com/4ian/GDevelop/issues/1449 )
One of the options I found was focused around outright ignoring the scaling issues with text and eliminating them altogether by always leaving text scale at 100%, but scaling the text font size instead?  This is font agnostic, and seems to work at all sizes.
Here's a code example where that's being done using Pixi: https://codepen.io/Tazy/pen/wJVExB
I found it in this thread: https://www.html5gamedevs.com/topic/29576-scalable-text-for-pixijs/
I actually have a bounty on this issue as I was hoping someone could just modify the Pixi Text extension for GDevelop, but (being the layman that I am) I didn't realize that this was a much larger issue with Pixi itself.
No idea if there are potential issues with this method, but it would seem to avoid all of the performance issues of sdf/msdf/etc.

@Silver-Streak If you mean increasing the font-size by the scale applied on the text display-object, then how would that be better in terms of performance? SDF stuff helps performance because you don't re-render at larger font-sizes.

@Silver-Streak Of course, your proposal will improve quality but I don't see how it will improve performance.

@Silver-Streak Of course, your proposal will improve quality but I don't see how it will improve performance.

Unless I'm misunderstanding, wouldn't changing the font size use less resources than scaling the entire object, or loading in another renderer? Again, I'm super layman when it comes to how Pixi's scaling actually works, it just made sense to me that leaving the font at native resolution but just changing the text size would be faster than scaling it. If that's incorrect, apologies for the distraction.

@Silver-Streak Scaling is implemented as a transform. You have a transformation matrix - and change the scale in that matrix is a trivial operation.
As of now, the text is rendered using Canvas 2D API into a canvas. Then it is copied onto the screen. Changing the scale will just change the screen coordinates to which the text in the canvas is mapped to.

Ahh, that makes sense. That's unfortunate, although to be fair my issue is more with the graphical rendering of fonts becoming blurry after being scaled, but I thought it'd be better performance too.
While I'd be excited for someone to implement it to fix the general text quality issues, I can totally understand wanting to improve performance as well.
Thanks for talking through it with me.

@Silver-Streak No problem. Implementing a quality-first text should not be hard in WebGL either, however. Some applications are implementing text by creating a mesh on their own. You can parse the text file, make a list of vertices, and tessellate it. Those vertices can be rendered through a mesh.
At high scales, this can cause slight "edges" - because ultimately, you are rendering the text as triangles. For even higher quality, you can use the "exact bezier curve" shader I was talking about in this thread - it will render the curves as curves and not triangles.
If you need help with text quality, I can help you guys out :)

@SukantPal I'm definitely not a contributor to GDevelop, (nor would you want me to be. I'm predominantly a Business Systems Analyst/DevOps in life, not a developer 😄 ) although I love that project and am active in the community. I also know there are other members of the community would love to see a solution for scaled text quality.
However, if you want to take a look at the issue in the post, I also have a bounty for it over on Bountysource as well open to all. I don't want to take up any more headspace here, though, as it's obvious there's a much deeper conversation going on around Pixi in general, and my suggestion wasn't as spot on as I thought it was.

@Silver-Streak I might take a look, since I've nothing better to do in corona-season

This thread should definitely be re-opened and kept alive - or at least have some sort of update every so often, as this is one of the most looked for issues.


(feat): Vendor and language carousel on landing page

This PR
Adds Carousel on the Landing page for Vendor and languages
Related Issues
Fixes #363
Notes
I have added devbox for local development; if this is not needed, I can drop that commit

@beeme1mr Can you please review this, I think the button can be improved, Please share your thoughts and suggestions

I will make the other changes later in the day

Hi @beeme1mr, I have updated the PR with the mentioned changes. Can you please review this again?

Hey @znonthedev, the changes look great!
There are just a few minor additions that should be made. Could you please:

Add a pointer cursor when hovering over an item in the carousel.
Update the contributing docs to briefly explain how to use Devbox for local development.

Thanks!

@znonthedev I'm not able to click and item in the carousel anymore. At least not in the current preview.

@znonthedev I'm not able to click and item in the carousel anymore. At least not in the current preview.

It was a minor bug, I fixed it. I should have tested it thoroughly. Should be okay by now


Complete install instructions

I'm only an end user of the Autotools so it took me a while to figure out how to convert the provided configure.ac to a configure script.
Also removed some parts from the README which referred to the libvmod example.

Hi @BartVB,
sorry for not paying attention to this. Merged

No problem. Thank you!


Opening and executing a notebook cell does not mark the tab as "open"

Open notebook
Execute a cell either via the "Execute All" or a single cell
Notice the editor tab is in italics, meaning it's not "open"
Navigate to a different file
:bug: notebook tab is gone

I would expect that when I run a cell that it stays open. It isn't until I perform an action, like edit text or delete cells, that it registers.

Now whenever users execute a cell or a document, the editor will be pinned immediately, similar to how content changes work.


[Backport to main] add ML task timeout setting and clean up expired tasks from cache (#662)

Description
Backport PR #662 to main branch
Issues Resolved
[List any issues this PR will resolve]
Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #770 (38f13ad) into main (c02e437) will decrease coverage by 1.45%.
The diff coverage is 76.89%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff              @@
##               main     #770      +/-   ##
============================================
- Coverage     84.78%   83.34%   -1.45%
- Complexity      988      998      +10
============================================
  Files            92       93       +1
  Lines          3542     3662     +120
  Branches        326      342      +16
============================================
+ Hits           3003     3052      +49
- Misses          402      463      +61
- Partials        137      147      +10

Flag
Coverage Δ

ml-commons
83.34% <76.89%> (-1.45%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...rg/opensearch/ml/plugin/MachineLearningPlugin.java
100.00% <ø> (ø)

...java/org/opensearch/ml/profile/MLModelProfile.java
46.29% <40.00%> (-0.65%)
:arrow_down:

...earch/ml/action/load/TransportLoadModelAction.java
83.78% <42.85%> (-5.54%)
:arrow_down:

...va/org/opensearch/ml/model/MLModelCacheHelper.java
92.06% <44.44%> (-3.70%)
:arrow_down:

...ain/java/org/opensearch/ml/task/MLTaskManager.java
59.39% <66.66%> (-7.63%)
:arrow_down:

...ain/java/org/opensearch/ml/model/MLModelCache.java
86.36% <71.42%> (-1.78%)
:arrow_down:

...n/java/org/opensearch/ml/model/MLModelManager.java
78.20% <74.75%> (-4.33%)
:arrow_down:

...arch/ml/action/forward/TransportForwardAction.java
90.76% <77.77%> (-0.14%)
:arrow_down:

.../ml/action/syncup/TransportSyncUpOnNodeAction.java
85.71% <81.25%> (-8.23%)
:arrow_down:

...java/org/opensearch/ml/utils/MLExceptionUtils.java
86.36% <86.36%> (ø)

... and 18 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


VS Code 1.9.1 on MacOSX

Hi The extension does not seem to work on maxosx. I have VS code 1.9.1. Are you aware of any command that need to be run after installing?

Hi,
the Extension currently supports VS Code 1.8.
I´m going to fix that later this day!


[T3A2][W14-A2]Tan Lok You

Ready for review

There are more places where feedbackToUser was used. for example, in the tests.

Some comments added. Feel free to add more comment if you have any questions. Once you are done, you may close the PR after reading comments.


Fix wallet encrypt/decrypt to return promise

refer to https://github.com/CityOfZion/neon-js/issues/460

I wonder why there isn't a unit test for this class.

Yeah I think I missed out the tests for this class

@snowypowers Unit test added

300 points MorganDream
150 points snowypowers for review

https://neoscan.io/transaction/44C84C9E7582DC00411BC9D9D075F6993BA7E203FFDDE366E21DFEBD125EA580


Newsletter 236 trranslate in french

Hello @bitschmidty ,
can you confirm 3 last active PR please ?

I'm sorry for all this stuff,
I don't understand why sometimes it's so complicated.
From master, I create a new branch, as you ask me
after I create the new file with newsletter to translate,
then create a PR
and after finished translation and a good check I submit to review.
What is wrong ?

@Copinmalin my guess is that your local master doesn’t match up with "origin/master" (from this repo). If you "git reset --hard origin/master" from your local master branch (assuming you dont have anything you’re working on there), I think that would help. Then new branches created from your local master branch would have the latest and also not have anything extraneous.

OK So I wait for merging my last PR and do again this hard reset.


[Bug] Switching database contexts causes logging error

When running a migration script, if the script changes database context, but does not return to the original database context, the migrate function will error out from being unable to find the SchemaVersionTable:

[Install-DBOScript] Script block number: 1; Message: Invalid object name 'dbo.SchemaVersions'.\n
[Install-DBOScript] System.Data.SqlClient.SqlException (0x80131904): Invalid object name 'dbo.SchemaVersions'.

Not sure the best approach - three part naming for the logging table? Open a new connection for the logging portion? I might be able to work on a PR for this depending on what approach is deemed desirable...

Switching database contexts is not a supported scenario in the underlying framework - DbUp. It never supported crossing the boundaries of a single database and the SchemaVersions table cannot be addressed in a three part naming manner in its current implementation.
As described here, the best solution seems to be to have independent deployment scenarios for each database, rather than switching DB contexts.

Ack! That is frustrating. Our use case involves making a change to the database, so generally switching out to a system database, which I'd rather not have to keep a separate migration workflow for. Thanks for the quick response!

I'll close this issue for now, but please, share the workaround if you find one!


Batching with labels

Hello,
you have created a great tool. Is it possible to save the node with the label in transactions / batching?

This example should be documented.


Added a belongs_to method to the formbuilder DSL

See: https://github.com/gregbell/active_admin/issues/878

Oh - it's been pointed out to me that I accidentally slipped in a datepicker method into that first commit. Should I redo this pull request?


⚠ Bofrost.de has degraded performance

In 14f3e26, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 2371 ms

Resolved: Bofrost.de performance has improved in eb85f25 after 9 minutes.


Implementation with PyTorch

Even with CPU thread limiting applied while using the provided code, the CPU consumption remains excessively high (around 1400%). Is it possible to implement this algorithm in PyTorch?

I have implemented a basic version of the algorithm in Javascript using Tensorflow.js You can probably convert it to PyTorch using that as a reference https://github.com/zqwitt/PaCMAP_TFjs


use shared libraries on FreeBSD

updated

Fixed the remaining issues https://github.com/D-Programming-Language/dmd/pull/4838, https://github.com/D-Programming-Language/druntime/pull/1333.

This might need an auto-tester update, b/c the shared phobos library is picked by default but the LD_LIBRARY_PATH isn't set accordingly.

requires https://github.com/D-Programming-Language/dmd/pull/4975

This is just about building and testing, not making them the default.

Updated, still blocked by the pending dmd PR though.

Updated, should work now.


[FEATURE] Community page (separate page)

Detailed description
We have to build a community page.

A little bit about community
Discord link
Twitter about our community

Context
The community page will help user to know more about the community
and will be able to join the community through the community page.
Possible implementation
Example page:

Additional information
If you need any help or have any doubts, can ask below.
Thanks
Code of Conduct

[X] I agree to follow this project's Code of Conduct

@kabir0x23, Can I take this?

@kabir0x23, Can I take this?

Hello @akshitarora921,
I am assigning you this issue

Hi, @kabir0x23
I see this issue has no assignee, and I'd love to work on it.
Assign it to me, please.

Question: Where do I the image/video for the background, and also the testimonials?

Hi, @kabir0x23
I see this issue has no assignee, and I'd love to work on it. Assign it to me, please.

awesome, i am assigning it to you.

Question: Where do I the image/video for the background, and also the testimonials?

you can leave the testimonials for now
the center section would be greate for now.
If you will have any questions, let us know.
Thank you.

Alright, I'll need the background image and the discord channel link.

Alright, I'll need the background image and the discord channel link.

discord link you can take from the footer.
image:

Alright, thank you.
I'm on it.

Hi @kabir0x23
Is this okay?

Hi @kabir0x23 Is this okay?

okay?
It's just Awesome

can you do one change?
discord color: papule
green looks good. but the thing is discord is purple will look different
What do you think?

green looks good too

can you do one change?
discord color: papule green looks good. but the thing is discord is purple will look different What do you think?

Hmmm, I'll do that now. Thank you.
So, what next?

Hmmm, not bad.

Hmmm, not bad.

Awesome
I think You can make a PR now

Hmmm, not bad.

Awesome I think You can make a PR now

Okay, I'll do that now.
Thank you.

Hmmm, not bad.

Awesome I think You can make a PR now

Okay, I'll do that now. Thank you.

thanks for the contribution. 😄

Okay, I'll do that now. Thank you.

thanks for the contribution. 😄

Thank you @kabir0x23
I'm so sorry. I got carried away and ran "git add ."
I hope it won't be a problem. I promise it won't happen again.

Thank you @kabir0x23
I'm so sorry. I got carried away and ran "git add ." I hope it won't be a problem. I promise it won't happen again.

It's okay, don't need to say sorry, it happens.
I will make changes from my side if some unwanted files got committed.

Thank you @kabir0x23
I'm so sorry. I got carried away and ran "git add ." I hope it won't be a problem. I promise it won't happen again.

It's okay, don't need to say sorry, it happens. I will make changes from my side if some unwanted files got committed.

Alright, thank you so much.
I've just made a PR, please review it.


Enabled DB encryption validation

Description
In the early stages of the development of kubecf, we had to disable the DB encryption validation as it was failing.
Fixes https://github.com/SUSE/kubecf/issues/24.
Motivation and Context
We should do the validation, which is the default behaviour with cf-deployment.
How Has This Been Tested?
Deployed locally and the api instance group doesn't fail anymore.
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] My code has security implications.
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.

@mook-as I missed a bit of the code. We actually need to perform the migration in the pre-start. It's fine now that the api instance group waits for the database to be ready.


Server-directed login flow using SFSafariViewController

Google dropped OAuth support for UIWebView on iOS recently. SFSafariViewController-based implementation is the right way to go.

-(void)loginWithProvider:(nonnull NSString *)provider customUrlScheme:(nonnull NSString *)urlScheme controller:(nonnull UIViewController *)controller animated:(BOOL)animated completion:(nullable MSClientLoginBlock)completion is going to replaced the equivalent WebView-based implementation in MSClient.
-(void)loginWithProvider:(nonnull NSString *)provider customUrlScheme:(nonnull NSString *)urlScheme parameters:(nullable NSDictionary *)parameters controller:(nonnull UIViewController *)controller animated:(BOOL)animated completion:(nullable MSClientLoginBlock)completion is going to replaced the equivalent WebView-based implementation in MSClient.

@shrishrirang

@shrishrirang Can you review my latest changes? I addressed everything except the potential issue of concurrent logins.


if memberlist is big, the selection window shows a unresponsive script error

When you open the selection window for a dimension that has a lot of members, the ui displays a message saying that the script is unresponsive.
In my case it is a list of products that the company sels, 40000 in total.
jpivot is slow when opening this list, but it does not give any error.

just fixed. please test and tell me if its better now :)


Create command creates new contact at the wrong index

Command result lists (last index + 1) but inserts new contact in a different position. It should insert at the end when the list is unsorted.
How to replicate:

Execute create command

Remove index comment


Loading indicator not working on iOS11

I've updated my iPhone to iOS11 and loading indicator is not working anymore.
Any help with that, please?
Thanks.

Which sdk are you using? I run an app that uses this module on an iPhone 7 with iOS 11 and the loading indicator appears.

6.2.2GA & Xcode9
Thanks!

I can confirm you that with the latest SDK/Xcode version the module is working fine. Check your configuration and in case you have other related issues, reopen this one. Thanks for your feedback.


Separation of header and c files ?

Hi, i can't figure out why nanovg_gl.h contains both definitions and declarations together, if i want to use nvgCreateGLES2 on multiple files so i have to include nanovg_gl.h (with -DNANOVG_GLES2_IMPLEMENTATION) in each files, now the problem occurs, there are so many files which have nvgCreateGLES2 compiled in and they're conflicting with each other in link time.
For the moment, i have separated definition and declarations in two separate file, am i correct?

i think the idea is that you only have the implementation in one file, ie get rid of the -DNANOVG_GLES2_IMPLEMENTATION, then pick one cpp file where you do:
#define NANOVG_GLES2_IMPLEMENTATION
#include "nanovg_gl.h"

every else you just include the header. (or... something like that :) )

I agree with one file for implementation, but combining both header and source file will make problems if you need using nvgCreateGLES2 on multiple files say multiple surfaces.

The one file idea is taken from stb libraries [1]. You should have only one file in your project where you define the implementation as @kritzikratzi mentioned.
[1] https://github.com/nothings/stb

Ran into the same problem. Perhaps this should be mentioned in the README?
I am guessing the reason the stb approach is used is so GL lib wrangling is left up to user.

what about using nvgluCreateFramebuffer (in nanovg_gl_utils.h) in multiple files, its impossible with this approach.

Hello,
Sorry to reopen that old issue but I struggle a lot with the same problem.
Has anything changed regarding the inclusion of nanovg_gl.h in several .cpp ?
I tried to write in a first .cpp :
    #include <nanovg.h>     #define NANOVG_GL3_IMPLEMENTATION     #include <nanovg_gl.h>
There is only one .cpp file which define that as @kritzikratzi suggests and then in my other .cpp file, I only put :
     #include <nanovg.h>          #include <nanovg_gl.h>
And this leads to the compile-time error   C3861 : ncgCreateGL3  identifier not found  from that second .cpp file.
i already tried to fix that by declaring the ncgCreateGL3 function  in my file without the definition, but the linker send a LNK2019 : unresolved external symbol for the definition.
Any idea of what is happening ?

Dude, there is no ncgCreateGL3 symbol in nanovg, so this is probably just a typo. Also maybe check #634.

@mulle-nat  I mean nvgCreateGL3  it's not a copy-past and I make a mistake when I wrote it here, sorry.

#define NANOVG_GL3

Thanks for your help! I don't really understand what happened or why adding that definition worked, but it fixed my problem.


fix(advanced search): update comparison operators in case of rich text (DEV-326)

resolves DEV-326

LGTM, I assume you've tested it? Or should I do it?

Thanks. Yes, I've tested it


[dua] fix 1.2 compile issue without DUA flag

To Resolve #5324

Will add workflow to build and test 1.2 FTD as mentioned in https://github.com/openthread/openthread/pull/5276#issue-455213582 in a separate PR soon.


Create a task specific for ContentSyncDown

To make it easier for users to understand how to sync down... create a specific task for ContentSyncDown instead of using DeplotTo that could be baffling.

Jobs done


Power consumption while idling

I do not have exact figures, power consumption compared to OEM Sony Android-10 feels considerably higher. It might be comparable to the previous aosp-9 port, and I know it will be difficult to trace it, besides it might originate down to aosp-10 layers.

I have a feeling it is higher than SFOS based on AOSP9, but I am not sure yet. Will look into whether I can make suspend stay for longer.

thanks, it's not a top priority issue, rather than a optisatiom one. i do not know how much is an aosp-10 issue or sfos tunning.

It may run hand-in-hand with framerates being slow. Maybe I need to look into CPU/GPU governors and see if all can be improved.

cat /proc/cpuinfo is almost identical to xa2, the only difference is the features lines that has more capabilities. i found tgat surprising as the two devices have different cpus

Looks like suspend stats are OK. At least offline/wlan only are very decent. With SIM card working, most of the wakeups coming from qrtr_0 (/d/wakeup_sources).
Will continue looking.

i can run some tests, what's the best way ti monitor activity? e.g. airplane mide, wifi only, cellular

No, not yet. let me check few things first

One observation, it seems that consumption while idling sometimes is really low and some other times it's draining fast.

available governors and frequencies for cpubw using ssh connection
[defaultuser@XperiaXZ3 soc:qcom,cpubw]$ cat governor
bw_hwmon

[defaultuser@XperiaXZ3 soc:qcom,cpubw]$ cat cur_freq
8132

[defaultuser@XperiaXZ3 soc:qcom,cpubw]$ cat available_frequencies
2288 4577 6500 8132 9155 12298 14236

[defaultuser@XperiaXZ3 soc:qcom,cpubw]$ cat available_governors
compute mem_latency bw_hwmon vidc-ar50-llcc vidc-ar50-ddr msm-vidc-llcc msm-vidc-ddr gpubw_mon bw_vbif msm-adreno-tz userspace powersave performance simple_ondemand

Yes, already checked most of them. Not all of them work for us, some lead to crash even. cpufreq governors are the one that is used to setup CPU frequencies and initialized in /vendor/etc/init/init.tama.pwr.rc . Although, some of the init sequence fails in SFOS and AOSP10 as well (error messages in dmasg).
Will have to check it out:
Jul 01 10:05:42 XperiaXZ2Compact droid-hal-init: Command 'write /sys/devices/system/cpu/cpu0/cpufreq/interactive/use_migration_notif 1' action=sys.boot_completed=1 (/vendor/etc/init/init.tama.pwr.rc:66) took 0ms and failed: Unable to write to file '/sys/devices/system/cpu/cpu0/cpufreq/interactive/use_migration_notif': open() failed: Permission denied
Jul 01 10:05:42 XperiaXZ2Compact droid-hal-init: Command 'write /sys/devices/system/cpu/cpu0/cpufreq/interactive/target_loads 83 1766400:95' action=sys.boot_completed=1 (/vendor/etc/init/init.tama.pwr.rc:72) took 0ms and failed: Unable to write to file '/sys/devices/system/cpu/cpu0/cpufreq/interactive/target_loads': Unable to write file contents: Invalid argument
Jul 01 10:05:42 XperiaXZ2Compact droid-hal-init: Command 'write /sys/devices/system/cpu/cpu0/cpufreq/interactive/max_freq_hysteresis 79000' action=sys.boot_completed=1 (/vendor/etc/init/init.tama.pwr.rc:74) took 0ms and failed: Unable to write to file '/sys/devices/system/cpu/cpu0/cpufreq/interactive/max_freq_hysteresis': open() failed: Permission denied
Jul 01 10:05:42 XperiaXZ2Compact droid-hal-init: Command 'write /sys/devices/system/cpu/cpu0/cpufreq/interactive/ignore_hispeed_on_notif 1' action=sys.boot_completed=1 (/vendor/etc/init/init.tama.pwr.rc:76) took 0ms and failed: Unable to write to file '/sys/devices/system/cpu/cpu0/cpufreq/interactive/ignore_hispeed_on_notif': open() failed: Permission denied
Jul 01 10:05:42 XperiaXZ2Compact droid-hal-init: Command 'write /sys/devices/system/cpu/cpu4/cpufreq/interactive/use_migration_notif 1' action=sys.boot_completed=1 (/vendor/etc/init/init.tama.pwr.rc:81) took 0ms and failed: Unable to write to file '/sys/devices/system/cpu/cpu4/cpufreq/interactive/use_migration_notif': open() failed: Permission denied
Jul 01 10:05:42 XperiaXZ2Compact droid-hal-init: Command 'write /sys/devices/system/cpu/cpu4/cpufreq/interactive/target_loads 83 1920000:90 2092800:95' action=sys.boot_completed=1 (/vendor/etc/init/init.tama.pwr.rc:87) took 0ms and failed: Unable to write to file '/sys/devices/system/cpu/cpu4/cpufreq/interactive/target_loads': Unable to write file contents: Invalid argument
Jul 01 10:05:42 XperiaXZ2Compact droid-hal-init: Command 'write /sys/devices/system/cpu/cpu4/cpufreq/interactive/max_freq_hysteresis 79000' action=sys.boot_completed=1 (/vendor/etc/init/init.tama.pwr.rc:89) took 0ms and failed: Unable to write to file '/sys/devices/system/cpu/cpu4/cpufreq/interactive/max_freq_hysteresis': open() failed: Permission denied
Jul 01 10:05:42 XperiaXZ2Compact droid-hal-init: Command 'write /sys/devices/system/cpu/cpu4/cpufreq/interactive/ignore_hispeed_on_notif 1' action=sys.boot_completed=1 (/vendor/etc/init/init.tama.pwr.rc:91) took 0ms and failed: Unable to write to file '/sys/devices/system/cpu/cpu4/cpufreq/interactive/ignore_hispeed_on_notif': open() failed: Permission denied

Right now working on idea of switching off power-hungry CPUs and keeping only low-power CPUs available while screen is off.

i thought checking the settings in the oem android, but required root access which i do not have. how android can achieve those power savings?

Re Android power savings: I don't know, unfortunately.

Out of the failures above, only target_load exists and fails. The rest of the API is missing, maybe updated with 4.14 kernel.

i csm flash it bavk to android if we can get tge info from there but i have the feeling the oem 'aosp' is very different to what we have?

stock != aosp android. as far as I know, even kernel versions are different.

I know, for XZ3 with OEM Android 10, version 52.1.A.3.137 and kernel version 4.9.186-perf+

While I am looking into how AOSP or Stock regulates consumption (some techniques don't work for us, such as app freezing), try to run attached script that will hotplug big core CPUs depending on your screen state. It will switch off the power-hungry CPUs while screen is off and will turn them on when you turn on the screen.
To  run, install dependencies
# install required python package
devel-su zypper in python3-gobject

In terminal on phone, start
# run script
devel-su python cpupower.py

Keep the terminal running that script for a while and see whether it improves the battery consumption.
To judge cpufreq governors, I will have to fix collectd CPU frequency plugin first. It seems to ignore hotplugging of CPUs and shows some odd data (frequencies that are not expected to be there at all).
cpupower.zip

Thanks, it's running now but will take some time to observe any difference. Is there any command line tool that displays battery level? I used to have an awk script that was grabbing that info but I forgot the command ;-(

Yes, it will take some time.
I should improve that script a bit further - looks like it resets interactive governor settings to default ones as soon as you switch on/off CPU. as a result it seems to get CPU stuck for longer in the high freq range.
Re battery: cat /sys/class/power_supply/battery/capacity

OK, looks like Linux kernel itself when calculating CPU frequency distribution doesn't take into account whether CPU is switched off or not. Not much I can do on collectd side of things and would have to use caution while interpreting the data.

14h now battery percentage drop 18% (64 -> 46) with little use. the script might have crashed as it shows half a line print and no updates.
collecting data ideally should not add to the load, tgat can be tricky on highly tuned systems. then the other option we have is to collect data over longer periods. i am wondering if tgere is a way to create a techical load, tgen at least by observing consumption we get more accurate measurements.

For CPUfreq scaling, Stock is using
H8324:/sys/devices/system/cpu/cpufreq/policy4/schedutil $ cat down_rate_limit_us
0
H8324:/sys/devices/system/cpu/cpufreq/policy4/schedutil $ cat hispeed_freq
1574400
H8324:/sys/devices/system/cpu/cpufreq/policy4/schedutil $ cat hispeed_load
90
H8324:/sys/devices/system/cpu/cpufreq/policy4/schedutil $ cat pl
1
H8324:/sys/devices/system/cpu/cpufreq/policy4/schedutil $ cat up_rate_limit_us
0

# for power efficient cpus
H8324:/sys/devices/system/cpu/cpufreq/policy0/schedutil $ cat down_rate_limit_us
0
H8324:/sys/devices/system/cpu/cpufreq/policy0/schedutil $ cat hispeed_freq
1209600
H8324:/sys/devices/system/cpu/cpufreq/policy0/schedutil $ cat hispeed_load
90
H8324:/sys/devices/system/cpu/cpufreq/policy0/schedutil $ cat pl
1
H8324:/sys/devices/system/cpu/cpufreq/policy0/schedutil $ cat up_rate_limit_us
0

For GPU bw:
H8324:/sys/class/devfreq/soc:qcom,gpubw # cat available_frequencies
0 381<PHONE_NUMBER> 1571 2086 2597 2929 3879 4943 5931 6881
H8324:/sys/class/devfreq/soc:qcom,gpubw # cat cur_freq
0
H8324:/sys/class/devfreq/soc:qcom,gpubw # cat max_freq
6881
H8324:/sys/class/devfreq/soc:qcom,gpubw # cat min_freq
0
H8324:/sys/class/devfreq/soc:qcom,gpubw # cat polling_interval
50
H8324:/sys/class/devfreq/soc:qcom,gpubw # cat governor
bw_vbif

Few observations: gpu is frequently on 0 frequency if you don't use and scroll much. According to trans_stat, when active, frequencies 762 and 1144 are used. Probably others kick in at some other cases. 0 is not shown in stats, but is shown as current frequency in trans_stat as well.
For CPU bw:
H8324:/sys/class/devfreq/soc:qcom,cpubw # cat available_frequencies
2288 4577 6500 8132 9155 12298 14236
H8324:/sys/class/devfreq/soc:qcom,cpubw # cat available_governors
compute mem_latency bw_hwmon msm-vidc-llcc msm-vidc-ddr gpubw_mon bw_vbif msm-adreno-tz cpufreq userspace powersave performance simple_ondemand
H8324:/sys/class/devfreq/soc:qcom,cpubw # cat cur_freq
2288
H8324:/sys/class/devfreq/soc:qcom,cpubw # cat governor
bw_hwmon
H8324:/sys/class/devfreq/soc:qcom,cpubw # cat max_freq min_freq
14236
2288
H8324:/sys/class/devfreq/soc:qcom,cpubw # cat polling_interval
50
H8324:/sys/class/devfreq/soc:qcom,cpubw # cat target_freq
2288
H8324:/sys/class/devfreq/soc:qcom,cpubw # cat trans_stat
     From  :   To
           :      2288      4577      6500      8132      9155     12298     14236   time(ms)
*      2288:         0      1133        93        62        20        38         6   3744270
       4577:      1049         0       474       111        32        91        50    142340
       6500:       139       300         0       175        25        30        16     56860
       8132:        65       122        42         0        43        79        42     37360
       9155:        23        38         7         4         0        28        33     11980
      12298:        60       116        31        18         9         0        93     41620
      14236:        17        98        38        23         4        61         0    118940
Total transition : 4938

H8324:/sys/class/devfreq/soc:qcom,cpubw/bw_hwmon # cat bw_step
190
H8324:/sys/class/devfreq/soc:qcom,cpubw/bw_hwmon # cat decay_rate
90
H8324:/sys/class/devfreq/soc:qcom,cpubw/bw_hwmon # cat dow
down_count  down_thres
H8324:/sys/class/devfreq/soc:qcom,cpubw/bw_hwmon # cat down_count
3
H8324:/sys/class/devfreq/soc:qcom,cpubw/bw_hwmon # cat down_thres
0
H8324:/sys/class/devfreq/soc:qcom,cpubw/bw_hwmon # cat guard_band_mbps
0
H8324:/sys/class/devfreq/soc:qcom,cpubw/bw_hwmon # cat hist_memory
20
H8324:/sys/class/devfreq/soc:qcom,cpubw/bw_hwmon # cat hyst_length
10
H8324:/sys/class/devfreq/soc:qcom,cpubw/bw_hwmon # cat hyst_trigger_count
3
H8324:/sys/class/devfreq/soc:qcom,cpubw/bw_hwmon # cat idle_mbps
1600
H8324:/sys/class/devfreq/soc:qcom,cpubw/bw_hwmon # cat io_percent
50
H8324:/sys/class/devfreq/soc:qcom,cpubw/bw_hwmon # cat mbps_zones
2288 4577 6500 8132 9155 10681
H8324:/sys/class/devfreq/soc:qcom,cpubw/bw_hwmon # cat sample_ms
4
H8324:/sys/class/devfreq/soc:qcom,cpubw/bw_hwmon # cat throttle_adj
0
H8324:/sys/class/devfreq/soc:qcom,cpubw/bw_hwmon # cat up_scale
250
H8324:/sys/class/devfreq/soc:qcom,cpubw/bw_hwmon # cat up_thres
10

@pagism - try to press few Enters in the terminal and see if new messages will start to show up. I have seen that terminal gets messed up once in a while.

Additional comment regarding stock - it doesn't use any CPU hotplugging, same as AOSP, I think. But Android workloads could be different from ours

I've ctr-c and restarted the script, seems to respond ok now. Hopefully does not require reboot(?).
Interesting, available frequencies in for GPU in xz3 do not go to 0, while in the stock one go:
[root@XperiaXZ3 soc:qcom,gpubw]# cat available_frequencies
381<PHONE_NUMBER> 1720 2086 2597 2929 3879 4943 5931 6881

the CPU frequencies are identical, available governors have some small differences too, and cpu stats look different:
[root@XperiaXZ3 soc:qcom,cpubw]# cat trans_stat
     From  :   To
           :      2288      4577      6500      8132      9155     12298     14236   time(ms)
       2288:         0     28233      3278      1385       277       243      1068  25876518
*      4577:     18157         0     14987      3118       229       218      1072   2085124
       6500:      9943      5378         0      3118       221       151<PHONE_NUMBER>
       8132:      4597      1902       562         0       252       366       219    624932
       9155:       355       247        44        62         0       127       226     68131
      12298:       348       320        44       107        56         0       672     93742
      14236:      1084      1702       354       108        26       442         0   2251933
Total transition : 105756

and for gpu:
[root@XperiaXZ3 soc:qcom,gpubw]# cat trans_stat
     From  :   To
           :       381       572       762      1144      1720      2086      2597      2929      3879      4943      5931      6881   time(ms)
*       381:         0         0         0      4115         0         0         0         0         0         0         0         0  29425330
        572:         0         0         0         0         0         0         0         0         0         0         0         0         0
        762:       311         0         0        12         0         0         0         0         0         0         0         0    485164
       1144:      3744         0       323         0        60         3         0         0         0         0         0         6   2566882
       1720:        60         0         0         2         0         1         0         0         0         0         0         0     48926
       2086:         0         0         0         3         0         0         3         0         0         0         0         0       773
       2597:         0         0         0         0         3         0         0         0         0         0         0         0      1427
       2929:         0         0         0         0         0         2         0         0         0         0         0         0       157
       3879:         0         0         0         1         0         0         0         2         0         0         0         0       387
       4943:         0         0         0         0         0         0         0         0         3         0         0         0       103
       5931:         0         0         0         0         0         0         0         0         0         3         0         0       147
       6881:         0         0         0         3         0         0         0         0         0         0         3         0      2115
Total transition : 8663

N.B. current notes were taken via an ssh connection, and the script running in a terminal, screen was off.

When switching to schedutil for CPUs 0-3 (policy0), I always get a crash. Sometimes, dmesg from pstore has something like
[ 1195.018006] healthd: battery l=91 v=4074 t=37.0 h=2 st=3 c=258 fc=2520000 chg=
[ 1197.415508] BUG: scheduling while atomic: sshd/8735/0x00000004

hm, not very informative to me unfortunately, can we get any help from xda community? or they are mainly now into aosp-11? makes me wondering how Jolla got relatively good quality aosp

They may use the same interactive governor as we do. I don't think this is altered from the one set by AOSP.
All in all, we are not too bad off. I will

check if the crash happens with AOSP. If it does, file issue upstream
update the script a bit and let's test it more. While Android does not seem to use hotplugging as far as I can see on Sony AOSP nor stock, it may make sense to use it on SFOS. If this POC works, I can write a small daemon for it. Will discuss it on SFOS channel as well, but first polish the script a bit.
GPU is separate issue and we will handle it separately. But, it maybe integrated with handling CPU as well - I will look whether to make the script aware of it.

Indeed it's that bad, even the GPU issue is ok-ish providing that it saves battery.
Actually without understanding much it seems that CPU policy is one scale higher than the stock version (4577 vs 2288), and GPU is similar (381 vs 0). Does that also explains the fact that tama devices on ssh are a lot more responsive than xa2?
Let's see what the others think, happy to try the new script. BTW I was thinking to upgrade the handler and print e.g. battery level or stats, or that will become heavy weighted?

Re CPU being higher: it is not in general, just snapshots from two cases.
Re responsive - it could be due to less CPU sleep, not sure. but CPU is way faster in Tama
Re GPU: will ask why they removed it

AOSP10 does allow to use schedutil and I observed a crash only once so far.
AOSP10 Issue filed: https://github.com/sonyxperiadev/bug_tracker/issues/705

hopefully we get something good from all this, thanks for your effort!

I run the script overnight, 5% drop in 8.5 hours (95 -> 90) not bad.

I think the daemon can help us a lot and it makes sense as well. There are more issues with schedutil (see bug in AOSP10 repo, reference above), so we will probably stick with the current governors.
I will tune the script a bit and then we can test it further.

Started making tests on suspend as well.
On clean SFOS install:

fully offline: suspend in excellent. Hitting 600 and 1200 seconds per suspend, or similar.
no network, 4G SIM: 6 second average
no network, 4G SIM: 7 second average
no network, 2G SIM: 67 seconds average

So, looks like 4/3G are leading to interrupts that wake up the device. Which is not surprising, I expect the same on AOSP

I have replaced the script with a small daemon that does the same: https://github.com/rinigus/zgovernor
It can be installed from Tama repos using zypper in zgovernor (just remember to refresh them).
On install, the daemon is automatically enabled and started. You could follow its messages using journalctl or systemctl status zgovernor
In addition to previous changes, it also switches power-efficient CPUs to conservative cpufreq governor on screen going off. As you could see in its config file (installed at /etc/zgovernor.ini and visible at https://github.com/rinigus/zgovernor/blob/master/example/zgovernor.ini), it will be easy to expand it to handle GPU as well.
Would be good to test it and see if it improves the battery consumption. At the same time, device shouldn't be slow either :)

Installed and running now.
upon device restart I get:
[root@XperiaXZ3 defaultuser]# systemctl status zgovernor
● zgovernor.service - Z Governor
   Loaded: loaded (/usr/lib/systemd/system/zgovernor.service; enabled; vendor preset: enabled)
   Active: inactive (dead)

is that right? I had to restart it manually

That was a bug in service file - thank you for reporting it! Please update, new version is uploaded

Saw a glitch in the daemon that prevented CPU sleep. Maybe similar (if it was) to your case where python script hanged somewhere. Will have to make it more robust.

This time it was /usr/bin/msyncd with long suspend inhibition for some reason, not the daemon. So, just would have to keep an eye on it.

power consumption  has improved  dramatically i am closing now 24h and the drop is 95 -> 73, it was 82 earlier but i used few apps the last hour

From the discussion on Sony channel:
Excessive idle drain is a known issue for Tama 4.14. According to Marijn:
Idle drain is at 12mA, bumps to 24mA at a random moment, and to 49 later It used to always go down to 6mA on 4.9 (just like Kumano on 4.14) That's the difference between 1%/h overnight and 1% for a full night (and during the day at idle)
On SFOS, according to my tests: drain for me was between 26mA-57mA, averaging on 32mA during a night with only calls accepted on sim (no net) and earlier 1-2% for a full night was a norm in these conditions, not on 4.14 anymore.

pity isn't it? disappointing quality of tama aosp is like that

it is pity, indeed. in the end, AOSP depends on BLOBs and their quality. So, it is probably down to the quality of those not AOSP itself.
Which we will have to take into account and discuss in some public forum on what to do with the port. SFO probably.

Running Apollo in flight-mode leads to battery drain 0.9%/h - 23mA on average. At the same time suspend was excellent, hitting ~600s per suspend on average.
Akari with SIM card in call accepting mode, no network, had drain ~1%/h - 29mA. Suspend time was about 9s per one suspend on average.
So, looks like drain is caused by something in device that does always consume energy regardless of suspend state.

my general feeling is very similar, is that issue at the aosp layer or further higher in the stack? should we run some tests running the aosp only?

From the discussion on Sony channel:
Excessive idle drain is a known issue for Tama 4.14. According to Marijn:
Idle drain is at 12mA, bumps to 24mA at a random moment, and to 49 later It used to always go down to 6mA on 4.9 (just like Kumano on 4.14) That's the difference between 1%/h overnight and 1% for a full night (and during the day at idle)
On SFOS, according to my tests: drain for me was between 26mA-57mA, averaging on 32mA during a night with only calls accepted on sim (no net) and earlier 1-2% for a full night was a norm in these conditions, not on 4.14 anymore.

Woah, hitting 6mA on idle on AOSP, that is great figure. If I understand it right, this is not the case with SFOS even with 4.9, correct? I am seeing ~22mA as average (never below 16) when only mobile network is on to accept calls. Running 4.9 on SFOS 4.0.1, Apollo.
Would it be possible to disable some sensors (like pressure sensor) and test if those are consuming power even when not in use? Like, disable from an even lower level than the OS itself?

I presume this is in flight mode. In flight mode on AOSP9-based SFOS I am quite sure we can get very low in this case.
@lal883: How do you measure power consumption? I let collectd do it's work for some time and take an average for few hours. Maybe should repeat that test on AOSP9 SFOS - would have to reflash then...

I run a script to record the current consumption to a file at 1 min intervals. It doesn't record when the phone is sleeping though, as I understand.
Collected data looks like this
(time | % | BMS reported remaining time | current)
10:38 - 51 % | 10 hr 17 min | 20 mA
10:42 - 51 % | 10 hr 17 min | 21 mA
10:47 - 51 % | 10 hr 17 min | 22 mA
10:50 - 51 % | 10 hr 17 min | 22 mA
10:55 - 51 % | 10 hr 17 min | 22 mA
10:58 - 51 % | 10 hr 17 min | 22 mA
11:03 - 51 % | 10 hr 17 min | 22 mA
I can do the collectd test and report back too. Will also check flight mode.

Good point regarding not recording at sleep. In this respect, drop in battery % is probably most reliable. Just requires longer measurement.

is it reasonable to expect a newer release of the aosp blobs? the current release is from dec 2020

I suspect that no, looks like AOSP10 is sunset and development shifted to AOSP11. I will test whether legacy GPU driver will work and maybe check out if there is something else out there.

Legacy MSM driver was no go - it has not been updated in the kernel to newer interface and we cannot use it. One more option out of the window!

maybe we have to use aosp11 blobs, but that then will have to port SFOS... or wait until xperia 10 iii is ported.
let's be realistic, I can live with GPU glitches, the increased power consumption is just about bearable I would say? the rest of the stuff and apps are working ok, and the camera is ok-ish, at least it does not crash.
is there any expectation that we might get an improved driver in the near future?

I don't expect we will get newer AOSP10 driver, I am quite sure it is as good as it is. As there have been no updates for AOSP10, that's all we can expect. And probably now we need to realize that the current state of AOSP10 would not improve much and discuss the options. I will probably open a new thread at FSO to make visible for all users.

For reference, on AOSP9 based SFOS port

Apollo in flight mode: 0.25%/h drain, 160s suspend on average. This is VS  0.9%/h on AOSP10 based port
Apollo with SIM card in 4G accepting calls only (no net): 0.9%/h, 5s suspend; This is VS 1%/h on different device (akari) in AOSP10. Not sure I can compare it like that, though, as devices are different and I would have to retest on the same device...

interesting, almost 4 times higher consumption in flight mode. but no difference with the sim enabled.
my usage has also on the wifi.
what tool do you use to collect the data?

Battery state could be different in different devices (my apollo vs my akari). So, as mentioned, I have to check with AOSP10 on Apollo as well.
I use collectd/SystemDataScope (obviously, as I ported collectd and wrote SDS), but probably something else can be used. Just measurements take time ...

thanks, it's important to use the same tools, i was a bit hesitant running collectd not knowning its overhead.

It should be minimal. But you could just record % on the paper as well :)

Observed the usage on Apollo, SFOS 4.0.1, Kernel 4.9, overnight. Had 4G network connected but not internet.
Printed the energy as well. Turns out the average current consumption is around 25mA looking at the energy figures.
`
01:13 - 16 % | 297133 uAh | 3 hr 26 min | 20 mA
01:24 - 16 % | 292934 uAh | 3 hr 26 min | 21 mA
01:31 - 16 % | 289519 uAh | 3 hr 26 min | 22 mA
01:44 - 16 % | 284273 uAh | 3 hr 26 min | 20 mA
01:59 - 16 % | 277877 uAh | 3 hr 26 min | 23 mA
02:15 - 16 % | 271402 uAh | 3 hr 26 min | 22 mA
Energy consumption (1:13 to 2:15) = 25731 uAh
02:29 - 16 % | 265436 uAh | 3 hr 26 min | 21 mA
02:50 - 16 % | 256871 uAh | 3 hr 26 min | 20 mA
03:16 - 16 % | 245899 uAh | 3 hr 26 min | 21 mA
Energy consumption (2:15 to 3:16) = 25503 uAh
03:51 - 15 % | 231519 uAh | 3 hr 13 min | 22 mA
04:19 - 15 % | 217880 uAh | 3 hr 13 min | 21 mA
Energy consumption (3:16 to 4:19) = 28019 uAh
04:35 - 15 % | 210962 uAh | 3 hr 13 min | 21 mA
05:03 - 15 % | 199933 uAh | 3 hr 13 min | 180 mA
Energy consumption (4:19 to 5:3) = 17947 uAh
05:29 - 15 % | 188796 uAh | 3 hr 13 min | 21 mA
05:52 - 14 % | 179697 uAh | 3 hr 1 min | 22 mA
06:18 - 14 % | 169058 uAh | 3 hr 1 min | 22 mA
Energy consumption (5:3 to 6:18) = 30875 uAh
06:36 - 14 % | 161465 uAh | 3 hr 1 min | 22 mA
06:54 - 13 % | 153819 uAh | 2 hr 48 min | 21 mA
07:09 - 13 % | 147663 uAh | 2 hr 48 min | 22 mA
Energy consumption (6:18 to 7:9) = 21395 uAh
`

I'll have to run the same test on my Apollo with AOSP10/aarch64 - then we can compare by battery %. Would have to see where I can get uAh data - any quick pointers?

@rinigus I was using the value recorded in "/sys/class/power_supply/battery/charge_counter"
Attaching the script that I use. Not very great, does the purpose, I believe. Records to a file "batterylog" in the same directory as the script.
batterymon.zip

Thanks!

on xz3, aosp10, zgovernor enabled, using collectd, in flight mode, user in deep sleeping mode too:
battery 29uA on average, min 25uA over 4h period.

@pagism : what about battery %/h? feeling what it was before? as uA are probably picked up during resume moments

@rinigus observing the battery % graph consumption is 1%/h
can we have a csv form of yhe graph too?

csv is maybe possible, but not trivial. easiest is to choose time window and then just calculate %/h from min/max values

Update after testing Apollo AOSP9 and AOSP10. This is an update from earlier message where I had to compare to Akari in one condition. Turned out that Akari and Apollo data are touch different, but difference is not major:
On AOSP9 based SFOS port

Apollo in flight mode: 0.25%/h drain, 160s suspend on average. This is VS 0.9%/h on AOSP10 based port
Apollo with SIM card in 4G accepting calls only (no net): 0.9%/h, 5s suspend; This is VS 1.17%/h in AOSP10, suspend times 5.5s.

So, when used with SIM, the difference was not that massive. Here, AOSP10 was using zgovernor, not used in AOSP9

The figures are not that bad, there is massive difference in flight mode, but practically the use of that mode is minimal so not an real issue.
The result with SIM card in is more interesting, that's 30% higher, however in practice that impact might not be that significant if we take into account normal use of the phone with active connections and screen on where consumption is a lot higher and will have more impact on battery.
So let's try to use the device regularly, under normal conditions and revisit the issue.

Agreed, such testing is pretty much what we do these days

Tested without network, just SIM, on AOSP10 and without zgovernor - so no CPU on/off switching. Result was 1%/h on Apollo, so marginally better than with zgovernor. Will have to test on Akari and see if with normal usage zgovernor actually helps at all. But that is rather difficult to compare, though.

I am going to test zgovernor changing GPU settings only. Got a phone responding in a bit choppy manner today and I suspect it is due to shifting most of its work to slow CPUs. Let's see whether it will hit the power consumption - I hope not much.


Getting alle unsubscribed contacts of list

Hi,
I like to get all unsubscribed contacts of a list.
I tried something like this:
function mailjetGetUnsubscribedContactsFromList(Mailjet $mj, $listID) {
$params = array(
"method" => "GET",
"ListID" => $listID,
"IsUnsubscribed" => true
);
$result = $mj->listrecipient($params);
if ($mj->_response_code == 201)
return "Okay";
else
echo "error - ".$mj->_response_code;
}
But the contact-ID is for listrecipient-function mandatory, too.
Is there anyway to solve my problem?

Hi @wfabian,
In order to get a list of all unsubscribed contacts of a list you need to filter the contact list. Here's a code snippet that does this:
function getUnsubscribedContactsFromList(Mailjet $mj, $listID) {
	$params = array(
		"method" => "GET",
		"ContactsList" => $listID,
		"Unsub" => true
	);

	$result = $mj->listrecipient($params);

    if ($mj->_response_code == 200)
       echo "success - got unsubscribed contact(s) ";
    else
       echo "error - ".$mj->_response_code;

	return $result;
}

I hope this helps. Don't hesitate to contact me if you need more help,
This code snippet and more could be found at https://github.com/mailjet/mailjet-apiv3-php-simple/blob/master/README.md
For more contact list filters you can have a look at our reference guide - http://dev.mailjet.com/email-api/v3/listrecipient/#filters
Cheers!

Thank's again, that solved my problem.

Hey @wfabian :airplane:
We just released a new PHP API wrapper.
It is now more PHP-ish (including namespace, PSR-0 compliance, and globally a better architecture).
All it requires is PHP 5.4
In addition to our API Guides, we would be happy to help you get started with it.
Guillaume, from Mailjet!


Action (as the example in README) is failing because of length undefined

This is the action:
on:
  release:
    types: [published]

jobs:
  github-releases-to-discord:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Github Releases To Discord
        uses<EMAIL_ADDRESS>        with:
          webhook_url: ${{ secrets.DISCORD_WEBHOOK }}
          color: '2105893'
          username: Release Changelog
          avatar_url: 'https://cdn.discordapp.com/avatars/487431320314576937/bd64361e4ba6313d561d54e78c9e7171.png'
          content: '||@everyone||'
          footer_title: Changelog
          footer_icon_url: 'https://cdn.discordapp.com/avatars/487431320314576937/bd64361e4ba6313d561d54e78c9e7171.png'
          footer_timestamp: true
          max_description: '4096'
          reduce_headings: true

This is the output

I'm not user whats failing, I'd apprecite any help 🙌

It seems that we were copying the last /github fragment and it was not being detected as keyword


Add cautionary assert for divide-by-zero

What
Adds a cautionary assert for divide-by-zero on get_permutated_index
Why

There is already is an implicit assert with the modulo

pivot = bytes_to_int(hash(seed + int_to_bytes1(round))[0:8]) % list_size

This produces

ZeroDivisionError: integer division or modulo by zero

My objective was to make it explicit.
Maybe I've been spending too much time in a language focused on safety.

Now I think about it, the spec is full of implicit asserts.
Happy to close this one.

Technically, we really should have an assert that index < list_size, which would implicitly require list_size > 0.

Technically, we really should have an assert that index < list_size, which would implicitly require list_size > 0.

I was just about to raise this actually! It still returns a result when index >= list_size.


Support security options for container creation in job

This will always fail circle CI tests bc their version of docker doesn't support this feature yet.

@rohansingh @gimaker

:+1: but guessing this will have to be rebased after #487 is merged.

yea


Teleprompter metric

Should the metric only return bool values or can it return a float? in the case of the latter, I assume DSPy will try to maximize it?

Please refer to this duplicate issue: #145

Answered in depth now at: #298


CI linux multiarch out of disk space

We've recently encountered a few "no space left on device" errors for the linux multiarch CI target.
Example 1:
#33 ERROR: failed to create temp dir: mkdir /tmp/containerd-mount2182378471: no space left on device
------
 > exporting to image:
------
error: failed to solve: failed to create temp dir: mkdir /tmp/containerd-mount2182378471: no space left on device
The push refers to repository [docker.io/envoyproxy/envoy-contrib-debug-dev]
An image does not exist locally with the tag: envoyproxy/envoy-contrib-debug-dev

Example 2:
#33 ERROR: mount callback failed on /tmp/containerd-mount335367424: mount callback failed on /tmp/containerd-mount2023061770: failed to write compressed diff: failed to create diff tar stream: failed to copy: write /var/lib/buildkit/runc-overlayfs/content/ingest/071fc2118cf86a76869e40667e29250e1f7baba41673234852e99dfc7ae4ac2b/data: no space left on device
------
 > exporting to image:
------
error: failed to solve: mount callback failed on /tmp/containerd-mount335367424: mount callback failed on /tmp/containerd-mount2023061770: failed to write compressed diff: failed to create diff tar stream: failed to copy: write /var/lib/buildkit/runc-overlayfs/content/ingest/071fc2118cf86a76869e40667e29250e1f7baba41673234852e99dfc7ae4ac2b/data: no space left on device
The push refers to repository [docker.io/envoyproxy/envoy-contrib-debug-dev]
An image does not exist locally with the tag: envoyproxy/envoy-contrib-debug-dev
##[error]Bash exited with code '1'.

cc @phlax @lizan you may have a better understanding of what's going on here.

yep - it used to clear disk space for all jobs, but was adding a lot of ci time - so i removed it on the basis that we can readd for any jobs that need it
that said, i think this is one of the jobs where it still does this - so i guess we can clear some more space


[Bug] API settings page is not getting updated on selecting a different API

Description
API settings page is not getting updated on selecting a different API. The correct page shows up after selecting a different page and then coming back to the settings page.
https://user-images.githubusercontent.com/1757421/108990504-f7e76b80-76bc-11eb-8b21-98bc1f7ace10.mov
Steps to reproduce the behaviour:

Go to Api1 settings.
Click on execute on load.
Select Api2.
See that settings page has not changed to reflect Api2 settings.

Important Details

Version: Cloud
OS: Linux
Browser: Chrome

I also faced this on Firefox on my Linux computer


ESP_GATT_INSUF_ENCRYPTION

Thanks a lot for your work Neil!
When using the BLE_Client example i get ESP_GATT_INSUF_ENCRYPTION in response from a sensor device (see below). The device is not using encryption, as i can read values from the nRF connect app.
in BLEUtils.cpp i implemented the read (similar to ESP_GATT_OK):
if (evtParam->read.status == ESP_GATT_INSUF_ENCRYPTION) {
  GeneralUtils::hexDump(evtParam->read.value, evtParam->read.value_len);
}
No success. Any suggestions, how to proceed?
Debug:
[D][BLERemoteCharacteristic.cpp:439] readValue(): >> readValue(): uuid: 00002a6e-0000-1000-8000-00805f9b34fb, handle: 25 0x19
[D][BLEDevice.cpp:143] gattClientEventHandler(): gattClientEventHandler [esp_gatt_if: 4] ... ESP_GATTC_READ_CHAR_EVT
[D][BLEUtils.cpp:1377] dumpGattClientEvent(): GATT Event: ESP_GATTC_READ_CHAR_EVT
[D][BLEUtils.cpp:1527] dumpGattClientEvent(): [status: ESP_GATT_INSUF_ENCRYPTION, conn_id: 0, handle: 25 0x19, value_len: 0]
[D][BLERemoteCharacteristic.cpp:467] readValue(): << readValue(): length: 0

Thanks for reply, i agree.
Do you have experience with bonding? Is it implemented in the cpp library?
For reference, i found this document that i will try in a few days: https://github.com/espressif/esp-idf/blob/master/examples/bluetooth/gatt_security_server/tutorial/Gatt_Security_Server_Example_Walkthrough.md

I cant say im expert, but know some about it (BLESecurity is my code).
First of all you have to understand how your sensor works. In my opinion best to do it is to use nRF connect on android. Its easy to use and easy to debug all messages or see how it is constructed from ble point of view (services, characteristics, descriptors etc). You will also see what kind of pairing/bonding it requires when you will try to read/write protected characteristic.

@hetlelid there has been no activity in this thread for several months. If your issue is solved, could you please close it to keep the issue list clean?
cc @chegewara

Closed #631