Conductor for B2B and Enterprise level Solution

Hi All,
I'm looking over the Netflix Conductor in order to use it as microservices orchestrator for various business cases. It looks pretty what is needed, but would like to check some things. I already walked through the doc and some github issues/question so many questions were resolved, but still some of them there which would be great to check with you:

Metrics/Logs Visualization: there is elasticsearch which as far as I got gathering workflows/tasks states changes and events. Is there any Kibana build outofthebox in Conductor which allows to watch over those details? Please advice.
Can elasticsearch be replaced with another tool like Scalyr for example? Could you please let me know what is required to do in order to switch it? If it requires code changes of conductor itself or matter of configs?
I assume with docker configs it is possible also to convert and create Kubernetes manifest files and run Conductor and it's throd parties as kubs cluster. Does it require any additional configurations or setups to do in order run cluster which will be able to process 100m workflows a day?
Regarding Zookeepr, is it enough just switch it on in configs. What is expected to be add in connections property? The link is broken in the doc to check some technical details.
Is it possible to use RabbitMQ as external events provider for the Conductor? In the doc it mentioned SQS, but I assume it is possible to use any MQ? If yes, does it require any conductor server code changes or matter of config?
HADR capability, as far as I got it is achieved with Dynomite and cluster scalability and the possibility to automatically store dumps on S3 or other cloud storage. Is that the right assumption?
Security: is there security support - authorization / authentication / encryption etc? Did not get from doc if there are some abilities with security. Please advice.
"Also there was a comment "The task execution is generally asynchronous but it can be enforced to run synchronously" - I found it in some blog article about conductor. So I wonder if it is mean with System tasks which looks like is executing in sync, but generally, it is the pure async solution to run worker tasks etc?
Distribution Execution Capability - if my kub cluster of Netflix will grow due to number of workflows to be executed (100k concurrently), so when it is scaled does it allow existing running workflows to start task execution on the new Conductor pods running or it will be running on the same instance where it was initiated? Maybe I missing something by explaining the topic, please let me know.
Finally, just thinking about the most critical point in the Conductor architecture - looks like it is Dynomite - if it down, then workflows are not executing and failing. So, something like single point of failure. But once it will be resurrected back - will conductor resume the workflow processing from the last stage where it was? Will it rerun tasks which stuck in progress or failed. Or how it will work, please provide some inputs on this.

Really appreciate your help and asnwers Team. Thanks!
Sorry in case some points are obvious or not clear.

@maxygo Thanks for considering Conductor for your use case.

Metrics/Logs Visualization: there is elasticsearch which as far as I got gathering workflows/tasks states changes and events. Is there any Kibana build outofthebox in Conductor which allows to watch over those details? Please advice.

We have ES5.x Kibana templates which we haven't published, but would be glad to do so. Also, we publish several metrics, which you can configure and visualize with tools like Atlas, Prometheus etc. Check this for more details: https://netflix.github.io/conductor/metrics/server/#publishing-metrics

Can elasticsearch be replaced with another tool like Scalyr for example? Could you please let me know what is required to do in order to switch it? If it requires code changes of conductor itself or matter of configs?

Indexing is optional and is extensible. However, you'd have to plugin your own module, say that implements Scalyr, similar to es5-persistence module in Conductor, which you can then configure in your conductor server set-up. You can similarly extend Conductor by plugging in your implementations for these interfaces: https://netflix.github.io/conductor/extend/#backend

I assume with docker configs it is possible also to convert and create Kubernetes manifest files and run Conductor and it's throd parties as kubs cluster. Does it require any additional configurations or setups to do in order run cluster which will be able to process 100m workflows a day?

Conductor is horizontally scalable. There are folks who have run it on Kubernetes (Gitter channel would be the good place to check about it), and you should be able to process 100M workflows with some minimal tuning of provided knobs.

Regarding Zookeepr, is it enough just switch it on in configs. What is expected to be add in connections property? The link is broken in the doc to check some technical details.

Zookeeper module is enabled by default. Take into consideration Zookeeper's scalablity, if you're planning to run Conductor at high scale. Also, consider redis-lock module for locking.

Is it possible to use RabbitMQ as external events provider for the Conductor? In the doc it mentioned SQS, but I assume it is possible to use any MQ? If yes, does it require any conductor server code changes or matter of config?

Yes, Someone just contributed AMQP bindings. You should be able to do the same with RabbitMQ. We don't have RabbitMQ bindings though.

HADR capability, as far as I got it is achieved with Dynomite and cluster scalability and the possibility to automatically store dumps on S3 or other cloud storage. Is that the right assumption?

There are several ways to achieve this. Conductor doesn't advocate one over the other and provides a generic solution which you can use to achieve your needs. For eg., Conductor can be set-up cross region with persistence layers that supports it, and achieve HADR

Security: is there security support - authorization / authentication / encryption etc? Did not get from doc if there are some abilities with security. Please advice.

AuthN and AuthZ is lacking in Conductor 2.0, and is one of the major features we'd like to work on for Conductor 3.0. We don't have an ETA yet for this.

"Also there was a comment "The task execution is generally asynchronous but it can be enforced to run synchronously" - I found it in some blog article about conductor. So I wonder if it is mean with System tasks which looks like is executing in sync, but generally, it is the pure async solution to run worker tasks etc?

Correct. Worker tasks would go through Task lifecycle which involves pushing tasks to queues, polling for and executing the task, which is asynchronous. However, system tasks can skip "pushing to queue" part and can be executed immediately, which is as synchronous as it can get.

Distribution Execution Capability - if my kub cluster of Netflix will grow due to number of workflows to be executed (100k concurrently), so when it is scaled does it allow existing running workflows to start task execution on the new Conductor pods running or it will be running on the same instance where it was initiated? Maybe I missing something by explaining the topic, please let me know.

Conductor is stateless about workflow execution. This helps to easily scale down or scale up the Conductor cluster. Each node in the cluster works independently and hence the need for global distributed lock to avoid race condition between two nodes trying to evaluate same workflow.

Finally, just thinking about the most critical point in the Conductor architecture - looks like it is Dynomite - if it down, then workflows are not executing and failing. So, something like single point of failure. But once it will be resurrected back - will conductor resume the workflow processing from the last stage where it was? Will it rerun tasks which stuck in progress or failed. Or how it will work, please provide some inputs on this.

Apart from Dynomite, Conductor also supports MySQL, Cassandra and few other databases. And if the persistence layer is down or transiently unavailable, the workflows will be terminated with a handled exception. However, we're working on a solution to recover the workflow from persistence failures, and keep it running, so that when the persistence layer resumes, the workflows can continue processing from where it left.
Also, when you mentioned 100M workflow execution, we're curious about the frequency. Can you provide us estimates about the volume and variety of your workflow executions per day.
Feel free to reach out with further questions or comments.

@kishorebanala Thanks a lot for the inputs. It was really helpful.

Also, when you mentioned 100M workflow execution, we're curious about the frequency. Can you provide us estimates about the volume and variety of your workflow executions per day.

There are not much details so far about this, we expect to have a different kind of workflows (not yet designed) which will involve parallel tasks execution (fork, join, exclusive join), conditional paths, subworkflows (it looks like a good solution to reuse similar workflow in more complex workflows), event handling.
As for the volume, it might be the case that the conductor should process up to 1-2 billion of workflows during a month. Most of the workflows would be able to process for 1-5 mins, and some of them might take several hours.
I also have some other few inputs:

You mentioned that the scalability of the conductor allows processing 100m during a day. Any suggestions/advice in the needed powers?
Use Case: workflow has the following tasks Simple Task 1 -> Simple Task 2 -> and next parallel execution with 1 Simple task and some task which should be executed only on some Customer Acceptance first; once Customer accepts we should trigger the next task to be executed. How this can be achievable? I assumed it might be dynamic workflows and event handling, but maybe you can suggest some better solution?
Dynomite is suggested for HA as per configs. Also, there is a possibility to dump workflows and tasks definitions right (to S3)? If so, there is also some mechanism to restore dump in case of need and distribute through the cluster?  If you will have a chance, could you please share the docs on it or suggestion to consider?
when you said conductor has few persistent solutions (MySQL, Cassandra). Does it mean that Conductor will not use Dynomite and instead will store all workflows and tasks definitions, statuses within the MySQL/Cassandra? So, how in that case Conductor Queue will work? Does it still involve dynomite / redis? Not sure I see clear vision on this.

Thank you also for gitter suggestion, I will check it and check some details here as well.

@maxygo No worries, glad to help!

You mentioned that the scalability of the conductor allows processing 100m during a day. Any suggestions/advice in the needed powers?

Well, we only load tested to the extend of processing around 10M workflows a day. While it is possible to scale Conductor to 100M workflows, I'd advise to invest into a load testing with synthetic data first and verify your use case. It might also help uncover potential bottlenecks at higher scale for your use case.

Use Case: workflow has the following tasks Simple Task 1 -> Simple Task 2 -> and next parallel execution with 1 Simple task and some task which should be executed only on some Customer Acceptance first; once Customer accepts we should trigger the next task to be executed. How this can be achievable? I assumed it might be dynamic workflows and event handling, but maybe you can suggest some better solution?

Yes, you can use event handler to complete a task. Other option is to periodically execute the same task by updating task with IN_PROGRESS state and reasonable callBackAfterSeconds.

Dynomite is suggested for HA as per configs. Also, there is a possibility to dump workflows and tasks definitions right (to S3)? If so, there is also some mechanism to restore dump in case of need and distribute through the cluster? If you will have a chance, could you please share the docs on it or suggestion to consider?

Dynomite is maintained by a separate team at Netflix, and they can better answer this question in their support channel.

When you said conductor has few persistent solutions (MySQL, Cassandra). Does it mean that Conductor will not use Dynomite and instead will store all workflows and tasks definitions, statuses within the MySQL/Cassandra? So, how in that case Conductor Queue will work? Does it still involve dynomite / redis? Not sure I see clear vision on this.

It is possible to use different databases for Queues and Workflow/Task persistence. Eg: You can use Dyno-queues for queues, and MySQL for persistence. This can be configured by plugging in the core interfaces I referenced above to one of the implementations in Conductor server.
Feel free to ask if you have further questions / comments.


Why is my charger only charging with 5A?

Dear,
My setup is almost working but I can't get more than 5A out of the charger. When I select 13A for example I get 13A available but nothing changes and the car keeps charging on just 5A (I can stop and start the charging process so I have communication) but I want to charge the car faster and its possible hardware wise. I have 'wiringMaxAmpsAllTWCs' at 40 so that shouldnt be an issue. Anyone knows how to solve this ?
Thanks in advance.
Best regards,
Dennis

I don't think it's an issue with this project, but I had the same thing happen to my Model 3.  After a few hours of having my home automation system adjust the TWC power and turning the car on and off, the car's internal amp limit got stuck at 6 no matter what the TWC did.  I had to replug the connector to reset it.  Manually changing the limit might have worked to.  Seems like it's a bug and/or feature in the car software.  I'm working on a solar power tracking algorithm that involves fewer commands sent to the car to see if that helps.

I don't think it's an issue with this project, but I had the same thing happen to my Model 3. After a few hours of having my home automation system adjust the TWC power and turning the car on and off, the car's internal amp limit got stuck at 6 no matter what the TWC did. I had to replug the connector to reset it. Manually changing the limit might have worked to. Seems like it's a bug and/or feature in the car software. I'm working on a solar power tracking algorithm that involves fewer commands sent to the car to see if that helps.

Allready fixed!

Allready fixed!

May I ask what you did to fix it, or how you determined it is or is not the fault of TWCManager?  I'm still having this issue with the latest code from the repository.

Allready fixed!

May I ask what you did to fix it, or how you determined it is or is not the fault of TWCManager? I'm still having this issue with the latest code from the repository.

Are you limiting the charge current in-car?


[DOC] User guide needs updating

Describe the issue linked to the documentation
user guide page is out of date
https://www.aeon-toolkit.org/en/latest/user_guide.html
not sure what goes in here. Package wide notebooks?
Suggest a potential alternative/fix
No response

We no longer have a user guide page, links to intro notebooks are on the index page.


Run the new-e2e-containers tests when pkg/clusteragent/admission/mutate is modified

What does this PR do?

Run the new-e2e-containers tests when pkg/clusteragent/admission/mutate is modified
Motivation

Action item after #incident-29616
Follows https://github.com/DataDog/datadog-agent/pull/27811 that got fixed with https://github.com/DataDog/datadog-agent/pull/28340

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

/merge


shortcut for blacklisting whole package

I think it'd be nice to have a shortcut identifier to blacklist a whole package from a given path.
like
{
  "target": "domain/**/*",
  "from": "~contracts"
}

would prevent any file within domain from importing from the contracts package.
Note:
I feel like this should already be possible with the node resolver, but I'm not able to get it to work without dragging in lots of other errors...
EDIT
had target and from flipped, just in case you're reading the comment history

I'm a bit confused. Generally one only lints first-party code - or do you mean ./domain/**/* the local files as opposed to the domain package?
Assuming that's what you mean, it seems like you can do this using the core https://eslint.org/docs/latest/rules/no-restricted-imports / https://eslint.org/docs/latest/rules/no-restricted-modules rules, and eslint overrides on that path?

the thing is that when using npm workspaces contracts is first party code.
So imagine I have a frontend package with a hexagonal architecture; I'd like to be able to prevent direct imports from the contracts package only in the domain layer. no-restricted-modules would blacklist that import everywhere in the frontend package, wouldn't it?
I can work around this by setting "target": "../node_modules/contracts" but that's a bit ugly :)

jeez, I see what you mean now... I've flipped "from" and "target"...

in that case, you don't want to be linting the contracts that's inside node_modules - you want to be linting the actual workspace's location.
Using eslint overrides, you'd prevent that import only in the files you wanted to prevent it in.

no, I'm not linting the files inside contracts, I've flipped the "from" and "target" originally - sorry for the confusion. I want to prevent importing from contracts in domain.
I've adjusted the original text to reflect this

OK - same solution applies tho.
Use eslint overrides to create a "rules" section (in the only eslintrc that should be in your repo, the one at the root) that only applies to files within domain. In that override, specify no-restricted-imports and no-restricted-modules so that files in contracts can not be required or imported.

yes, that works, same as directly pointing to the node_modules folder. However, I'd like to keep my rules for enforcing my architecture as close together as possible.
Is there a reasoning for not implementing the shortcut other than "there are existing solutions"? Don't get me wrong, nothing wrong with that reasoning, I just thought it'd be neat to have a shortcut.

It’s added complexity that is already achievable. The slippery slope is that every rule has to add such a shortcut - why go down that road when overrides exist?


[0.9.5] OOM and timeout issue

I applied influxDB(0.9.5) to my production environment, but It dies too frequently yelling out of memory error.
(and there are bunch of timeout logs, both tsm1 and bz1)
input : 30000 points/sec across 20 kinds of table(serie?)
Is there any solution or tip for performance?
If I'm using influxDB in wrong way, please let me know.
hardware spec and error logs are below.
hardware

cpu /

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 24
On-line CPU(s) list: 0-23
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 45
Stepping: 7
CPU MHz: 1999.995
BogoMIPS: 3999.44
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 15360K
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23

storage/
SSD enough space

ram/
64GB

Error log is like this(OOM case)
Nov 26 18:30:11 LGRNGDM7513 kernel: influxd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Nov 26 18:30:11 LGRNGDM7513 kernel: influxd cpuset=/ mems_allowed=0-1
Nov 26 18:30:11 LGRNGDM7513 kernel: Pid: 33863, comm: influxd Not tainted 2.6.32-504.16.2.el6.x86_64 #1
Nov 26 18:30:11 LGRNGDM7513 kernel: Call Trace:
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff810d41b1>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff81127410>] ? dump_header+0x90/0x1b0
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff8122ee4c>] ? security_real_capable_noaudit+0x3c/0x70
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff81127892>] ? oom_kill_process+0x82/0x2a0
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff811277d1>] ? select_bad_process+0xe1/0x120
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff81127cd0>] ? out_of_memory+0x220/0x3c0
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff8113460f>] ? __alloc_pages_nodemask+0x89f/0x8d0
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff8116c8ba>] ? alloc_pages_current+0xaa/0x110
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff81124807>] ? __page_cache_alloc+0x87/0x90
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff811241ee>] ? find_get_page+0x1e/0xa0
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff811257a7>] ? filemap_fault+0x1a7/0x500
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff8114ec14>] ? __do_fault+0x54/0x530
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff8106d534>] ? enqueue_task_fair+0x64/0x100
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff8114f1e7>] ? handle_pte_fault+0xf7/0xb00
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff81064bf0>] ? wake_up_state+0x10/0x20
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff810b240c>] ? wake_futex+0x3c/0x60
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff8114fe89>] ? handle_mm_fault+0x299/0x3d0
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff8104d096>] ? __do_page_fault+0x146/0x500
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff811d7bb4>] ? ep_poll+0x314/0x350
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff81529ba6>] ? schedule+0x176/0x3a0
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff8100bc6e>] ? invalidate_interrupt3+0xe/0x20
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff8153041e>] ? do_page_fault+0x3e/0xa0
Nov 26 18:30:11 LGRNGDM7513 kernel: [<ffffffff8152d7d5>] ? page_fault+0x25/0x30
Nov 26 18:30:11 LGRNGDM7513 kernel: Mem-Info:
Nov 26 18:30:11 LGRNGDM7513 kernel: Node 0 DMA per-cpu:
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 0: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 1: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 2: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 3: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 4: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 5: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 6: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 7: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 8: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 9: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 10: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 11: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 12: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 13: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 14: hi: 0, btch: 1 usd: 0Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 15: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 16: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 17: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 18: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 19: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 20: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 21: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 22: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 23: hi: 0, btch: 1 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: Node 0 DMA32 per-cpu:
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 0: hi: 186, btch: 31 usd: 30
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 1: hi: 186, btch: 31 usd: 1
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 2: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 3: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 4: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 5: hi: 186, btch: 31 usd: 1
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 6: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 7: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 8: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 9: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 10: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 11: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 12: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 13: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 14: hi: 186, btch: 31 usd: 30
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 15: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 16: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 17: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 18: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 19: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 20: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 21: hi: 186, btch: 31 usd: 1
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 22: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 23: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: Node 0 Normal per-cpu:
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 0: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 1: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 2: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 3: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 4: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 5: hi: 186, btch: 31 usd: 2
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 6: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 7: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 8: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 9: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 10: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 11: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 12: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 13: hi: 186, btch: 31 usd: 1
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 14: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 15: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 16: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 17: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 18: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 19: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 20: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 21: hi: 186, btch: 31 usd: 8
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 22: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 23: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: Node 1 Normal per-cpu:
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 0: hi: 186, btch: 31 usd: 3
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 1: hi: 186, btch: 31 usd: 13
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 2: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 3: hi: 186, btch: 31 usd: 31
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 4: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 5: hi: 186, btch: 31 usd: 31
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 6: hi: 186, btch: 31 usd: 7
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 7: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 8: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 9: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 10: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 11: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 12: hi: 186, btch: 31 usd: 2
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 13: hi: 186, btch: 31 usd: 20
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 14: hi: 186, btch: 31 usd: 30
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 15: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 16: hi: 186, btch: 31 usd: 32
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 17: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 18: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 19: hi: 186, btch: 31 usd: 18
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 20: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 21: hi: 186, btch: 31 usd: 31
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 22: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: CPU 23: hi: 186, btch: 31 usd: 0
Nov 26 18:30:11 LGRNGDM7513 kernel: active_anon:15257901 inactive_anon:1020508 isolated_anon:64
Nov 26 18:30:11 LGRNGDM7513 kernel: active_file:77 inactive_file:0 isolated_file:0
Nov 26 18:30:11 LGRNGDM7513 kernel: unevictable:0 dirty:0 writeback:0 unstable:0
Nov 26 18:30:11 LGRNGDM7513 kernel: free:41026 slab_reclaimable:3970 slab_unreclaimable:12377
Nov 26 18:30:11 LGRNGDM7513 kernel: mapped:229 shmem:0 pagetables:43864 bounce:0
Nov 26 18:30:11 LGRNGDM7513 kernel: Node 0 DMA free:15732kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15336
Nov 26 18:30:11 LGRNGDM7513 kernel: lowmem_reserve[]: 0 3211 32248 32248
Nov 26 18:30:11 LGRNGDM7513 kernel: Node 0 DMA32 free:117564kB min:1616kB low:2020kB high:2424kB active_anon:2033420kB inactive_anon:508452kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(
Nov 26 18:30:11 LGRNGDM7513 kernel: lowmem_reserve[]: 0 0 29037 29037
Nov 26 18:30:11 LGRNGDM7513 kernel: Node 0 Normal free:14600kB min:14624kB low:18280kB high:21936kB active_anon:28037220kB inactive_anon:1752276kB active_file:72kB inactive_file:0kB unevictable:0kB isolated(anon):256kB i
Nov 26 18:30:11 LGRNGDM7513 kernel: lowmem_reserve[]: 0 0 0 0
Nov 26 18:30:11 LGRNGDM7513 kernel: Node 1 Normal free:16208kB min:16276kB low:20344kB high:24412kB active_anon:30960964kB inactive_anon:1821304kB active_file:236kB inactive_file:0kB unevictable:0kB isolated(anon):0kB is
Nov 26 18:30:11 LGRNGDM7513 kernel: lowmem_reserve[]: 0 0 0 0
Nov 26 18:30:11 LGRNGDM7513 kernel: Node 0 DMA: 1*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15732kB
Nov 26 18:30:11 LGRNGDM7513 kernel: Node 0 DMA32: 613*4kB 417*8kB 344*16kB 203*32kB 89*64kB 19*128kB 6*256kB 6*512kB 3*1024kB 7*2048kB 17*4096kB = 117564kB
Nov 26 18:30:11 LGRNGDM7513 kernel: Node 0 Normal: 2666*4kB 26*8kB 4*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 15032kB
Nov 26 18:30:11 LGRNGDM7513 kernel: Node 1 Normal: 3005*4kB 36*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 16420kB
Nov 26 18:30:11 LGRNGDM7513 kernel: 100188 total pagecache pages
Nov 26 18:30:11 LGRNGDM7513 kernel: 99585 pages in swap cache
Nov 26 18:30:11 LGRNGDM7513 kernel: Swap cache stats: add 631336, delete 531751, find 16326/27899
Nov 26 18:30:11 LGRNGDM7513 kernel: Free swap = 0kB
Nov 26 18:30:11 LGRNGDM7513 kernel: Total swap = 2097148kB
Nov 26 18:30:11 LGRNGDM7513 kernel: 16777215 pages RAM
Nov 26 18:30:11 LGRNGDM7513 kernel: 296970 pages reserved
Nov 26 18:30:11 LGRNGDM7513 kernel: 689 pages shared
Nov 26 18:30:11 LGRNGDM7513 kernel: 16432193 pages non-shared
Nov 26 18:30:11 LGRNGDM7513 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Nov 26 18:30:11 LGRNGDM7513 kernel: [ 3081] 0 3081 4589 41 0 0 0 irqbalance
Nov 26 18:30:11 LGRNGDM7513 kernel: [ 3126] 0 3126 1092 1 14 0 0 mcelog
Nov 26 18:30:11 LGRNGDM7513 kernel: [ 3165] 0 3165 5600 2 10 0 0 xinetd
Nov 26 18:30:11 LGRNGDM7513 kernel: [ 3393] 0 3393 129460 125 0 0 0 dsm_sa_datamgrd
Nov 26 18:30:11 LGRNGDM7513 kernel: [ 3571] 0 3571 69809 25 0 0 0 dsm_sa_eventmgr
Nov 26 18:30:11 LGRNGDM7513 kernel: [ 3608] 0 3608 159947 52 17 0 0 dsm_om_shrsvcd
Nov 26 18:30:11 LGRNGDM7513 kernel: [ 3721] 0 3721 5473 11 2 0 0 ipmievd
Nov 26 18:30:11 LGRNGDM7513 kernel: [ 3903] 0 3903 1016 2 0 0 0 mingetty
Nov 26 18:30:11 LGRNGDM7513 kernel: [ 3905] 0 3905 1016 2 18 0 0 mingetty
Nov 26 18:30:11 LGRNGDM7513 kernel: [139409] 0<PHONE_NUMBER> 1 14 -17 -1000 udevd
Nov 26 18:30:11 LGRNGDM7513 kernel: [140363] 0<PHONE_NUMBER>9 15 0 -17 -1000 sshd
Nov 26 18:30:11 LGRNGDM7513 kernel: [140391] 0<PHONE_NUMBER>6 148 1 0 0 snmpd
Nov 26 18:30:11 LGRNGDM7513 kernel: [140761] 0<PHONE_NUMBER>7 51 1 0 0 rsyslogd
Nov 26 18:30:11 LGRNGDM7513 kernel: [140820] 28<PHONE_NUMBER>37 71 1 0 0 nscd
Nov 26 18:30:11 LGRNGDM7513 kernel: [140855] 0<PHONE_NUMBER> 1 2 0 0 atd
Nov 26 18:30:11 LGRNGDM7513 kernel: [119639] 10000<PHONE_NUMBER> 9 15 0 0 noms_nsight
Nov 26 18:30:11 LGRNGDM7513 kernel: [119640] 10000<PHONE_NUMBER>20 4329 10 0 0 noms_nsight
Nov 26 18:30:11 LGRNGDM7513 kernel: [105449] 10000<PHONE_NUMBER> 9 15 0 0 noms_nsight
Nov 26 18:30:11 LGRNGDM7513 kernel: [105450] 10000<PHONE_NUMBER>02 4040 12 0 0 noms_nsight
Nov 26 18:30:11 LGRNGDM7513 kernel: [77302] 0 77302 5572 30 0 0 0 box
Nov 26 18:30:11 LGRNGDM7513 kernel: [77304] 0 77304 98797 174 1 0 0 box
Nov 26 18:30:11 LGRNGDM7513 kernel: [77343] 0 77343 2204 6 0 0 0 box
Nov 26 18:30:11 LGRNGDM7513 kernel: [77344] 0 77344 26058 32 15 0 0 box
Nov 26 18:30:11 LGRNGDM7513 kernel: [78490] 0 78490 28742 19 0 0 0 crond
Nov 26 18:30:11 LGRNGDM7513 kernel: [78535] 38 78535 6787 33 12 0 0 ntpd
Nov 26 18:30:11 LGRNGDM7513 kernel: [66967] 0 66967 5572 30 0 0 0 box
Nov 26 18:30:11 LGRNGDM7513 kernel: [66969] 0 66969 98796 199 1 0 0 box
Nov 26 18:30:11 LGRNGDM7513 kernel: [33699] 0 33699 23403698 16169787 13 0 0 influxd

Each serie has 2~3 tags.

How many possible values do the tags have? RAM usage is highly dependent on the series cardinality. Is your cardinality approaching 10 million series?

I applied influxDB(0.9.5) to my production environment, but It dies too frequently yelling out of memory error.

Is this a fresh install or an upgrade?

(and there are bunch of timeout logs, both tsm1 and bz1)

Are you intermixing the two storage engines, or have you conducted separate tests with each?

input : 30000 points/sec across 20 kinds of table(serie?)

Probably 20 measurements is what you mean. 30k pps is not a high load at all for this machine.
What do the InfluxDB logs show?

It's fresh installed one, about 20 measurements, and I conducted test tsm1 and bz1 separately.
If the performance is highly dependent about series cardinality, maybe that's the point what I should look into, because I'm running several continuous queries to measurements which have 2k~6k number of cardinality.

If the performance is highly dependent about series cardinality

Write and query performance is not very dependent on series cardinality, only the RAM usage strongly correlates.

I'm running several continuous queries to measurements which have 2k~6k number of cardinality

CQs are just as expensive to run as a standard query, and running queries does take RAM. Can you share the output from SHOW CONTINUOUS QUERIES?

Thank you for your kind answer.
SHOW CONTINUOUS QUERIES doesn't return any result, but there are running CQs obviously, something like this.
CREATE CONTINUOUS QUERY "coinhist1m_4_cq" ON rangers_ts
BEGIN
SELECT
sum(freeBalanceVariation) as free_sum, sum(paidBalanceVariation) as paid_sum, count(flowType) as cnt
INTO "coinhist1m_4"
FROM release_PlayerCoinHistory
GROUP BY _flowType, time(1m)
END
Every CQ is much like above, grouping by one tag field, counting number of rows.
Can you advise me tuning the query, please?
*Another question. : )
The CQ above doesn't generate sum(freeBalanceVariation) as free_sum, sum(paidBalanceVariation) as paid_sum columns, but only count(flowType) as cnt column.
Can you advise me what am I missing, please?

SHOW CONTINUOUS QUERIES doesn't return any result

That is difficult to understand. If that returns nothing, there are no CQs installed on the system. Are you executing the query via the Admin UI? If so, please be aware it is not intended to be the primary way to query the database. There are some bugs on SHOW queries where the results are not displayed. Use the CLI or direct curl commands, or examine the actual HTTP response in developer tools to find the output.
Can you run curl -G 'http://localhost:8086/query' --data-urlencode 'q=show continuous queries' or run the same command from the CLI, which is the influx binary?

Every CQ is much like above, grouping by a tag and time, counting number of rows. Is there any possible performance/memory leak on the query? Please let me know.

I don't see anything in that CQ that suggests a problem, although if there are enough CQs running it might cause contention.

The CQ above doesn't generate sum(freeBalanceVariation) as free_sum, sum(paidBalanceVariation) as paid_sum columns, but only count(flowType) as cnt column. Can you advise me what am I missing, please?

Please open a new issue for this, and paste in the results of querying the coinhist1m_4 measurement.

It wasn't caused by CQs, but the some incogitant user of admin tool.
Some user used query without 'where time' phrase.
It seems there's no problem now. Thank you for your help.


 Use a.any() or a.all().")
E           ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all().

../../../anaconda/envs/autograd-sparse/lib/python3.6/site-packages/scipy/sparse/base.py:288: ValueError

Not sure why this is happening, though.

It looks like the problem is the line vspace(vjp_y) == x_vs.  What are the types of each side?  It looks like they should both be vspaces, but one side isn't.

Thanks for looping back, @duvenaud! :smile:
I ran a debugger while running the tests, and this is what I am seeing:
x_vs is the following:
(Pdb) x_vs
TupleVSpace_{'shape': (SparseArrayVSpace_{'t': <class 'scipy.sparse.csr.csr_matrix'>, 'value': <5x5 sparse matrix of type '<class 'numpy.float64'>'
        with 5 stored elements in Compressed Sparse Row format>, 'shape': (5, 5), 'dtype': dtype('float64')}, SparseArrayVSpace_{'t': <class 'scipy.sparse.csr.csr_matrix'>,'value': <5x5 sparse matrix of type '<class 'numpy.float64'>'
        with 0 stored elements in Compressed Sparse Row format>, 'shape': (5, 5), 'dtype': dtype('float64')})}

and
(Pdb) vjp_y
(<5x5 sparse matrix of type '<class 'numpy.float64'>'
        with 0 stored elements in Compressed Sparse Row format>, matrix([[-0.02496764, -1.01590897, -0.59913901,  0.16105753, -0.3390676 ],
        [-0.02118735, -0.86209239, -0.50842466,  0.13667215, -0.28773011],
        [-0.0223457 , -0.90922478, -0.5362213 ,  0.14414431, -0.30346092],
        [-0.02142915, -0.87193107, -0.51422709,  0.13823193, -0.29101385],
        [-0.02235796, -0.90972352, -0.53651544,  0.14422337, -0.30362737]]))
(Pdb) vspace(vjp_y)
*** AttributeError: 'VSpace' object has no attribute '__dict__'

It appears that the issue is when passing vjp_y into the vspace() function. However, the error trace is kind of opaque -- I'm not quite sure how to debug this one though. I hope to treat this as a learning opportunity on autograd; what would be the first things you would think to debug here? I was stuck yesterday when I hit this problem.


case insensitive string replacement

Hello!
Thank you so much for this contribution. It is really useful to me.
I might have come across a bug. In
func process_svg_path()
you use the replacen() function.
This leads to all the letters being upper-case (Godot 4.0), and therefore the match statement cannot distinguish between relative and absolute positions.
A replace(), instead, solves this problem.
Kind regards,
Joel

Thanks Joel, I'll look into it.


v1.1 not available on https://plugins.gradle.org

Hi guys,
thanks for the 1.1 release.
Wenn will it be available on https://plugins.gradle.org/plugin/com.jfrog.bintray?
Best regards,
Peti

The package is linked to the gradle-plugins repo, so it must be something with the sync of the portal.

Most expedient thing to do would be to ask a message to the Gradle org on Bintray. We have a better process for dealing with this there.

Done, thanks!

Looks like v1.1 is missing the 'gradle-plugin' attribute:
https://bintray.com/jfrog/jfrog-jars/gradle-bintray-plugin/1.1/view
Compare with:
https://bintray.com/jfrog/jfrog-jars/gradle-bintray-plugin/1.0/view

OMG, what a fail. Sorry!
cc @eyalbe4 @liorhson

The property has been added to version 1.1.
We'll make sure this does not happen in the future.
Our apologies!

Works now... thanks!


Regenerated jcstress playlist for jcstress and pushed its generator

The initial set of tests is obtained by jcstress.jar -l The generator is:
mixing targets with  to few targets
ensuring no test is run twice
targets are as short as possible where still fully descriptive
regexes for -t runtime are fully quified
Similarly as there is exclude list to not shorten groups where missleading preffix would remain, can be added include list, which would connect remaining tests with 1 or two targets to artificial group. But it was jsut 23 tests in time of this commit

Will resolve https://github.com/adoptium/aqa-tests/issues/5278 if (ever) agreed on

It seems the jcstress issue got staled a bit. Opening from draft for review.

Hello!
In past week, I was playing a lot with the generator, and before we proceed with play-list generation, and I think there is quite a few factors to consider.

experinece: in my runs in RH, the stadnart runtime of all between jcstress2020 and 2024 got aprox 100x times longer

possible solutions is to not run some targets and/or limit the number of cores
important note is, that limiting to 1 core is no go, because it eliminates to much tests.

The runtime is quite rising with the number of allowed cores, so keeping on default ( using all machine cores ) leads to incredibly unpredictable results.

So I have added the optional -c switch to have predictable number of cores generated. Eg -c 2 will mean that all machines will use two cores.
See eg this results I generated .

 1 core : Total time: 9 minutes [0d+00:09:35]
 2 cores: Total time: 16458 minutes [11d+10:18:34]
 3 cores: Total time: 33695 minutes [23d+09:35:21]
 4 cores: Total time: 149339 minutes [103d+16:59:21]
 8 cores: Total time: 149339 minutes [103d+16:59:20]

not sure why 4 and 8 are same, My machine have 4cores with hyper threading, so  8 virtual. Maybe the jcstress is using only real ones or  it estimated 103days as to much . Anyway Imagine it on 100core Arm...

the single jcstress test do not have constant length.
When I did some basic grouping,  I realised, that some group with eg 100 tests runs much longer then some group with 2000 tests.

it is deterministic
It seems that generated tests are those fast ones
It is well visible on following figure:

Results gathered: 14; 100% time of longest group, n% time of ideal group
small.groups.1 with 853tests took 117791s [1d+08:43:11] (100%)(+66%)
org.openjdk.jcstress.tests.memeffects.basic-001 with 816tests took 117504s [1d+08:38:24] (99%)(+66%)
org.openjdk.jcstress.tests.atomics-001 with 694tests took 98783s [1d+03:26:23] (83%)(+40%)
org.openjdk.jcstress.tests.acqrel.varHandles.byteBuffer-001 with 672tests took 96767s [1d+02:52:47] (82%)(+37%)
org.openjdk.jcstress.tests.locks-001 with 646tests took 93023s [1d+01:50:23] (78%)(+31%)
org.openjdk.jcstress.tests.acqrel-001 with 588tests took 84671s [0d+23:31:11] (71%)(+20%)
small.groups.2 with 597tests took 83807s [0d+23:16:47] (71%)(+18%)
org.openjdk.jcstress.tests.acqrel.varHandles-001 with 504tests took 72575s [0d+20:09:35] (61%)(+2%)
org.openjdk.jcstress.tests.atomicity.varHandles-001 with 714tests took 68543s [0d+19:02:23] (58%)(-3%)
small.groups.3 with 528tests took 65915s [0d+18:18:35] (55%)(-7%)
org.openjdk.jcstress.tests.atomicity-001 with 612tests took 64800s [0d+18:00:00] (55%)(-9%)
org.openjdk.jcstress.tests.seqcst.volatiles with 2131tests took 8928s [0d+02:28:48] (7%)(-88%)
org.openjdk.jcstress.tests.seqcst.sync with 2131tests took 8927s [0d+02:28:47] (7%)(-88%)
small.groups.4 with 57tests took 5471s [0d+01:31:11] (4%)(-93%)
Total time: 16458 minutes [11d+10:18:25]
Ideal avg time: 1175 minutes [0d+19:35:36] (100%)
Max seen  time: 1963 minutes [1d+08:43:11] (166%)
Min seen  time: 91 minutes [0d+01:31:11] (7%)
Avg differecne from longest: 59%
Avg differecne from ideal: 60%

You can see that:
org.openjdk.jcstress.tests.seqcst.volatiles with 2131tests took 8928s [0d+02:28:48] (7%)(-88%)
org.openjdk.jcstress.tests.seqcst.sync with 2131tests took 8927s [0d+02:28:47] (7%)(-88%)

Are much shorter then others, despite having much more tests. My Generator can split it, but how, depends on how mch targets we wish to have. This example is with LIMIT=500 and not splitting the huge groups.
Also it is visible that spiting of thsoe groups have sense only since some time per group (in this case, 2 cores, it would be 2.5 hours)
All those times are based on estimations jcstress itself is doing. I have verified that those estimations quite precisse enough. Worst I had seen was +-25% of real run compared to estimate one, but usually less then 10%.
So the questions:

how much cores to use?
How much can the basic split time be?

longstory short  estiamtion (with two cores) are as follows (the times of groups are in last brackets):

* split_exl Limit 10 - 603 groups, from those  7 "small groups" (0.5hours each. %like longest/ideal %17%/? (6m-2.5h)
* split_all Limit 10 - 603 groups, from those  7 "small groups" (0.5hours each. %like longest/ideal %68%/81? (26m-38mm)
* split_exl Limit 50 - 128 groups, from those  6 "small groups" (~2.5hhours each. %like longest/ideal %60/85% (45m-3.5h)
* split_allLimit 50 - 128 groups, from those  6 "small groups" (~2.5hhours each. %like longest/ideal %60/85% (45m-3.5h)
* Limit 100 - 60 groups, from those  7 "small groups" (~4.5hours each. %like longest/ideal %63/79% (2.5h-7h)

Although I will provide full statistics later, It seems that having  more then 150 of groups is impractical and having longer then 3hours per is impractical to. Thusl number of tests i group (limit) should be around 50. As in this are is the glorifed 2.5 hours for "unsplittable" groups, theirs split depends on selected cores and limits

jcstress-20240222. -c 1 -l | wc -l
4489
jcstress-20240222.jar -c 2 -l | wc -l
4489
jcstress-20240222.jar -c 4 -l | wc -l
4489

Bad luck, -l do not honour -c

Full statistics of cores x runs with split possibilities:
https://jvanek.fedorapeople.org/jcstress/details.html
The smallest (10)  groups with large namesapces split are still running, and will run till the end of today, but I doubt that this group wil be used. Will udpate it once finshed.

Hello!
In past week, I was playing a lot with the generator, and before we proceed with play-list generation, I think there is quite a few factors to consider.
* experinece: in my runs in RH, the stadnart runtime of all between jcstress2020 and 2024 got aprox 100x times longer

  * possible solutions is to not run some targets and/or limit the number of cores

After short discussion in https://bugs.openjdk.org/browse/CODETOOLS-7903749, dropping out some tests is no go.
And I forget about -tb.  That narrows the whole case. I still need to investiagte impac of -c  in case -tb is used, but in all cases, what is the maximal  time jcstress are supposed to consume?
Split to 60 tasks where each will run one hour? 30 tasks? 30 minutes? Less, More?

https://bugs.openjdk.org/browse/CODETOOLS-7903749 also suggestys to keep -c missing, thus on max (thus all available cores). I may have two issues:

if some of the machines will have less then 4 cores, then some generated target may end with FATAL: No matching tests.

this may be workaroudned by  generating only targets for -c 2 or 3 but that will skip other interesting tests.
waits for https://bugs.openjdk.org/browse/CODETOOLS-7903748

Full statistics of cores x runs with split possibilities: https://jvanek.fedorapeople.org/jcstress/details.html The smallest (10) groups with large namesapces split are still running, and will run till the end of today, but I doubt that this group wil be used. Will udpate it once finshed.

Updated https://jvanek.fedorapeople.org/jcstress/details.html  but I guess with -tb it no longer metters. For curious, it is very interesting

I had improved the generator so it is using -tb now.
I think best solution is current default or close - Generate list for all cores (so not waiting for https://bugs.openjdk.org/browse/CODETOOLS-7903748) , use all cores on any setup, set -tb to agreed time. And set number of parallel groups. The generated playlist now have 138 groups. 30minutes each, is some 3days of HW time (20s per test)
So the question simplified - Howmuch HW time do you wish to use for jcstress?
I'm now running the OUTPUT=do;LIMIT=500, with (now) default time of 30m per group and all splitting enabled (so some 20 fairly sized groups). That is some 6 hours to finish. and aprox 1-2s per test. Will see, how the enforced time (30m) will fit with final runs.

hm:(  https://bugs.openjdk.org/browse/CODETOOLS-7903748

I had improved the generator so it is using -tb now.
I think best solution is current default or close - Generate list for all cores (so not waiting for https://bugs.openjdk.org/browse/CODETOOLS-7903748) , use all cores on any setup, set -tb to agreed time. And set number of parallel groups. The generated playlist now have 138 groups. 30minutes each, is some 3days of HW time (20s per test)
So the question simplified - Howmuch HW time do you wish to use for jcstress?
I'm now running the OUTPUT=do;LIMIT=500, with (now) default time of 30m per group and all splitting enabled (so some 20 fairly sized groups). That is some 6 hours to finish. and aprox 1-2s per test. Will see, how the enforced time (30m) will fit with final runs.

@smlambert "So the question simplified - Howmuch HW time do you wish to use for jcstress?" is really up to you. If it would be mine infra, I would give whole jcstress 12-24 houirs, split to some 12-24 chunks one hour each....

So the question simplified - How much HW time do you wish to use for jcstress?" is really up to you. If it would be mine infra, I would give whole jcstress 12-24 hours, split to some 12-24 chunks one hour each....

The answer to this depends upon how often these tests find defects.  Anyone using AQAvit could choose to run them if they want, though I imagine this set being most useful for developers making changes to GC or some such.  The important aspect of these tests are that they are available as an optional set of tests to run.
Listing options to consider to retain flexibility / choice for people using AQAvit:

Parameterize the execution time (default of 30m per group)
Disable long running test targets, and leaving them to be run manually via make _disabled.jcstressxxxx where jctressxxxx are targets that are very long.
Consider running these tests in a separate test job, on a less regular basis, or based on certain types of OpenJDK code changes.

So the question simplified - How much HW time do you wish to use for jcstress?" is really up to you. If it would be mine infra, I would give whole jcstress 12-24 hours, split to some 12-24 chunks one hour each....

The answer to this depends upon how often these tests find defects. Anyone using AQAvit could choose to run them if they want, though I imagine this set being most useful for developers making changes to GC or some such. The important aspect of these tests are that they are available as an optional set of tests to run.
Listing options to consider to retain flexibility / choice for people using AQAvit:
1. Parameterize the execution time (default of 30m per group)

(even) if the execution time is parametrized - maybe even more important if it is - then how much groups to split it on?
In theory, we may get rid of split at all, and simply relay on -tb . Simply run all the jcstress in one hugebulk.
1hour after commit, 3hours on daily build and 24 hours  on release build. But that is slightly going agaisnt the principle of waht is longer then 20minutes, should be split.
But splitting it, is dangerous. Because -tb is not reliable. If one huge testsuite of jcstress, who shoudl run for 24hours, would run for 36 hours,  or instead of 3 hours, 5 hours.. then it is moreover ok. But if it is split to 10 groups, each for hour,  and half of them will go wild, then it will cause much more mayhem.

2. Disable long running test targets, and leaving them to be run manually via `make _disabled.jcstressxxxx` where jctressxxxx are targets that are very long.

There is nothing like this.  All tests are equally iportant.
Currenty I'm splitting really by number of actors and arbiters (so tests). Whcih is probably the moust precisse way it can be done, except really run that. ut individual testcases really do not have constant timeframe.  I can go even more deep, and really leave it run for the 23 days,  and try to split it by real time frame. (I do not want!!!-) )

3. Consider running these tests in a separate test job, on a less regular basis,

Thats what the split to individual targets is doing, isnt it? Less regularly/shorter time/ dev x always - that  is up to you. Most likely I would like to have short brainstorming  gcall on this topic. My expereicne with jcstress and your expereicne with aqa and adoptium HW are hard to mix together.

or based on certain types of OpenJDK code changes.

This is beyond my expereince on both fields. Unless you really know how to do that, I would leave it in future work.

Yes, apologies, I am making notes for myself in the comment I left, not expecting you to have a response.
I am not convinced that the jcstress tests should be run in our regular builds, as I have no metrics or data regarding their effectiveness in finding defects when compared against the amount of resources they take to run.

Yes, once I review the generator and playlist more fully, I will want to have a quick call, likely Friday looking at the calendar.

Yes, apologies, I am making notes for myself in the comment I left, not expecting you to have a response.

Thats perfectly ok :)

I am not convinced that the jcstress tests should be run in our regular builds, as I have no metrics or data regarding their effectiveness in finding defects when compared against the amount of resources they take to run.

They were extensively  catching hard to find issues for several years, but last two.,maybe three years they are completely silent. Most of the bugs were found during shenandaoh and aarch64  development, and jfr backporting (with jfr on, which is not applicable this case anyway)
As they are concurrency tests, the issues were very often not deterministic, but the report was usually precise enough to guide us. The non deterministic nature unluckily imply the need of long duration (longer duration, more likley the issue will be hit (and also more cores is implied by this)).
generally the nature of tests is really unique, and is covering very huge are not covered elsewhere.

The initial reason we added 'a short set of jcstress tests' initially was to show developers how to run and adapt the

That is interesting, but this feature is in jcstress itself. Once CI hits the issue, the exact test is printed out, so developer will then loop only that single test, and as single test is in milliseconds, the develpre can loop it and be sure they hit the issue.
That circles back to CI, being capable to hit the issue, thus run all.

tests for their own personal use and to only run a very short few hours of jcstress in our regular runs to ensure they kept 'working', not to exercise the exhaustive set regularly. The way the current PR stands, all of the test targets will get triggered as part of our EA builds (occuring weekly), consuming likely more resources than we have or wish to devote to these tests.

Yes, current case is some 24hours of HW time, only possibly able to run in parallel in 138 vms/cotnianers/chroots whatever. I assume it is no go.

4. Create/generate documentation. Parameterize the test target name and create a jcstress_custom target to allow developers to select what they run based on the changes they are making to the OpenJDK codebase.

After this discussion, I'm even more inclined to really simply run whole jcstress, with -tb 3hours daily/weekly, and  3x-4x more  on rampdowns or simialr events .
For the parts of jdk itself, this feature in jcstress itself. I can generate all 4000 targets  (really per case) to enable the uttermost granularity if aqavit is about to help with those, or really strictly by top level group (soem 400 targets, with 1-2000 tests each), but am not sure how useful ti would be.
In the moment there will be mechanism to map individual tests to jdk changesets, then is if course getting absolutely another dimension.

After this discussion, I'm even more inclined to really simply run whole jcstress, with -tb 3hours daily/weekly, and 3x-4x more on rampdowns or simialr events .

Yep, let's settle on a plan after a Friday call.  I will book something in the calendar to make sure time is reserved.

Hmm.. small change in thinking, despite all assuring I had, I'm unable to make -tb work.:  https://bugs.openjdk.org/browse/CODETOOLS-7903750?focusedId=14681341&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14681341

Summary of our Friday discussion ( thank you for the chat @judovana ):

Generate targets based on jcstress groups and create them as <disable> targets so they are there for developer use

Add some doc, and potentially a lightning talk to describe the tests and how to run the disabled targets

Add a single target for running jcstress which is enabled to run with time budget 1 hour (even though it may not be honoured)

Before merging, we will test the PR with these changes on all platforms to monitor the behaviour of the enabled target, if any platforms behave 'poorly' (as in, hang or timeout or crash), we will exclude on those platforms for now in order to merge the PR and with the idea to investigate platform failures under a separate issue

first "-tb 1h" all grinder run https://ci.adoptium.net/view/Test_grinder/job/Grinder/10399/console
Am wondering how -tb will be honored, and how resutls will be saved

Attemt 2 on intel: https://ci.adoptium.net/view/Test_grinder/job/Grinder/10410

Rebuild with 10410 with TIME_LIMIT=50 to see how long it takes to run: https://ci.adoptium.net/view/Test_grinder/job/Grinder/10411/
The default TIME_LIMIT is 10 hours, which 10410 hits and then is aborted.

thanx!
It seems as bad as the aarch64 I got at first:
07:27:44  (Time: overtime 01:33:25, 1 tests in flight, 30 ms per test)
07:27:44  (Sampling Rate: N/A)
07:27:44  (JVMs: 0 starting, 1 running, 0 finishing)
07:27:44  (CPUs: 4 configured, 8 allocated)
07:27:44  (Results: 3030696 planned; 0 passed, 0 failed, 0 soft errs, 3506 hard errs)

And it have jsut 8 cores...

thanx!
It seems as bad as the aarch64 I got at first:
07:27:44  (Time: overtime 01:33:25, 1 tests in flight, 30 ms per test)
07:27:44  (Sampling Rate: N/A)
07:27:44  (JVMs: 0 starting, 1 running, 0 finishing)
07:27:44  (CPUs: 4 configured, 8 allocated)
07:27:44  (Results: 3030696 planned; 0 passed, 0 failed, 0 soft errs, 3506 hard errs)

And it have jsut 8 cores...

filled https://bugs.openjdk.org/browse/CODETOOLS-7903754

And worse:
14:31:03
14:31:03    Messages:
14:31:03      Unrecoverable error while running
14:31:03      java.lang.IllegalStateException: Failed: 22
14:31:03      	at org.openjdk.jcstress.os.AffinitySupport$Linux.set(AffinitySupport.java:135)
14:31:03      	at org.openjdk.jcstress.os.AffinitySupport$Linux.bind(AffinitySupport.java:108)
14:31:03      	at org.openjdk.jcstress.os.AffinitySupport.bind(AffinitySupport.java:37)
14:31:03      	at org.openjdk.jcstress.tests.seqcst.volatiles.S1__S1_S2__S2__S2_S1_Test_jcstress$JcstressThread_actor3.jcstress_iteration_actor3(S1__S1_S2__S2__S2_S1_Test_jcstress.java:556)
14:31:03      	at org.openjdk.jcstress.tests.seqcst.volatiles.S1__S1_S2__S2__S2_S1_Test_jcstress$JcstressThread_actor3.internalRun(S1__S1_S2__S2__S2_S1_Test_jcstress.java:549)
14:31:03      	at org.openjdk.jcstress.infra.runners.CounterThread.run(CounterThread.java:44)
14:31:03
14:31:03  (Time: 1d+overtime 08:36:44, 1 tests in flight, 30 ms per test)
14:31:03  (Sampling Rate: N/A)
14:31:03  (JVMs: 0 starting, 1 running, 0 finishing)
14:31:03  (CPUs: 4 configured, 8 allocated)
14:31:03  (Results: 3030696 planned; 0 passed, 0 failed, 0 soft errs, 45993 hard errs)

Next to being some 2400% late, ot started to fail.

Rerun againm: https://ci.adoptium.net/view/Test_grinder/job/Grinder/10432/console ; I kept the TIME_LIMIT=50
14:39:51  (Time: 00:58:55 left, 1 tests in flight, 30 ms per test)
14:39:51  (Sampling Rate: 82.80 K/sec)
14:39:51  (JVMs: 0 starting, 1 running, 0 finishing)
14:39:51  (CPUs: 8 configured, 8 allocated)
14:39:51  (Results: 491088 planned; 55 passed, 0 failed, 0 soft errs, 0 hard errs)

Looks healthy

Rerun againm: https://ci.adoptium.net/view/Test_grinder/job/Grinder/10432/console ; I kept the TIME_LIMIT=50
14:39:51  (Time: 00:58:55 left, 1 tests in flight, 30 ms per test)
14:39:51  (Sampling Rate: 82.80 K/sec)
14:39:51  (JVMs: 0 starting, 1 running, 0 finishing)
14:39:51  (CPUs: 8 configured, 8 allocated)
14:39:51  (Results: 491088 planned; 55 passed, 0 failed, 0 soft errs, 0 hard errs)

Looks healthy

Rerun againm: https://ci.adoptium.net/view/Test_grinder/job/Grinder/10432/console ; I kept the TIME_LIMIT=50
14:39:51  (Time: 00:58:55 left, 1 tests in flight, 30 ms per test)
14:39:51  (Sampling Rate: 82.80 K/sec)
14:39:51  (JVMs: 0 starting, 1 running, 0 finishing)
14:39:51  (CPUs: 8 configured, 8 allocated)
14:39:51  (Results: 491088 planned; 55 passed, 0 failed, 0 soft errs, 0 hard errs)

Looks healthy

ok, was killed after 50hours.

I had noted that non of the tests is keeping artifacts. I guess htat is not intentional.

Despite usptram speaking differently, I think it is crucial to limit cores to 4. its the only way I saw it end in real time, and still do probably all reasonable combinations

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10462/  with cores limited to -c 4 Before reaching this, I had 4 aborted builds which were just hard errors. That will need another investigations

I had noted that non of the tests is keeping artifacts. I guess htat is not intentional.

@smlambert  Any thoughts about that? jcstress2024 is generating tarballs with results

Any thoughts about that? jcstress2024 is generating tarballs with results

If the test aborts, I do not think we attempt to keep artifacts.  Will have to check.
Or are you referring to a test that runs to completion and does not keep artifacts?

on latest playbook, nothing finished so far.  I dont know what are the habbits, if builds fails  or is aborted (becasue there is no reason to run sdomehting, which print sut "error" since the launch, maybe the results shoudl be gathered anyway?
But tbh, I'm not sure if results are produced by jcsrtess itself when it is aborted in such way (if it would run in VM, it would have no chance to do anythgn anyway)
So yes, in aborted run. Nothing nowhere I guess.

20:22:46  (Time: overtime 03:47:37, 2 tests in flight, 30 ms per test)
20:22:46  (Sampling Rate: 68.79 K/sec)
20:22:46  (JVMs: 0 starting, 1 running, 1 finishing)
20:22:46  (CPUs: 4 configured, 4 allocated)
20:22:46  (Results: 3030696 planned; 19342 passed, 0 failed, 0 soft errs, 0 hard errs)

not good

-c 3 was killed after two days: https://ci.adoptium.net/view/Test_grinder/job/Grinder/10498/r
Now trying -c 2 https://ci.adoptium.net/view/Test_grinder/job/Grinder/10501/

@judovana - as we wait for your suggested upstream changes to jcstress, we could consider also disabling the jcstress.all target so we can merge this PR.  Once changes upstream are accepted and we can re-enable the jcstress.all target with appropriate settings so it can be run in a timely manner.

10:30:56  (Time: overtime 16:28:58, 2 tests in flight, 30 ms per test)
10:30:56  (Sampling Rate: 135.63 K/sec)
10:30:56  (JVMs: 0 starting, 1 running, 0 finishing)
10:30:56  (CPUs: 2 configured, 2 allocated)
10:30:56  (Results: 337416 planned; 88910 passed, 0 failed, 0 soft errs, 0 hard errs)

Looks better then usually.
My upstream changes are not going to solve it. They are walking around, trying to be a bit more precise, but not lowering the total time.   Give me a bt more time please. I still keep experimenting above it, and maybe it will go somewhere.

Killed by bug in infra:
11:13:16  (Time: overtime 17:11:18, 2 tests in flight, 30 ms per test)
11:13:16  (Sampling Rate: 135.71 K/sec)
11:13:16  (JVMs: 0 starting, 1 running, 0 finishing)
11:13:16  (CPUs: 2 configured, 2 allocated)
11:13:16  (Results: 337416 planned; 92563 passed, 0 failed, 0 soft errs, 0 hard errs)
11:13:16
11:13:22  Cannot contact test-docker-ubi9-x64-1: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
11:19:55  wrapper script does not seem to be touching the log file in /home/jenkins/workspace/Grinder@tmp/durable-82ef8d76
11:19:55  (JENKINS-48300: if on an extremely laggy filesystem, consider -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=86400)

restart:  https://ci.adoptium.net/view/Test_grinder/job/Grinder/10505/

10:47:24  (Time: 1d+overtime 21:19:37, 1 tests in flight, 30 ms per test)
10:47:24  (Sampling Rate: 78.73 K/sec)
10:47:24  (JVMs: 0 starting, 1 running, 0 finishing)
10:47:24  (CPUs: 2 configured, 2 allocated)
10:47:24  (Results: 337416 planned; 162967 passed, 0 failed, 0 soft errs, 0 hard errs)

Looks good - for the test-  but not good for timeout. Anyway I think it is the way to go. To limit the -c to 2 and give smallest possible time budget. But still we must count with aprox 3 days on slowest machines.

As https://ci.adoptium.net/view/Test_grinder/job/Grinder/10505 was intel, trying now arm with -c 2:
https://ci.adoptium.net/view/Test_grinder/job/Grinder/10506/

All targets are now disabled, and cores are set to default of 2.
https://ci.adoptium.net/view/Test_grinder/job/Grinder/10679/console

Will add  also -Djcstress.console.printIntervalMs handling

wait a sec, this is funny

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10680/ had  just -Djcstress.console.printIntervalMs=3600000 set instead of default (-Djcstress.console.printIntervalMs=15000) And it have some unexpected impact:
Original  ETA and samplign rate
20:38:07  (Time: 00:59:59 left, 1 tests in flight, 30 ms per test)
20:38:07  (Sampling Rate: 147.79 K/sec)
20:38:07  (JVMs: 0 starting, 1 running, 0 finishing)
20:38:07  (CPUs: 2 configured, 2 allocated)
20:38:07  (Results: 337416 planned; 1 passed, 0 failed, 0 soft errs, 0 hard errs)

new ETA and sampling rate:
21:28:01  (Time: 11d+17:05:01 left, 1 tests in flight, 2969 ms per test)
21:28:01  (Sampling Rate: 7.62 M/sec)
21:28:01  (JVMs: 0 starting, 1 running, 0 finishing)
21:28:01  (CPUs: 2 configured, 2 allocated)
21:28:01  (Results: 337416 planned; 94 passed, 0 failed, 0 soft errs, 0 hard errs)

Sampling rate is much higher, but Eta is absolutely out.  And looking to the passed tests, counting, the delay is real.
The

Gosh, I had reset time budget. Fixing

ok, fixed. https://ci.adoptium.net/view/Test_grinder/job/Grinder/10681/console looks good.
I'm not aware of any more issues (except the time needed). Feel free to merge.  I will now move to next step in https://github.com/adoptium/aqa-tests/issues/5261 - to improve https://github.com/adoptium/TKG/blob/master/scripts/getDependencies.xml so it is using jcstress-20240222.jar from  dependency pipeline. (it will need to regenerate te playlist.xml once done)

hmm.. that https://ci.adoptium.net/view/Test_grinder/job/Grinder/10681/console looks good! It is in the half after twelve hours!

Just for record, the https://ci.adoptium.net/view/Test_grinder/job/Grinder/10681/console finished in 20 hours :)
I think that one can be deployed under some circumsatnces.


Support simple arcade expression in the label

Chris mentioned this today, if we could add support for simple arcade expression, as for example, if a field value is empty, it will not show the value or show a custom text returned by a arcade expression.
Test App: https://sg-em-devext.mapsdevext.arcgis.com/apps/mapviewer/index.html?webmap=e77a0dc95d064ef593b257f76c9a0f5a
Test Map: https://sg-em-devext.mapsdevext.arcgis.com/apps/mapviewer/index.html?webmap=e77a0dc95d064ef593b257f76c9a0f5a
Currently the export function is skipping this expression in both pdf labels and csv

FYI @chris-fox
@MikeTschudi do you have any idea of the level of effort for this? if it is lot of work we can push it to next release.

@MikeTschudi that's awesome!

Verified


Topic search should use exact match

Gitea version (or commit ref): 1.10.0+dev-190-g852876a98
Can you reproduce the bug at https://try.gitea.io:

[x] Yes: https://try.gitea.io/explore/repos?q=it&topic=1
[ ] No
[ ] Not relevant

Description
Current behavior: Topic search includes all repositories with a topic containing the search string
Expected behavior: When clicking on a topic, you want to search that particular topic, not any topics containing that topic. For example, looking for topic "it" shall not give repositories with topic "git" or "gitea".
I think generic searches could contain any repository with search term in name or part of topic, as today.

Currently the search function does not make a distinction between name, description and topic. We could force a search by whole words for all fields, or we could make a distinction for the topic, like in topic:<word>. We could even do both. Is there a preference for this?

@guillep2k The search function actually doesn't look into description today, something I'm also missing.
In the long term I think it would be a good idea to support filters in repo search, similar that GitHub does for issue search. Search within name and description should probably still be "free text" search.
Something to consider is whether whole words shall always be used for topic search, or only when you specifically search for topic (topic=1). I did a test implementation and noticed a behavior that can potentially be confusing for the user; In the interactive (ajax) search on the dashboard, there can be more results as you type (instead of narrowing down). Although I don't see a problem with that.

I see, I hadn't thought about the incremental search. It looks like it should be handled differently than the "proper" search.

Closing issue as related PR is merged.


⚠ Turnos 24x7 has degraded performance

In 6354fd5, Turnos 24x7 (https://reservas-dev.alternativasinteligentes.com/) experienced degraded performance:

HTTP code: 200
Response time: 1039 ms

Resolved: Turnos 24x7 performance has improved in 2407069.


break line not supported

hi
break line or new line or new paragraph not working for me
myfirsttext\nmysecondtext is work good
myfirsttext \n mysecondtext not work
the \n between space not work

Can you be a bit more specific what you mean?


X509_check_private_key() crash with EC key from engine

(cf. https://github.com/OpenSC/libp11/issues/98)
When the PKCS#11 engine has only private key and no public key, I get a crash in EC_POINT_cmp():
Starting program: /home/dwmw2/git/openconnect/ossl/.libs/openconnect -c pkcs11:token=foo1\;pin-value=1234\;object=ecnopub localhost:4443 --servercert sha1:a82547f68f44d6351bef6cacd1d7b96e84f9dfa3
...
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff6c30a58 in EC_POINT_cmp (group=0x6d7680, a=0x6d6660, b=0x0, ctx=0x0)
    at ec_lib.c:1001
1001        if ((group->meth != a->meth) || (a->meth != b->meth)) {
Missing separate debuginfos, use: dnf debuginfo-install krb5-libs-1.14.1-7.fc23.x86_64 libp11-0.3.0-1.fc23.x86_64 libtasn1-4.8-2.fc23.x86_64 lz4-r131-1.fc23.x86_64 opensc-0.15.0-4.fc23.x86_64 pcsc-lite-libs-1.8.15-1.fc23.x86_64 softhsm-2.1.0-1.fc23.x86_64 stoken-libs-0.90-2.fc23.x86_64
(gdb) bt
#0  0x00007ffff6c30a58 in EC_POINT_cmp (group=0x6d7680, a=0x6d6660, b=0x0,
    ctx=0x0) at ec_lib.c:1001
#1  0x00007ffff6c3aacc in eckey_pub_cmp (a=<optimized out>, b=0x6d44c0)
    at ec_ameth.c:224
#2  0x00007ffff6caee51 in X509_check_private_key (x=<optimized out>,
    k=k@entry=0x6d44c0) at x509_cmp.c:333
#3  0x00007ffff6ffda9c in ssl_set_pkey (c=0x649bc0, pkey=0x6d44c0)
    at ssl_rsa.c:223
#4  0x00007ffff7bcdf99 in load_pkcs11_key (vpninfo=0x644a90)
    at ../openssl-pkcs11.c:584
#5  0x00007ffff7bca00d in load_certificate (vpninfo=0x644a90)
    at ../openssl.c:889
#6  0x00007ffff7bcb65c in openconnect_open_https (vpninfo=0x644a90)
    at ../openssl.c:1633
#7  0x00007ffff7ba21a0 in do_https_request (vpninfo=0x644a90,
    method=0x7ffff7bd2bdf "POST",
    request_body_type=0x7ffff7bd2e60 "application/x-www-form-urlencoded",
    request_body=0x645660, form_buf=0x7fffffffd960, fetch_redirect=0)
    at ../http.c:875
#8  0x00007ffff7bbba70 in cstp_obtain_cookie (vpninfo=0x644a90)
    at ../auth.c:1285
#9  0x00007ffff7ba5b06 in openconnect_obtain_cookie (vpninfo=0x644a90)
    at ../library.c:182

https://www.openssl.org/docs/manmaster/crypto/EVP_PKEY_cmp.html
says:
"Since OpenSSL private keys contain public key components too the function EVP_PKEY_cmp() can also be used to determine if a private key matches a public key."
But that is not always possible.
With PKCS#11 the private key does not have an attribute for the EC_POINT. There in no requirement that PKCS#11 has access to the match public key. When using a hardware token, that has an EC key it does not have to expose it. But the token can still do ECDSA sign.
The OpenSC libp11 can create a EVP_KEY and provide a ECDSA_METHOD (ECKEY_METHOD (in 1.1)  that does have the EC_POINT.
In eckey_pub_cmp:223
*pb = EC_KEY_get0_public_key(b->pkey.ec);
returns NULL, and then calls
r = EC_POINT_cmp(group, pa, pb, NULL);
that then segfaults.
Possible solutions:
EC_KEY_get0_public_key(b->pkey.ec); returns NULL assume the keys do match and even copy the EC_POINT  to the private key.

Why assume? We can perform a signature using the privkey and validate it using the cert.

You mean add another signature operation to be done on a possibly slow smart card?
Who is "we" in this? There are multiple pieces of software involved that each have a piece of the information and expectations of what they need.
OpenSSL expects it private  EVP_PKEY  to contain the ec_point.
The token, that does not have or does not expose the ec_point for a private key.
PKCS#11 does not return the ec_point as an attribute for private key, and specifically says to get the ec_pont from the public key.
libp11 that gets all its information from the token via PKCS#11 API.
You, the user, said you did not want to put the pubkey on the token.
If you put the pubkey on the token, the problem goes away.
The openconnect could try and add the ec_point to the EVP_KEY by parsing the cert you are reading from a file anyway. But that only fixes the issue for openconnect.
libp11 could fail to load the private key if it can not also find the ec_point via the pubkey.
If we add this you will have to  put the pubkey on the token anyway.
In any case OpenSSL should not segfault. But what should OpenSSL do if the the key does not have the ec_point? Why is it comparing the keys to see if they match? Sounds like to me they added extra checks but if they did not check and just went ahead things should work. (Should be verified that that is the case.) But they may stick by their statement that  "OpenSSL private keys contain public key components" and fail the compare.
If they take that stance, you will have to patch openconnect or add the pubkey to the token.
I have not run into this situation. The main card I have interest in is the PIV card. The only way to find out what keys are on the card, the type of the key, the parameters, the size and the ec_point is to read the certificate object for each key. The OpenSC driver reads the certificate to get the information and then presents via PKCS#11 the certificate, public key and private key. libp11 then reads the public key and private key and get the ec_point from the public key to place into the EVP_PKEY used by OpenSSL.
So why not just put the public key on the token?

"Why not just put the public key on the token?" No reason... except that this is part of my certificate "torture test". I have discovered a case where OpenSSL+libp11 not only doesn't work, but crashes in a situation where the GnuTLS build of openconnect works fine. You're basically asking me to remove that particular unit test from the suite because it fails. That isn't really how test suites are supposed to work :)
And this unit test wasn't even one of the ones where I was being deliberately nasty — I expected this one to pass. It happened on my first test of EC keys, when all I did was import them with p11tool in the manner which seemed natural. Now, I'm happy to go and prod @nmav and make sure that future versions of p11tool will import the pubkey as a separate object. But right now it's not an unlikely occurrence.
It would be better for users if the match is actually checked, and they get a coherent error message about a mismatch instead of a later SSL negotiation failure and an opaque 'TLS alert received' message. And if it's slightly slower, only in the case where the pubkey isn't present on the token, I think that's tolerable.

I've added the failing test case to OpenConnect in commit e048030f8. In fact the easiest way to reproduce is not to use OpenConnect at all but just to use its test harness.
• git clone git://git.infradead.org/users/dwmw2/openconnect
• cd openconnect
• ./autogen.sh && ./configure — no need to make it; you only need softhsm2.conf to be created by  the configure script.
• cd tests
• SOFTHSM2_CONF=softhsm2.conf gdb --args openssl s_client -engine pkcs11 -keyform engine -key 'pkcs11:token=openconnect-test1;object=EC;pin-value=1234' -cert certs/ec-cert.pem -connect localhost:443
This assumes that SoftHSMv2 is installed correctly with a p11-kit module file. You can add 'HOME=.' to the test run if not which will make p11-kit pick up the file from  tests/.config/pkcs11/modules/softhsm2.module. You might also need to edit that file to point to the right path for ibsofthsm2.so.

You, the user, said you did not want to put the pubkey on the token.
If you put the pubkey on the token, the problem goes away.

Note that there are cards which have no public keys in them, only certificates.

Errr... The certificate contains a public key, anything else makes the cert useless...

Yes. The cert contains a public key. But in X509_check_private_key() you appear to be comparing the public key associated with the privkey, with the public key in the cert. Even when the former doesn't exist and you crash with a NULL pointer dereference...

@dwmw2  said: "No reason... except that this is part of my certificate "torture test". OK, then the tests have uncovered a problem, OpenSSL segfaults. it has also uncovered issues between different packages and assumptions they have made based on the standards they use. A user could fall into one of these issues.

@dengert In case you hadn't noticed, I'm trying to resolve all those issues, tilting at one windmill at a time. With a key imported into my token I expect to be able to use it from any application in my system by its RFC7512 ID. So all these (and more) should JustWork™:
 openconnect -c 'pkcs11:token=foo;id=%01;pin-value=1234' vpn.example.com
 curl -E 'pkcs11:token=foo;id=%01;pin-value=1234' https://www.example.com
 nmcli con modify OfficeWifi 802-1x.private-key 'pkcs11:token=foo;id=%01;pin-value=1234'

@nmav said: "Note that there are cards which have no public keys in them, only certificates."
I agree.  But how should this  be handled?
Should libp11 look for a matching certificate on the card if no public key can be found?
Should the PKCS#11 module look for the certificate to extrace the public key if no public is found on the card? Should it do this just to get the ec_point?
Should the card driver do the same?
As I said the opensc PIV driver does. It needs it internally. It was also the card used to develop the OpenSC EC support, as well as the EC changes to libp11, and the opensc-engine and the mods to get OpenSSL to support an engine for ECDSA and ECDH.
During that development, it was always assumed the card (or card driver) would present a public key  for each private key and libp11 could always get the ec-point from the public key to fill in the EVP_KEY.
Other drivers may or may not. That is upto the driver developer.

The way things currently are, openssl requires an engine to fill in the public key components in the EVP_PKEY before returning it. If the pkcs11 engine doesn't, it's currently faulty vis-a-vis openssl. Simple as that.
You can certainly propose a change regarding this, but you must understand that this is a big semantic change, and there is no way you will see it before openssl 1.2 at the earliest. Among others, it will require an changed engine API.
Side note: I agree that a pkcs11 client should be incorporated into openssl. I cannot predict in what version that would happen, that depends entirely on what API changes that would require...

@dwmw2 And you seam have the attention of representatives from most of the windmills.
But I have not seen anyone from softhsm which represents a card, card driver, and PKCS#11 module of its own.

Guys, name calling and blame game isn't very productive and really has no place here. Can we choose the path of moving forward instead?

@levitte  Even if a pkcs11 client is incorporated into openssl  ECC,  DSA and  GOST R 34.10-2001 PKCS#11 private keys do not contain the "DER-encoding of ANSI X9.62 ECPoint value Q", "Public value y" or  "64 bytes for public key; 32 bytes for each coordinates X and Y of elliptic curve point P(X, Y) in little endian order" respectively.
Its only RSA PKCS#11 private keys that contain CKA_MODULUS and CKA_PUBLIC_EXPONENT.

@levitte
Sorry, I am not trying to name call or place blame. I am trying to point out there are differences caused by interpretation of various standards and how this lead to this situation and how it could be resolved.

@levitte  Even if a pkcs11 client is incorporated into openssl  ECC,  DSA and  GOST R 34.10-2001 PKCS#11 private keys do not contain the "DER-encoding of ANSI X9.62 ECPoint value Q", "Public value y" or  "64 bytes for public key; 32 bytes for each coordinates X and Y of elliptic curve point P(X, Y) in little endian order" respectively.

Yes, understood.  However, is that data possible to retrieve given the same token?  In that case, that's what an openssl engine is currently required to do when returning the EVP_PKEY.  If that information isn't possible to retrieve, then we do have a much more serious problem for sure.

The data may not be retrieval from the token. That depends on the token, how it was initialized and the how far the PKCS#11 module will (or can) go to get the information from the public key or certificate if they are on the token.  There are cases where data from the card may be cached or stored externally as in the PIV retired key case. NIST defined a Key History Object that can contain a offCardCertURL where these certs can be found.  The OpenSC PIV driver will not currently download the URL, but if the user has downloaded it to ~/.eid/cache/ the driver will read the certs as if they were on the card. So it is very token specific.
The engine and libp11 usually only get data from the PKCS#11 module. But the certificate or public key may be available to OpenSSL via some other means. i.e. User has stored the certificate in a file that is read via application that calls OpenSSL. I believe this is what @dwmw2  is doing in his torture test, by including the parameter --servercert sha1:a82547f68f44d6351bef6cacd1d7b96e84f9dfa3
And that cert and the private key from libp11 are being passed to X509_check_private_key.
In most cases the the user considers these to be the matching pair. But a "torture test" should also include a test were they don't match.

With regard to native PKCS#11 support in OpenSSL, I think we already have a vague consensus on the path forward. We've talked in the not-so-distant past about merging something that looks a bit like the existing libp11, "after 1.1". Which I suppose means I need to pull my finger out and start looking at specifics some time soon :)
I have already obtained permission from the major contributors to libp11, for reusing their work under the OpenSSL licence.
That's kind of orthogonal to this specific issue... to which I think the answer has to be that when the private key is in hardware, you can't rely on having any additional information about it. When you're asked to use a given private key with SSL_CTX_use_PrivateKey(), you can reasonably assume that you can perform a 'sign' operation with it.  If you want to validate that it matches the X.509 certificate you were also asked to use for that same SSL_CTX, then sign some dummy data and validate that signature.
As for fixing this in 1.1 and earlier, vs. declaring it a more intrusive semantics change that has to wait for 1.2... if it is such a hard and fast rule that private keys must always have public keys attached, why does EVP_PKEY_missing_parameters() even exist? Could we not at least prevent X509_check_private_key() from segfaulting somehow in the older releases, to allow this use case to work? Changing semantics to always require a pubkey when previously we didn't, would be a change we can't make. But making it more tolerant shouldn't be so bad, surely?
Note, again, that this was not intended to be the "torture" part of my torture test — this was my first simple attempt at importing an EC key into SoftHSM with p11tool and trying to use it. I didn't deliberately omit the public key; that's just how it happened by default.

@dengert: No, the --servercert sha1:… is not relevant here; that's just telling the SSL client to accept the server's certificate with that fingerprint. Given that the server in our test suite is using a cert signed by our own test CA, I could have provided a --cafile argument. Or even just --no-cert-check although I'm seriously tempted to remove that option completely... people keep using it :)
Back on-topic... we use the -c argument to openconnect to set the client cert, and also perhaps the -k argument to specify the private key, if it isn't found in the same file or by the same (RFC7512) PKCS#11 URI as the cert. There is no option to provide a public key separately from the cert + privkey.

OK.  But the EC_POINT can be obtained from the pubkey that is in the cert.  But in what cases would it do this? Only if the private key was from an engine? Or flag from an engine that it was OK to do this?

Note, again, that this was not intended to be the "torture" part of my torture test — this was my first simple attempt at importing an EC key into SoftHSM with p11tool and trying to use it. I didn't deliberately omit the public key; that's just how it happened by default.

May not be that accurate. I bet that the public key was there, but it was marked private (i.e., needed PIN to see it) by default. https://github.com/opendnssec/SoftHSMv2/pull/222

No I really this it was absent. I had already added --no-mark-private to work around https://github.com/OpenSC/libp11/issues/99
Importing with sofrhsm2-util results in a public key object. Importing with p11tool seems not to. The openconnect suite has all three variants now. Two of which fail for the OpenSSL/libp11 build.

Hmmm...  a quick, cheap and dirty way would be for the pcks11 engine to copy the EVP_PKEY_METHOD of a newly allocated EVP_PKEY of the right type, then set it's own comparison functions with the help of EVP_PKEY_asn1_set_param and/or EVP_PKEY_asn1_set_public, and possibly replace other method functions that would otherwise require the public key components to be present in the EVP_PKEY structure.  I haven't tested this myself, but since the EVP_PKEY_METHOD type exists, it can as well be used for this type of thing.

Hm, cute idea but...
int X509_check_private_key(X509 *x, EVP_PKEY *k)
{
    EVP_PKEY *xk;
    int ret;

    xk = X509_get_pubkey(x);

    if (xk)
        ret = EVP_PKEY_cmp(xk, k);
    ...

int EVP_PKEY_cmp(const EVP_PKEY *a, const EVP_PKEY *b)
{
    ...
    if (a->ameth) {
    ...
        if (a->ameth->pub_cmp)
            return a->ameth->pub_cmp(a, b);

... we use the EVP_PKEY_METHOD of the key we found in the X509, not the engine key we're comparing with.
In the meantime, we could at least make it not crash, by being a little more defensive:
--- a/crypto/ec/ec_ameth.c
+++ b/crypto/ec/ec_ameth.c
@@ -224,6 +224,8 @@ static int eckey_pub_cmp(const EVP_PKEY *a, const EVP_PKEY *b)
     const EC_GROUP *group = EC_KEY_get0_group(b->pkey.ec);
     const EC_POINT *pa = EC_KEY_get0_public_key(a->pkey.ec),
         *pb = EC_KEY_get0_public_key(b->pkey.ec);
+    if (!group || !pa || !pb)
+        return -2;
     r = EC_POINT_cmp(group, pa, pb, NULL);
     if (r == 0)
         return 1;
@@ -394,6 +396,8 @@ static int ec_cmp_parameters(const EVP_PKEY *a, const EVP_PKEY *b)
 {
     const EC_GROUP *group_a = EC_KEY_get0_group(a->pkey.ec),
         *group_b = EC_KEY_get0_group(b->pkey.ec);
+    if (!group_a || !group_b)
+        return -2;
     if (EC_GROUP_cmp(group_a, group_b, NULL))
         return 0;
     else

That won't fix it though; that'll just make it fail more gracefully than a segfault. We'll end up in the -2 case in X509_check_private_key(), setting the X509_R_UNKNOWN_KEY_TYPE error.
Now maybe the answer is for that -2 case in X509_check_private_key() to try the key, by signing some dummy data and then validating it against the X509.
However, I see this is actually handled differently for RSA keys which have the same problem. In that case we never end up in X509_check_private_key() at all, because we have this check in ssl_set_pkey():
#ifndef OPENSSL_NO_RSA
        /*
         * Don't check the public/private key, this is mostly for smart
         * cards.
         */
        if ((pkey->type == EVP_PKEY_RSA) &&
            (RSA_flags(pkey->pkey.rsa) & RSA_METHOD_FLAG_NO_CHECK)) ;
        else
#endif
        if (!X509_check_private_key(c->pkeys[i].x509, pkey)) {

So maybe the cleanest and simplest fix here is to extend that RSA_METHOD_FLAG_NO_CHECK case to be something we can set generically for any EVP_PKEY.

OK. But the EC_POINT can be obtained from the pubkey that is in the cert. But in what cases would it do this? Only if the private key was from an engine? Or flag from an engine that it was OK to do this?

@dengert No, at this point we don't have "the" certificate. There exist hardware devices which only do keys, and don't have certificate storage at all. I believe the OpenPGP cards work like this, because they're not really designed with X.509 in mind. And the OpenSSL TPM engine certainly does.
Using a key from an engine and a certificate from elsewhere is a perfectly valid use case. In X509_check_private_key() we have "a" certificate and a key, and we are checking that they are indeed a match. We cannot simply copy the information from one to the other and then say "oh look, they do match!". That's kind of missing the point :)
We can perform a dummy signature using the key, and validate it using the cert. Or, as noted above with the RSA_METHOD_FLAG_NO_CHECK flags, we can simply bypass the check altogether.

The issue appears to come down to associating a private key on a token with a certificate that may not be stored on the token. It may be in a certificate store or provided by the user or application. With RSA it was easy, use the modulus. But PKCS#11 does not define a private key with a EC_POINT, it says get it from the public key (or public key in the certificate.)  It does suggest that matching cert, public key and private key all use the same CKA_ID so they can be matched up.  But when the certificate to be used is not obtained via PKCS#11 and the private key does not have the EC_POINT there is no way to match them up without additional matching information. There is no unique value between the certificate and the private key.
The additional information may be know to the user who could provide a single private key and single certificate which the user knows match.
A single sign and multiple verify operations could be used to match a private key and certificate, but on slow tokens, and the need for the PIN  in my option this is not a viable option. (It could also be the token has said the EC key can only be used for ECDH and not ECDSA.)
Libp11 is the bridge that used the CKA_ID to match a key to a certificate or public key in order to get the EC_POINT to present as part of the private key to OpenSSL or other software so they can use the EC_POINT to match keys and certificates.
So in other words OpenSSL private keys can not have a NULL EC_POINT.  If the private key is coming from PKCS#11 the certificate or public key must come from libp11.  If they are not on the token, then libp11 needs an alternate source of the certificate or public key and be provided with the CKA_ID or  to be used to match to a private key on the token.
Maybe rfc7512  and p11-kit could provide a way to get the certificate from a file, and the private key form the token so libp11 then gets the EC_POINT from the certificate and adds it to the private EVP_KEY
used withing OpenSSL.

Maybe rfc7512 and p11-kit could provide a way to get the certificate from a file, and the private key form the token so libp11 then gets the EC_POINT from the certificate and adds it to the private EVP_KEY used withing OpenSSL.

The PKCS#11 URI is just a standard form for an identifier string (really a search term) for locating PKCS#11 objects by various criteria including CKA_ID, CKA_LABEL and fields from CK_TOKEN_INFO. Something like this doesn't really live there — not even if we'd suggested it during the years when it was an Internet-Draft, and before its final publication as RFC7512.
As for exposing the certificate to libp11... I'd prefer not to. Applications already expect that certificates and keys can come from different locations, and provide config file or command line options to specify them separately. The key may come from libp11 or an engine, while the certificate comes from somewhere else entirely. The existing code model is that the certificate and key for SSL client authentication only come together when you install them into the SSL_CTX with SSL_CTX_use_certificate() and SSL_CTX_use_PrivateKey(). (Or into the SSL if you must, but the principle is the same).
So I do not like the idea of somehow providing the certificate to libp11 "in advance", just for it to copy over the EC_POINT data that OpenSSL doesn't actually need; it only uses it to check if the certificate and the key are the same.
It would be much better just to fix OpenSSL not to require the EC_POINT in the EVP_PKEY object. After all, it already copes with precisely the equivalent situation for RSA keys in an engine, via the RSA_METHOD_FLAG_NO_CHECK flag. That's all we need.
If we really can't do that in existing released versions of OpenSSL, then let's work around it in the application — which does already have the certificate. It sucks, but then applications already have to have hundreds of lines of code even just to coherently load certificates from various types of files, let alone PKCS#11. I was already on the verge of BSD-licensing the OpenConnect code to do so and publishing it as a library alongside my certificate torture tests — the addition of this would just be a drop in the ocean :)
It would go something like:
•  Sign dummy data with the EVP_PKEY.
• Validate signature against the X509 and bail out if it matches
• o2i_ECPublicKey(EVP_PKEY_get1_EC_KEY(pkey), X509_get_pubkey(x509))
Of course, OpenSSL could do that too if it wanted, when it found no EC_POINT in X509_check_private_key().

I thing you may be on to something here.
My argument was assuming that OpenSSL or some application might try and find a certificate from a set of certificates by expecting the EC_POINT to match.  I don't see any code in in OpenSSL that is doing this , but I may have missed it.
The fact that for RSA they have the RSA_METHOD_FLAG_NO_CHECK used only in ssl_set_pkey and comment says it is there "mostly for smart cards" and implies this problem has been seen before for RSA keys. It was in 0.9.7.
Here is another flag that looks promising: EC_PKEY_NO_PUBKEY
[https://www.openssl.org/docs/man1.0.2/crypto/d2i_ECPrivate_key.html ]
but when reading a ECPrivateKey, they generate the EC_POINT from the private key, I don't think that would work with any tokens.

I have worked around this in OpenConnect thus:
http://git.infradead.org/users/dwmw2/openconnect.git/commitdiff/85ff68a9722b
It's just another hundred-odd lines of code where I already have over a thousand lines in my SSL_CTX_ffs_please_OpenSSL_just_use_these_certs_the_user_asked_me_to_use() functions... it's almost in the noise. But I will be revisiting this whole thing with a pointy stick :)

Yes. The crash is fixed, and the remaining issue is dealt with in #1551