[Bug] NullPointerException when the login fails using the Jakarta EE module

Search before asking

[X] I had searched in the issues and found no similar issues.

Environment
I am running a PoC using Tomcat Embedded (10.1.23) with CDI (Weld 5.1.2) + JSF (Mojarra 4.0.7)
Shiro version
I am using Shiro 2.0.0 with Jakarta classifier
    implementation('org.apache.shiro:shiro-jakarta-ee:2.0.0:jakarta') {
        exclude(group: 'org.apache.shiro', module: 'shiro-core')
        exclude(group: 'org.apache.shiro', module: 'shiro-cdi')
        exclude(group: 'org.apache.shiro', module: 'shiro-web')
    }
    implementation('org.apache.shiro:shiro-cdi:2.0.0:jakarta') {
        exclude(group: 'org.apache.shiro', module: 'shiro-core')
    }
    implementation('org.apache.shiro:shiro-web:2.0.0:jakarta') {
        exclude(group: 'org.apache.shiro', module: 'shiro-core')
    }
    implementation 'org.apache.shiro:shiro-core:2.0.0:jakarta'

What was the actual outcome?
I am getting a NullPointerException when the login fails. The reason is that the LOGIN_WAITTIME_ATTR_NAME request attribute was not set.
https://github.com/apache/shiro/blob/f532a2112993a5f7c47c286dcbd3f34e4e1cc906/support/jakarta-ee/src/main/java/org/apache/shiro/ee/filters/Forms.java#L161
There is code to set the value in AuthenticationFilterDelegate.preHandle(). I debugged it, and the method never is called. I am not sure if this is due to my custom configuration. I tried setting up the SecurityManager and my custom realm using CDI and EnvironmentLoaderListener (shiro.ini), but in both cases, I got the same exception.
What was the expected outcome?
The method AuthenticationFilterDelegate.preHandle() is executed and adds the LOGIN_WAITTIME_ATTR_NAME param to the request or assumes a default value when the attribute is null.
How to reproduce

Setup a project using shiro-jakarta-ee 2.0.0
Add a realm (in my case it is a custom class inherited from AuthenticatingRealm) using CDI https://shiro.apache.org/jakarta-ee.html#configuring_cdi_without_jakarta_ee_module_or_shiro_ini
Add a JSF login page https://shiro.apache.org/jakarta-ee.html#jakarta_faces_jsf_features
Fails the login by throwing an AuthenticationException from the AuthenticatingRealm

Debug logs
SEVERE: Servlet.service() for servlet [FacesServlet] in context with path [] threw exception [Cannot invoke "java.lang.Integer.intValue()" because the return value of "org.omnifaces.util.Faces.getRequestAttribute(String)" is null] with root cause
java.lang.NullPointerException: Cannot invoke "java.lang.Integer.intValue()" because the return value of "org.omnifaces.util.Faces.getRequestAttribute(String)" is null
	at org.apache.shiro.ee.filters.Forms.login(Forms.java:161)
	at org.apache.shiro.ee.filters.Forms$AuthenticationMethods.login(Forms.java:84)
	at org.apache.shiro.ee.filters.Forms$AuthenticationMethods.login(Forms.java:73)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.apache.el.parser.AstValue.invoke(AstValue.java:252)
	at org.apache.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:266)
	at org.jboss.weld.module.web.util.el.ForwardingMethodExpression.invoke(ForwardingMethodExpression.java:40)
	at org.jboss.weld.module.web.el.WeldMethodExpression.invoke(WeldMethodExpression.java:50)
	at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:70)
	at com.sun.faces.application.ActionListenerImpl.getNavigationOutcome(ActionListenerImpl.java:74)
	at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:62)
	at jakarta.faces.component.UICommand.broadcast(UICommand.java:205)
	at jakarta.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:858)
	at jakarta.faces.component.UIViewRoot.processApplication(UIViewRoot.java:1332)
	at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:56)
	at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:72)
	at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:131)
	at jakarta.faces.webapp.FacesServlet.executeLifecyle(FacesServlet.java:691)
	at jakarta.faces.webapp.FacesServlet.service(FacesServlet.java:449)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:196)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:391)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:896)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1736)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
	at java.base/java.lang.Thread.run(Thread.java:833)

Hi,
Thank you for the report. This is probably a bug, as the particular combination of Tomcat / Weld etc. has not been tested (at least not by the Shiro team)
Is there a reproducer we can try?
Just guessing here, but can you try setting authc.loginFailedWaitTime = 0 in your shiro.ini manually to see if that helps?
Thank you!

Ok, scratch my last comment. Looking at your issue further and it's a misconfiguration.
You are trying to use Jakarta EE module functionality while specifically disabling it's initialization.
Jakarta EE module is intended to be zero-conf. No configuration is required to run it.
What's described in https://shiro.apache.org/jakarta-ee.html#configuring_cdi_without_jakarta_ee_module_or_shiro_ini
is intended to initialize and use the CDI module, without Jakarta EE module.
Just disregard what's in https://shiro.apache.org/jakarta-ee.html#configuring_cdi_without_jakarta_ee_module_or_shiro_ini as it doesn't apply in your case.
I can see how that section can be confusing. We will move that down in the docs so its not percieved as the preferred configuration method.
There is no configuration required besides shiro.ini which is described in the docs.

Docs have been updated.
Please feel free to reopen the issue or file another ones if any arise.
Thank you


Location names for US cities do not include states

15:14 PSchwisow: !weather 60002
15:14 Phergie: Antioch, US | Rain | Temp: 24C | Humidity: 83% | Sunrise: 05:40:09 | Sunset: 19:57:57

There is more than one Antioch!

Cheers @PSchwisow & @tonsit, apologies for the late response, I've had a few offline things taking up all my free time recently.
I'll spin the change up at some point soon to test and if all's good I'll get it merged

Proposed fix from alternate provider in #8


Datepicker Pro input throws error when using keyboard with picker-only option

Reproduction
https://formkit.link/26abe9131c18c7dc46f05b6233934960
Describe the bug
I have this setup:
<FormKit
    type="datepicker"
    name="dob"
    label="Date of birth"
    format="DD MMMM YYYY"
    value-format="YYYY-MM-DD"
    :sequence="['year', 'month', 'day']"
    picker-only
    validation="required"
/>

And clicking through the options works fine, but if you press any key on the keyboard, it throws a RangeError.
RangeError: Invalid time value
Environment
• OS: MacOS
• Browser Chrome
• Version latest

This bug is now fixed on the @next tag, thanks for the report 👍


Criteo jax OOMs on host

Criteo jax runs out of host memory after about 5K - 7K steps.
Description
Total RAM on the machine: 250GB
Traceback:
I0318 07:48:39.290826<PHONE_NUMBER>83264 checkpoints.py:356] Saving checkpoint at step: 6925
I0318 07:49:18.171090<PHONE_NUMBER>83264 checkpoints.py:317] Saved checkpoint at /experiment_runs/timing/criteo1tb_jax/trial_1/checkpoint_6925
I0318 07:49:18.647214<PHONE_NUMBER>83264 checkpoint_utils.py:240] Saved checkpoint to /experiment_runs/timing/criteo1tb_jax/trial_1/checkpoint_6925.
I0318 07:50:22.682096<PHONE_NUMBER>07264 logging_writer.py:48] [7000] global_step=7000, grad_norm=0.0072265081107616425, loss=0.12346725910902023
I0318 07:52:55.673196<PHONE_NUMBER>36352 logging_writer.py:48] [7100] global_step=7100, grad_norm=0.0090188542380929, loss=0.12465228140354156
I0318 07:58:27.361398<PHONE_NUMBER>83264 spec.py:298] Evaluating on the training split.
I0318 08:09:34.849290<PHONE_NUMBER>83264 spec.py:310] Evaluating on the validation split.
I0318 08:13:38.652261<PHONE_NUMBER>83264 spec.py:326] Evaluating on the test split.
I0318 08:16:59.559698<PHONE_NUMBER>83264 submission_runner.py:362] Time since start: 28759.92s, 	Step: 7124, 	{'train/loss': 0.12170044394100414, 'validation/loss': 0.12363606741573034, 'validation/num_examples': 89000000, 'test/loss': 0.1263589119942319, 'test/num_examples': 89274637}
I0318 08:16:59.582683<PHONE_NUMBER>07264 logging_writer.py:48] [7124] global_step=7124, preemption_count=0, score=9349.471796, test/loss=0.126359, test/num_examples=89274637, total_duration=28759.921956, train/loss=0.121700, validation/loss=0.123636, validation/num_examples=89000000
I0318 08:17:05.913339<PHONE_NUMBER>83264 checkpoints.py:356] Saving checkpoint at step: 7124
Traceback (most recent call last):
  File "submission_runner.py", line 602, in <module>
    app.run(main)
  File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "submission_runner.py", line 575, in main
    score = score_submission_on_workload(workload,
  File "submission_runner.py", line 510, in score_submission_on_workload
    timing, metrics = train_once(workload, global_batch_size,
  File "submission_runner.py", line 377, in train_once
    checkpoint_utils.save_checkpoint(
  File "/algorithmic-efficiency/algorithmic_efficiency/checkpoint_utils.py", line 225, in save_checkpoint
    flax_checkpoints.save_checkpoint(
  File "/usr/local/lib/python3.8/dist-packages/flax/training/checkpoints.py", line 366, in save_checkpoint
    target = serialization.msgpack_serialize(target)
  File "/usr/local/lib/python3.8/dist-packages/flax/serialization.py", line 334, in msgpack_serialize
    return msgpack.packb(pytree, default=_msgpack_ext_pack, strict_types=True)
  File "/usr/local/lib/python3.8/dist-packages/msgpack/__init__.py", line 38, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 294, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 304, in msgpack._cmsgpack.Packer.pack

Steps to Reproduce
docker run -t -d \
-v /home/kasimbeg/data/:/data/ \
-v /home/kasimbeg/experiment_runs/:/experiment_runs \
-v /home/kasimbeg/experiment_runs/logs:/logs \
--gpus all --ipc=host \
us-central1-docker.pkg.dev/training-algorithms-external/mlcommons-docker-repo/base_image \
-d criteo1tb \
-f jax \
-s reference_algorithms/target_setting_algorithms/jax_nadamw.py \
-w criteo1tb \
-t reference_algorithms/target_setting_algorithms/criteo1tb/tuning_search_space.json \
-e timing \
-m 10666 \
-b true

Source or Possible Fix
The RAM usage at the first step is already quite high (~150GB, compared to ~100GB for the pytorch workload).
So far we've ruled out checkpointing function by testing manual deletion of variables inside checkpointing method.
I noticed the RAM usage significantly increases just from initializing and iterating over the dataset, 50GB from initialization and at a rate of about 20GB per 1000 batches.
Interestingly, the pytorch workload does not OOM on host. For comparison, over the course of a run the RAM use increases from 100 GB (after initialization etc) to 150 GB.

I profiled the jax and pytorch workloads without docker for comparison. It seems like the memory increases between evals, but mostly during evals when it's evaluated on the train_eval, validation and test sets.
The biggest jump for both workloads is when it calls eval for the first time (100GB vs 50GB for Jax vs Pytorch). The following biggest jumps in usage are during evals (~15GB for Jax vs 5GB for pytorch initially), this number seems to slightly decrease over time.
Jax:
I0321 20:50:12.885326<PHONE_NUMBER>91392 submission_runner.py:231] Starting train once: RAM USED (GB) 3.931410432
I0321 20:50:12.885688<PHONE_NUMBER>91392 submission_runner.py:240] After Initializing dataset: RAM USED (GB) 3.931410432
I0321 20:50:18.567030<PHONE_NUMBER>91392 submission_runner.py:252] After Initializing model: RAM USED (GB) 7.714209792
I0321 20:50:21.465607<PHONE_NUMBER>91392 submission_runner.py:261] After Initializing metrics bundle: RAM USED (GB) 7.714996224
I0321 20:50:22.133624<PHONE_NUMBER>91392 submission_runner.py:312] Before starting training loop and logger metrics bundle: RAM USED (GB) 7.751073792
I0321 20:52:38.932373<PHONE_NUMBER>91392 submission_runner.py:334] After dataselection batch at step 0: RAM USED (GB) 45.78729984
I0321 20:52:59.151939<PHONE_NUMBER>91392 submission_runner.py:372] Before eval at step 1: RAM USED (GB) 53.63308544
I0321 21:10:50.153927<PHONE_NUMBER>91392 submission_runner.py:391] After eval at step 1: RAM USED (GB)<PHONE_NUMBER>04
I0321 21:20:33.790335<PHONE_NUMBER>91392 submission_runner.py:372] Before eval at step 487: RAM USED (GB)<PHONE_NUMBER>52
I0321 21:38:35.544191<PHONE_NUMBER>91392 submission_runner.py:391] After eval at step 487: RAM USED (GB)<PHONE_NUMBER>4
I0321 21:48:17.558385<PHONE_NUMBER>91392 submission_runner.py:372] Before eval at step 967: RAM USED (GB)<PHONE_NUMBER>8
I0321 22:05:10.913231<PHONE_NUMBER>91392 submission_runner.py:391] After eval at step 967: RAM USED (GB)<PHONE_NUMBER>96
I0321 22:14:55.042828<PHONE_NUMBER>91392 submission_runner.py:372] Before eval at step 1449: RAM USED (GB)<PHONE_NUMBER>56

pytorch
I0321 21:36:13.839403<PHONE_NUMBER>64960 submission_runner.py:230] Starting train once: RAM USED (GB) 5.049655296
I0321 21:36:13.839647<PHONE_NUMBER>64960 submission_runner.py:239] After Initializing dataset: RAM USED (GB) 5.049954304
I0321 21:36:26.142259<PHONE_NUMBER>64960 submission_runner.py:251] After Initializing model: RAM USED (GB) 14.66904576
I0321 21:36:26.143074<PHONE_NUMBER>64960 submission_runner.py:260] After Initializing metrics bundle: RAM USED (GB) 14.66904576
I0321 21:38:38.603623<PHONE_NUMBER>64960 submission_runner.py:333] After dataselection batch at step 0: RAM USED (GB) 53.544239104
I0321 21:38:40.801852<PHONE_NUMBER>64960 submission_runner.py:371] Before eval at step 1: RAM USED (GB) 57.1206656
I0321 21:57:13.853280<PHONE_NUMBER>64960 submission_runner.py:390] After eval at step 1: RAM USED (GB)<PHONE_NUMBER>24
I0321 22:06:23.800114<PHONE_NUMBER>64960 submission_runner.py:371] Before eval at step 467: RAM USED (GB)<PHONE_NUMBER>76
I0321 22:26:06.325435<PHONE_NUMBER>64960 submission_runner.py:390] After eval at step 467: RAM USED (GB)<PHONE_NUMBER>12
I0321 22:35:20.300642<PHONE_NUMBER>64960 submission_runner.py:371] Before eval at step 909: RAM USED (GB)<PHONE_NUMBER>56
I0321 22:54:42.722589<PHONE_NUMBER>64960 submission_runner.py:390] After eval at step 909: RAM USED (GB)<PHONE_NUMBER>32

Reducing the eval batch size for jax seems to help close the gap:
I0321 23:54:21.646273<PHONE_NUMBER>72000 submission_runner.py:299] Saving flags to /home/kasimbeg/experiments/debug_criteo_small_eval_batch/criteo1tb_jax/trial_1/flags_0.json.
I0321 23:54:21.674989<PHONE_NUMBER>72000 submission_runner.py:304] After checkpoint and logger metrics bundle: RAM USED (GB) 7.71524608
I0321 23:54:21.675219<PHONE_NUMBER>72000 submission_runner.py:311] Before starting training loop and logger metrics bundle: RAM USED (GB) 7.71524608
I0321 23:54:21.675282<PHONE_NUMBER>72000 submission_runner.py:312] Starting training loop.
I0321 23:57:01.052532<PHONE_NUMBER>72000 submission_runner.py:333] After dataselection batch at step 0: RAM USED (GB) 45.890441216
I0321 23:57:21.980293<PHONE_NUMBER>14944 logging_writer.py:48] [0] global_step=0, grad_norm=7.941814422607422, loss=0.7395287752151489
I0321 23:57:21.994342<PHONE_NUMBER>72000 submission_runner.py:350] After update parameters step 0: RAM USED (GB) 53.810757632
I0321 23:57:21.994553<PHONE_NUMBER>72000 submission_runner.py:371] Before eval at step 1: RAM USED (GB) 53.810757632
I0321 23:57:21.994624<PHONE_NUMBER>72000 spec.py:298] Evaluating on the training split.
I0322 00:06:08.399080<PHONE_NUMBER>72000 spec.py:310] Evaluating on the validation split.
I0322 00:10:16.798121<PHONE_NUMBER>72000 spec.py:326] Evaluating on the test split.
I0322 00:14:07.295299<PHONE_NUMBER>72000 submission_runner.py:380] Time since start: 180.32s,      Step: 1,        {'train/loss': 0.7395033004794426, 'validation/loss': 0.7445032359550562, 'validation/num_examples': 89000000, 'test/loss': 0.7416469248707223, 'test/num_examples': 89274637}
I0322 00:14:07.295766<PHONE_NUMBER>72000 submission_runner.py:390] After eval at step 1: RAM USED (GB) 95.224762368

The interleave, shuffle and prefetch tf operations in the pipeline that use RAM are.
I reduced the number of prefetch batches 2 and tried to set it to AUTOTUNE but that did not slow down the memory growth.

Using tcmalloc instead of libmalloc (see tf issue) seems to eliminate the memory increases:

Will verify w full run.

Tested on full run and the memory usage flattened at around step 400 to ~122GB. It stayed at about ~122GB for the rest of the run:
Before eval train evaluation at step 8000: RAM USED (GB)<PHONE_NUMBER>08
Before iteration over eval_train split for eval at step 8000 over 170: RAM USED (GB)<PHONE_NUMBER>08
After iteration over eval_train split for eval at step 8000 over 170: RAM USED (GB)<PHONE_NUMBER>28
Before eval train at step 8000: RAM USED (GB)<PHONE_NUMBER>28
Before valid evaluation at step 8000: RAM USED (GB)<PHONE_NUMBER>28
Before iteration over validation split for eval at step 8000 over 170: RAM USED (GB)<PHONE_NUMBER>28
After iteration over validation split for eval at step 8000 over 170: RAM USED (GB)<PHONE_NUMBER>08
After valid evaluation at step 8000: RAM USED (GB)<PHONE_NUMBER>08
Before test evaluaton at step 8000: RAM USED (GB)<PHONE_NUMBER>08
Before iteration over test split for eval at step 8000 over 171: RAM USED (GB)<PHONE_NUMBER>08
After iteration over test split for eval at step 8000 over 171: RAM USED (GB)<PHONE_NUMBER>04
After test evaluaton at step 8000: RAM USED (GB)<PHONE_NUMBER>04

Action items:

update docker startup script to install tcmalloc and set $LD_PRELOAD variable
reduce total memory footprint by  tuning shuffle buffer size, interleave params or removing interleave and just shuffle file numbers instead

Fixed pull/376


Implementing todos from the list

See commits for description (commit-by-commit review is advised)
(Relates to #4)

perfect, I love it !


Add website

add samsung.com

Samsung has support documents and other non-shopping stuff on that same domain, so this is blocked until the list supports wildcards.


load commercial salmon data

a shell script to load the commercial salmon data from 1996 - 2022 into postgres
a dim model in stahl to do some basic preprocessing of things

Something to pay attention to is how the postgres credentials are passed in the shell script. Right now I just reference the environment variables (and they are local since I was testing this in my own variety) including the password at the top of the script, but idk if this is the right way to do it. Would love an explanation of the standard procedure for this kind of thing.

a shell script to load the commercial salmon data from 1996 - 2022 into postgres
a dim model in stahl to do some basic preprocessing of things

Something to pay attention to is how the postgres credentials are passed in the shell script. Right now I just reference the environment variables (and they are local since I was testing this in my own variety) including the password at the top of the script, but idk if this is the right way to do it. Would love an explanation of the standard procedure for this kind of thing.

Also for some reason I can't quite figure out the files from the ashannon/scopuli/product_timeseries_metrics_smoketest branch are committed in this PR. I guess ignore that?


1311 enhancement generic controllers

WIP PR

@Goldziher I would like to take a stab at re-creating the Django tutorial for Litestar, which may then be used as a stepping point for a "Migrating from Django" guide.
Does it make sense to wait for this to be done first? If so, is there anything I can do to help with this?
Thanks! :)

@Goldziher I would like to take a stab at re-creating the Django tutorial for Litestar, which may then be used as a stepping point for a "Migrating from Django" guide.
Does it make sense to wait for this to be done first? If so, is there anything I can do to help with this?
Thanks! :)

That's awesome 😎. I suggest you start - I need to do a few more hours on this, and then it would be great if you could actually test it properly.

@Goldziher you mean start by using this branch as a basis? Or start with the way Starlite works today?
Assuming the former, I'll admit the limited tests and docs make it daunting to know where to start :sweat_smile: but I am willing to take a stab at it :)

@Goldziher you mean start by using this branch as a basis? Or start with the way Starlite works today?
Assuming the former, I'll admit the limited tests and docs make it daunting to know where to start :sweat_smile: but I am willing to take a stab at it

Well, I should get you wait. I need a few more hours to bring this PR to a functional state with some tests, and that will make it easier for you

@Goldziher you mean start by using this branch as a basis? Or start with the way Starlite works today?
Assuming the former, I'll admit the limited tests and docs make it daunting to know where to start 😅 but I am willing to take a stab at it :)

ok, still no documentation at all and the tests are not going to clarify usage, but this should be fully operational at this point. I will complete the tests and documentation in the coming days.

tests are in place, @benjamin-kirkbride - take a look and tell me what you think. I will be adding docs as a final step before review.

Ok, I will merge this branch into a feature branch.
I implemented DTO logic, but it uncovers issues with the DTO stuff. So I will do this in a followup.


travis_webhooks: Removed the unnecessary ignore_pull_request option.

Removed the &ignore_pull_request option from the TravisCI integration, including the /integrations documentation for it.
As this functionality is now fully covered by our standard event filtering framework, and having both event filtering and a custom filtering option is confusing.
Fixes:  #30934

Self-review checklist

[ ] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[ ] Explains differences from previous plans (e.g., issue description).
[ ] Highlights technical choices and bugs encountered.
[ ] Calls out remaining decisions and concerns.
[ ] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[ ] Each commit is a coherent idea.
[ ] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[ ] Visual appearance of the changes.
[ ] Responsiveness and internationalization.
[ ] Strings and tooltips.
[ ] End-to-end functionality of buttons, interactions and flows.
[ ] Corner cases, error conditions, and easily imagined bugs.

In general, please push changes to an existing PR, rather than starting a new one (#30993).


[Steam Tags Importer] Exception Importing Steam Game from Non-Steam Store

Extension name
Steam Tags Importer
Bug Description
When I try to import some games from non-Steam stores that are also available on Steam I get an error. This happens reliably for the same game. I've reproduced it with several games across several stores.
To Reproduce
Find an itch.io game that also exists on Steam (I've consistently reproduced this on Fidel Dugneon Rescue), select it, and import tags from the Extensions menu. This results in a dialog indicating that an exception occurred (see the screenshot).
I've also reproduced this with Creatorcrate from the Humble Trove.
Extensions log
extensions.log
Screenshots

This issue should be fixed for v3.0
I'm no longer developing and maintaining the extension. The extension is now developed by Jeshibu over at https://github.com/Jeshibu/PlayniteExtensions, if there are new feature requests or bug reports please open an issue in that repository


questions about load dataset for localhost,  make own policy and use headless eval mode

Hello, I'm trying to download a data set on hugging face to the local and then call this data set from the local. For example, 'aloha_sim_insertion_scripted_image' , its format is many 'episode_000000.parquet' files . Then how to load this format by LeRobotDataset() func or other ways?
Second, I want to create my own policy. After I parse the code framework, I think I may need to create my policy code file by mimicking the following files:

lerobot/common/policies/act/configuration_act.py
lerobot/common/policies/act/modeling_act.py
However, I am having some difficulties in making my own policy now, and I want to create a new policy to implement my idea, which is to introduce the concept of comparative learning. That is to say, the policy enables the agent to learn the correct samples and stay away from the wrong samples. I would like to ask you what should be modified to complete this idea.

I really need examples of this, and it would be very helpful if you could give me detailed advice!
Finally, my server is headless, which means that when evaluating a policy, there is no way to call mujujo to view the evaluation, so can our code framework support headless mode and save the evaluation video?
As a new researcher in this field, it would be great if I could further communicate with you about the above issues. Thank you very much!
Best wishes : )

I use xvfb-run python lerobot/scripts/train.py  command solved the last problem


How to show custom dropdown on clicking on particular cell

Look at canPerformAction in MMViewController.m in the example folder.


Remove OpenMlsProvider dependency of PublicGroup.

It was impossible to instantiate a PublicGroup without implementing a full StorageProvider (via OpenMlsProvider), this PR removes that requirement.

Looks like some tests fail on matching the correct error.


Urbdrc rewrite

This PR is a major rewrite of the USB redirection channel.

Removes the insane thread use
Fixes the (very) misleading log messages

hope somebody will find it useful ;)

@bmiklautz @hardening @mfleisz @nfedera anyone want to review/merge this?

When will this be merged? This is one of the most wanted PRs to fix usb redirection


Storage Adapter repo tree that PCS is using has build errors.

Type of issue

[x] Bug
[ ] New feature
[ ] Improvement

Description
The version of the storage adapter that the remote monitoring repository is pointing to has build errors when master does not.
...
Steps to reproduce

git clone --recurse on this repo
cd storage-adapter
./scripts/build

Expected behavior
Tests run with no error messages
Current behavior
Error message
✘ jwendl  /c/Source/GitHub/azure-iot-pcs-remote-monitoring-java/storage-adapter  ➦ 0b16dd5 ●  ./scripts/build
Running tests...
/c/Source/GitHub/azure-iot-pcs-remote-monitoring-java/storage-adapter/build.sbt:29: error: not found: value assemblyMergeStrategy
assemblyMergeStrategy in assembly := {
^
/c/Source/GitHub/azure-iot-pcs-remote-monitoring-java/storage-adapter/build.sbt:58: error: not found: value PlayJava
.enablePlugins(PlayJava)
^
sbt.compiler.EvalException: Type error in expression
[error] sbt.compiler.EvalException: Type error in expression
[error] Use 'last' for the full log.
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? q
Known workarounds
Clone pcs-storage-adapter-java manually at master and try to make that work.
Possible solution
Context and Environment

Operating System: Windows 10 using WSL Ubuntu 16.04
GitHub branch: remote monitoring master
Java JDK/JRE:
openjdk version "1.8.0_162"
OpenJDK Runtime Environment (build 1.8.0_162-8u162-b12-0ubuntu<IP_ADDRESS>-b12)
OpenJDK 64-Bit Server VM (build 25.162-b12, mixed mode)

@jwendl Can you confirm if this issue is still there?

@jwendl We are going to close this issue. Please let us know if you are still running into issues.


Investigate inclusion in OpenFinD3FC

Investigate and identify if the showcase's structure and API can be improved for easier inclusion in the OpenFin integration.
https://github.com/owennw/OpenFinD3FC
/cc @owennw @bjedrzejewski @ColinEberhardt

I expect we may have to revisit the current integration and make some (hopefully small) changes as the projects progress, but as #537 and https://github.com/owennw/OpenFinD3FC/pull/256 have been merged, I'll close this.


Support DataType in puppet function (Resolve.hs)

Some puppet function can now take a type as its argument. For instance:
lookup('classes', Array[String], 'unique')

Currently resolveFunction accepts Expression only.
Would you mind helping with this ? Either with code or by outlining  the solution ?
Thanks

A PValue can be a DataType, but an UnresolvedValue can't. A case should be added in this type, and the terminalG function should be extended to parse types. This looks straightforward, but it might break parsing in unforeseen ways :)

One you add a DataType case in UnresolvedValue, warnings should appear in several places. Fix them mechanically, and this should be good.

I have had a try in https://github.com/bartavelle/language-puppet/pull/222 but the added tests for lookup with a datatype are still failing (see pending tests).
I guess what I have (blindly) done in the terminalG function is not correct ?
https://github.com/bartavelle/language-puppet/pull/222/commits/4bfb48faf7eb4d54f4343e120f53317557fff1be

Looks like I would have done. You get parse errors?

You need to put it above ResourceReference, and hope nobody used a resource reference with the same name as a type ...

I have tried this already but it did not make any difference. I have just pushed the change now (the pending is still failing).

If I change the test for     it "should succeed with two arguments, the second on being a datatype" (checkSuccess ["hostname", Terminal (UDataType (DTString Nothing Nothing))] "pure") it works.
So probably something missing with the IsString instance ?

Works fine outside the test ... So there is no need to change any "IsString" instance ...

Is it expected to use the whole Terminal (UDataType (DTString Nothing Nothing) in that context ? The problem might come from a partial IsString implementation for Expression, is that so ?

It should not be partial! It should translate any string to the corresponding unresolved string. Where is the failing test?

The test is here: https://github.com/bartavelle/language-puppet/blob/master/tests/Function/LookupSpec.hs#L21
If I replace Terminal (UDataType (DTString Nothing Nothing) by 'String' the test is failing.

It would mean the following puppet code:
lookup("hostname", "String")

Which I suppose should fail with puppet too?

Ah you are right. It should be lookup ("hostname", String).
So I guess the way the test is currently written is correct.


CI for Node

Failing Tests

Game › Game › Initializes with two players
Game › Game › Starts the game with a random player
Game › nextPlayer › Sets the current player to be whoever it is not


expected_cumulative_transactions will calculate actual transactions incorrectly

While I am using plot_cumulative_transactions, I think I found a bug in utils.expected_cumulative_transactions.
Let's say I want to build a beta_geo_fitter model, and freq='D'.  First I will call utils.summary_data_from_transaction_data to summarize my transaction.  That function will treat transactions which happened on the same day as one transaction when it calculates frequency.
Later on, when I call plot_cumulative_transactions, I see that it calls utils.expected_cumulative_transactions, but that function will treat multiple transactions happening on the same day as multiple transactions (see the line below).
act_transactions = (transactions_current.groupby(customer_id_col).size() - 1).sum()
As a result, the actual and expected lines I got from plot_cumulative_transactions look very different.  I worked around this by calling df.drop_duplicates([customer_id_col, date_col]) on my transaction dataframe first before I call plot_cumulative_transactions.

@patng323 Nice catch. I'll fix that soon and will add a test for that function utils.expected_cumulative_transactions

@patng323  IIUC you are talking about that line in function expected_cumulative_transactions. Some lines before there is reindexing which leads to one record per freq period. So it seems now that there are no any bugs...
Could you provide an example dataset?

@aprotopopov I tried and I am afraid the reindexing line you mentioned will not reduce the data to one record to freq period.
Please note that back in summary_data_from_transaction_data, it is actually this line which did that reduction trick.

The expected_cumulative_transactions is a bit strange: everywhere else we deal with a summary dataset, but here we ask for the raw transactions.
Anyways, the function is broken, but I have a fix like what @patng323 suggests.


Discuss whether to: Run (approved subset of) specification-tests against various servers

We should run the specification-tests against:

NSS
PSS
Solid-Nextcloud
CSS runs them in CD/CI
TrinPod
pod.inrupt.com -> see https://gitter.im/solid/test-suite?at=61fa633b0779373db8c4c7e9

And add the results into the table.

It wasn't clear to me in the meeting on Monday whether all tests in https://github.com/solid-contrib/specification-tests have already been approved by the spec editors team, I know they went to a stricter review process than the jest-based tests, but I seem to remember @csarven saying they are currently not (all) approved, so maybe the set of approved tests is still empty now?

Please do not publish results from the specification-tests for any servers without permission from the implementers. I have made a public commitment to that effect on a number of occasions in blog posts and at SolidWorld. I made it clear to people that I didn't feel the specs or tests were ready for that. In time, it may be appropriate to publish results, or provide a platform for implementers to publish their results, but for now I committed to two things:

If I ran the tests, I would make results available to the implementers via private channels.
I would help any implementers to run the test harness themselves in order for them to get their own reports.

With regards to review status of tests, unfortunately I had to leave the meeting before that was asked. Some tests have been approved by editors, others are not yet reviewed. The status is shown in the reports and defined in the test manifests:

https://github.com/solid-contrib/specification-tests/blob/main/protocol/solid-protocol-test-manifest.ttl
https://github.com/solid-contrib/specification-tests/blob/main/web-access-control/web-access-control-test-manifest.ttl

There is a --status switch available in the test harness to enable it to only run approved tests though in practice people are running all the tests since even unreviewed tests can provide valuable insights and an opportunity to query the test designs. This could, of course, lead to noise if people keep seeing failing tests so where there are known issues or debates the intent is to capture these so that they also get included in the reports. An example of this is here: https://github.com/solid-contrib/specification-tests/blob/main/protocol/requirement-comments.ttl
There is also an opportunity for server developers to add notes about their implementations and to skip inapplicable aspects of tests (preferably with comments on why) here: https://github.com/solid-contrib/specification-tests/blob/main/test-subjects.ttl. The reports show cases like this by marking tests as inapplicable or untested using the EARL vocab.
There is a proposed process for contributing tests and getting them reviewed here: https://github.com/solid-contrib/specification-tests/blob/main/CONTRIBUTING.md
I suggest we close this issue but if an implementer still wants us to publish results, open a separate issue/PR in those specific cases.
Is that ok?

Thanks @edwardsph
That answers the question about which tests are approved and also makes it clear we don't agree on this as a task, so let's turn it into a discussion item. I'll change the title to that effect and let's find time to address it (whether in tomorrow's meeting or elsewhere), and we'll work together to find a way forward with this disagreement.

After discussing with Pete just now I'm withdrawing this proposal.
We can use the specification-tests to find gaps in the jest-based tests, but then if we do find a gap, we can just add a jest-based test there. That way, the reports on solidservers.org are only about the output of the jest-based tests.
In the long run, the goal is for the jest-based tests to be fully replaced by the specification-tests, and for solidservers.org to be fully replaced by information published on solidproject.org.


Question about the depth in dataset

Hello there,
I hope this message finds you well. I'm reaching out to discuss an issue we've encountered while working with your provided dataset. We've noticed that there appears to be a discrepancy in the depth information stored as PNG files.
Upon closer examination, we found that the depth values were encoded into the RGB channels with different operations for each channel:
The red channel (high byte) was calculated using division by 256.
The green channel (middle byte) was calculated using modulo 255 after division by 256.
The blue channel (low byte) was calculated using modulo 255 directly on the depth value.
This inconsistency makes it challenging for us to accurately reconstruct the depth information from the dataset. The non-uniform division and modulo operations lead to a loss of precision and potential data corruption when attempting to decode the depth values back from the RGB channels.
Red channel: r = (f / 256) / 256
Green channel: g = (f / 256) % 255
Blue channel: b = f % 255
We kindly request that you consider updating the dataset to ensure that the depth information is consistently and accurately encoded across all RGB channels. This would greatly improve the dataset's usability and allow for reliable reconstruction of depth data.
Thank you for your attention to this matter, and we look forward to your response.
Best regards

Thank you, I fixed the code, updated the dataset and run some training with an image resolution 128x128. I also renamed some of the tasks, added additional data as well as a script to visualize an episode. I am currently running more tests and will update the webpage soon. Training longer seems to help more :)


trackpad horizontal and vertical scroll

Give the option to choose if vertical mousewheel in an horizontal iscroll will scroll (default right now) or not. If set to false, then horizontal mousewheel will scroll horizontally.

@cubiq did this ever get looked at?

@cubiq, I am running into the same issue this pull request is solving. I see that there are two other pull requests to fix this issue: #1028, #961. Are there plans to get this fixed? What needs to happen to get it fixed?

@cubiq I would love to see this pulled in as well. This seems like it would be an issue an any trackpad that supports scrolling. More and more mouses on Windows are also adding "Magic Mouse" like support. It seems like this problem will only increase.


added Surface Pro 4 and brick docking station files

Hi Dan, can you add more of a description to this pull request so it's clear what these changes are for? For example,  how are these different from the current back and bottom plates?


Add unit tests for parseClassParameters gluster method

Add unit tests for parseClassParameters in glusterfs provisioner

Oops, sorry, I noticed this PR when I already finished my unit tests. Nice try, although there is fake clientset just for this purposes, there is no need to inject ugly code into glusterfs.go.


Parsing http response status-line without reason-phrase

To prevent instantiating a Status each time an empty reason-phrase is received.
See : https://github.com/http4s/blaze/issues/29

After a little thinking I don't know how I feel about this. Making a single object isn't a big issue to me (if we are worried about a single instance per request this is certainly not the library to be using) and while the reason phrase is essentially meaningless I don't see an advantage to inferring one as opposed to just omitting it.

I think we should decide whether reason is important at all.  If it is (my feeling is that it is), maybe always infer it.
How do we feel about seeing
HTTP/1.1 201 wheeeeeeeeee!!!
come out of the library? :)

@refried, I don't actually mind so long as it was the users intent. Just because it doesn't have meaning for the protocol doesn't mean a user can't leverage it to convey meaningful information.

If the user set a specific message, then it worth to keep it, but if the message is empty, doesn't it make sense to take the default message from the specifications?
In my fix, the reason-phrase is kept as it is unless it is empty (in which case, the default value from the specification is taken).

@aaubert: perhaps there is meaning to an empty phrase, much the same way Some(Seq.empty) != None. I'm believe it right to not make assumptions about the meaning of the message and to give the user exactly what the server sent them so long as its not dangerous, overly resource extensive, or against spec in an egregious way.

@bryce-anderson I wasn’t thinking straight before; I agree with you now.

I agree.  The blaze bugfix was necessary, but with that in place, we should return what we received.

I think this is probably not the behavior we want. Thanks @aaubert for the PR both here and in http4s/blaze.


logger.close & transport.close should return promises

Please tell us about your environment:

winston version? ^3.2.1

[ ] winston@2
[x] winston@3

node -v outputs: v12.16.2
Operating System? (Windows, macOS, or Linux) macOS
Language? (all | TypeScript X.X | ES6/7 | ES5 | Dart) TypeScript 3.8.3

What is the problem?
I have a seeding script that runs locally and in builds. It relies on a winston logger that can be configured with multiple transports, including a remote sink over UDP (papertrail), and a daily rotate file transport. When the main function completes and is done with the logger the last line executes logger.close(). The code hangs waiting for the transports to close.
It's very important for the seeding script to exit cleanly and accurately. The build should fail if there is a seeding error - which is based on the return code.
Also: I could not find any clear documentation on how to shut down your logger (and it's transports). I spent a lot of time today dissecting winston code to figure out what is happening.
Lacking a contract for shutting down, each Transport handles things differently.

The daily-rotate-file transport emits 'finish'
The SysLog transport emits 'closed'

My workaround below takes advantage of this, but since it relies on the implementation detail and not a contract, the code could break when we upgrade winston.
What do you expect to happen instead?

The contract for logger.close() and Transport.close should each return a promise.
logger.close should invoke close on each transport (which it does) ...and then return a Promise.all which resolves when every transport's returned close promise has resolved. Which it doesn't.
Establish clear documentation on how to shut down a Logger, including it's transports.

Other information
Here is a workaround... the winston logger is wrapped in a nestjs LoggerService
LoggerService Snippet
  async close() {
    const promises: Promise<any>[] = [];

    // close all transports -- transports dont use promises...
    // syslog close function emits 'closed' when done
    // daily-rotate-file close function emits 'finish' when done
    for (const transport of this.logger.transports) {
      if (transport.close) {
        const promise = new Promise(resolve => {
          transport.once('closed', () => {
            resolve();
          });
          transport.once('finish', () => {
            resolve();
          });
        });
        promises.push(promise);
        // transport.close();  <-- invoked by logger.close()
      }
    }

    this.logger.close();
    return Promise.all(promises);
  }

Calling Code Snippet:
    try {
      await logger.close();
    } catch (err) {
      console.log(err);
      process.exit(1);
    }
    process.exit(0);

Thank you for posting your close() function - it works like a charm for me (also in a wrapper with a console, daily-rotate-file, and custom sentry transport). 👍
Waiting for a log to close is basic and the lack of these contracts is a weakness that makes the library difficult to deal with, so thanks for posting your workaround.

@mlandisbqs I need to wait the transporters to log the messages before closing, but there is no documentation on how to know when a transporter has delivered a message successfuly, how would you do that?
Thanks!


⚠ Bofrost.be has degraded performance

In 6fb9414, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 2391 ms

Resolved: Bofrost.be performance has improved in c75e8c5.


Scene Tree can now display full instanced scenes hierarchy

Scene Tree can now display full instanced scenes hierarchy, useful to keyframe any parameter from any instanced scene children, no matter how deep they are in he hierarchy.

The root of an instanced scene get the same color as its children. There is no reason to treat it differently, they are all part of the instanced scene. This way you get an instant overview of which nodes are part of an instanced scene and which are not.

Can this be merged now? Would sometimes be handy for animating some node properties on instanced scenes.

I believe this has been obsoleted by the changes @reduz did on subscene preview (for 1.1?) and editing for 2.0.


Gdown fail when downloading a lot of files...

I'm using "gdown" to download 700 files, but after sometime, "gdown" still downloading the file with failing...
I'm writing a log file and getting "403" error.
urllib3.connectionpool DEBUG https://doc-0c-60-docs.googleusercontent.com:443 "GET /docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/sbe24qem3pbp0ok09d3p3jud34r78gdo/1655400600000/03896324383948142479/*/1IgH_8zRxyT43o0SeQPBCcKadssad5737aakOZ8WnZH?e=download HTTP/1.1" 403 1103
Can I use some "try/except" or something to get the response code ?
I think google are blocking my downloads, because when I try to download again the same file that failed befora, work well, but after certain time, this error back again.
So if i can check the response code, I can do some tricky to wait or try again...

Do you use gdown.download?
If so it returns None when it encounters some errors e.g.,
https://github.com/wkentaro/gdown/blob/1bf9e20442a0df57eec3e75a15ef4115dbec9b2f/gdown/download.py#L168-L180


incorrect initial configuration

first of all, congratulations on your project. it is really interesting and useful.
I am having problems in the first configuration phase.
when I add the integration to home assistant it starts loading the configuration.
This upload takes a very long time until it goes into error.
in the logs it looks like this:
Logger: homeassistant.util.package
Source: util/package.py:99
First occurred: 12:37:20 (1 occurrences)
Last logged: 12:37:20
Unable to install package music-assistant==1.1.19: error: subprocess-exited-with-error × python setup.py bdist_wheel did not run successfully. │ exit code: 1 ╰─> [22 lines of output] running bdist_wheel running build running build_py running egg_info writing src/greenlet.egg-info/PKG-INFO writing dependency_links to src/greenlet.egg-info/dependency_links.txt writing requirements to src/greenlet.egg-info/requires.txt writing top-level names to src/greenlet.egg-info/top_level.txt reading manifest file 'src/greenlet.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' no previously-included directories found matching 'docs/_build' warning: no files found matching '.py' under directory 'appveyor' warning: no previously-included files matching '.pyc' found anywhere in distribution warning: no previously-included files matching '.pyd' found anywhere in distribution warning: no previously-included files matching '.so' found anywhere in distribution warning: no previously-included files matching '.coverage' found anywhere in distribution adding license file 'LICENSE' adding license file 'LICENSE.PSF' adding license file 'AUTHORS' running build_ext building 'greenlet._greenlet' extension error: command 'gcc' failed: No such file or directory [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for greenlet error: subprocess-exited-with-error × Running setup.py install for greenlet did not run successfully. │ exit code: 1 ╰─> [24 lines of output] running install /usr/local/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( running build running build_py running egg_info writing src/greenlet.egg-info/PKG-INFO writing dependency_links to src/greenlet.egg-info/dependency_links.txt writing requirements to src/greenlet.egg-info/requires.txt writing top-level names to src/greenlet.egg-info/top_level.txt reading manifest file 'src/greenlet.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' no previously-included directories found matching 'docs/_build' warning: no files found matching '.py' under directory 'appveyor' warning: no previously-included files matching '.pyc' found anywhere in distribution warning: no previously-included files matching '.pyd' found anywhere in distribution warning: no previously-included files matching '.so' found anywhere in distribution warning: no previously-included files matching '.coverage' found anywhere in distribution adding license file 'LICENSE' adding license file 'LICENSE.PSF' adding license file 'AUTHORS' running build_ext building 'greenlet._greenlet' extension error: command 'gcc' failed: No such file or directory [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: legacy-install-failure × Encountered error while trying to install package. ╰─> greenlet note: This is an issue with the package mentioned above, not pip. hint: See above for output from the failure. WARNING: You are using pip version 22.0.3; however, version 22.1.1 is available. You should consider upgrading via the '/usr/local/bin/python3 -m pip install --upgrade pip' command.
I have been fighting for 3 days. I can't understand what the problem is. I would be very able if you were able to help me out.
Luigi

duplicate of #268
One of the dependencies (greenlet) can not be found nor built for your platform.
Please let me know the details of your architecture (such as processor/model etc)

@ginoromano first of all, I can't read all of above because it is not in English but do note that there's an error in there that your setup is unsupported. No idea if that is related/the cause but I thought good to mention.
Unfortunately there's not a lot I kind do. One of the requirements for Music Assistant is "databases", which in turn has a requirement called "sqlalchemy". SQLAlchemy has a requirement "greenlet" and that is failing to setup on your setup due to absent prebuilt wheels and it is not able to build because of missing compiler/dev tools.
The very strange thing however is that Home Assistant itself requires the exact same version of sqlalchemy so that package should already be installed. What you could do is check the "deps" subfolder of your "config" folder. Maybe there's some old stuff in there.

I'm afraid there is not a lot I can do to help here. You are running an unsupported configuration, as clearly stated in your screenshots (in red font even). If you're running an unsupported config such as architecture or host OS, there is a chance you run into trouble and without being rude I want to say to you that if you're technical enough to go for the supervised route instead of HA OS, you also have to deal with these kind of compatibility issues.
That said, as a last resort I have now pinned version 1.1.1. of greenlet in the manifest of Music Assistant, this may help (or not, I don't know) and that was the last thing I could try. This small fix is available in version 1.1.22 of Music Assistant, now available on HACS.
If that still doesn't work and you really want to give MA a try, I suggest you to get a supported configuration by either installing HA OS on your device or changing other bits that make the text "Unsupported" go away.
I hope you understand my point of view.
Thanks!


#47: Ansible 2.4 compatibility

I've changed a lot of import_tasks for include_tasks in order to prevent pitfalls when the variables used inside will not be yet computed at the parsing time. In short: whenever we have the variables and/or conditions I prefer to have the includes processing at the runtime.
http://docs.ansible.com/ansible/latest/playbooks_reuse.html#dynamic-vs-static

@lokeoke final call: are we dropping the Ansible =< 2.4 support?

I don't see any reason why to keep it, updating is easy and smooth.


registered_commands to accept partial commands

I like to register a command with arguments and arguments could change, e.g.
ls *.so
ls /home/paperino
I like to create one unique registered_commands for all of them, something like
profiles:

name: ls args
registered_commands:

ls *

where the star is intended as a regular expression to match a set of arguments

lemme know if the latest release works for you @michelepagot


Nada/P2PS-4250/fix: trackjs error with analytics

Pull Request Test Coverage Report for Build<PHONE_NUMBER>2
Details

4 of 5 (80.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.006%) to 85.564%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/hooks/custom-hooks/useDerivAnalytics.ts
4
5
80.0%

Totals

Change from base Build<PHONE_NUMBER>8:
-0.006%

Covered Lines:
6531

Relevant Lines:
7097

💛  - Coveralls


New example - filling up a glass of water

Signed-off-by: Tihomir Surdilovic<EMAIL_ADDRESS>Many thanks for submitting your Pull Request :heart:!
Please specify parts this PR updates:

[x ] Specification
[ ] Schema
[x ] Examples
[ ] Extensions
[ ] Roadmap
[ ] Use Cases
[ ] Community
[ ] TCK
[ ] Other

What this PR does / why we need it:

updates info on expression functions
adds the filling up glass of water example
Special notes for reviewers:

Additional information (if needed):

depends on https://github.com/serverlessworkflow/specification/pull/278
parsed example: https://github.com/tsurdilo/specification/tree/fillglassexample/examples#Filling-a-glass-of-water


Method to increase steelhax installation success rate

After a failed steelhax installation "corrupted save data has been deleted", instead of pressing ok and redoing the process,  just closing the game without pressing ok, going directly to steelhax installer and reinstalling steelhax seems to have a higher succes rate, with commonly working after 1-3 tries. I suggest to edit the steelhax reinstallation part in DSiWare-Downgrade section IV part 15 with this for easier time setting up steelhax.

This seems to help for me too, I think that's because when you press ok the game creates additional data to replace the corrupt data, but exiting directly to home menu instead allows you to install the exploit again without the new data being created.

thanks a lot for sharing this info!

Works as stated by @FrozenChen "corrupted save data has been deleted" >> press home before  prompted to create new mi....re-install steelhax, launch game, HBL will launch after 3DS splash screen loads.

Fixed


Unable to Change Email Address on Account

If your account has an email address that you no longer have access to, changing the email address on the account is impossible. The confirmation message for changing the email address will be sent to the old address. If, for some reason, you cannot access it, you won’t be able to update the email address. And as a bonus, the Bluesky support team does not respond to email requests.
What platform(s) does this occur on?
iOS, Android, Web (Desktop), Web (Mobile)
What version of the app are you using?
latest

Twitter also does not allow you to update email addresses if you don't have access to your email.
Github allows you to add "Primary", "Backup" emails, so I agree that it is desirable if email is an array of strings and you can set one of those to be "Primary" and "Backup", so you won't get locked out in case you lose your bluesky login password.

Twitter also does not allow you to update email addresses if you don't have access to your email.

Xitter also have more options to login and MFA.


fix: ensure that updatedAt is returned as string value

In db we store it as number and we return invalid value to client.

Verified in datasync-starter


scope stack doesn't allways display or doesn't display proper scope or doesn't handle scopes correctly in certain instances

Example for displaying incorrectly:
if a decision is a targeted_decision towards a state the FROM scope should be a state, but in the tooltip says FROM is a country scope. It does handle it as as state scope though.

Example for not handling:
in on_actions a on action like on_unit_leader_promote_from_ranks_veteran will allow any trigger or effect in it's effect block. Like it allows a trigger that requires a country scope and a trigger that requires a state scope at the same time on the same level.

I've been trying to work around by editing the hoi4 cwt rules but haven't had any luck with getting the on_actions to work

I've managed to get on_action working with scopes but it required removing the subtypes. In general it seems like scopes and subtypes don't always work nicely.
Another issue I found is specifically with FROM.FROM not giving the correct scope.

On a related note the autocomplete for hoi4 suggest prevprev or fromfrom instead of prev.prev and from.from.

A note for a future myself - to revert https://github.com/Kaiserreich/cwtools-hoi4-config/commit/ede26ae586dd16b45c6010dd3ef36f8bbdeef2af once this issue is fixed


[Serialization] Pass along allow errors/skip bodies flags

Building swiftinterface's should use the allow errors and skip bodies
flags of the main compilation. Consider a swiftinterface A that depends
on a module B. If B is built allowing errors then deserialization while
building the module for A should not fail if there are in fact errors.
Resolves rdar://84515859.

@swift-ci please test


Get moveit transform

This uses RobotState::getGlobalLinkTransform() to get the transform from MoveIt's planning frame to the user's desired planning frame, instead of tf2. This should update faster and be more accurate.

Should have been opened against the official MoveIt repo


How to wrap object fields

Not everything needs to be wrapped, for example math objects are ok. Fields with naming like xCount need to be write protected. However object like Mesh hold collections that need to be wrapped in order to implement range checks for example.
First one is easy, remove export marker and write procs, as explained in the manual:
proc glyphCount*(x: Front): int32 = x.glyphCount

My solution for the last one is to return hidden pointers (this has the cost of introducing temporaries) and define array access operators on distinct types, one for each field:
type
  MaterialMaps* = distinct Material

template maps*(x: Material): MaterialMaps = MaterialMaps(x)

proc `[]`*(x: MaterialMaps, i: MaterialMapIndex): MaterialMap =
  # implement range checks here
  result = Material(x).maps[i]

proc `[]`*(x: var MaterialMaps, i: MaterialMapIndex): var MaterialMap =
  result = Material(x).maps[i]

proc `[]=`*(x: var MaterialMaps, i: MaterialMapIndex, val: MaterialMap) =
  Material(x).maps[i] = val

Tested and it works. Pros: Accidentally creating a copy: let x = x.maps is prevented by the compiler (no =copy defined for distincts), also the syntax remains the same. Cons: temporary pointer variables are introduced by the C back end.

Also worth exploring dot operators: https://nim-lang.github.io/Nim/manual_experimental.html#special-operators-dot-operators

Needs to support .[] and .[]= in order to work properly.

I even changed the parser to accept .![] but the solution is simple. Just use templates! Updated the issue.

I'm not sure I completely understand the problem. Even with range checks - usually you just pass a model to a function and this function parses it into a suitable C struct so we rarely have to interact with the internals unless we want something very specific. And at this point we might as well risk a little bit since these cases should be rare and it will be relatively simple to isolate the bug should it happen.
I don't know the main use case for this so the above can be wrong. Could you clarify the need for this? This change looks a bit involved so I wouldn't include it unless it drastically improves the ease of using the library. And there's always a reason that C works fine with it, do we really want to put more safeguards by sacrificing the familiarity with C.

Could you provide the resulting syntax after your suggested change? It looks like model.materials[Albedo] where I would expect model.materials.maps[Albedo] as in C

The second one! Actually for ModelAnimation.framePoses I used m.framePoses[i, j] because it's ptr UncheckedArray[ptr UncheckedArray[Transform]]

It totally went over my head that `[]`, `[]=` both suffer from https://github.com/nim-lang/Nim/issues/19351 We need .copy pragma for procs as araq suggested.

It totally went over my head that `[]`, `[]=` both suffer from nim-lang/Nim#19351 We need .copy pragma for procs as araq suggested.

Does .copy pragma imply that there will be an allocation and copy somewhere? I didn't see a comment from Araq on this topic

.copy means the parameter type can't be a hidden pointer, there is a heuristic like sizeof(param) > 3*sizeof(float) that changes parameter types for non var proc parameter.


Error while executing on Docker

There's an error shown while executing the sample through docker

@XaserAcheron @aaguiarz Got the exact same error.
The sample used is the one downloaded in this quickstart: https://auth0.com/docs/quickstart/spa/ember

Hmm, that sample's definitely out of date, unfortunately. :(
There's a slightly-more-minimal quickstart guide in the README.md of this repo, but it's up to date. Give that a shot and see if it works out for y'all. It's all I can suggest from here, I'm afraid.


EPERM: operation not permitted, rmdir

[x] I have read the contribution documentation for this project.
[x] I agree to follow the code of conduct that this project follows, as appropriate.
[x] I have searched the issue tracker for an issue that matches the one I want to file, without success.

Please describe your issue:
When I run electron-forge make, while "Preparing to Package Application for arch: x64" I will consistently get:

EPERM: operation not permitted, rmdir 'C:\Users\MYUSERNAME\AppData\Local\Temp\electron-packager\win32-ia32\kingdomsim-win32-ia32\resources\app'
Error: EPERM: operation not permitted, rmdir 'C:\Users\MYUSERNAME\AppData\Local\Temp\electron-packager\win32-ia32\kingdomsim-win32-ia32\resources\app'

The same project runs make just fine on Mac. Haven't tried on Linux. I am running Command Prompt in Administrator mode.
What command line arguments are you passing?
electron-forge make

What does your config.forge data in package.json look like?
"forge": {
      "make_targets": {
        "win32": [
          "squirrel"
        ],
        "darwin": [
          "zip"
        ],
        "linux": [
          "deb",
          "rpm"
        ]
      },
      "electronPackagerConfig": {
        "packageManager": "npm",
        "icon": "./textures/icon",
        "out": "./releases"
      },
      "electronWinstallerConfig": {
        "name": "kingdomsim",
        "setupIcon": "./textures/icon"
      },
      "electronInstallerDebian": {},
      "electronInstallerRedhat": {},
      "github_repository": {
        "owner": "",
        "name": ""
      },
      "windowsStoreConfig": {
        "packageName": "",
        "name": "kingdomsim"
      }
    }

Here's the link to my most recent commit.

I am running Command Prompt in Administrator mode.

That's your problem, as a general rule.

Never run any tooling as sudo or Administrator unless it specifically asks you to.  Most tooling is not designed to handle permission changes, inconsistencies or failures.

Delete all the admin controlled stuff it made in your Temp directory and run it again as your user, not an administrator. You wouldn't run it with sudo on your mac, don't use admin on windows 😀


iter8-KUI: HIL Experiments with iter8 initial campaign

Create an simple blog article for the inital iter8-KUI campaign

Is it time to close this epic? @sushmarchandran @sriumcp @fabolive


Add support for Swift 3.0, including the swift package manager

Maintains iOS support, and the general .xcodeproj so that dynamic frameworks can still be built.
Renames the ZeroMQ C module to CZeroMQ, to be more idiomatic (see https://github.com/apple/swift-package-manager/blob/master/Documentation/SystemModules.md), and for the swift package manager, use my repo to provide CZeroMQ, since SwiftPM REALLY wants a git repo for that external module. 😞
Also removes the declaring parameters as var stuff in Socket.swift, because SE-0003.
Restructures folder structure to be Sources and Tests, as opposed to SwiftZMQ and SwiftZMQTests, because Swift Package Manager.

We already have https://github.com/Zewo/CLibzmq module maps. is it good enough? can you link to it? If not can you please send a PR to it? and maybe add some instructions for linux?

Whoa hey. Didn't know of that. Thanks, I'll update everything to use CLibzmq.

I can rename it to CZeroMQ. It seems to be more idiomatic:

Note we do not call the module CLibjasper. In general, avoid the lib prefix unless the authors of the package typically always refer to it that way. A good rule of thumb is to look at the header files, here we can see the header is called simply "jasper.h". In the event of non-typical headers (eg jpeglib.h) refer to the project homepage, the authors of the JPEG library refer to it as “The JPEG library” and not “libjpeg” or “jpeglib”. Pay attention to capitalization; it is CJPEG and not CJpeg, because JPEG is an acronym and is typically spelled all-caps. It is CJasPer and not CJasper because the project itself refers to the library as “JasPer” in all their documentation.

Additionally, push a tag to it. The SwiftPM really wants a tag for that major version.

I just went ahead and renamed it.

Is it good? give me an OK if I can tag it, or send a PR to make any necessary changes and then I'll tag it. If you could please add instructions for linux installation.

Tag it once https://github.com/Zewo/CZeroMQ/pull/3 is merged. Thanks!

merged and tagged


afreq is still too slow

We're doing a lot of unnecessary parsing in the current version and run time, even with pure python, should be much better.

https://github.com/hall-lab/svtools/pull/111 takes me from ~26 hours or so on ~5000 samples to about 7.25 hrs on our cluster.


Update Russian language

Thanks @WebVPF! :)

@LukeTowers @bennothommo I want to warn you.
Today someone said that you can install paid October plugins in Winter CMS.
They say it's a violation of the license.
Threatened with legal action.
In the Russian-speaking community of October, they are asked not to raise a fuss for the time being.
I will delete this comment after you read it.

Interesting, sounds like a bug in the API code on our side, installing paid plugins should work, but only if you have a valid project ID for the product as determined by the October marketplace. Shoot me an email or message me on discord with the details and I'll look into it and fix it if it's an issue on our side. Thanks for the report @WebVPF!

@LukeTowers I didn't install paid plugins. I'm just reporting that this was reported to Sam and Bobkov. They threaten to sue. This is written by Lavtsevich (Lovata) and Biz-Mark

@LukeTowers https://t.me/octobercms_rus/202867


Add more i18n sniffs

This PR is to add another test to the I18nSniff.php to ensure that multiple placeholders are orderable.
e.g. from the PHP: sprintf page:
__( 'The %s contains %d monkeys' ); is a problem because the variables will need to be the other order in some languages, and the string should be:
__( 'The %1$s contains %2$d monkeys' );
The PR includes tests and a fixer.
One thing worth discussing is that the fixer could potentially get it wrong if:

It's a pluralised string
The order of the placeholders is different between the two strings
The placeholders are unordered.

The test file gives an example of the sort of string I'm talking about:
_n( 'There is %d monkey in the %s', 'In the %s there are %d monkeys', $number, 'my-slug' );
My assessment here is that this is a bug in the code, and this sniff won't fix it, but it also won't make it any worse than it already was.  I was tempted to not apply the fix in plural cases, but think the good outweighs the bad, so for now the fix does run on plurals.

This looks great, thanks @deBhal! In regard to the plurals and placeholders in the wrong order, I don't think that we really need to worry about that. As you say, it is a bug in the plugin itself. Though I suppose we could check if the order is the same in both of the strings, and report in error if it isn't. In some cases it would be undetectable though, if the user was just using %s both times instead of %d.

Thanks for the quick feedback!
I've added another new sniff for the singular problem described in https://codex.wordpress.org/I18n_for_WordPress_Developers#Plurals
_n( 'This is a subtle problem', 'These are subtle problems', $number ).
"Singular" means exactly one in English, but that's not the case in several languages.
I've also added a warning for mismatched placeholders, which will catch at least some of the cases we talked about above.
I'm adding these to this PR rather than opening separate ones because the way the tests work would force annoying re-edits if they came from separate PRs.
Quick question:  I've used $this->compare_single_and_plural_arguments(), but the existing code uses call_user_func() instead of calling a member function directly.  Is there a reason for this?  PHP versions or something?

"Singular" means exactly one in English, but that's not the case in several languages.

Thanks for this, I didn't realize that (or maybe had just forgotten).

Quick question: I've used $this->compare_single_and_plural_arguments(), but the existing code uses call_user_func() instead of calling a member function directly. Is there a reason for this? PHP versions or something?

I'm pretty sure that this is just a leftover from when the code called one of several different methods in that spot, but it has been refactored to just call that one now. So there is really no reason for call_user_func() to be used there anymore, it is just a relic.

Thanks, @GaryJones.  I've fixed that comment and several others, and squashed the feedback changes into logical commits.

I've added one more test that checks for translations that are just placeholders ( '%s just wastes everyone's time).
I'm not expecting to add any more tests now, but I'll still respond to feedback.


Error generating epicov data

[covizu@BEVi ~]$ tail batch.log
...
🏄 [12:20:25.736335] start BQ.1.1, 73245 entries
🏄 [12:27:05.707608] Parsing output files
Failed to retrieve metadata for accession EPI_ISL_18554387

https://github.com/PoonLab/covizu/blob/12a9d08b70d754a30b72d84636affb28d3934765/covizu/utils/gisaid_utils.py#L327-L341

Can we grep for this accession number in that provision file?

This accession number existed in a previous provision file, but not in the latest file. Investigating how this accession number is showing up in the current clusters.json file

Sometimes sequences are retracted in the database so the record that appeared in a previous provision file would no longer appear in subsequent files.

https://github.com/PoonLab/covizu/blob/22f4f4cac124292e1f404c83f7068c3e4033fd55/batch.py#L320-L338


Migrate docs client webapp to React / Typescript / Material UI

This large PR migrates the docs client from a Javascript/jQuery/Bootstrap-based architecture to Typescript/React/MaterialUI.
It adds a separate docs-client project which is a standard NodeJS / webpack based build, run using gradle-node-plugin. Currently it's a separate JAR, but if it makes more sense to add it into the core JAR pretty sure we can do that too.
I changed the URLs a little, /method -> /methods, /namedType -> /structs + /enums. Since struct and enum pages are quite different, it made sense to separate the URLs out. The old URls are still supported and redirect to the new ones.
The TypeScript configuration is as strict as possible, including whitespace / semantic linting based on AirBnb style + Prettier. It's a pretty common configuration nowadays.
The README covers development concepts, let me know if there's anything else to add.
To try out the client, I'd recommend pulling the branch and just stuffing a Thread.sleep(Long.MAX_VALUE) in one of ThriftDocServiceTest or GrpcDocServiceTest.
For #654

@anuraaga Please rebase your PR so that Travis CI builds do not fail anymore due to JDK download failure.

Miscellaneous issues found:

There's no message when there are no parameters/exceptions/members, e.g. 'This method does not have parameters.'
'Use these HTTP headers for all functions.' checkbox does not work.

From aesthetic PoV, I think it looks worse. :sweat_smile: But I guess we can improve on that later.

Thanks - I found in the old code the only one that shows "There are no..." is the method parameters, the rest are just hidden so tried to reproduce that.
Also, sticky headers should be working if you submit the form. In the old code, it coincidentally sort of worked even if you didn't submit because state wasn't reinitialized, but if you went to a page with debug form to one without and back, they'd be gone. Is it better if the sticky kicks in before hitting submit button? Having three sources of the information (URL, JS state, specification) isn't terrible but a bit complicated.
And yeah happy to tweak aesthetic in this or followup PR (maybe a material design designer sits next to you? :D). In particular, I'm thinking of adding Paper in certain places, I think the grey background makes it look weird in many places.

Thanks - I found in the old code the only one that shows "There are no..." is the method parameters, the rest are just hidden so tried to reproduce that.

I see. It would be still nicer if 'there are no...' is shown for exceptions and fields as well.

Also, sticky headers should be working if you submit the form. In the old code, it coincidentally sort of worked even if you didn't submit because state wasn't reinitialized, but if you went to a page with debug form to one without and back, they'd be gone. Is it better if the sticky kicks in before hitting submit button?

I think so. A user may have filled all the headers and then realized that he or she was on a wrong service method. :-)

And yeah happy to tweak aesthetic in this or followup PR.

No worries. I'm not very sure what changes would make it look better yet anyway. Let me play with it a little bit once merged. :bowing_man:

Thanks - think I addressed the comments. Also added some Paper so it's less grey and I'm sure we'll come up with more aesthetic tweaks in the future :)

Thanks! It looks much better. :-)
Some more bugs:

If you choose Enums -> ConsistencyLevel and scroll down the right pane, the left pane looks truncated.
The left pane does not disappear into hamburger even if you shrink the width of the window.
The HTTP headers text area should be visible if it's not empty. For example, if you tick the 'Use these HTTP headers for all functions' checkbox in HelloService.hello() and switch to SleepService.sleep(), the HTTP headers text area should be visible.

As you see below, the left pane's height is not synchronized with the right pane's height.

Thanks for the thorough check, had forgotten about the navigation. Now it's responsive, and also fixed the headers textbox so it shows when there are headers.
And after lots of playing, figured out the CSS magic to make the heights work I think!

Great job! :-)
But, in mobile layout, after I click the hamburger, I find no way to hide the method list. I also see excessive top margin. Perhaps appbar should be there so a user can click the hamburger again to hide the method list?

Ok render the mobile drawer at 80% width so clicking off of it can close it, this seems to be what e.g., Google Maps does. I also had the top-space just copied from the material-ui.com example, not sure why they had it seems fine without it :) Not just trying to figure out why their example renders sticky subheaders correctly but not mine, if I can't figure it out I'll just disable it.

Got the method names to not overlap - think it's ready for another look

More minor issues. :pray:
Could we reduce the left margin of the hamburger button so it is aligned with the page content?

Could we reduce the left margin of the left pane so that method names are placed better? Also, when a cursor hovers at a method name, the gray selection does not cover the row completely, leaving white background on the left of the method name.
Could we reduce the font size of the items in the left pane? It looks too big comparing to the right pane content. On the other hand, could the service names in the left pane be bigger?
I see a performance issue when pressing the hamburger button. It takes about a second since I clicked the button for the left pane to show up. Could we make it appear sooner?

Fixed the method margin, it was introduced by extra ul tags that should have been div.
Tweaked the fonts so service name is larger and method names are smaller. Typography is a complex problem, and hoping to use the defaults as much as possible for now and customize it better in the future - WDYT?
And I'm not sure if I'm seeing the same performance issue, it feels like around 2-300ms for me, but I tried adding a prop that might make it faster for slower devices.

Thanks a lot for the detailed review!

As a preview, looking forward to sending https://github.com/anuraaga/armeria/pull/5 with pluggable auth support after this is in :)

@minwoox @hyangtack Gentle ping :stuck_out_tongue_winking_eye:

Finally-! Thank you so much for your patience and contribution, @anuraaga !


Optimize quoting process

The current approach of generating a quote for an order involves estimating the order as it is as well as estimate an "artificial" order buying WETH for the sell_token of the user order.
Let's call the first order buy_token_estimate. This will have whatever order kind the user specified.
Let's call the second one native_token_estimate. This will always be a buy order.
Having at least one buy order for every user order regardless its order kind poses a problem for PriceEstimators which can't handle buy orders (e.g. OneInchPriceEstimator).
The code section best shows the main pain point.
Now the question is: Is there an architecture which gives us enough flexibility to actually use PriceEstimators which can't handle buy orders?
There are different solutions which different trade-offs.
Improve Caching native_token_estimate
As you can see here we already cache previously computed FeeData to skip computing it again for identical orders.
When we compute the sell_token_price we always estimate a buy order with a constant amount where WETH is the buy_token. Only the order kind and sell_token is variable.
Given that information one option could be to add another code path which only fetches the sell_token_price for the user order. This should result in many cache hits where every cache hit means that a price estimator doesn't have to estimate a buy order.
I think this could be a general improvement but in the particular example of the OneInchSolver it could make the estimator viable at all.
Forwarding Estimations Based on Order Kind
Given the fact that not every estimator is able to estimate every order kind, we could simply forward order estimations to different estimators based on the order kind. Where this should happen is unclear. One option could be to have 2 distinct PriceEstimator hierarchies (basically copy the one we have but with out the OneInchPriceEstmator). Alternatively we could introduce a MuxPriceEstimator which takes 2 estimators (1buy, 1sell) to construct and forwards
queries appropriately.
Give PriceEstimators a Fallback
This is basically a different flavor of the previous solution. Instead of deciding outside of the PriceEstimator where to forward quotes to, a ParentPriceEstimator could do that by having a fallback ChildPriceEstimator for scenarios the parent can't handle. This pattern is already widely used in the code base but not for this particular use case.
The downside is that it would result in doing some work twice. Imagine giving the OneIncPriceEstimator the BaseLinePriceEstimator as a fallback. Now the BaseLinePriceEstimator would estimate an order twice for every order OneInchPriceEstimator can't handle (once as the fallback and once as the standalone BaseLinePriceEstimator).
Use Rich Price API
Computing the value of a token like we do is a common use case. That's why zeroex and paraswap have a rich price API which does just that. This approach would essentially make computing the sell_token_price a responsibility of the price estimator. This is not really a solution for the problem at hand, but this change is really desirable for zeroex and paraswap so a solution enabling OneInchPriceEstimtator shouldn't make it more difficult to implement this change for zeroex and paraswap. This could be a bigger redesign of the price estimation API. This design would probably go well with the previous idea.
One thing we should keep in mind is how we can properly utilize our existing cache for estimators which are not paraswap and zeroex. Those would have to implement the rich price API on their own while still caching as much as possible.

One imo reasonable suggestion we already thought about is always using a "local" price estimator for the native token order. That would fix this issue too and is kind of like your fallback idea. The local estimator would be quasimodo or baseline.

One imo reasonable suggestion we already thought about is always using a "local" price estimator for the native token order

For reference, here is the issue: https://github.com/gnosis/gp-v2-services/issues/1542

Some general comments:

Improve Caching native_token_estimate
I think this could be a general improvement but in the particular example of the OneInchSolver it could make the estimator viable at all.

While I agree, I do see using our DB cache as a little orthogonal WRT to making the 1Inch price estimates work. While it would allow estimates in many situations, it doesn't solve the problem of 1Inch price estimator not being able to compute the sell token price in ETH. If we do implement this, I still think we need something to make the 1Inch price estimator work even without a cache.

Give PriceEstimators a Fallback

Another flavour of this is to have MinFeeCalculator take two price estimators, one for computing order estimates (so sell_token for buy_token price and gas estimates) and another for computing sell token ETH price estimates (so buy 1ETH with sell_token price).
This allows us to configure the two separately, so we can configure 1Inch price estimator is only used for the order estimate. Additionally, this also allows us to implement #1542 by configuring the sell token ETH price estimator to just use Quasimodo and/or Baseline solvers.

Personally, I'm the most fan of the "rich API" approach. There are risks however (for example, we don't know where the ETH and USD prices come from in 0x, 1Inch and ParaSwap API, so it might cause us to under-quote fees for illiquid tokens).

Also worth metioning is @fleupold's comment here: https://github.com/gnosis/gp-v2-services/issues/1571#issuecomment-1014766988. Specifically, this is about the /auction endpoint and the suggestion to use all our price estimators (0x, ParaSwap, 1Inch) for computing the "external price" array.
This might have an impact on what we chose to use for "sell token ETH price estimator" in this particular case.

The caching idea is indeed no solution which makes the OneInchPriceEstimator work for every scenario, but it could work around the problem enough to make it viable.
Always using quasimodo for the native_token_estimate sounds very nice from a maintenance perspective. Not sure how big the consequences for our and the users costs might be if we don't have estimator competition for the native_token_estimate, though.
I guess this change should be easy enough to get the OneInchPriceEstimator going soon while being easy to throw away once we tackle the rich price API.

Just to summarize what we discussed in the meeting today:
We decided to add a new component specialized specifically for computing token prices in ETH (more specifically, native token price, so xDai on Gnosis Chain).
This component can hopefully be implemented in terms of the current PriceEstimating interface (and use the same underlying price estimators). Also, critically, it should be cached (keyed on the token we are estimating the price for) and shared with the code that computes the /auction external price vector. Furthermore this cache can be updated on a periodic basis (together with how we update the solvable_orders).
We also decided to not to do use rich API responses

We also decided to not to do use rich API responses

This means that even with the /auction endpoint we won't increase the number of tokens that are tradable (we have a lot of unsupported tokens because of "no pool")? Is there another plan to support those without adding more native liquidity to the system?

With the merge of #1615 this issue can be considered done.


Minimal working example for Enright test

Hello,
I am learning a way to use libeltopo.a library in C program, but so far examples directories are huge and too much interconnected. A minimal example which I would like to achieve is to make Enright test, for octahedron with Euler time integration and in between that all fancy stuff which are present in figures 4.4, 4.5 from [Brochu and Bridson 2009].
How can I write it or on which source files should I look to extract minimal working example?

Under talpa/drivers/ there is a file called EnrightDriver.h which does the Enright test. I'm using RK4 rather than forward Euler, but you can easily modify that part of the code.

I don't understand where the functions defined in eltopo.h are used for integrating EnrightDriver::set_predicted_vertex_positions from enrightdriver.h. Actually it might be sufficient to see how ElTopoStaticOperationsOptions and ElTopoDefragInformation are initialized for running the enrightdriver.h.

Ahh, right, those drivers use the El Topo data structures directly (e.g. SurfTrack). For an example on using the API from eltopo.h, see main.cpp in talpa/. See specifically el_topo_wrapper_static_operations and el_topo_wrapper_integrate.

that looks promising ;) Only where can I find definition of g_surf for checking out default parameters?

Hello. Some days ago I got back on track to interface this library, which after intense focus resulted in minimal working example. As I see that issue is still open and it might be usefull for other newcommers, I report my working example:
// A real minimal example. Compile it with
// g++ surftrackexample3.cpp libeltopo_release.a -llapack -lblas -lstdc++ -lm -I../common -I../eltopo3d -o test
//C hello world example

#define NSIZE 20000

#include <stdio.h>
#include <math.h>
#include <vector>
#include <vec.h>
#include <surftrack.h>
#include <subdivisionscheme.h>

#include "mesh.c"

int printmesh(int ii, int Nverticies, int Ntriangles, double * verticies, int * triangles) {

  char buf[256];
  snprintf(buf, sizeof buf, "%s%d%s", "output/", ii, ".obj");

  FILE *fp;

  fp = fopen(buf, "w+");
  for (int i=0;i<Nverticies;i++){
    fprintf(fp,"v %f %f %f \n",verticies[3*i],verticies[3*i+1],verticies[3*i+2]);
  }

  for (int i=0;i<Ntriangles;i++) {
    fprintf(fp,"f %d %d %d\n",triangles[3*i]+1,triangles[3*i+1]+1,triangles[3*i+2]+1);
  }

  fclose(fp);
}

struct SurfaceMesh
{
  double * verticies;
  int Nverticies;
  int * triangles;
  int Ntriangles;
};

void improvemesh(SurfaceMesh msh,SurfaceMesh * outmsh)
{
  std::vector<Vec3d> vs;
  std::vector<double> masses;

  for ( int i = 0; i < msh.Nverticies; ++i )
    {
      vs.push_back( Vec3d( msh.verticies[3*i], msh.verticies[3*i + 1], msh.verticies[3*i + 2] ) );
      masses.push_back( 0.5 );
    }

  std::vector<Vec3st> ts;
  for ( int i = 0; i < msh.Ntriangles; ++i )
    {
      ts.push_back( Vec3st( msh.triangles[3*i], msh.triangles[3*i + 1], msh.triangles[3*i + 2] ) );
    }

  SurfTrackInitializationParameters parameters;

  parameters.m_proximity_epsilon = 1e-4;
  parameters.m_friction_coefficient = 0.0;
  parameters.m_min_triangle_area = 1e-7;
  parameters.m_improve_collision_epsilon = 2e-6;
  parameters.m_use_fraction = false;
  parameters.m_min_edge_length = 0.05;
  parameters.m_max_edge_length = 0.2;
  parameters.m_max_volume_change = 0.1;
  parameters.m_min_triangle_angle = 35;
  parameters.m_max_triangle_angle = 180;
  parameters.m_use_curvature_when_splitting = false;
  parameters.m_use_curvature_when_collapsing = false;
  parameters.m_min_curvature_multiplier = 1.0;
  parameters.m_max_curvature_multiplier = 1.0;
  parameters.m_allow_vertex_movement = false;
  parameters.m_edge_flip_min_length_change = 1e-8;
  parameters.m_merge_proximity_epsilon = 1e-5;
  parameters.m_collision_safety = true;
  parameters.m_allow_topology_changes = false;
  parameters.m_allow_non_manifold = false;
  parameters.m_perform_improvement = true;

  parameters.m_subdivision_scheme = new ButterflyScheme();

  SurfTrack surface_tracker( vs, ts, masses, parameters );
  surface_tracker.m_verbose = false;

  surface_tracker.improve_mesh();
  surface_tracker.topology_changes();
  surface_tracker.defrag_mesh();

  // Creating usual array objects

  int Nverticies_out = surface_tracker.get_num_vertices();
  //double verticies_out[Nverticies_out*3];

  for ( int i = 0; i < Nverticies_out; ++i )
    {
      outmsh->verticies[3*i+0] = surface_tracker.get_position(i)[0];
      outmsh->verticies[3*i+1] = surface_tracker.get_position(i)[1];
      outmsh->verticies[3*i+2] = surface_tracker.get_position(i)[2];
    }

  int Ntriangles_out = surface_tracker.m_mesh.num_triangles();
//  int triangles_out[Ntriangles_out*3];
  for ( int i = 0; i < Ntriangles_out; ++i )
    {
      const Vec3st& curr_tri = surface_tracker.m_mesh.get_triangle(i);
      outmsh->triangles[3*i + 0] = curr_tri[0];
      outmsh->triangles[3*i + 1] = curr_tri[1];
      outmsh->triangles[3*i + 2] = curr_tri[2];
    }

  printf("Nverticies=%d  Ntriangles=%d   volume=%f \n",Nverticies_out,Ntriangles_out,surface_tracker.get_volume());

  outmsh->Nverticies = Nverticies_out;
  outmsh->Ntriangles = Ntriangles_out;
}

int main()
{
  printmesh(0,Nverticies,Ntriangles,verticies,triangles);

  SurfaceMesh msh;
  msh.verticies = verticies;
  msh.Nverticies = Nverticies;
  msh.triangles = triangles;
  msh.Ntriangles = Ntriangles;

  double verticiesout[NSIZE];
  int trianglesout[NSIZE];
  SurfaceMesh mshout;
  mshout.verticies = verticiesout;
  mshout.triangles = trianglesout;

  improvemesh(msh,&mshout);
  printmesh(101,mshout.Nverticies,mshout.Ntriangles,mshout.verticies,mshout.triangles);

  printf("Hello world\n");
  return 0;
}

Here #define mesh.c deffines mesh of sphere, but printmesh puts mesh to .obj format.


Integration by copying the binaries code sign error

I followed the Integration by copying the binaries instructions, but get the errors below when building the app.   Any suggestions?
code object is not signed at all
In subcomponent: AppCenterCrashes.framework
Command CodeSign failed with a nonzero exit code
Previously, I was using the hockeyappSDK without issue.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: bdaabafa-4dc0-7db9-cf7a-fafa24667c62
Version Independent ID: b033ae6a-2ff6-59f6-7e0f-2d05f4a0ae34
Content: Get Started with macOS - Visual Studio App Center
Content Source: docs/sdk/getting-started/macos.md
Service: vs-appcenter
GitHub Login: @elamalani
Microsoft Alias: emalani

Was able to eliminate many of these errors by adding other Apple frameworks to the project, but I'm still left with following errors I cannot resolve...
Undefined symbol: std::__1::__vector_base_common::__throw_length_error() const
Undefined symbol: std::__1::__basic_string_common::__throw_length_error() const
Undefined symbol: std::logic_error::logic_error(char const*)
Undefined symbol: std::length_error::~length_error()
Undefined symbol: std::exception_ptr::exception_ptr(std::exception_ptr const&)
Undefined symbol: std::exception_ptr::~exception_ptr()
Undefined symbol: std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >::assign(char const*)
Undefined symbol: std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >::reserve(unsigned long)
Undefined symbol: std::set_terminate(void ()())
Undefined symbol: std::current_exception()
Undefined symbol: std::rethrow_exception(std::exception_ptr)
Undefined symbol: std::terminate()
Undefined symbol: typeinfo for char const
Undefined symbol: typeinfo for std::length_error
Undefined symbol: typeinfo for std::exception
Undefined symbol: vtable for __cxxabiv1::__class_type_info
Undefined symbol: vtable for __cxxabiv1::__pointer_type_info
Undefined symbol: vtable for __cxxabiv1::__vmi_class_type_info
Undefined symbol: vtable for std::length_error
Undefined symbol: operator delete(void*)
Undefined symbol: operator new(unsigned long)
Undefined symbol: ___cxa_allocate_exception
Undefined symbol: ___cxa_begin_catch
Undefined symbol: ___cxa_current_exception_type
Undefined symbol: ___cxa_demangle
Undefined symbol: ___cxa_end_catch
Undefined symbol: ___cxa_free_exception
Undefined symbol: ___gxx_personality_v0
Undefined symbol: _deflate
Undefined symbol: _deflateEnd
Undefined symbol: deflateInit2

found the issue.   Unlike hockeyappSDK, I had to remove the AppCenter frameworks from the Copy File Build Phase.

found the issue.   Unlike hockeyappSDK, I had to remove the AppCenter frameworks from the Copy File Build Phase.