AttributeError: module 'tensorflow.python.eager.backprop' has no attribute '_record_gradient'

Hello
I am trying to get shap_values using DeepExplainer but it keep giving this error
"AttributeError: module 'tensorflow.python.eager.backprop' has no attribute '_record_gradient'"

same error with tensorflow 2.7 and python 3.8.11.
Any solutions that do not require reverting tensorflow?

@juan-carlos-calvo I haven't found any, sorry. Although I had no side effects from reverting, and shap works great!

yeah, but I need functionality only available in the latest tensorflow version :/. I'm trying with some of the other explainers for now.

given that importing keras works differently in later versions (tf.keras), not supporting recent tensorflow versions puts stress on the rest of the codebase

I tried to fix it and opened the pull request: #2355

still happening in kaggle kerner
python version: Python 3.7.12
shap==0.40.0

tensorflow @ file:///opt/conda/conda-bld/dlenv-tf-2-6-cpu_1639877225671/work/tensorflow-2.6.2-cp37-cp37m-linux_x86_64.whl
tensorflow-addons==0.14.0
tensorflow-cloud==0.1.14
tensorflow-datasets==4.3.0
tensorflow-estimator==2.6.0
tensorflow-gcs-config==2.6.0
tensorflow-hub==0.12.0
tensorflow-io==0.21.0
tensorflow-metadata==1.5.0
tensorflow-probability==0.14.1
tensorflow-serving-api==2.7.0
tensorflow-transform==1.5.0

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/tmp/ipykernel_54/3488049008.py in <module>
      6 # ...or pass tensors directly
      7 # e = shap.DeepExplainer((model.layers[0].input, model.layers[-1].output), background)
----> 8 shap_values = e.shap_values(correct[1:5])
      9
     10 # plot the feature attributions

/opt/conda/lib/python3.7/site-packages/shap/explainers/_deep/__init__.py in shap_values(self, X, ranked_outputs, output_rank_order, check_additivity)
    122             were chosen as "top".
    123         """
--> 124         return self.explainer.shap_values(X, ranked_outputs, output_rank_order, check_additivity=check_additivity)

/opt/conda/lib/python3.7/site-packages/shap/explainers/_deep/deep_tf.py in shap_values(self, X, ranked_outputs, output_rank_order, check_additivity)
    306                 # run attribution computation graph
    307                 feature_ind = model_output_ranks[j,i]
--> 308                 sample_phis = self.run(self.phi_symbolic(feature_ind), self.model_inputs, joint_input)
    309
    310                 # assign the attributions to the right part of the output arrays

/opt/conda/lib/python3.7/site-packages/shap/explainers/_deep/deep_tf.py in run(self, out, model_inputs, X)
    363
    364                 return final_out
--> 365             return self.execute_with_overridden_gradients(anon)
    366
    367     def custom_grad(self, op, *grads):

/opt/conda/lib/python3.7/site-packages/shap/explainers/_deep/deep_tf.py in execute_with_overridden_gradients(self, f)
    399         # define the computation graph for the attribution values using a custom gradient-like computation
    400         try:
--> 401             out = f()
    402         finally:
    403             # reinstate the backpropagatable check

/opt/conda/lib/python3.7/site-packages/shap/explainers/_deep/deep_tf.py in anon()
    359                     v = tf.constant(data, dtype=self.model_inputs[i].dtype)
    360                     inputs.append(v)
--> 361                 final_out = out(inputs)
    362                 tf_execute.record_gradient = tf_backprop._record_gradient
    363

/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in __call__(self, *args, **kwds)
    883
    884       with OptionalXlaContext(self._jit_compile):
--> 885         result = self._call(*args, **kwds)
    886
    887       new_tracing_count = self.experimental_get_tracing_count()

/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in _call(self, *args, **kwds)
    931       # This is the first call of __call__, so we have to initialize.
    932       initializers = []
--> 933       self._initialize(args, kwds, add_initializers_to=initializers)
    934     finally:
    935       # At this point we know that the initialization is complete (or less

/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in _initialize(self, args, kwds, add_initializers_to)
    758     self._concrete_stateful_fn = (
    759         self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
--> 760             *args, **kwds))
    761
    762     def invalid_creator_scope(*unused_args, **unused_kwds):

/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _get_concrete_function_internal_garbage_collected(self, *args, **kwargs)
   3064       args, kwargs = None, None
   3065     with self._lock:
-> 3066       graph_function, _ = self._maybe_define_function(args, kwargs)
   3067     return graph_function
   3068

/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _maybe_define_function(self, args, kwargs)
   3461
   3462           self._function_cache.missed.add(call_context_key)
-> 3463           graph_function = self._create_graph_function(args, kwargs)
   3464           self._function_cache.primary[cache_key] = graph_function
   3465

/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   3306             arg_names=arg_names,
   3307             override_flat_arg_shapes=override_flat_arg_shapes,
-> 3308             capture_by_value=self._capture_by_value),
   3309         self._function_attributes,
   3310         function_spec=self.function_spec,

/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes, acd_record_initial_resource_uses)
   1005         _, original_func = tf_decorator.unwrap(python_func)
   1006
-> 1007       func_outputs = python_func(*func_args, **func_kwargs)
   1008
   1009       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in wrapped_fn(*args, **kwds)
    666         # the function a weak reference to itself to avoid a reference cycle.
    667         with OptionalXlaContext(compile_with_xla):
--> 668           out = weak_wrapped_fn().__wrapped__(*args, **kwds)
    669         return out
    670

/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
    992           except Exception as e:  # pylint:disable=broad-except
    993             if hasattr(e, "ag_error_metadata"):
--> 994               raise e.ag_error_metadata.to_exception(e)
    995             else:
    996               raise

AttributeError: in user code:

    /opt/conda/lib/python3.7/site-packages/shap/explainers/_deep/deep_tf.py:243 grad_graph  *
        out = self.model(shap_rAnD)
    /opt/conda/lib/python3.7/site-packages/keras/engine/base_layer.py:1037 __call__  **
        outputs = call_fn(inputs, *args, **kwargs)
    /opt/conda/lib/python3.7/site-packages/keras/engine/functional.py:415 call
        inputs, training=training, mask=mask)
    /opt/conda/lib/python3.7/site-packages/keras/engine/functional.py:550 _run_internal_graph
        outputs = node.layer(*args, **kwargs)
    /opt/conda/lib/python3.7/site-packages/keras/engine/base_layer.py:1037 __call__
        outputs = call_fn(inputs, *args, **kwargs)
    /opt/conda/lib/python3.7/site-packages/keras/layers/convolutional.py:249 call
        outputs = self._convolution_op(inputs, self.kernel)
    /opt/conda/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:206 wrapper
        return target(*args, **kwargs)
    /opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/nn_ops.py:1138 convolution_v2
        name=name)
    /opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/nn_ops.py:1268 convolution_internal
        name=name)
    /opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/nn_ops.py:2722 _conv2d_expanded_batch
        name=name)
    /opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/gen_nn_ops.py:973 conv2d
        data_format=data_format, dilations=dilations, name=name)
    /opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:522 _apply_op_helper
        preferred_dtype=default_dtype)
    /opt/conda/lib/python3.7/site-packages/tensorflow/python/profiler/trace.py:163 wrapped
        return func(*args, **kwargs)
    /opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:1566 convert_to_tensor
        ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
    /opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:2023 _dense_var_to_tensor
        return var._dense_var_to_tensor(dtype=dtype, name=name, as_ref=as_ref)  # pylint: disable=protected-access
    /opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:1421 _dense_var_to_tensor
        return self.value()
    /opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:576 value
        return self._read_variable_op()
    /opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:683 _read_variable_op
        result = read_and_set_handle()
    /opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:674 read_and_set_handle
        self.handle, self._dtype)
    /opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/gen_resource_variable_ops.py:491 read_variable_op
        "ReadVariableOp", _inputs_flat, _attrs, _result)
    /opt/conda/lib/python3.7/site-packages/shap/explainers/_deep/deep_tf.py:26 custom_record_gradient
        out = tf_backprop._record_gradient("shap_"+op_name, inputs, attrs, results)

    AttributeError: module 'tensorflow.python.eager.backprop' has no attribute '_record_gradient'

if the latest version of TensorFlow version is installed please add the below lines -
import tensorflow as tf
tf.compat.v1.disable_v2_behavior()
This will solve the problem.

if the latest version of TensorFlow version is installed please add the below lines -
import tensorflow as tf tf.compat.v1.disable_v2_behavior()
This will solve the problem.

Agreed, this solved for me! Also, you can refer to the following SO answer: https://stackoverflow.com/a/66844039/2627859

@kaii55 Awwwwwwwwwesome!! You're the greatest engineer.
I have gotten some weird, because there are many fraud information.
But your suggestion works very well!!
Thank you.

Hi, since this has been fixed in release 0.41, shouldn't this issue be closed? @slundberg
Otherwise this might be misleading: the fact that the issue is still open made me believe it was not fixed. Hence, I did not even bothered looking for an updated version of the package whereas in reality an easy pip install --upgrade shap would have sufficed.

@synchro-- I have just updated to .41 (with tf 2.8) and I still need this fix
shap.explainers._deep.deep_tf.op_handlers["AddV2"] = shap.explainers._deep.deep_tf.passthrough
to run shap on my model. It's not inconvenient once you know about this solution, but maybe not yet fixed, right?


updated long descriptions for python modules

Coverage remained the same at 71.413% when pulling 0ce9f94f974e37121b8c7e2df93dcefb30078014 on akmorrow13:adam_0.25.0 into<PHONE_NUMBER>1d03fb18f9ba5c605a0079cad20f79 on bigdatagenomics:master.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/788/
Test FAILed.

Jenkins, retest this please.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/791/
Test FAILed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/792/
Test PASSed.


Added support to ArchetypePublishedContent for Umbraco extension methods

For the IPublishedContent implementation, added support for GetIndex(), so that extension methods, such as IsFirst(), IsLast(), etc will work as expected.
Unfortunately we wont be able to support the Siblings(), (or FollowingSibling, PrecedingSibling) extension methods, as these perform an internal query to the node's parent's children - which in the context of Archetype data is not possible access.
Added unit-tests to support this enhancement.
Resolves #329

:+1: I'll test this out before the next release!

:+1: Thanks!


Reorganize the config?

As mentioned in this comment, the config screen is starting to get a little busy and if we're not careful it could soon be overcrowded and confusing.
This inspired me to begin work on https://github.com/shedaniel/cloth-config/pull/195, but in the meantime it might be a good idea to consider how the current config could be improved.
Perhaps we could make use of tabbed category pages, collapsible sub-categoried (either expanded or collapsed by default) and other config entry types?
In the future we could also make use of dependencies to hide/disable some options when relevant.

I like this idea. However, I have been considering switching to YACL since shedaniel has stated that they are done working on Cloth Config and aren't planning on adding any new features. What do you think?

Done


Turn on RAW_IO on Windows

For queue transfers on IN endpoints

As I noted on your prior PR, I'm not sure if this is safe in edge cases.
While users shouldn't, and have no reason to, create multiple Queues for the same endpoint, or create a Queue while individual TransferFutures are pending, there's nothing stopping one from doing so with the current API, so it's possible for this to be called with other transfers pending on the endpoint.
@martinling In the libusb discussions around RAW_IO everyone seemed to think that WinUsb_SetPipePolicy was not allowed while there are pending transfers. As far as I've seen, the Microsoft documentation doesn't say one way or another. Do you know of any documentation on this, or have you encountered problems doing it? I am specifically looking for anything suggesting that it is forbidden as in Rust unsafe -- even returning an error or causing pending transfers to fail would be fine here. So it requires a much weaker guarantee than libusb was looking for when wanting to set it on a transfer-by-transfer basis.
In my limited testing of this branch on Windows 10, it seems to work fine:
    let bulk_in_pending = interface.bulk_in(0x81, RequestBuffer::new(16000));
    let mut queue = interface.bulk_in_queue(0x81);
    queue.submit(RequestBuffer::new(64));
    block_on(bulk_in_pending).into_result().unwrap();
    block_on(queue.next_complete()).into_result().unwrap();

[2024-06-01T20:27:12Z DEBUG nusb::platform::windows_winusb::transfer] Submit transfer 0x1ea3881f9e0 on endpoint 81 for 16001 bytes IN
[2024-06-01T20:27:12Z DEBUG nusb::platform::windows_winusb::device] WinUsb_SetPipePolicy succeeded to enable RAW_IO on endpoint 0x81
[2024-06-01T20:27:12Z DEBUG nusb::platform::windows_winusb::transfer] Submit transfer 0x1ea3881fd70 on endpoint 81 for 64 bytes IN
[2024-06-01T20:27:12Z DEBUG nusb::platform::windows_winusb::transfer] Handling completion for transfer 0x1ea3881fd70
[2024-06-01T20:27:12Z DEBUG nusb::platform::windows_winusb::transfer] Handling completion for transfer 0x1ea3881f9e0
[2024-06-01T20:27:12Z DEBUG nusb::platform::windows_winusb::transfer] Transfer 0x1ea3881f9e0 on endpoint 81 complete: 16001 bytes transferred
[2024-06-01T20:27:12Z DEBUG nusb::platform::windows_winusb::transfer] Transfer 0x1ea3881fd70 on endpoint 81 complete: 64 bytes transferred

I also considered just unconditionally enabling RAW_IO for all IN endpoints when opening the interface, before anyone could have submitted transfers. The documentation already notes "The requested length must be a multiple of the endpoint’s maximum packet size" with the intention of avoiding overflows, so this should be fine, however I should have enforced that in code. Probe-rs is submitting odd-sized reads via the TransferFuture API in at least one place.
Really, I should just go ahead with the breaking change and design the v0.2 transfer interface that guarantees exclusive access, but if this is reasonably safe in that edge case I don't want to block this indefinitely like libusb.

@kevinmehall

As far as I've seen, the Microsoft documentation doesn't say one way or another. Do you know of any documentation on this, or have you encountered problems doing it?

I don't have anything specific. It was more a case of thinking that behind the scenes, this clearly switches on or off some DMA stuff in the driver, and it would be very tricky for the OS to do that safely whilst existing transfers are in flight. What happens when you have an odd-size transfer in flight when the policy is changed? Maybe it just defers the change until the next transfer starts - but what if there is more than one odd-size transfer already queued up? So it'd need to do something like putting a special entry in the transfer queue saying "when you get to here, turn the DMA stuff on" and defer that work to that point.
And that all just seemed like a lot of work, that nobody would bother to do unless there was a very specific need for it, and surely if they had gone to all that trouble someone would at least note in the documentation that this was allowed and handled properly.
From experience, it seemed much more Microsoft-like to me that they would just implicitly expect programs to only call SetPipePolicy before making transfers, and not really document that assumption.
What also came up in the libusb discussions was that there is already this comment in the libusb backend code, noting that calling WinUsb_SetPipePolicy to change the SHORT_PACKET_TERMINATE option can cause problems with transfers in flight.
That comment was added by @hjelmn in this commit; perhaps he might be able to comment on his findings.
In general the feeling was that in the absence of information confirming it was safe to change policy with transfers in flight, it was better to just assume that it isn't. I certainly wouldn't want to guarantee it was safe in the Rust sense of the word, particularly for something that's messing with DMA.


With both machine-readable and verbose parameters step level reporting is not output

With both machine-readable and verbose parameters step level reporting is not output
As the IntelliJ plugin adds machine-readable to any spec, it is not possible to get step level output to the test runner in IntelliJ even if verbose is added as a parameter.   IntelliJ 2017.3.5 and IntelliJ-Gauge plugin v3.9
This was not an issue in Gauge 0.9.3, Gauge-Java 0.6.5, IntelliJ plugin 3.4 and IntelliJ 2017.1.5  last known working versions where step-level reporting was output with the verbose parameter in IntelliJ.   Example below takes IntelliJ out of the equation, though I cannot say for sure what the output was at a command line (non-IntelliJ) in Gauge 0.93 + Java 0.6.5
Expected behavior
When verbose is specified, step level reporting is expected in the console output.
In IntelliJ when a spec is run, this step level output would be human-friendly to read.
Actual behavior
When both verbose and machine-readable are specified, step level reporting is not included in the console output
Steps to reproduce

run command:   gauge run --machine-readable --verbose specs/example.spec
observe in output that individual steps are not output.

Sample output when both machine-readable and verbose are specified
C:\dev\GaugeJavaInit>gauge run --machine-readable --verbose specs/example.spec
{"type":"suiteStart"}
{"type":"specStart","id":"C:\\dev\\GaugeJavaInit\\specs\\example.spec","name":"Specification Heading","filename":"C:\\dev\\GaugeJavaInit\\specs\\example.spec","line":1}
{"type":"scenarioStart","id":"C:\\dev\\GaugeJavaInit\\specs\\example.spec:13","parentId":"C:\\dev\\GaugeJavaInit\\specs\\example.spec","name":"Vowel counts in single word","filename":"C:\\dev\\GaugeJavaInit\\specs\\example.spec","line":13,"result":{"time":0}}
{"type":"scenarioEnd","id":"C:\\dev\\GaugeJavaInit\\specs\\example.spec:13","parentId":"C:\\dev\\GaugeJavaInit\\specs\\example.spec","name":"Vowel counts in single word","filename":"C:\\dev\\GaugeJavaInit\\specs\\example.spec","line":13,"result":{"status":"pass","time":1}}
{"type":"scenarioStart","id":"C:\\dev\\GaugeJavaInit\\specs\\example.spec:21","parentId":"C:\\dev\\GaugeJavaInit\\specs\\example.spec","name":"Vowel counts in multiple word","filename":"C:\\dev\\GaugeJavaInit\\specs\\example.spec","line":21,"result":{"time":0}}
{"type":"scenarioEnd","id":"C:\\dev\\GaugeJavaInit\\specs\\example.spec:21","parentId":"C:\\dev\\GaugeJavaInit\\specs\\example.spec","name":"Vowel counts in multiple word","filename":"C:\\dev\\GaugeJavaInit\\specs\\example.spec","line":21,"result":{"status":"pass","time":0}}
{"type":"specEnd","id":"C:\\dev\\GaugeJavaInit\\specs\\example.spec","name":"Specification Heading","filename":"C:\\dev\\GaugeJavaInit\\specs\\example.spec","line":1,"result":{"status":"pass","time":0}}
{"type":"suiteEnd","result":{"status":"pass","time":0}}
Successfully generated html-report to => C:\dev\GaugeJavaInit\reports\html-report\index.html

Gauge version
Gauge version: 0.9.7
Commit Hash: bc9784c

Plugins
-------
csharp (0.10.1)
flash (0.0.1)
html-report (4.0.3)
java (0.6.6)
screenshot (0.0.1)
xml-report (0.2.0)

The problem were having is that after moving from Gauge 0.9.3 to 0.9.7, the --verbose tag does not work as expected in IntelliJ.   The IntelliJ plugin is adding --machine-readable and there is no way around.   When specifying --machine-readable and --verbose, I'd expect the step level reporting to still be output, but to be shown in JSON format like the other messages.  The IntelliJ Gauge plugin would be responsible for taking this and presenting it in the IntelliJ test runner in a human-friendly format.
Given our QA staff uses the step level reporting to track progress of longer running specs and debug failing specs, not being able to  get step level reporting in IntelliJ is definite problem.

This issue was moved to getgauge/Intellij-Plugin#315


Distributed builds

I have two different web apps and their dependencies in a yarn monorepo. Many of the dependencies are used by both apps so I wanted a way to reuse the build output of dependencies for app builds in different CI jobs.
I'm willing to work on it too if you think it's a good idea :)
Current setup

Job 1: run yarn workspace app-1 build, which builds deps first and then bundles app-1
Job 2: run yarn workspace app-2 build, which builds deps first and then bundles app-2

Ideal setup

Job 1: run yarn workspace dep-x build, which builds dep-x and its dependencies
Job 2: run yarn workspace dep-y build, which builds dep-y and its dependencies
Job 3 (after 1 & 2 finish): run yarn workspace app-1 build which would not have to build dep-x or dep-y.
Job 4 (after 1 & 2 finish): run yarn workspace app-2  build which would not have to build dep-x or dep-y either.

I could build all dependencies in one job and then run separate app build jobs in parallel if I pass build artifacts and yarn.build.json, but I have no way of merging the build cache automatically as in the explained ideal setup.
Easy solution
Splitting the build cache into separate per-package status files (instead of using just one) would allow me to pass success status to following pipeline steps.
Instead of having .yarn/yarn.build.json, we'd have something like .yarn/yarn.build/dep-x.json, dep-y.json, app-1.json, etc., describing the status of each package individually.
Alternate solution
Being able to configure custom checks for the plugin in .yarnrc.yml, so I could skip the build from subsequent jobs if output artifacts exist or any other condition I could use to detect if the build exists based on directory state.
This approach could also allow for "force rebuild" conditions as well (if we depend on something external).
Hacky solution
Write to .yarn/yarn.build.json directly on the pipeline jobs to update its state based on checking which files are there before running build. This isn't really a suggestion for yarn.build but a hacky workaround to get what I need working. I don't really think it's ok to manipulate your plugin's internal state from outside or if it's schema could change in future versions.

I think --skipIfOutputExists for build is a good addition to solve for this.
I am still a bit curious why you can't build everything once, and then split off and do other tasks with the results. Mainly because it breaks the soundness of the whole solution.
For example if you run yarn build in the top directory, it will determine the optimal plan to build everything, and parallelise it as much as possible.

I'm going to close this issue, but feel free to re-open or comment if you are still having trouble or have more information.


Copying to clipboard doesn't work on Linux

I'm trying to extract the AndroidManifest.xml. With the latest v8.0 release, since exporting doesn't work, I tried selecting text and copying it to the clipboard. On Goobuntu (Ubuntu 14.04 LTS), neither Ctrl+C nor Ctrl+Insert place the selected text in the clipboard.
openjdk version "1.8.0_112-google-v7"
OpenJDK Runtime Environment (build 1.8.0_112-google-v7-152501353-152497516)
OpenJDK 64-Bit Server VM (build 25.112-b01, mixed mode)

Will investigate, on Mac copy/paste work

Does not work on Windows (10) either with the latest Java 8.

Can also confirm does not work on Windows 10.


Add AuthorizationMiddleware and update sandbox

https://github.com/aspnet/Routing/issues/537
PR adds first cut of AuthorizationMiddleware. Largely a copy from Ryan's original implementation, which is an adaption of AuthorizationFilter. Highlevel changes:

I added a check for IAllowAnonymous metadata so it is more inline with MVC's AuthorizationFilter
Convention builder for adding IAuthorizeData as metadata to endpoints. Example: https://github.com/aspnet/Routing/compare/master...jamesnk/authz-middleware?expand=1#diff-c65f61210298d7cca90708f6e5f60b46R55

Run the routing sandbox sample to try it out.
@rynowak where is the best long term home for the middleware? BasicMiddleware? An existing authorization package? New authorization package?
@blowdart It is very similar to MVC's AuthorizationFilter. Is this a good template for doing authz in middleware?
Will add unit tests once we're happy with the overall approach and it is moved to its final home.

I don't get what you're trying to do here, but generally everything would go though the IAuthoizationService rather than calling evaluate on the policy directory, as that way people can swap the entire thing out for something like PolicyServer

@rynowak where is the best long term home for the middleware? BasicMiddleware? An existing authorization package? New authorization package? @blowdart It is very similar to MVC's AuthorizationFilter. Is this a good template for doing authz in middleware?

I think it's probably worth a short meeting to discuss these kinds of details.

I don't get what you're trying to do here, but generally everything would go though the IAuthoizationService rather than calling evaluate on the policy directory, as that way people can swap the entire thing out for something like PolicyServer

@blowdart do you have a sample of this somewhere? This code was lifted pretty much verbatim from MVC so if it's wrong here then it's wrong there.

Replaced with https://github.com/aspnet/Security/pull/1894


Doesn't update hosts file

Hi,
Nice tool but unfortunately it doesn't works for me.
I see that the program is triggered as a new line apparead in the file the first time i started an hyperv vm, but this is only a comment, and after other trials (shutdown and restart vm) nothing happenend.
the line who appeared is below :
[HYPERV2HOSTS] Updated at Tue Jan 31 18:48:44 2023
Windows 10/64
br

It's indeed not working all the time.  Two things to check:

You need to run the tool as administrator.

It finds the VM guest IP address by the following powershell command:

Get-VMNetworkAdapter -VMName linux | Select VMName, IPAddresses

If you run the command in powershell as administrator and it's not there, then something else is wrong.
PS, I just changed the hosts file format.  The original order in line 99 was wrong.

Hi,
The user which run the task is member of group admin
The powershell command returns nothing but this is normal as you pass an argument "-VMName linux" and my VM is not named "linux". if i change the argument with my vm name, the command is ok and returns VM's IP Address.
According to readme, i understand that this program should list all vms, so the name shouldn't be fixed.
Do you mean that it is only functional for one unique vm whose name is linux ? It seems that's true since i can see this powershell command in source code (i can't compile ccp source code, no tools, and last time i did c++ was almost 20 years ago).
Perhaps, coud you run "Get-VM | Get-VMNetworkAdapter | Select VMName, IPAddresses" and iterate on the results ?
br

I've added a new release here: https://github.com/aaalgo/hyperv2hosts/releases/tag/1.0.2
You need to supply the guest name with the command line argument "-n vmname" or "--name vmname".  If you want to change the hostname, you can use "--host-fmt vm_{}", where {} will be replaced with guest name.
I'm not sure if this will work.  I chose C++ so I can build something standalone, but maybe a python script will be better for the purpose.


Expose codec flags

I needed to set AV_CODEC_FLAG_GLOBAL_HEADER to get the correct encoder extradata for an RTMP application, here are the changes I made to achieve that.
A few notes:

The set of flags defined in avcodec.h differs between ffmpeg 4.x and 5.x, here I have included flags from both versions
The flags are split into two sets, same as in libavcodec, but they could also be merged into one set in the Rust interface
The names of the flags are not very descriptive, and their function could change in future libavcodec versions. It might make sense to include a way to pass in an arbitrary flag value in addition to the predefined ones, just to cover all use cases.

Hi @JaceSn and thank you for your pull request. However, I belive the added functionality isn't needed. You can do the same thing using the VideoEncoderBuilder::set_option() method (or AudioEncoderBuilder::set_option() for audio encoders).
For example:
AudioEncoder::builder("aac")?
    .set_option("flags", "+global_header")
    ...

As you mentioned, the flags themselves are quite unstable and they tend to change between FFmpeg versions.  The options API on the other hand is a bit more stable and you can also use it to set codec-specific options that are not available within AVCodecContext. So unless there is a good reason to expose a certain AVCodecContext field, the options API should be preferred.
If you're wondering how to list available options, you can find them either directly in the FFmpeg code (for example http://ffmpeg.org/doxygen/5.1/libavcodec_2options__table_8h_source.html#l00047) or you can simply list them using the ffmpeg command:
ffmpeg -h full

@operutka thank you for taking the time to look over the issue and share your expertise. I can confirm that the suggested method works exactly as advertised and is a much cleaner and more versatile solution in general.
Feel free to close the PR, though perhaps the title could also be edited to be a bit more descriptive for the benefit of the next person to come along looking for the same functionality.

Ok, I'm closing the PR then. Feel free to edit the title, if you want.


inference wav worse than demo wav

Thank you for release.
After run "python inference.py --checkpoint_path checkpoints/default/model.pt", it generate wav, it worse than demo
little_star_generated_e2e.zip
.

Hello @zhangsanfeng86, thanks for opening this issue.
As noted in the original paper, we used an in-house variant of HiFi-GAN, which was additionally fine-tuned on the Children's Song dataset. The code base, as provided in this repository, uses the universal HiFi-GAN checkpoint, so the quality of synthesized results is likely going to be perceptibly worse than those on the demo webpage. For better results, I would recommend that you first try fine-tuning the HiFi-GAN vocoder on CSD.
Please let me know if you have any more questions or concerns. Thanks!

OK， thank you for quick and helpful reply !


Fixed #27129 New functional to django/template/defaultfilters.py was …

…added
Customisation classes of ul and li elements in method unordered_list
was implemented in this commit.

As I said on the ticket, I don't think it's possible to do a simple, generic implementation that will make everyone happy. Better to leave this to a custom template tag, I think.


Feature Request!

Hey there! I read this extensive reddit post that you commented on, about how Vein Shrouds and Kidnapper Foxes worked, and I gotta say it was super enlightening!
Having read, and hopefully understanding that, I think this mod would be absolutely perfect if we could in some way cleanse a planet by killing the origin Vein Shroud. I'd say you could set a config on whether cleansing a planet of its Vein Shrouds should be permanent, as in, once a planet becomes infected, and you kill the origin shroud, that planet will never become infected in that run again, or temporary, where it can become infected again, but I think this, especially paired well with other mods like Late Game Upgrades, that significantly make the Weed Killer more useful overall, would make the preventative counter play for Kidnapper Foxes really fun!

Currently I'm waiting to see what happens with the Kidnapper Fox in later updates before I continue working on this mod.
v60 looks to be releasing very soon and apparenly the fox will be disabled in that update.
I'll definitely consider this idea though! (Assuming the mechanic doesn't get completely overhauled.)

Yeah I made this request before finding that out myself haha,


Add platform implementation for Advertiser

Problem
The DiscoveryManager is using a different interface for publishing services.
Summary of Changes
Make discovery manager use the same interface.

@hawk248 @jelderton @saurabhst @BroderickCarlin ?

@hawk248 @jelderton @saurabhst @BroderickCarlin ?


Benchmarks to run on comits of a Pull Request

Description
Added a new file pr_benchmarks.yml to run benchmarks on pull request commits and then comment on the pull request with the results.
Type of change

[ ] New feature (backwards compatible)
[ ] New feature (breaking API changes)
[ ] Bug fix
[x] Other (Documentation, CI, ...)

Contribution checklist

[ ] Add our license to new files.
[ ] Check that your code adheres to our coding style.
[ ] Write unittests.
[ ] Run the unittests and check that they’re passing.
[x] Run the benchmarks to make sure your change doesn’t introduce a regression.

Since the comment action cannot be triggered from a forked repository, you can checkout the comment on benchmarks from the PR on my forked repository: Comment Link

@phschiele on: [ pull_request_target ] does not seem to trigger the pr_benchmarks workflow.


fix health insights LRO poller

In the case of the HealthInsights LRO, we were getting LROPoller[None] instead of LROPoller[OnoPhenotypeResult]. This was because for this operation's response, there was no inner response with a body, we just had to get the information from getLroMetadata. I don't think this is a tsp issue, basically just made the code where if it's an LRO, we allow for the case where there might be no body from inner response. When I wrote the code, I made it similar to the code in typespec-autorest, and that's likely what we'll have in tcgc
discussion:
https://teams.microsoft.com/l/message/19:meeting_MWYwYzEyMTEtYmQwZS00MTM5LWFhYjItMDU2N2M3ZDhmY2Vi@thread.v2/1688165451777?context={"contextType"%3A"chat"}

@catalinaperalta could you tell some context about removing 200 response for the lro template of healthinsights: https://github.com/Azure/azure-rest-api-specs/commit/1aee3a868488b4ccdfd0050ed756c583af357450?

@catalinaperalta could you tell some context about removing 200 response for the lro template of healthinsights: Azure/azure-rest-api-specs@1aee3a8?
cc: @MaryGao

Having the | TResponse was a temporary workaround to unblock the service for their first preview release. That was the only way we could get the pollers to emit the correct poller objects. However, the service does not return a 200 and the service needs their TypeSpec to be correct so we have worked with the TypeSpec team to fix the template to accurately represent the LRO and now we needed the fix in the SDK emitter.

@catalinaperalta could you tell some context about removing 200 response for the lro template of healthinsights: Azure/azure-rest-api-specs@1aee3a8?
cc: @MaryGao

Having the | TResponse was a temporary workaround to unblock the service for their first preview release. That was the only way we could get the pollers to emit the correct poller objects. However, the service does not return a 200 and the service needs their TypeSpec to be correct so we have worked with the TypeSpec team to fix the template to accurately represent the LRO and now we needed the fix in the SDK emitter.

@catalinaperalta But current template definition does not have a body definition. Could you provide the real payload of this operation?

@catalinaperalta could you tell some context about removing 200 response for the lro template of healthinsights: Azure/azure-rest-api-specs@1aee3a8?
cc: @MaryGao

Having the | TResponse was a temporary workaround to unblock the service for their first preview release. That was the only way we could get the pollers to emit the correct poller objects. However, the service does not return a 200 and the service needs their TypeSpec to be correct so we have worked with the TypeSpec team to fix the template to accurately represent the LRO and now we needed the fix in the SDK emitter.

@catalinaperalta But current template definition does not have a body definition. Could you provide the real payload of this operation?

@tadelesh they dont have a body in their initial 202. Here is an example the service team added: https://github.com/Azure/azure-rest-api-specs/blob/3eac7ab53bcbcba33b70e8cf9c053f3de6dc6ec1/specification/cognitiveservices/HealthInsights/healthinsights.openapi/examples/2023-03-01-preview/SuccessfulOncoPhenotypeRequest.json#L53

@catalinaperalta is their following polling response and final response also 202? From the example, it seems 200: https://github.com/Azure/azure-rest-api-specs/blob/feature/health-lro/specification/cognitiveservices/HealthInsights/healthinsights.openapi/examples/2023-03-01-preview/SuccessfulOncoPhenotypeResponse.json.
@iscai-msft I've changed init operation response to JSON type.
@catalinaperalta @iscai-msft According to current template define, the logical result we get from getLroMetaData is the wrapped definition: https://github.com/Azure/azure-rest-api-specs/blob/3d741204b727f52998941324d713374f843b11aa/specification/cognitiveservices/HealthInsights/healthinsights.oncophenotype/model.oncophenotype.tsp#L20C7-L20C27. And the final step response should be wrapped with OperationStatus: https://github.com/Azure/typespec-azure/blob/5dd7d00e6cab257e225493c9ab235ead227e4eb0/packages/typespec-azure-core/lib/foundations.tsp#L106C39-L106C69.

@iscai-msft I've changed init operation response to JSON type.
@catalinaperalta is their following polling response and final response also 202? From the example, it seems 200: https://github.com/Azure/azure-rest-api-specs/blob/feature/health-lro/specification/cognitiveservices/HealthInsights/healthinsights.openapi/examples/2023-03-01-preview/SuccessfulOncoPhenotypeResponse.json.
@catalinaperalta @iscai-msft According to current template define, the final response should be wrapped with OperationStatus, and the logical result we get from getLroMetaData is OncoPhenotypeResult. These two are all wrapped object with job id and status info. I don't think it is a right definition. I think service behavior should be using OncoPhenotypeResult as response and using OncoPhenotypeResults as final results. Since OncoPhenotypeResult is different from azure core OperationStatus, service template should define customized OperationStatus.

The final response is a 200 with the final payload. Initial response is a 202 with no body.
About the second comment, the service would like to return OncoPhenotypeResult from the poller object. Or is this a separate issue?

@catalinaperalta So, I think there are two separated problem here:

According to current template as follows, the response only has a 202 definition, though TResponse will be treated as the final logic result, I doubt no 200 defined is a right way, is it a workaround for now?

@post
op TLongRunningRpcOperation<
  TParams,
  TResponse
> is Azure.Core.Foundations.Operation<
  TParams & RepeatabilityRequestHeaders,
  (Foundations.AcceptedResponse<Foundations.LongRunningStatusLocation<TResponse> &
    Foundations.RetryAfterHeader> &
    RepeatabilityResponseHeaders),
  RepeatabilityRequestHeaders & RepeatabilityResponseHeaders,
  Azure.Core.Foundations.ErrorResponse
>;

Also from the template, you could see Foundations.LongRunningStatusLocation<TResponse> is defined as:

/**
 * Metadata for long running operation status monitor locations.
 * @template TStatusResult The type of the operation status result.
 */
@doc("Metadata for long running operation status monitor locations")
model LongRunningStatusLocation<TStatusResult = never> {
  @pollingLocation
  @doc("The location for monitoring the operation state.")
  @TypeSpec.Http.header("Operation-Location")
  operationLocation: ResourceLocation<OperationStatus<TStatusResult>>;
}

The TstatusResult here should be operation status result, not including operation status info, the operation status info has already defined in the OperationStatus which is:
/**
 * Provides status details for long running operations.
 * @template TStatusResult The type of the operation status result.
 * @template TStatusError The type of the operation status error. If not provided, the default error is used.
 */
@doc("Provides status details for long running operations.")
model OperationStatus<TStatusResult = never, TStatusError = Foundations.Error> {
  @key("operationId")
  @doc("The unique ID of the operation.")
  id: string;

  @doc("The status of the operation")
  status: OperationState;

  @doc("Error object that describes the error when status is \"Failed\".")
  error?: TStatusError;

  @doc("The result of the operation.")
  result?: TStatusResult;
}

Current logic should be right for a correct tsp, other things I think is depending on tsp right definition for lro and getLroMetaData right logic.

@tadelesh I worked on this template with @markcowl and @johanste to define the Health Insights operation behavior. The reason we're passing OncoPhenotypeResult to LongRunningStatusLocation is because we are defining Operation-Location based polling for the service. They have defined their response as including the status monitor properties and return those upon a successful response. Their original swagger that was not tsp based also defined it this way. Mark also helped me check that you're getting the right information from getLroMetadata. I dont think there's anything else missing here to support this scenario.

@catalinaperalta As I described here, our emitter must know the final logical result is the body directly or wrapped by OperationStatus. With current health insights definition, Python emitter will treat the lro as the later one, but I don't think the real payload has the wrapper.


Reduced website size by 75%

by removing big useless params from truffle artifacts

Do not merge until ethereum-abi missing module error from travis build is resolved

Travis automatic deployment:
https://pr469--dxreact.review.gnosisdev.com

Travis automatic deployment:
https://pr469--dxreact.review.gnosisdev.com

Travis automatic deployment:
https://pr469--dxreact.review.gnosisdev.com

Travis automatic deployment:
https://pr469--dxreact.review.gnosisdev.com

closing in favour of #497


Failed Graphs On New Device

I've set up the receiver on a new device - a CHIP $9 computer board.  It's all working fine save three of the charts not functioning - core temperature, and the two disk charts.  I suspect this is due to the microprocessor not supporting a core temperature (though another chip on the board does) and it having inbuilt flash, rather than microSD or HDD.  df reports it as 'ubi0:rootfs' of a 'ubifs' type.

Hi, thanks for the report and while I'm not aware of anyone having tried the adsb-receiver scripts on a CHIP it's great to know that it mostly works :)
From some quick research it should be fairly eàsy to get equivalent temp and file system graphs from tgis platform, and perhaps add a framework to help detect other non-Pi platforms!
If memory serves me correctly I have a CHIP lying around somewhere that I hadnt found a use for so will see if I can track it down to help testing...
In the mean time are you able to share the steps you took to install adsb-receiver on the CHIP so we can add any relevant steps to the documentation and/or install scripts?

(links from my initial research)
This should get the CPU temp, or somewhere close:
https://bbs.nextthing.co/t/how-to-check-the-temperature-of-chip/1738/41
Looks like there may be a setting required to disable some kind of overload protection:
https://bbs.nextthing.co/t/c-h-i-p-rtl-sdr-webserver-a-no-go/13466
And to set a static Mac address to ensure starts reported to feeder sites is attributed correctly:
https://bbs.nextthing.co/t/solved-can-i-disable-on-board-ethernet-gadget-adapers-or-assign-a-fixed-mac/5427

That first link I already had loaded, actually, haha, to see what was available myself.  It's the AX209 power controller, not the micro, but it's what we got.
For this setup, I've actually got a powered USB hub to a ProStick - doing it straight resulted in the board shutting down.  Had to use a 2.0 hub as well, a 3.0 hub was also drawing too much juice.  Hadn't tried the overload protection disable, but I've got 3 more and was intending to with another unit eventually.  Haven't had any mac address problems with them shifting on me yet though.
As for the setup, I installed the main headless image and set up wireless, at which point I installed:

dump1090-mutability
flightware/piaware tracker
planefinder tracker
Advanced setup with remote MySQL database

I also manually specified my gain in the dump1090-mutability file, but I doubt that'll have had any impact here.

So it looks like support for "/sys/class/thermal/thermal_zone0/temp" disappeared at some point and the nearest approximation is "/sys/class/hwmon/hwmon0/temp1_input"...
Would you mind sharing the output of 'cat "/sys/class/hwmon/hwmon0/temp1_input' to confirm this is a 5 digit number that looks like 1/100's of a degree C?
I am updating the graph generation with basic hardware detection and the assumption the above value is usable on all devices which list "Hardware : Allwinner sun4i/sun5i Families" in /proc/cpuinfo...  but if not we may need to look into using  'axp209 --temperature' ;)

I have pushed what I hope should be a fix into my 2.5.1 branch, would you be able to test cloning my fork with:
git clone -b 2.6.0 https://github.com/Romeo-Golf/adsb-receiver.git
And then updating by running the install script with:
cd adsb-receiver && ./install -b 2.6.0

Attempted this, but no dice - system still showing 2.5.0 as version.  Could be my attempt, though.

Hi,
Unfortunately the portal updates for 2.6.0 are still pending; however the graph creation script in my 2.6.0 branch should detect the CHIP platform and generate the temperature/disk graphs using the same file names as they would be on Raspberry Pi... is anything at all displayed for these graphs in the portal performance graphs tab?
Thanks.

Ah, found the problem on this - it flips itself back to the 'master' branch.  The install flag doesn't seem to do anything.  Even manually switching to 2.6.0, the install script runs it back to 'master'.

Using the 2.6.0 branch code the install script accepts the -b flag to override the hard coded reference to the master branch...
It could be there are remnants of the old code so if your happy to try a full reinstall then please try the following:
rm -rf ~/adsb-receiver
git clone -b 2.6.0 https://github.com/Romeo-Golf/adsb-receiver.git
cd ~/adsb-receiver
git checkout origin/2.6.0
./install.sh -b 2.6.0
Thanks.

Executed.  Doesn't seem to have changed anything, however, the graphs aren't being generated.
Let me know what you'd like me to try - I'll keep at it.
As a note too, for these devices, we may want to examine file usage.  After it had run without restart for about 10 days the whole thing locked up on writing the adsb json with "too many files".  A restart fixed it, but it should be checked.

Thanks for the update, would you mind sharing the output of the following:
sudo bash -x /home/pi/adsb-receiver/build/portal/graphs/make-collectd-graphs.sh 24h
Not sure about the 'too many files' issues, if it happens again would you mind trying to capture the exact error text?
Thanks.

    Nothing is under hwmon folder on the CHIP.

Another note on this - we may not be able to get the IOPS stats at all.  Apparently, collectd relies on /proc/diskstats to be populated, but the file is empty.  This is something that there's an issue on the CHIP forums about, but no response or any other information.

hmm, how about /sys/class/thermal/ ?
Any chance you could post on their forum to see if we can get a defiantive answer?  :)
https://bbs.nextthing.co/t/how-to-check-the-temperature-of-chip/1738/45

Okay, color me annoyed, and possibly having to reinstall the whole thing.
I went to another one of my CHIPs that is newer (the one I'm on is an Alpha Chip, the early ones from Kickstarter).  The newer models do in fact have hwmon on them, and the requisite files.  /proc/diskstats is also populated, but whether it's the ubi filesystem or something else, all entries are 0.
Getting those charts to work on a CHIP may be a bit harder than we originally thought, at least IOPS.
I'll do some reading and see if I can't find out if it's a side effect of the older hardware or what.

Interesting.. I'll try to dig mine out but it was also from KS so may also be an early hardware revision :(

So... reading further,  I've confirmed that as of right now collectd does not support ubi-type filesystems.  Which are the ones made for embedded NAND flash style memory.  They just don't report the same kind of stats apparently, yet, to the kernel, so we won't get that.
Still hunting on other info for the CHIP thermal stuff, but the one comment I saw about it had zero responses.  Everyone seems to rely on the ax209 for thermal.


Fix Bug: Fix the calculation of frame duration

Hey,
Today the frame's duration is based on the difference between the current frame timestamp and the previous frame timestamp.
The frame's duration should be the difference between the next frame timestamp and the current frame timestamp because the new frame replaces the current frame

Thanks for raising the pull-request. Looks good to me.  Apologise for the delay, as I'm bit busy with some work!!


refactor: generic extension

Refactors the chain extension to maximise flexibility, ensuring that it is highly configurable and so that anything which is runtime specific is defined there - e.g. versioning.
A high level overview of the components:
[contract environment: contract ⇄ pop-api] ⇄ [runtime: pallet-contracts ⇄ pop-chain-extension ⇄ pallet-api ⇄ fungibles]

Codecov Report
Attention: Patch coverage is 29.96109% with 180 lines in your changes missing coverage. Please review.

Project coverage is 30.15%. Comparing base (6482a43) to head (e70b83d).

Files
Patch %
Lines

extension/src/functions.rs
0.00%
54 Missing :warning:

runtime/devnet/src/config/api/versioning.rs
66.37%
39 Missing :warning:

pallets/api/src/extension.rs
0.00%
23 Missing :warning:

runtime/devnet/src/config/api/mod.rs
0.00%
19 Missing :warning:

extension/src/decoding.rs
0.00%
17 Missing :warning:

extension/src/lib.rs
0.00%
11 Missing :warning:

primitives/src/lib.rs
0.00%
11 Missing :warning:

extension/src/matching.rs
0.00%
6 Missing :warning:

@@             Coverage Diff              @@
##           daan/api     #218      +/-   ##
============================================
- Coverage     34.08%   30.15%   -3.93%
============================================
  Files            34       38       +4
  Lines          3013     2928      -85
  Branches       3013     2928      -85
============================================
- Hits           1027      883     -144
- Misses         1954     2013      +59
  Partials         32       32

Files
Coverage Δ

runtime/devnet/src/config/contracts.rs
60.00% <ø> (ø)

extension/src/matching.rs
0.00% <0.00%> (ø)

extension/src/lib.rs
0.00% <0.00%> (-8.11%)
:arrow_down:

primitives/src/lib.rs
0.00% <0.00%> (-62.80%)
:arrow_down:

extension/src/decoding.rs
0.00% <0.00%> (ø)

runtime/devnet/src/config/api/mod.rs
0.00% <0.00%> (ø)

pallets/api/src/extension.rs
0.00% <0.00%> (ø)

runtime/devnet/src/config/api/versioning.rs
66.37% <66.37%> (ø)

extension/src/functions.rs
0.00% <0.00%> (ø)

@peterwht @chungquantin as per your comments above, I addressed them in https://github.com/r0gue-io/pop-node/pull/218/commits/2152867b8894e36a17b6773e1bc3ada1f7966437 and resolved the conversation. Please could you kindly take a look at the commit and let me know if any concerns?

https://github.com/r0gue-io/pop-node/pull/236
https://github.com/r0gue-io/pop-node/pull/237

As for the open questions regarding whether to convert the DispatchError in e.g. the ReadState function. My view is that in the end it is also a pallet with an index. We do account for changes in a pallet index, so why not include those as well? I think treating all pallets the same is the easiest and safest.


Add screenshots to documentation

To get a quick visual indication on how the UI might look for the user, we should add some screenshots.

Fixed.


[KBM] Ignore IME keys while checking keyboard state for Shortcut Remaps

Summary of the Pull Request
What is this about?
As mentioned in the comment at https://github.com/microsoft/PowerToys/issues/6951#issuecomment-706626338, it was found that VK_KANJI sometimes gets set and doesn't get released while pressing Alt+` on Japanese IME. To prevent this from blocking shortcut remaps, this PR adds all the IME related keys as per the Virtual Key Codes doc to the ignore list.
PR Checklist

[X] Applies to #6951 https://github.com/microsoft/PowerToys/issues/6951#issuecomment-706626338
[X] CLA signed. If not, go over here and sign the CLA
[X] Tests passed

Validation Steps Performed
How does someone test & validate?

Add some shortcut to shortcut remapping
Switch to Japanese IME keyboard layout
Open a text editor, and do the following steps:

Press Alt
Press ` (At this point the Japanese IME should get changed)
Release Alt
Release `
Press the shortcut you remapped (earlier this wouldn't work until you hit and release Alt+` normally again)

/azp run


[nri-statsd] Parameterise replicas to set it dynamically

Is this a new chart
No
What this PR does / why we need it:
With this we can dynamically set replicas count in the deployment.
Which issue this PR fixes

Improvement: Add replicaCount parameter.

Special notes for your reviewer:
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[X] Chart Version bumped
[X] Variables are documented in the README.md
[X] Title of the PR starts with chart name (e.g. [mychartname])

cc: @bambash

Hi @thilinapiy this has been released at https://github.com/newrelic/helm-charts/releases/tag/nri-statsd-1.0.1


Prometheus exporter fix for v2.x

Prometheus exporter fix for v2.x
This PR brings fixes for several issues in existing Prometheus exporter plugin implementation. It brings fixes into 2.x branch and all the following text is relevant mostly to this branch although I think similar series of fixes is needed for master branch as well (this can be addressed by followup PR depending on feedback from plugin maintainers).
Why I did this?
My main goal was to make sure Prometheus exporter plugin can work well with Elasticsearch cluster that is using SearchGuard plugin for security. As of writing I was focusing on Elasticsearch 2.x but my interest going forward is to make sure Prometheus exporter plugin will work well with later ES versions (5.x, ...) too.
While I was doing my evaluation with Elasticsearch 2.x and SearchGuard I hit couple of issues (see below for details). I realized that fixing all those issues is easier with one monolithic PR rather than several independent PRs. Part of the story was that changes required significant architecture reimplementation.
Issues fixed in this PR
[#57] thread context handling is buggy
This issue is caused by the fact that Client instance is shared and reused for other REST calls. In connection with SearchGuard plugin this represents major security issue (and I think the same can apply to other security solutions, this may not be SG exclusive). Please follow up the #57 for further details.
The solution is to not keep and reuse client instance.
[#58] Do not execute a blocking operation on a networking thread
This issue actually stems from the fact that plugin is using blocking client calls in the REST layer. In normal situation users are not hit by this issue because Asserts are not enabled. However, when running ES with Assets enabled (like when implementing integration tests using Elasticsarch test framewrok) the execution leads to exception.
The correct solution is to move the plugin logic to lower layers in Elasticsearch. From REST layer to Transport layer and use strictly async calls. At the same time it is very important to make sure that specific actions are executed locally on the node that receives the initial REST call. This PR brings all of this.
One of the tradeoffs here is that for each REST call a new instance of PrometheusMetricsCollector is created. But I do not think this is really expensive especially given Prometheus server is typically setup to scrape metrics once per 10 seconds or so. May be even there is a way how we can save some cycles and refactor & reuse objects that handle individual metrics registeration.
[new feature] Possibility to control access to node REST API with SearchGuard
One of the benefits of moving the logic from REST to Transport layer is the possibility to implement Elasticsearch Action which enables much better access control to plugin REST API (/_prometheus/metrics) when using SearchGuard.
Right now users accessing REST API have to have permissions for all behind-the-scene calls. Specifically, they have to be granted permissions for cluster health and node stats (and possibly others for 5.x branch). In SearchGuard config it looks like this:
sg_role_both_perms:
  cluster:
    - cluster:monitor/health
    - cluster:monitor/nodes/stats

If user is missing any of these then the REST call fails. On the other hand, any user that has these perms is allowed.
This PR intorduces new permission for prometheus metrics REST API only. In SearchGuard it is now possible to do this:
sg_role_prometheus_perm:
  cluster:
    - cluster:monitor/prometheus/metrics

All users having this single permission are allowed in and no other permission is required. This also means that if in the future the prometheus plugin will start accessing data from other resources in Elasticsearch the configuration for SearchGuard does not need to change.
I did test this feature using the same code that I used to demonstrate [#57]. That repo now contains two new branches case_1_fixed and case_2_fixed and both are using custom build of fixed Prometheus exporter plugin. It can bee seen in associated Travis logs (here for case_1_fixed and here for case_2_fixed) that both cases now work correctly.
[#63] Why using node name instead of node ID for label?
Strictly speaking, this is a new feature rather than an issue. See #63 for details.
[new feature] Introduction of integration tests
I added integration tests. I think they are self explanatory. One of the key aspectes they test is the fact that Prometheus exporter returns data only for the node that received the HTTP call (the nodes stats).
Use mvn clean verify to run both unit tests and integration tests.
Anything else...?
While I was on it I did some other minor updates here and there. For example I updated the java_client version and did some improvements in logging.
Where to go from here?

I would love to hear back
I will be happy to answer any questions
I want to help get the same fixes to other branches
I would like to suggest adding integration with Travis (now that integration tests are in place it makes a lot of sense)

@vvanholl @davidkarlsen any comments or concerns on this please?

Hi @lukas-vlcek, I just want to notify you I am currently reading your PR

For your information the master branch now use Gradle and I have my own Jenkins server which make the builds and the artifacts.
For the moment it is a private part but I want to make it public to allow testing on external PRs.

Do you plan porting changes to master 5.x branch ?
... the master branch now use Gradle

Yes I do. And I noticed the Gradle and I haven't been using it much in the past but that is life :-) I will try to find a way how to port those pieces of maven magic around integration tests to Gradle.

I have my own Jenkins server which make the builds and the artifacts. For the moment it is a private part but I want to make it public to allow testing on external PRs.

Well, I guess/hope it should be possible just add another CI service. Or does GitHub support only a single CI service integration? In other words, if GitHub supports more CI services then I do not see any issues if you keep your Jenkins and just add another one for unit/integration tests only.
I also noticed you are using something for reviews (you do not like GitHub reviews?).

How and when the async part is triggered ?

Check the TransportNodePrometheusMetricsAction class. There are two nested async actions (kind of callback interfaces). And then you can see that TransportNodePrometheusMetricsAction is registered to REST action in PrometheusExporterPlugin like this:
actionModule.registerAction(NodePrometheusMetricsAction.INSTANCE, TransportNodePrometheusMetricsAction.class);

And a few supporting classes for serialization of requests and responses are needed.

"I have my own Jenkins server which make the builds and the artifacts.
For the moment it is a private part but I want to make it public to allow testing on external PRs." - how about using travis: https://travis-ci.org/ - it's quite versatile and easy, free for opensource, and integrates well with github.

You can use both Jenkins and Travis or even Circle CI at the same time. In fact GitHub can handle multiple hooks.

For reviews I personally rely only on Github, but I noticed some externals PRs are linked to other reviewing systems.

Thanks @vvanholl

@lukas-vlcek Does this change exist for 5.5.2.x?  What is required to make it available if it does not?

It does not. One of the most critical issues have been fixed in this plugin itself and that is this plugin no longer holds and reuses a single ES client for all the requests. This was causing issues with SearchGuard because the ES client is connected with initial request context (meaning it could have been using wrong user for later requests).
But there were other features that this fix implemented and IMO would be very useful to bring them into ES 5.x version. The most useful one is the ability to better control access to the REST API in SearchGuard (ie cluster:monitor/prometheus/metrics). But this requires some non-trivial modifications and it is not possible to simply cherry-pick it from 2.x branch for the following reasons:

ES 5.x is likely to introduce API breaking changes. This shouldn't be hard to figure out. However, to make sure that it works correctly it is critical to have valid unit tests, see below...
I implemented integration tests using ESIntegTestCase, but it is quite unclear if that would be good approach for ES 5.x and forward, see https://github.com/vvanholl/elasticsearch-prometheus-exporter/issues/66
ES 5.x switched from Maven to Groovy which I think will have impact on how integration tests needs are executed. And I did not have a time to investigate this in detail.

The other problem I see here is that bringing all this into the plugin will change the code from relatively high-level ES API use to much lower level API (making it more sensitive to internal API changes in ES). And I am not sure we can promise the plugin developers our quick help when they will want release plugin from the most recent ES version while are behind. May be they will not need our help... I do not know, but you know how tricky it can be to implement unit test for ES, right? (Notice, for this plugin it is critical to be able to start several ES nodes in unit tests)
Is there any particular feature you are looking for Jeff for the ES 5.x?


LoRa App Server: unknown method MigrateChannelConfigurationToGatewayProfile

Is this a bug or a feature request?
Bug
What did you expect?
What happened?
No Connection to LoRa App Server Web Interface, no data from devices.
What version are your using?
0.21.1
How can your issue be reproduced?
I don't know.
Could you share your log output?
Jun 26 15:47:59 DietPi lora-app-server[8002]: time="2018-06-26T15:47:59+03:00" level=error msg="finished client unary call" error="rpc error: code = Unimplemented desc = unknown method MigrateChannelConfigurationToGatewayProfile" grpc.cod
e=Unimplemented grpc.method=MigrateChannelConfigurationToGatewayProfile grpc.service=ns.NetworkServer grpc.time_ms=1.903 span.kind=client system=grpc
Jun 26 15:47:59 DietPi lora-app-server[8002]: time="2018-06-26T15:47:59+03:00" level=error msg="gateway-profile migration failed" error="migrate gateway-profiles for network-server error: migrate channel-configuration to gateway-profile e
rror: rpc error: code = Unimplemented desc = unknown method MigrateChannelConfigurationToGatewayProfile"

root@DietPi:~# systemctl status lora-app-server.service
● lora-app-server.service - LoRa App Server
   Loaded: loaded (/lib/systemd/system/lora-app-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-06-26 15:47:58 EEST; 31min ago
     Docs: https://docs.loraserver.io
 Main PID: 8002 (lora-app-server)
   CGroup: /system.slice/lora-app-server.service
           └─8002 /usr/bin/lora-app-server

Jun 26 16:19:49 DietPi lora-app-server[8002]: time="2018-06-26T16:19:49+03:00" level=warning msg="finished client unary call" error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure" grpc.code=Unavailable grpc.met
hod=MigrateChannelConfigurationToGatewayProfile grpc.service=ns.NetworkServer grpc.time_ms=0.063 span.kind=client system=grpc
Jun 26 16:19:49 DietPi lora-app-server[8002]: time="2018-06-26T16:19:49+03:00" level=error msg="gateway-profile migration failed" error="migrate gateway-profiles for network-server error: migrate channel-configuration to gateway-profile e
rror: rpc error: code = Unavailable desc = all SubConns are in TransientFailure"
Jun 26 16:19:51 DietPi lora-app-server[8002]: time="2018-06-26T16:19:51+03:00" level=warning msg="finished client unary call" error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure" grpc.code=Unavailable grpc.met
hod=MigrateChannelConfigurationToGatewayProfile grpc.service=ns.NetworkServer grpc.time_ms=0.08 span.kind=client system=grpc
Jun 26 16:19:51 DietPi lora-app-server[8002]: time="2018-06-26T16:19:51+03:00" level=error msg="gateway-profile migration failed" error="migrate gateway-profiles for network-server error: migrate channel-configuration to gateway-profile e
rror: rpc error: code = Unavailable desc = all SubConns are in TransientFailure"
Jun 26 16:19:53 DietPi lora-app-server[8002]: time="2018-06-26T16:19:53+03:00" level=warning msg="finished client unary call" error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure" grpc.code=Unavailable grpc.met
hod=MigrateChannelConfigurationToGatewayProfile grpc.service=ns.NetworkServer grpc.time_ms=0.076 span.kind=client system=grpc
Jun 26 16:19:53 DietPi lora-app-server[8002]: time="2018-06-26T16:19:53+03:00" level=error msg="gateway-profile migration failed" error="migrate gateway-profiles for network-server error: migrate channel-configuration to gateway-profile e
rror: rpc error: code = Unavailable desc = all SubConns are in TransientFailure"
Jun 26 16:19:55 DietPi lora-app-server[8002]: time="2018-06-26T16:19:55+03:00" level=warning msg="finished client unary call" error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure" grpc.code=Unavailable grpc.met
hod=MigrateChannelConfigurationToGatewayProfile grpc.service=ns.NetworkServer grpc.time_ms=0.112 span.kind=client system=grpc
Jun 26 16:19:55 DietPi lora-app-server[8002]: time="2018-06-26T16:19:55+03:00" level=error msg="gateway-profile migration failed" error="migrate gateway-profiles for network-server error: migrate channel-configuration to gateway-profile e
rror: rpc error: code = Unavailable desc = all SubConns are in TransientFailure"
Jun 26 16:19:57 DietPi lora-app-server[8002]: time="2018-06-26T16:19:57+03:00" level=warning msg="finished client unary call" error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure" grpc.code=Unavailable grpc.met
hod=MigrateChannelConfigurationToGatewayProfile grpc.service=ns.NetworkServer grpc.time_ms=0.061 span.kind=client system=grpc
Jun 26 16:19:57 DietPi lora-app-server[8002]: time="2018-06-26T16:19:57+03:00" level=error msg="gateway-profile migration failed" error="migrate gateway-profiles for network-server error: migrate channel-configuration to gateway-profile e
rror: rpc error: code = Unavailable desc = all SubConns are in TransientFailure"

You're running two versions that are incompatible or the configuration is pointing to the wrong server / port (so that the receiver does not know how to handle the API call).
To be better with component compatibility, The 1.x version was released. See also: https://forum.loraserver.io/t/release-lora-server-v1-0-lora-app-server-v1-0/1397
This could also be helpful: https://forum.loraserver.io/search?q=unimplemented desc %3D unknown method


Apply burnt fuel inventory with checking

The only change here is control.lua with 8 extra lines.
I am not used to Github.  There is only 1 actual commit, but somehow here listed 3 commits (including my sync attempts).

Looks like dimonft included these changes in his latest release.  Might want to close the pull request.


Bug when using stack's docker integration (command_ hangs)

I'm building a stack project using shake. Stack is configured to use docker (see here). Building (stack build) works at the shell but not through shake. Shake gets stuck at the first stack command it tries to run. The command completes fully, i.e. all the expected output is displayed, but then shake gets stuck and doesn't run any further actions.
Here is a minimal reproduction:

clone https://github.com/SandiDusic/shake-stack-docker-bug-repro
edit: optionally, uncomment the line in my-app/stack.yaml that changes the docker image. I can't try it out right now, but I think the problem persists with any image, so there's no need to download the default giant fpco/stack-build images.
in the build-system directory, build the build system with stack build
run the build system with stack exec build -- build
the project (my-app) builds completely, but then shake hangs (putQuiet "done building" never gets run)
try running stack build directly in my-app/ - it works (though NB it might show no output if it doesn't have to recompile, at least it doesn't for me)
try removing docker: enable: true from my-app/stack.yaml. the build system will work.

Info about my system:
Docker version 19.03.5-ce, build 633a0ea838
Linux pc 4.19.88-1-MANJARO #1 SMP PREEMPT Thu Dec 5 11:04:44 UTC 2019 x86_64 GNU/Linux
Version 2.1.3, Git revision 636e3a759d51127df2b62f90772def126cdf6d1f (7735 commits) x86_64 hpack-0.31.2

I am having some other troubles with stack's docker integration (https://github.com/commercialhaskell/stack/issues/5118) so my setup might be defective somehow.

I don't have a Linux environment so will struggle to reproduce this. I suggest running shake with -VVV which will print lots of extra information - in particular it might give you a clue what step it freezes on. If you upload that trace it should give us a hint. If you change the command to run anything other than stack does it work? If so, that sounds like Stack is defective (if not, worth investigating further). Perhaps add more verbosity arguments to stack itself too?

If you change the command to run anything other than stack does it work?

Well, stack itself works, if it's not running a docker. echo works as well. So does docker, for example replacing the command_ with:
command_ [] "docker" ["run", "--rm", "alpine", "ls", "/bin"]

I suggest running shake with -VVV which will print lots of extra information - in particular it might give you a clue what step it freezes on.

Here's the whole output with max verbosity on both shake and stack:
https://gist.github.com/SandiDusic/01bf60a1cc1044656b5035f03a790b98
Notice that I terminated manually at the end, the command hanged before that. I also updated the repo, adding the -v option to stack so this output can be seen.

If so, that sounds like Stack is defective

I wouldn't know how to open a useful issue with them, aside from linking this one. Any idea what could make a program misbehave when run from shake? Stack does work fine when running the command manually.

It seems Stack is a red herring here; I believe this is the same issue as https://github.com/ndmitchell/shake/issues/748.

Ah, thanks for tracking this down @gergoerdi  - closing this report in favour of that one, which also has a PR and explains the root cause.


Preview component causes email send to fail - with mailSend only

Describe the Bug
if we add:  Preview Component
  <Tailwind>
				  <Head />
				 <Preview>
					Heading text
				  </Preview>
				  <Body className="bg-white font-sans">
					  <{...}
				 </Section>

we get this error:

      'Error sending email with mailSend:' {
  headers: {
    'cache-control': 'no-cache, private',
    'cf-cache-status': 'DYNAMIC',
    'cf-ray': '8c3f1b8fbdd782aa-IAD',
    connection: 'close',
    'content-type': 'application/json',
    date: 'Mon, 16 Sep 2024 07:22:50 GMT',
    server: 'cloudflare',
    'strict-transport-security': 'max-age=31536000; includeSubDomains',
    'transfer-encoding': 'chunked',
    'x-apiquota-remaining': '985',
    'x-apiquota-reset': '2024-09-17T00:00:00Z',
    'x-ratelimit-limit': '120',
    'x-ratelimit-remaining': '119'
  },
  body: {
    message: 'The html field contains invalid input.',
    errors: { html: [Array] }
  },
  statusCode: 422
}

when we remove this component the email sends successfully.
the emails ends successfully with resend with the preview component in the email
Which package is affected (leave empty if unsure)
@react-email/preview
Link to the code that reproduces this issue
using latest resend package
To Reproduce
create a email and add preview component using mail send as the agent
Expected Behavior
should not get an error
What's your node version? (if relevant)
20

What if you move the <Preview> to inside the <Body>?

What if you move the <Preview> to inside the <Body>?

This solved the issue. thank you

@gabrielmfern I have discovered that if the preview contains more than 52 characters, then the email arrives completely blank.
Moving it inside <Body> seems to fix the issue, however all the examples I looked at had the preview outside the body.
It would be helpful to update documentation to be clear on this.

Actually, I've just run into this issue again where it's breaking even when the Preview component is inside the body, again only when the text is more than 52 chars long. It seems this issue is not resolved, can we reopen this ticket?

@Hainesy I think that issue is separate to this one, could you open one for that? Also, what email client is having it arrive blank?

Had the same issue, it's happening when Preview content has more than 53 chars

@gabrielmfern In my case emails were not even sent: The html field contains invalid input, (i'm using nodemailer to send)

@fafifox How did the rendered HTML look like then? I also find it weird that the error happens with more than 53 characters when the maximum preview length is 150. 🤔


Stats Pages

Stats for staff vs. public

Building this in to the admin/dashboard project for now.


Feign client issues

When using the feign client and properly adding the annotations (@EnableFeignClients(basePackages = "com.github.cc007.headsplugin") on the application class and @FeignClient(name="..." , url="...") on the client interfaces)
I get the following exception:
Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'org.springframework.cloud.openfeign.FeignContext' available
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:343) ~[?:?]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:335) ~[?:?]
        at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1123) ~[?:?]
        at org.springframework.cloud.openfeign.FeignClientFactoryBean.getTarget(FeignClientFactoryBean.java:243) ~[?:?]
        at org.springframework.cloud.openfeign.FeignClientFactoryBean.getObject(FeignClientFactoryBean.java:235) ~[?:?]
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:171) ~[?:?]
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:101) ~[?:?]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1674) ~[?:?]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getObjectForBeanInstance(AbstractAutowireCapableBeanFactory.java:1216) ~[?:?]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:257) ~[?:?]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[?:?]
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:277) ~[?:?]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.addCandidateEntry(DefaultListableBeanFactory.java:1467) ~[?:?]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1424) ~[?:?]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1207) ~[?:?]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1164) ~[?:?]
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:593) ~[?:?]

I have added the spring cloud dependency management and openfeign dependency (using the Greenwich.RELEASE version). All fixes that google suggests for the missing FeignContext bean didn't work, so I feel like it could be something in mcspring-boot or spigot that prevents it from working.

Hi, can you provide your pom.xml?

 I only had the HttpClient in my dependencyManagement, not my dependencies. It did however expose another issue where it couldn't find a bean that is definitely in the scanned packages:
Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'com.github.cc007.headsplugin.rest.clients.MineSkinClient' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true)}
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.raiseNoMatchingBeanFound(DefaultListableBeanFactory.java:1655) ~[?:?]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1214) ~[?:?]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1168) ~[?:?]
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:593) ~[?:?]

My Application class looks like this:
@SpringBootApplication(scanBasePackages = {"com.github.cc007.headsplugin"})
public class Application
{}
and I have a `FeignConfig` class like this:

@Configuration
@EnableFeignClients
public class FeignConfig
{}

The @EnableFeignClients scans the current package (com.github.cc007.headsplugin.config) by default, which is not the same of the client (com.github.cc007.headsplugin.rest.clients), you missed the @EnableFeignClients(basePackages = "com.github.cc007.headsplugin")

thx. I had that before, but I removed it when trying to fix the issues with http client. thx for your help.
But now im back to the issue I started with. Something causes the autoconfigure from feign to not be automatically included.

Is the feign auto configuration located at META-INF/spring.factories inside the jar file, or somewhere else?

spring factories seems to only contain your autoconfig...
my pom.xml files look like this:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
		 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<parent>
		<groupId>com.github.cc007</groupId>
		<artifactId>headsplugin-parent</artifactId>
		<version>2.0.0-SNAPSHOT</version>
		<relativePath>../headsplugin-parent</relativePath>
	</parent>

	<artifactId>headsplugin-core</artifactId>

	<packaging>jar</packaging>
	<dependencies>
		<dependency>
			<groupId>org.bukkit</groupId>
			<artifactId>bukkit</artifactId>
		</dependency>

		<dependency>
			<groupId>dev.alangomes</groupId>
			<artifactId>spigot-spring-boot-starter</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-starter-openfeign</artifactId>
		</dependency>

		<dependency>
			<groupId>org.modelmapper</groupId>
			<artifactId>modelmapper</artifactId>
		</dependency>

		<dependency>
			<groupId>org.apache.httpcomponents</groupId>
			<artifactId>httpclient</artifactId>
		</dependency>

		<dependency>
			<groupId>org.projectlombok</groupId>
			<artifactId>lombok</artifactId>
		</dependency>

	</dependencies>
	<build>
		...
	</build>
</project>

With dep management for versions in the parent.

Could it be because the spring-boot-maven-plugin isn't added? Since it is a plugin, the jar doesn't have a main method and therefore that plugin doesn't work properly.

This seems to be caused by the manual build configured. Since the spring-boot-maven-plugin is not used, the spring.factories is not merged in the final artifact.
Try using the official plugin:
<build>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
        </plugin>
    </plugins>
</build>

And add a empty main method (as required by the spring-boot-maven-plugin) in yout plugin class:
public static void main(String[] args) {
}

I will change the documentation to suggest this way.

 I recommended a manual configuration to make the integration of spring and spigot less "painful", but I think there's no other way rather than using the official maven plugin.

Yea, it would be a hassle to go through all your dependencies and check if there is any spring.factories file and then add those files to your context.

When using the plugin after having added the main method, the program does build, but the plugin now breaks, because the resources aren't added anymore.

This is because the files were moved to BOOT-INF\classes

This is because the files were moved to BOOT-INF\classes

I noticed that now, I'm trying to find a workaround...

I think finding a way to move the contents of BOOT-INF/classes to the root of the jar and add the lib file to the root as well. then you can use something similar to https://bukkit.org/threads/tutorial-use-external-library-s-with-your-plugin.103781/ to load the libraries
It would of course be nice if you could make a maven plugin that does the file moving for you and processes and edits the onEnable to automatically load the jars :D

If you get this working, I think it is also not necessary anymore to shade the jar anymore

Also important to note: since the maven jar plugin, maven shade plugin and the spring boot maven plugin all act on the package goal, it matters in which order these are added to the build section in the pom. Something that you could add to your "known issues" section in your wiki.

Also important to note: since the maven jar plugin, maven shade plugin and the spring boot maven plugin all act on the package goal, it matters in which order these are added to the build section in the pom. Something that you could add to your "known issues" section in your wiki.

After some digging, I managed to make the shading works:
<build>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.1.0</version>
            <dependencies>
                <dependency>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-maven-plugin</artifactId>
                    <version>${spring.version}</version>
                </dependency>
            </dependencies>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                </execution>
            </executions>
            <configuration>
                <transformers>
                    <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                        <resource>META-INF/spring.handlers</resource>
                    </transformer>
                    <transformer
                            implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer">
                        <resource>META-INF/spring.factories</resource>
                    </transformer>
                    <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                        <resource>META-INF/spring.schemas</resource>
                    </transformer>
                    <transformer
                            implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
                </transformers>
                <shadedArtifactAttached>true</shadedArtifactAttached>
                <shadedClassifierName>shaded</shadedClassifierName>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>2.3.2</version>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>
    </plugins>
</build>

I think finding a way to move the contents of BOOT-INF/classes to the root of the jar and add the lib file to the root as well. then you can use something similar to https://bukkit.org/threads/tutorial-use-external-library-s-with-your-plugin.103781/ to load the libraries
It would of course be nice if you could make a maven plugin that does the file moving for you and processes and edits the onEnable to automatically load the jars :D

I don't think that writing code to dynamic load  libraries is the right way, I want this project to be a "out of the box" support for spring, adding so much boilerplate code will not help starting new projects. I hope that this maven configuration will work for all cases.

In this case, since you're not running a goal in the spring-boot-maven-plugin, that plugin can be removed I think.
Also, with that I get the following error:
Caused by: java.lang.ClassNotFoundException: org.springframework.cloud.bootstrap.BootstrapImportSelectorConfiguration
        at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:340)
        at org.springframework.util.ClassUtils.forName(ClassUtils.java:275)
        at org.springframework.beans.factory.support.AbstractBeanDefinition.resolveBeanClass(AbstractBeanDefinition.java:438)
        at org.springframework.context.annotation.ConfigurationClassPostProcessor.enhanceConfigurationClasses(ConfigurationClassPostProcessor.java:400)

Some starters don't support a custom classloader (bad implementation), a definitive workaround for this is changing the thread class loader (which spring uses by default):
public class TestPlugin extends JavaPlugin {

    private ClassLoader defaultClassLoader;

    @Override
    public void onEnable() {
        defaultClassLoader = Thread.currentThread().getContextClassLoader();
        Thread.currentThread().setContextClassLoader(getClassLoader());
        // init code
    }

    @Override
    public void onDisable() {
        // disable code
        Thread.currentThread().setContextClassLoader(defaultClassLoader);
    }

}

Ok, now I added that, here is the next issue :D
[04:46:55 INFO]: 2019-06-01 04:46:55.464  WARN 11288 --- [  Server thread] s.c.a.AnnotationConfigApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'mineSkinApiTestCommand': Unsatisfied dependency expressed through field 'mineSkinCl
ient'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.github.cc007.headsplugin.rest.clients.MineSkinClient': FactoryBean threw exception on object creation; nested exception is org.springframework.boot.context.properties.ConfigurationPropertiesBindException: Error creating bean with name 'feign.client-org.springfr
amework.cloud.openfeign.FeignClientProperties': Could not bind properties to 'FeignClientProperties' : prefix=feign.client, ignoreInvalidFields=false, ignoreUnknownFields=true; nested exception is org.springframework.boot.context.properties.bind.BindException: Failed to bind properties under 'feign.client.config' to java.util.Map<java.lang.String, org.springframework.cloud.open
feign.FeignClientProperties$FeignClientConfiguration>

[04:46:55 INFO]: 2019-06-01 04:46:55.474  INFO 11288 --- [  Server thread] ConditionEvaluationReportLoggingListener :

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.

[04:46:55 INFO]: 2019-06-01 04:46:55.479 ERROR 11288 --- [  Server thread] o.s.b.d.LoggingFailureAnalysisReporter   :

***************************
APPLICATION FAILED TO START
***************************

Description:

Failed to bind properties under 'feign.client.config' to java.util.Map<java.lang.String, org.springframework.cloud.openfeign.FeignClientProperties$FeignClientConfiguration>:

    Reason: No converter found capable of converting from type [org.bukkit.configuration.MemorySection] to type [java.util.Map<java.lang.String, org.springframework.cloud.openfeign.FeignClientProperties$FeignClientConfiguration>]

Action:
Update your application's configuration

Can you provide your config.yml?

I found a mapper implementation: convertMapsToSections in YamlConfiguration:72

# plugin specific config
...

# feign config
feign:
  client:
    config:
      default:
        connectTimeout: 5000
        readTimeout: 5000
        loggerLevel: basic

I added a conversor from ConfigurationSection to Map by default on 0.18.1, will be available ASAP (https://github.com/Alan-Gomes/mcspring-boot/commit/41b5b7321fdb1782e46055c20bacdf0ac4ce5654#diff-e99133dfd499da62a0f4c9dc230e9eb4R33)
Thanks!

This seems to be a separate issue regarding config, since without this part in the config, the server starts allright and Feign is working like a charm. I'll add a new issue and consider this one closed.

Oh, you already fixed it. No need for that issue then. I have another one for you though :D

Hello!
Sorry for bringing back to life this issue, but something wrong has been mentioned in this comment which can be misleading in fact:

Some starters don't support a custom classloader (due to a bad implementation), a definitive workaround for this is changing the thread class loader (which spring uses by default):
public class TestPlugin extends JavaPlugin {

    private ClassLoader defaultClassLoader;

    @Override
    public void onEnable() {
        defaultClassLoader = Thread.currentThread().getContextClassLoader();
        Thread.currentThread().setContextClassLoader(getClassLoader());
        // init code
    }

    @Override
    public void onDisable() {
        // disable code
        Thread.currentThread().setContextClassLoader(defaultClassLoader);
    }

}

It should work without any problems, since each plugin has it's own class loader.

Modifying the context class loader is definitively not the perfect way to go forward with Bukkit. It's right that it seems to be the default first place where the Spring framework take a look at when searching a class loader before loading and resolving any bean. It's mentioned in the ClassUtils#getDefaultClassLoader() method that is used typically by the DefaultResourceLoader class (when looking at the implementation):
	/**
	 * Return the ClassLoader to load class path resources with.
	 * <p>Will get passed to ClassPathResource's constructor for all
	 * ClassPathResource objects created by this resource loader.
	 * @see ClassPathResource
	 */
	@Override
	@Nullable
	public ClassLoader getClassLoader() {
		return (this.classLoader != null ? this.classLoader : ClassUtils.getDefaultClassLoader());
	}

But there is certainly a cleaner way to customize the bean class loader by customizing the ApplicationContext at initialization time before starting to process any bean:
  @Override
  public void onEnable() {
    applicationContext =
        new SpringApplicationBuilder(MyApplication.class)
            .bannerMode(Mode.OFF)
            .initializers(appContext -> appContext.setClassLoader(getClassLoader()))
            .build()
            .run();
  }

Still, I'm not able to provide guarrantee that if it will work for all the Spring Boot starters (e.g. Feign here) since the default behavior can be overridden. You mentioned it typically: Some starters don't support a custom classloader (due to a bad implementation. I would be curious to learn more about your experience on this front.
But anyway, one thing which is wrong is saying that the context class loader is dedicated to a specific plugin. The thread is shared between all the plugins which implies that it's the same for the context class loader. I have made some experimentations on my side and here are my findings:

[21:53:53] [Server thread/INFO]: [TemplateSpringPlugin] [STDOUT] Context classloader: java.net.URLClassLoader@759ebb3d
[21:53:53] [Server thread/INFO]: [TemplateSpringPlugin] [STDOUT] System classloader: jdk.internal.loader.ClassLoaders$AppClassLoader@2a139a55
[21:53:53] [Server thread/INFO]: [TemplateSpringPlugin] [STDOUT] Plugin's classloader: PaperPluginClassLoader{libraryLoader=java.net.URLClassLoader@5fe36d7e, seenIllegalAccess=[], loadedJavaPlugin=TemplateSpringPlugin v0.1.0-SNAPSHOT}

As you can see, the PaperMC class loader dedicated to the plugin is not the same than the context class loader one. So, it's better to keet context class loader untouched in order to avoid potential conflicts with other plugins (especially when they are relying on Spring Boot while trying to alter all together the context class loader).
But anyway thanks a lot for this message since it gave me an hint for the solution to implement in order to make Spring Boot working with Bukkit! :)

I've moved to using Dagger2 about 2 years ago, but I hope your comments will help anyone else with similar issues.


Make ComplementQuery functional

For now it is quite useless

We removed this query so I am going to close this issue :)


Track which tasks failed on which agents

... and don't schedule them on the same agents again.

Looks good, merge when ready.


Create a "Credits" page

Icons:
https://pixabay.com/illustrations/under-construction-construction-sign-2408062/
https://pixabay.com/vectors/error-page-was-not-found-news-http-1349562/
https://pixabay.com/illustrations/thanks-thankyou-gratitude-grateful-5709753/
https://freesvg.org/red-coming-soon-stamp-vector-graphics

Done. It will be improved with the time, but for now it's ok.