Small noise_variance value leads to weird InvalidArgumentError exception

Here's what should be a minimal, reproducible example:
import numpy as np
import gpflow

x = np.linspace(0.0, 0.5)
y = np.sin(x) + 1e-2*np.random.standard_normal(x.shape)

# var_val = 1e-2
var_val = 1e-7

model = gpflow.models.GPR(data = (x,y),
                          kernel = gpflow.kernels.RBF(),
                          noise_variance = var_val)

gpflow.utilities.print_summary(model)

If the noise_variance argument in the above code is set to 1e-2, things work as expected, and I get the following result:
╒═════════════════════════╤═══════════╤══════════════════╤═════════╤═════════════╤═════════╤═════════╤═════════╕
│ name                    │ class     │ transform        │ prior   │ trainable   │ shape   │ dtype   │   value │
╞═════════════════════════╪═══════════╪══════════════════╪═════════╪═════════════╪═════════╪═════════╪═════════╡
│ GPR.kernel.variance     │ Parameter │ Softplus         │         │ True        │ ()      │ float64 │    1    │
├─────────────────────────┼───────────┼──────────────────┼─────────┼─────────────┼─────────┼─────────┼─────────┤
│ GPR.kernel.lengthscales │ Parameter │ Softplus         │         │ True        │ ()      │ float64 │    1    │
├─────────────────────────┼───────────┼──────────────────┼─────────┼─────────────┼─────────┼─────────┼─────────┤
│ GPR.likelihood.variance │ Parameter │ Softplus + Shift │         │ True        │ ()      │ float64 │    0.01 │
╘═════════════════════════╧═══════════╧══════════════════╧═════════╧═════════════╧═════════╧═════════╧═════════╛

However, if it is set to 1e-7, then I get the following traceback:
Traceback (most recent call last):
  File "./test.py", line 14, in <module>
    noise_variance = var_val)
  File "/Users/jjramsey/venvs/gpflow2/lib/python3.7/site-packages/gpflow/models/gpr.py", line 49, in __init__
    likelihood = gpflow.likelihoods.Gaussian(noise_variance)
  File "/Users/jjramsey/venvs/gpflow2/lib/python3.7/site-packages/gpflow/likelihoods/likelihoods.py", line 404, in __init__
    self.variance = Parameter(variance, transform=positive(lower=variance_lower_bound))
  File "/Users/jjramsey/venvs/gpflow2/lib/python3.7/site-packages/gpflow/base.py", line 82, in __init__
    unconstrained_value = self.validate_unconstrained_value(value, dtype)
  File "/Users/jjramsey/venvs/gpflow2/lib/python3.7/site-packages/gpflow/base.py", line 165, in validate_unconstrained_value
    return tf.debugging.assert_all_finite(unconstrained_value, message=message)
  File "/Users/jjramsey/venvs/gpflow2/lib/python3.7/site-packages/tensorflow_core/python/ops/numerics.py", line 67, in verify_tensor_all_finite_v2
    verify_input = array_ops.check_numerics(x, message=message)
  File "/Users/jjramsey/venvs/gpflow2/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_array_ops.py", line 902, in check_numerics
    _ops.raise_from_not_ok_status(e, name)
  File "/Users/jjramsey/venvs/gpflow2/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 6606, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: gpflow.Parameter: unconstrained value of passed value has NaN or Inf and cannot be assigned. : Tensor had NaN values [Op:CheckNumerics]

My software and hardware configuration are as follows:

GPflow version: 2.0.0, from PyPI
Tensorflow version: 2.1.0, from PyPI
Tensorflow Probability version: 0.9.0, from PyPI
Does your tensorflow use GPU: No
Python version: 3.7.7
OS: macOS 10.13.6
Macbook configuration: MacBook Pro (15-inch, 2017), Processor: 2.9 GHz Intel Core i7, Memory: 16 GB 2133 MHz LPDDR3, Graphics: Radeon Pro 560 4096 MB, Intel HD Graphics 630 1536 MB

I was also seeing this with variance 1e-6

I believe this may be due to the lower bound on the positive constraint in the likelihood. From the source:
class Gaussian(ScalarLikelihood):
    r"""
    The Gaussian likelihood is appropriate where uncertainties associated with the data are
    believed to follow a normal distribution, with constant variance.
    Very small uncertainties can lead to numerical instability during the
    optimization process. A lower bound of 1e-6 is therefore imposed on the likelihood variance
    by default.
    """

    def __init__(self, variance=1.0, variance_lower_bound=1e-6, **kwargs):
        super().__init__(**kwargs)
        self.variance = Parameter(variance, transform=positive(lower=variance_lower_bound))

This ensures that the likelihood (noise) variance can only take values strictly larger than variance_lower_bound. This lower bound exists because if the noise variance is optimized to arbitrarily small numbers, the numerics become unstable. GPflow 1 had an implicit lower bound on the noise variance, i.e. you could set it to 1e-20, but it would still be 1e-6. In GPflow 2 we decided to explicitly error.
How could we improve documentation/behaviour to make this more obvious & self-explanatory?

How could we improve documentation/behaviour to make this more obvious & self-explanatory?

I suggest we document it where it's possible to specify the variance but not the variance_lower_bound, e.g. in GPR. There's a bit explaining it in the docs for Gaussian which is good, perhaps it could be explicit in the __init__, sth like
        """
        :param variance: The variance of this Gaussian. Must be greater than
            ``variance_lower_bound``.
        :param variance_lower_bound: The lower (exclusive) bound of ``variance``.
        :param kwargs: Keyword arguments forwarded to :class:`ScalarLikelihood`.
        :raise InvalidArgumentError: If ``variance`` is not greater than ``variance_lower_bound``.
        """

That may be a workaround, but it looks like something is wrong in the Parameter constructor. Take this code for example:
import gpflow
import tensorflow_probability as tfp

tfb = tfp.bijectors
lower_bound = 1e-6

# Should be Softplus + Shift
bijector = tfb.Chain([tfb.Shift(gpflow.utilities.to_default_float(lower_bound)), tfb.Softplus()])

x_in = 1e-7
x_out = bijector.forward(x_in)

print("x_in = {}".format(x_in))
print("x_out = {}".format(x_out))
print("bijector.inverse(x_out) = {}".format(bijector.inverse(x_out)))

The output of this is as one would expect, with the bijector apparently successfully doing its job:
x_in = 1e-07
x_out = 0.6931482305599465
bijector.inverse(x_out) = 1e-07

However, if I try to make a GPFlow parameter from x_in and the bijector, then I get about the same traceback as before:
Traceback (most recent call last):
  File "test2.py", line 18, in <module>
    x_param = gpflow.base.Parameter(x_in, transform = bijector)
  File "/Users/jjramsey/venvs/gpflow2/lib/python3.7/site-packages/gpflow/base.py", line 82, in __init__
    unconstrained_value = self.validate_unconstrained_value(value, dtype)
  File "/Users/jjramsey/venvs/gpflow2/lib/python3.7/site-packages/gpflow/base.py", line 165, in validate_unconstrained_value
    return tf.debugging.assert_all_finite(unconstrained_value, message=message)
  File "/Users/jjramsey/venvs/gpflow2/lib/python3.7/site-packages/tensorflow_core/python/ops/numerics.py", line 67, in verify_tensor_all_finite_v2
    verify_input = array_ops.check_numerics(x, message=message)
  File "/Users/jjramsey/venvs/gpflow2/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_array_ops.py", line 902, in check_numerics
    _ops.raise_from_not_ok_status(e, name)
  File "/Users/jjramsey/venvs/gpflow2/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 6606, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: gpflow.Parameter: unconstrained value of passed value has NaN or Inf and cannot be assigned. : Tensor had NaN values [Op:CheckNumerics]

It looks like there's something in Parameter's validate_unconstrained_value that leads to a false positive in the NaN check.

@jjramsey you've used .forward in your test, but the code uses .inverse, as the hidden unconstrained value on the Parameter will be the inverse of the bijector (I think)

Ah, you're right. The forward output of Softplus + Shift is constrained to be a value greater than Shift, so it corresponds to the constrained value. I got it backwards. Never mind.

How could we improve documentation/behaviour to make this more obvious & self-explanatory?

At the very least, a bad value for the variance (or any other argument) should be filtered out at the earliest level. As a negative example, here's the current constructor for the Gaussian likelihood:
    def __init__(self, variance=1.0, variance_lower_bound=1e-6, **kwargs):
        super().__init__(**kwargs)
        self.variance = Parameter(variance, transform=positive(lower=variance_lower_bound))

Even though variance is supposed to be greater than variance_lower_bound, there's no check that this is the case. Instead, the constructor should be something like this:
    def __init__(self, variance=1.0, variance_lower_bound=1e-6, **kwargs):
        if variance <= variance_lower_bound:
            raise ValueError("Likelihood variance must be strictly greater than {}".format(variance_lower_bound))

        super().__init__(**kwargs)
        self.variance = Parameter(variance, transform=positive(lower=variance_lower_bound))

With this exception message, it's more obvious what the problem is and how to fix it.
Furthermore, the value of this lower bound should be documented. Ideally, it should be some constant in GPFlow that a user can query, maybe something like gpflow.config.default_noise_variance_lower_bound().

Furthermore, the value of this lower bound should be documented. Ideally, it should be some constant in GPFlow that a user can query, maybe something like gpflow.config.default_noise_variance_lower_bound().

There is actually gpflow.config.default_positive_lower_bound() (which by default is 0, and explicitly overridden by the variance_lower_bound in Gaussian()). I'm a bit hesitant to add more specific configurables - should this be default_gaussian_likelihood_noise_variance? or should it apply to e.g. StudentT as well (but in what way - it's not exactly comparable)? What's the value of making this lower bound configurable - if you are thinking about the lower bound, you can also set it manually on the concrete parameters?

I think the Parameter.assign() check is important, as TFP has quite subtle behaviour around invalid values passed through bijectors. Note the difference between
import gpflow
pos = gpflow.utilities.positive(lower=1e-6)
print(pos.forward(pos.inverse(1e-10)).numpy())  # -> "1e-10"
print(pos.forward(pos.inverse(1e-10) + 0.0).numpy())  # -> "nan"

because the bijectors do caching on tensor identity. It's easy to accidentally end up with nan or -inf in the unconstrained_variable representation, at which point your optimizer is either going to behave weirdly or at the very least not optimize your parameter even though it is set trainable (as inf or nan minus any finite gradient is still inf or nan!).
I think it's better to fail earlier - which is why I clarified the error message on this check (see #1408); please let me know if that error message would have made it easier for you to debug yourself or how we could phrase it even better. I've also included your suggestion of explicitly checking in the Gaussian constructor.

What's the value of making this lower bound configurable - if you are thinking about the lower bound, you can also set it manually on the concrete parameters?

It's less about making it configurable and more about guarding against changes in the lower bound on your end, a way of avoiding "magic numbers". The lower bound doesn't need to be in gpflow.config per se; that was just for the sake of having a concrete example. I don't really care much if it's configurable; I only care about being agnostic as to the lower bound's exact value.
If I have a script that reads
noise_variance = (noise_var_init_est if noise_var_init_est > 1.1e-6 else 1.1e-6)

in order to ensure that my initial value for the noise variance doesn't encroach on the lower bound that GPFlow has set for that variance, then I've essentially hardcoded the assumption that GPFlow will always have a lower bound less than 1.1e-6, an assumption that may change in the future. Such code would be brittle if later editions of GPFlow were to increase that lower bound.
However, if I can have code roughly like this,
noise_variance = (noise_var_init_est
                  if noise_var_init_est > 1.1*gpflow.NOISE_VAR_LB
                  else 1.1*gpflow.NOISE_VAR_LB)

then I'm no longer hardcoding an assumption about the value of the lower bound of the noise variance in GPFlow. There's also a bit of self-documentation there as well; it's more obvious that this code is hedging against a lower bound set by GPFlow itself, rather than user's own guestimate of what a good lower bound should be.
Note that I don't care that much if the constant in question is called gpflow.NOISE_VAR_LB. It could even be a class constant like, say, gpflow.likelihoods.Gaussian.default_variance_lower_bound.

I think the Parameter.assign() check is important

Of course, there's no reason for it to go away. It's certainly useful as a guard against bugs. However, it is no substitute for a check at a higher level that indicates the reason for a problem. The following error message, which would be from the Parameter.assign() check, still leaves me guessing as to the reason for a problem:

"gpflow.Parameter: the value to be assigned is incompatible with this parameter's transform (the corresponding unconstrained value has NaN or Inf) and hence cannot be assigned."

At this point, I don't know why the value is incompatible with this parameter's transform. I might not even be sure what this "transform" even means at this point. It's an error message that suggests a weird bug that involves getting into the weeds of GPFlow's internals, not something that's trivially solvable.
On the other hand, if I see an error message like, "Argument noise_variance must be strictly greater than gpflow.NOISE_VAR_LB (= 1e-6)", then I have a far clearer idea of where the problem lies.

It could even be a class constant like, say, gpflow.likelihoods.Gaussian.default_variance_lower_bound.

Great idea! Hadn't thought of that :+1:

I'd say that as a general rule, invalid user input to a constructor should be rejected directly by that constructor, rather than passing it along to other constructors. That way, when the constructor sees invalid input, it can report an intelligible error message that explains why the input is invalid and how to fix it.

Yes, I agree, where we don't do that already and can easily do it, we definitely should! (If you're already aware of more places in our codebase where that might apply, it'd be great if you could contribute a pull request to improve it!)
I've tried to incorporate all your feedback in #1408 , could you check the changes there please and see if there's anything I missed, or whether you can think of better ways of phrasing messages etc.?

I think that looks like it's on about the right track.

Hi @jjramsey, I hope you are satisfied with the resolution of your issue - please let us know if not, or if you've got any further feedback! Thanks for helping us improve GPflow :+1:


chore: writing unit tests for todo repository

Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes # (issue)
Type of change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Checklist:

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@ab2cfe6). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main      #17   +/-   ##
=======================================
  Coverage        ?   41.70%
=======================================
  Files           ?        9
  Lines           ?      223
  Branches        ?        0
=======================================
  Hits            ?       93
  Misses          ?      127
  Partials        ?        3

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ab2cfe6...9f00e88. Read the comment docs.


INFO: Page 1: Scraping http://favicon.ico/ { url: 'http://favicon.ico/' } D:\code\jina-reader-hargup\reader\backend\functions\node_modules\tld-extract\index.js:43       throw new Error("Invalid TLD " + JSON.stringify({parts, tld_level, allowUnknownTLD}));

the following error occurred when crawling：

i got the same error, Can you tell me how you solved this problem?

I got the reason.
This error occurs in the Chrome browser and no problem in the curl code.
Maybe the wrong header of  browser request cause the error


Stress testing classes for throttling service.

There are two main sets of changes in this PR:
Stres Test for Throttling

There are two stress tests: MRStressTest and LocalStressTest.
A Stressor is a class that sends requests to a Limiter. Different stressors can simulate different usage patterns.
The LocalStressTest uses and embedded throttling server and multiple threads to run a stressor. The requests do not go through the network, but the stress test can add artificial latency representing the network latency. Every 15 seconds, the stress test logs statistics about the throttling.
The MRStressTest uses a real throttling server. It spawns multiple mappers each one running a stressor. The mappers emit their effective permit use rate every 15 seconds, which then gets collected by the reducer to compute the global effective QPS every 15 seconds.

Throttling algorithm updates

Some changes to the batching algorithm. The previous algorithm was keeping the permit requests open for a long time (~7 seconds under high contention) because it allowed up to 7 seconds to acquire even a small number of permits. The new algorithm uses a short timeout instead, generally just short enough to satisfy the minimum number of permits requested. Larger permit allocations only happen if there are permits stored (meaning low contention).

@ydai1124 addressed comments.


Clear category_ids on category delete

When category deleted we need to remove it from category_ids[] for every product.

Fixed in Pull Request #543


Do not ignore src/ when publishing NPM

See: https://github.com/tensorflow/tfjs/issues/1339

This change is

I'd like to better understand the usability before we make this decision. See https://github.com/tensorflow/tfjs/issues/1339#issuecomment-471193352 for an alternative and let me know if that works for you. Thanks!

HI @dsmilkov , thank you for the reply and not closing this PR.
As I replied in the other issue, I will lose the interest to debug the tfjs library if I can not found the source code when I'm working in the terminal.

For example: how could I get to know what's wrong from my following error trace log?
# npm i @tensorflow/tfjs-node@next

><EMAIL_ADDRESS>install /notebooks/node_modules/@tensorflow/tfjs-node
> node scripts/install.js

* Downloading libtensorflow
[==============================] 2922683/bps 100% 0.0s
* Building TensorFlow Node.js bindings
+<EMAIL_ADDRESS>removed 21 packages, updated 1 package and audited 140 packages in 20.35s
found 0 vulnerabilities

root@94a0ba4be94e:/notebooks# node -r ts-node/register translation.ts dist/fra.txt
2019-03-09 16:56:43.196524: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
Number of samples: 10000
Number of unique input tokens: 69
Number of unique output tokens: 93
Max sequence length for inputs: 16
Max sequence length for outputs: 59
Saved metadata at:  /tmp/translation.keras/metadata.json
Orthogonal initializer is being called on a matrix with more than 2000 (262144) elements: Slowness may result.
Orthogonal initializer is being called on a matrix with more than 2000 (262144) elements: Slowness may result.
Epoch 1 / 20
(node:3814) UnhandledPromiseRejectionWarning: Error: Not yet implemented
    at NodeJSKernelBackend.KernelBackend.zerosLike (/notebooks/node_modules/@tensorflow/tfjs-core/src/kernels/backend.ts:597:11)
    at environment_1.ENV.engine.runKernel.$x (/notebooks/node_modules/@tensorflow/tfjs-core/src/ops/tensor_ops.ts:467:50)
    at /notebooks/node_modules/@tensorflow/tfjs-core/src/engine.ts:206:22
    at Engine.scopedRun (/notebooks/node_modules/@tensorflow/tfjs-core/src/engine.ts:167:19)
    at Engine.runKernel (/notebooks/node_modules/@tensorflow/tfjs-core/src/engine.ts:202:10)
    at zerosLike_ (/notebooks/node_modules/@tensorflow/tfjs-core/src/ops/tensor_ops.ts:467:21)
    at Object.zerosLike (/notebooks/node_modules/@tensorflow/tfjs-core/src/ops/operation.ts:46:24)
    at Object.$x (/notebooks/node_modules/@tensorflow/tfjs-core/src/ops/unary_ops.ts:349:21)
    at _loop_1 (/notebooks/node_modules/@tensorflow/tfjs-core/src/tape.ts:171:43)
    at Object.backpropagateGradients (/notebooks/node_modules/@tensorflow/tfjs-core/dist/tape.js:158:9)
(node:3814) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:3814) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
root@94a0ba4be94e:/notebooks#


feat: Create one branch per overlay combination when aggregating PR changes

Previous PR would also allow a single PR to get the aggregated changes, regardless of the overlay being updated. This PR fixes that behavior by adding the overlay name to the branch that is created.

Changes merged in #26


Written file is not detected by Gradle and so not packaged in the final APK

I'm not fully sure if it's a Gradle or Android Gradle plugin issue, but as I can reproduce using Gradle 6.7 and not upgrading the AGP, I'll give it a try here, please point me out if it's not the case.
We've written a Gradle plugin that writes a file to the build folder (assets) so this file is gonna be packaged in the final APK (Android App), this file contains a hash of the mappings file for stack traces symbolization on the server.
So this file needs to be written in the build folder after the AGP minifyReleaseWithR8 task as we need to compute the hash first, but before the AGP packageRelease task, as the file needs to be packaged in the final APK.
All the tasks seem to be executed in the right order.
I'd guess that this is something related to incremental tasks and its caching system (input/output etc...).
Expected Behavior
The written file makes it to the final APK
Current Behavior
The file doesn't make it to the final APK, only after the 2nd run (assembleRelease), but If I do clean assembeRelease, it doesn't work sometimes.
Context
This used to work on AGP <= 4.0.x and Gradle <= 6.6.x, and now either upgrading Gradle to 6.7.x or AGP to 4.1.x looks like the file is written to the build folder correctly but not in the final APK.
Also, this is the original issue: https://github.com/getsentry/sentry-android-gradle-plugin/issues/37 with more context of my findings
Steps to Reproduce
A demo is available https://github.com/mrrobot97/SentryGradlePluginIssueDemo
./gradlew clean assembleRelease -> the file is not written in the assets folder in the final APK
once more ./gradlew assembleRelease -> the file is written in the assets folder in the final APK
Your Environment

Build scan URL:
1st run: clean assembleRelease https://scans.gradle.com/s/a6usdxgw62kpu
2nd run: assembleRelease https://scans.gradle.com/s/mzq33uilc4rpk

any update on this? thanks :)

any update on this? thanks :)

I am experiencing something similar starting with Gradle 6.7. Gradle 6.6 still works fine.
The use-case is having a script that pushes files into the intermediates/merged_assets//out path.
The task copying of the file has a setup as follows:
copyStuff.dependsOn {
    tasks.matching { task -> task.name.startsWith('merge') && task.name.endsWith('Assets') }
}

tasks.withType(JavaCompile) {
    compileTask -> compileTask.dependsOn copyStuff
}

copyStuff

With gradle 6.6 and lower, this leads to a file in the specified path and it also ending up in the apk under the assets.
With gradle 6.7+ (also tried 6.8), the file is present in the intermediates, but does not end up in the apk.
If there is any specific change that makes this behavior obvious, please let me know.

I am experiencing something similar starting with Gradle 6.7. Gradle 6.6 still works fine.
The use-case is having a script that pushes files into the intermediates/merged_assets//out path.
The task copying of the file has a setup as follows:
copyStuff.dependsOn {
    tasks.matching { task -> task.name.startsWith('merge') && task.name.endsWith('Assets') }
}

tasks.withType(JavaCompile) {
    compileTask -> compileTask.dependsOn copyStuff
}

copyStuff

With gradle 6.6 and lower, this leads to a file in the specified path and it also ending up in the apk under the assets.
With gradle 6.7+ (also tried 6.8), the file is present in the intermediates, but does not end up in the apk.
If there is any specific change that makes this behavior obvious, please let me know.

I also encountered the same problem. Our business is the newly generated file in the compilation stage, which will be copied to the asset directory and finally packaged into the apk. But after upgrading to gradle 6.7, it was found that the files compiled and copied to the build directory were not packaged into the apk.

I also encountered the same problem. Our business is the newly generated file in the compilation stage, which will be copied to the asset directory and finally packaged into the apk. But after upgrading to gradle 6.7, it was found that the files compiled and copied to the build directory were not packaged into the apk.

Hi there, please make sure that the sentry plugin and task are declaring the output files in the task.
Here is how you can do that: https://docs.gradle.org/current/userguide/more_about_tasks.html#sec:task_input_output_annotations

This is not a bug on Gradle's part. There are a couple of things missing from the Sentry plugin:

It does not register the file as an output
It does not depend on the correct task (the correct one would be merge<variant>Assets)
It does not add the file to the source set

Thanks @ansman for investigating this. I will close this out.

Hi there, please make sure that the sentry plugin and task are declaring the output files in the task.
Here is how you can do that: https://docs.gradle.org/current/userguide/more_about_tasks.html#sec:task_input_output_annotations

Hey :) thanks for your message, yep we've noticed that and we're already planning a full rewrite of this plugin using the newest AGP and Gradle APIs, what I'm missing here is only why this used to work and now it does not anymore.
Looking at https://docs.gradle.org/6.7/userguide/upgrading_version_6.html#changes_6.7 I also don't see any explicit breaking changes.


StateAttribute: ClonedAttribute may overwrite original uniform

StateAttribute::apply may contain per attribute uniform,
and as TypeMember is the same between cloned and original, uniform cache will return the same uniform for both, and will then be overwritten if not handled with an early out return code
if (!this._enable)return ;

(need to update example shader generator sample to advertize the possible behaviour)

Should not be a problem, I added extra check in State.js with this PR https://github.com/cedricpinson/osgjs/pull/382 that would help to find the real problem


Add rez info command

A simple new command to be able to get information of the current package without the need to be in a rez environment.  ( rez env rez)
For example, we will use that from Jenkins pipelines to get information of the current package to build the name of the artifacts.
usage:
>rez info name
usdBase

>rez info version
0.8.1

>rez info --separator  ", " requires
stdlib-4.4.6+<5, tinyxml-2.5.2+<2.6, cpp_btree-1.1, libcurl-7

>rez info variants
CentOS-6.2+<7 AL_boost-1.55.0
CentOS-6.2+<7 boost-1.55.0

>rez info --raw  variants
[['CentOS-6.2+<7', 'AL_boost-1.55.0'], ['CentOS-6.2+<7', 'boost-1.55.0']]

I checked the Travis build error, it does not seem it is coming from the changes in this PR

Someone asked me today what is the difference from this with rez search.
In order to use rez search, you need to know the name of the package.
The scenario where we use this is on some Jenkins jobs that build muti rez package git repositories. When a commit happens in one of those repos, we only want to build the rez package that changed (and it's downstream dependencies).
From the commit, we found the information of the paths that were changed and if it under a valid package (contains package.py|yaml + CmakeLists.txt) we build that package, but we need information from the current package being built i.e we need the name and version to create the artifact name or the dynamic name of the stages in the Jenkins scripted pipeline.
The other thing is that is a bit faster as it just opens the file and gets the info it needs. It is basically a command to avoid cat package* | grep | sed lines. And of course, something that works across all platforms.
If you think there is no need for a new command, and there is a lot of overlap with rez search, I can think on extending the rez search capabilities to include the case where you want info from the current package without needing the name of the package.

I'd be interested in something similar, though I would see it more like rez-view . --format yaml maybe? From Jenkins you can easily open a YAML file with readYaml def packageContent = readYaml file: <path to file.yaml>. And I guess we could even add support for a JSON format in case someone would not be able to easily parse YAML from a CI infra.
I personally feel rez-info would cause confusion with rez-view. But I might be wrong.
What do you guys think?

I agree that updating rez-view is probably the best approach here. I would:

add options to print a specific field;
add option to view the "current" package (meaning: The unbuilt package in
cwd with package.py present).

This has been raised before, specifically wrt the need to get access to a
package's attributes from a build script during build. Makes sense.
A
On Sun, Oct 14, 2018 at 4:06 AM JeanChristopheMorinRodeoFX <
<EMAIL_ADDRESS>wrote:

I'd be interested in something similar, though I would see it more like rez-view
. --format yaml maybe? From Jenkins you can easily open a YAML file with
readYaml
https://jenkins.io/doc/pipeline/steps/pipeline-utility-steps/#readyaml-read-yaml-from-files-in-the-workspace-or-text def
packageContent = readYaml file: . And I guess we could
even add support for a JSON format in case someone would not be able to
easily parse YAML from a CI infra.
I personally feel rez-info would cause confusion with rez-view. But I
might be wrong.
What do you guys think?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nerdvegas/rez/pull/467#issuecomment-429558392, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABjqSvZ2wfQqsDGEanpFhCyEkE7F771fks5ukh2ugaJpZM4RE8gY
.

I noticed that rez-view already has a -c (--current) flag but is used for the current context.
Also,  you will need the ability to extract just one field (raw and formatted), you can copy the code from this PR.
Our use cases are all in the test, so as long as those test pass I do not mind consolidating it into rez-view

Here is the new PR: https://github.com/nerdvegas/rez/pull/539/files, 2 weeks later that what I initially said. Anyway, the important thing is it's there.
@fnaum You are welcome to test my branch. Let me know if there is a difference with what you had (in theory it's good, but if you could confirm to me, that would be great).

Great work!
Can we add indices to the formatted variants dumps?
I wrote something similar with indices so that I can lookup what to feed rez-build --variants XXX with.
I guess at this point we can also consider hashed variants for build packages, which would help https://github.com/nerdvegas/rez/issues/754#issuecomment-538110357
Maybe like:
>rez info variants
0: CentOS-6.2+<7 AL_boost-1.55.0 [HASHIFAVAILABLE]
1: CentOS-6.2+<7 boost-1.55.0


Updating Velero CR configured plugins doesn't update deployment

After installing Velero (creating a Velero CR) with one plugin (aws), I then went and edited the CR with an additional plugin (openshift) and the Velero deployment was not patched to include the new initContainer.
I have not confirmed if this is reproducible or if it was just my environment with Noobaa (Noobaa could have been stalling the deployment update).

Confirmed this a bug even outside of the Noobaa use case.


Replace radio on supplier trading status page

Trello: https://trello.com/c/aYM0E0zY/896-2-replace-radios-with-govuk-frontend-radios-component-in-confirm-company-details-journey
As part of moving to Design System 3, we need to replace our use of toolkit/forms/selection-buttons.html with govukRadios from the design system. This should make this page more accessible.
As part of this, we're also moving away from our customised DMRadioField to the standard wtforms equivalent.
We've added a helper method in CompanyTradingStatusForm to get the items in the right format for govukRadios. We're not sure this is the right way to do this - we'll refactor as necessary when we come to replace the other radios in the supplier details journey.

Rebased to bring in @domoscargin 's (much neater) test updates from #1395

Thanks. Spacing all looks sensible, so I'm going to merge


Does not convert all chunks

i noticed that this converter doesn't convert chunks which dont have a path/way from spawn. I have a lot of people in my survival server who are using the nether to go away/travel. (not sure how to call it) and if those chunks are not connected, your converter tool doesn't convert them. could you add that the converter converts all chunks even if they are in the millions?
if am wrong let me know what could be the issue why not everything got converted.... thanks

There's no connectivity check, the tool will convert all chunks that are in the MCA file. There was an issue which region files which dimensions exceeds 2 digits were not being converted, may be you aren't seeing the chunks because of it. This PR will fix the region issue: #79
If the issue still exists in the next version, please attach a MCA file so I can test with it.

well I could send you a dl of the complete map private

ok ill give the new version a try


v2.1.0-rc3 Errors When Checking for Update

Brief Description:
Operating System (OS and version):
Windows 10 Home.
v2.1.0 release candidate 3
This may be expected behavior. Since this is a release canddiate, it might be expected to fail when checking for updates. I've been monitoring the console log in the Windows RC. I noticed this pop up. It seems to be related to checking for updates:
C:\Users\Chris\AppData\Local\OpenBazaar2\app-2.1.0\resources\app.asar\prod\start.js:900 Checking for update at https://updates2.openbazaar.org:5001/update/win64/2.1.0
C:\Users\Chris\AppData\Local\OpenBazaar2\app-2.1.0\resources\app.asar\prod\start.js:900 Command failed:<PHONE_NUMBER>
System.AggregateException: One or more errors occurred. ---> System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.WebClient.GetWebResponse(WebRequest request, IAsyncResult result)
   at System.Net.WebClient.DownloadBitsResponseCallback(IAsyncResult result)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.Utility.<LogIfThrows>d__38`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.FileDownloader.<DownloadUrl>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Squirrel.UpdateManager.CheckForUpdateImpl.<CheckForUpdate>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.UpdateManager.<CheckForUpdate>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.Update.Program.<Download>d__7.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at Squirrel.Update.Program.executeCommandLine(String[] args)
   at Squirrel.Update.Program.main(String[] args)
   at Squirrel.Update.Program.Main(String[] args)
---> (Inner Exception #0) System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.WebClient.GetWebResponse(WebRequest request, IAsyncResult result)
   at System.Net.WebClient.DownloadBitsResponseCallback(IAsyncResult result)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.Utility.<LogIfThrows>d__38`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.FileDownloader.<DownloadUrl>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Squirrel.UpdateManager.CheckForUpdateImpl.<CheckForUpdate>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.UpdateManager.<CheckForUpdate>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.Update.Program.<Download>d__7.MoveNext()<---

This is working as intended, release candidates cannot automatically update.


Ctrl+[ does not work on MAC

Ctrl+[ does not work on MAC
Version 0.5.7

same issue.

Will add this one ASAP.


User Guide not updated with the new commands and prefixes

User Guide still contain the old commands and discarded commands such as addinfo command.
Let's update user guide with the new commands for adding and flagging
clients.
Note: UG Section on deleting and editing by name will be done by Vish.
This should close #69. Nice.

Codecov Report

Merging #70 (da28a71) into master (f42d1ea) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master      #70   +/-   ##
=========================================
  Coverage     69.53%   69.53%
  Complexity      425      425
=========================================
  Files            75       75
  Lines          1408     1408
  Branches        147      147
=========================================
  Hits            979      979
  Misses          381      381
  Partials         48       48

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f42d1ea...da28a71. Read the comment docs.


Add Big Patent dataset

More info on the dataset: https://evasharma.github.io/bigpatent/
There's another raw version of the dataset available from tfds. However, they're quite large so I don't have the resources to fully test all the configs for that version yet. We'll try to add it later.

@lhoestq reduced the dummy data size to around 19MB in total and added the dataset card.

@lhoestq so I ended up removing all the nested JSON objects in the gz datafile and keep only one object with minimal content: {"publication_number": "US-8230922-B2", "abstract": "dummy abstract", "application_number": "US-201113163519-A", "description": "dummy description"}.
They're reduced to 35KB in total (2KB per domain and 17KB for all domain), hopefully, it's small enough.


RFCs 5 and 11 metadata is not complete

RFC 5 does not have an implementation-tracking issue listed.
RFC 11 does not have an RFC PR or implementation-tracking issue listed.

I think this has basically since been taken care of.


Add support for multiple exclusion labels

Fix #458
Divider between labels: ;

looks good. one suggestion inline and can you also include documentation updates?

Documentation updated

Thanks for the contribution 🎉

Thanks for the contribution 🎉

My pleasure. Looking forward to contribute more!!


NRatings stopped working on my Windows7 PC yesterday.

As the subject line states, I cannot seem to get the app to open anymore.  I've never had an issue before and have the most current version of the program (v5.1.6).  Will this be fixed soon?

Do you get any error messages?
The thing that changed in 5.1.6 is that now the local browser is used for
the authentication part instead of the embedded browser (WebView2) because
the latter doesn't work anymore with Google Authentication since a recent
change on their side.
In order for the authentication mechanism to work with the local browser, a
specific port (64646) on your PC is opened on startup of NRatings to listen
to incoming authentication callbacks. In 99.9% of the cases this port is
probably free but maybe you'll need to check if you have any other specific
application running on your PC that is using this port. Try doing a port
scan on your PC to see which ports are in use.
Let me know your findings
On Wed, 18 Aug 2021 at 00:24, JRFranklin62 @.***> wrote:

As the subject line states, I cannot seem to get the app to open anymore.
I've never had an issue before and have the most current version of the
program (v5.1.6). Will this be fixed soon?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/64Soft/NRatings.Client/issues/23, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA3NCTHRU2ROGXD3LITUAYDT5LOR5ANCNFSM5CKZAIPQ
.

Here is the error message that I get when I try to open NRatings.
[image: Image2.jpg]
It doesn't appear that my PC has any ports or any way to add them.
[image: Image1.jpg]
Is there a way for me to resolve this?
Thanks,
J.R. Franklin
On Wed, Aug 18, 2021 at 2:32 AM 64Soft @.***> wrote:

Do you get any error messages?
The thing that changed in 5.1.6 is that now the local browser is used for
the authentication part instead of the embedded browser (WebView2) because
the latter doesn't work anymore with Google Authentication since a recent
change on their side.
In order for the authentication mechanism to work with the local browser, a
specific port (64646) on your PC is opened on startup of NRatings to listen
to incoming authentication callbacks. In 99.9% of the cases this port is
probably free but maybe you'll need to check if you have any other specific
application running on your PC that is using this port. Try doing a port
scan on your PC to see which ports are in use.
Let me know your findings
On Wed, 18 Aug 2021 at 00:24, JRFranklin62 @.***> wrote:

As the subject line states, I cannot seem to get the app to open anymore.
I've never had an issue before and have the most current version of the
program (v5.1.6). Will this be fixed soon?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/64Soft/NRatings.Client/issues/23, or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AA3NCTHRU2ROGXD3LITUAYDT5LOR5ANCNFSM5CKZAIPQ
.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/64Soft/NRatings.Client/issues/23#issuecomment-900856535,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AVH26L7X3YQJC3VBIP3W363T5NHWTANCNFSM5CKZAIPQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
.

Cant seem to see the images you posted

They are jpgs.  Let me try this again.  I'll attach them instead of
inserting them into the email.
Hopefully, you;ll be able to open them now and see them.
Thank,
J.R.
On Wed, Aug 18, 2021 at 1:24 PM 64Soft @.***> wrote:

Cant seem to see the images you posted
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/64Soft/NRatings.Client/issues/23#issuecomment-901293535,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AVH26L27TIQYL3M2Z6LNM3DT5PUFZANCNFSM5CKZAIPQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
.

Nope they are still not attached. Please instead of going via email, attach them directly in the Github issue thread: https://github.com/64Soft/NRatings.Client/issues/23

Here is the error message I am getting:

It doesn't appear that my PC has any ports or any way to add them.:

Is there a way to fix this?
As you may remember, we are the guys from Ground Pounder Designs who provide all of the Whelen Modifieds for NR2003.  So I really need to figure out how to get NRatings working again.  As always, thanks for your awesome ratings tool and for any help you can provide me in getting this working.  I'm not the most PC savvy guy.
J.R. Franklin

Okay.  I posted the pics at the GitHub issue forum.
On Wed, Aug 18, 2021 at 5:54 PM 64Soft @.***> wrote:

Nope they are still not attached. Please instead of going via email,
attach them directly in the Github issue thread: #23
https://github.com/64Soft/NRatings.Client/issues/23
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/64Soft/NRatings.Client/issues/23#issuecomment-901454784,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AVH26L2WZHDF6MGTR6ML2KTT5QTYTANCNFSM5CKZAIPQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
.

Ok thanks now I see them.
Do you get this error the moment you start up NRatings or does it happen when you perform a specific action within NRatings ? Also, you might want to go into Windows Event Viewer and see if you find any errors there in the "Application" or "System" tabs under "Windows Logs" at the timestamp you get the error. If you do, please post the error details.

You could also try to download Advanced-Port Scanner https://www.advanced-port-scanner.com/ and with NRatings closed, do a scan of "localhost" of "All TCP ports" and see in the resulting list on the right if port 64646 is mentioned there. If it is, it means the port is in use by some other app and we'll need to investigate further to find out which one.

In the meantime, I'll try to start up a Windows 7 VM and see if I can reproduce the error. I do my development and testing on Windows 10.

I get that message immediately after clicking on the shortcut icon.

v5.1.7 is being released shortly and it will tell you on startup of NRatings if the required port is already occupied by another app. If that is the case, you'll need to investigate which one. You can use above mentioned port scanner or TCPView for that (https://docs.microsoft.com/en-us/sysinternals/downloads/tcpview)


💡 Feature Request - Provide steps to evaluate for 'cannot-be-validated-with-arg' items

Describe the solution you'd like
Updating the manual steps for the "ImpactedResources" tab on the WARA Action Plan spreadsheet can be quite time consuming.  I appreciate that there are links provided in the APRL site, but with 3,600 manual tests to validate for my WARA, going through the links to find how to validate is going to be quite consuming.  I recommend that we add the manual validation steps for each of the 'cannot-be-validated-with-arg' APRL items.
Describe alternatives you've considered
Only option I know of is to review the APRL links
Additional context
For example, look at this Restrict default access to Azure NetApp Files volumes.  Doing this for all the manual checks is going to take me a long time.

Hi @terrymandin, thanks for bringing this up. We are currently discussing potential options for this. However, as we currently have other high priority items, we don't have an anticipated ETA.


[Bug] [4.4.0] [mc:1.16.5] Furnaces sometimes stop working and allow item duplication

Sometimes furnaces stop smelting, destroying them (digging) drops there contents.

The process can now be repeated...
Installed mods and versions:
minecraft: 1.16.5
forge: 36.1.4 (1.16.5)
fastfurnace: 4.4.0 (1.16.4)
placebo: 4.4.1 (1.16.4)

others:
appliedenergistics2-8.3.0-alpha.2.jar
astralsorcery-1.16.4-1.13.9.jar
BiomesOPlenty-1.16.4-<IP_ADDRESS>1-universal.jar
chiselsandbits-0.3.4-RELEASE.jar
CodeChickenLib-1.16.4-<IP_ADDRESS>8-universal.jar
curios-forge-1.16.5-<IP_ADDRESS>.jar
Cyclic-1.16.5-1.2.2.jar
EnderStorage-1.16.4-<IP_ADDRESS>-universal.jar
envirocore-1.16.5-<IP_ADDRESS>.jar
envirotech-1.16.5-<IP_ADDRESS>.jar
ExtremeReactors2-1.16.4-2.0.23.jar
FastWorkbench-1.16.4-4.5.1.jar
ferritecore-2.0.3-forge.jar
ftb-backups-<IP_ADDRESS>.jar
JAMD-1.1.9-beta+mc-1.16.5.jar
jei-1.16.5-<IP_ADDRESS>.jar
jeiintegration_1.16.5-<IP_ADDRESS>.jar
JustEnoughResources-1.16.4-<IP_ADDRESS>.jar
observerlib-1.16.4-1.5.1.jar
Patchouli-1.16.4-50.jar
SereneSeasons-1.16.5-<IP_ADDRESS>-universal.jar
theoneprobe-1.16-3.0.7.jar
valkyrielib-1.16.5-<IP_ADDRESS>.jar
woot-1.16.5-<IP_ADDRESS>.jar
ZeroCore2-1.16.4-2.0.21.jar

java:
$ java --version
openjdk 11.0.10 2021-01-19
OpenJDK Runtime Environment (build 11.0.10+9-Ubuntu-0ubuntu1.18.04)
OpenJDK 64-Bit Server VM (build 11.0.10+9-Ubuntu-0ubuntu1.18.04, mixed mode, sharing)

after further investigation, this might not be a fastfurnace bug.
related: https://github.com/AppliedEnergistics/Applied-Energistics-2/issues/5069


A720695315750402 sellout insider entry point

Add entry point for sellout.insider

Coverage remained the same at 82.696% when pulling 9ef06082ce25a55debeab256c94c6881fb8fbd7a on Scandie:a720695315750402_sellout_insider_entry_point into fa1b3884bc2165078883995f0d674934c9545589 on openprocurement:master.


webgui: enable compression

Hi!
Noticed that the gui returns the content uncompressed, although the mod_deflate module is enabled.
I think lighty expects deflate.allowed-encodings to select a method from a match with the browser request.
after adding the option, content is returned gziped.
perhaps building lighty with --with-brotli would be good  but I did not find it in the ports..
ref. https://redmine.lighttpd.net/projects/lighttpd/wiki/Mod_deflate
thanks!
ps. I'm not sure about the need to add a cron for old\stale cache entries purge.

Depending on the characteristics of the site, one option is to prune or clear the cache each time lighttpd is started, or to do it on shutdown.
Unless the content being returned is larger and mostly static (not frequently changed), the difference between brotli and gzip is often small.  YMMV.

thanks!


Azure Support

Hi! 👋  I'd like to help in adding support for running pachyderm on Microsoft Azure ☁ . From my understanding, in addition to deploying a standard k8s cluster, what will be needed is:

[x] data disk for rethinkdb
[x] azure storage implementation similar to that of google
[x] pachctl command line arguments updates
[x] doc updates
[ ] data disk for etcd, should be identical to the situation for rethink
[x] make target for easily spinning up the cluster, similar to make amazon-cluster

Let me know if I'm missing anything else, and how best to go about adding this support. Thanks.
--
Updated above list to reflect the new comments below.

Hey @jpoon! This would be great and we're here to help! I think that basically covers it!

Really excited to finally get Azure support :)
Two very minor additions:

[ ] data disk for etcd, should be identical to the situation for rethink
[ ] make target for easily spinning up the cluster, similar to make amazon-cluster

In terms of how best to go about adding this support, I'd start with the storage driver. That's probably the most interesting bit. From there we should be able to get something up and running to test and start building to other parts.

data disk for etcd, should be identical to the situation for rethink

is this separate of the etcd cluster needed for kubernetes?

Yes, Pachyderm currently uses etcd and rethink for different things because they have different consistency models. Unfortunately k8s won't give applications access to the internal etcd cluster because it would be a potential security hole.
We're working on getting rid of etcd but currently we still need it.

How do I deploy my modified pachd image to my k8s cluster?
I tried:

make docker-build to build the necessary docker images on my local dev machine
pachctl deploy microsoft (... and some other flags I added to the cmd line ...) --dev which makes it use a pachyderm/pachd:local image

But the pods fail to pull said image. Do I need to setup my own docker registry?

Yup registry is needed.
Steps needed to get it working would be:
docker tag pachyderm/pachd:local jpoon/pachd:local
docker push jpoon/pachd:local
pachctl deploy microsoft --dry-run --dev | sed 's/pachyderm\/pachd/jpoon/pachd' | kubectl create -f -

Sorry that's a bit awkward.

My code changes to add Azure storage support worked on the first try which makes me suspicious. Are there automated tests I can run against my deployment?

Yes, you can run them with go test ./src/server.
They can be a bit flaky if you try to run them all at once so you may want to play with how much parallelism the tests have.

With https://github.com/pachyderm/pachyderm/pull/974, that should be the last of it. Looking at the kubernetes manifest, the etcd services are using a hostpath volume so no further changes are needed there.

Closed with https://github.com/pachyderm/pachyderm/pull/974

@jdoliner Awesome possum. When should we expect the next patch release? As mentioned on the email, it'd be great to have one with these bits prior to Nov. 7 so no rush.

Awesome possum indeed :). Should have the release out early next week.


Remove test that crashes ArtNet receiver

GAH.
I did not intend to check in ArtNet until I'd run it under fire, but I got distracted.  :-/

+1
On Fri, May 4, 2018 at 6:18 AM Tom Swirly<EMAIL_ADDRESS>wrote:

GAH.
I did not intend to check in ArtNet until I'd run it under fire, but I got
distracted. :-/
You can view, comment on, or merge this pull request online at:
https://github.com/ManiacalLabs/BiblioPixel/pull/789
Commit Summary

Remove test that crashes ArtNet receiver

File Changes

M bibliopixel/control/artnet.py
https://github.com/ManiacalLabs/BiblioPixel/pull/789/files#diff-0
(1)

Patch Links:

https://github.com/ManiacalLabs/BiblioPixel/pull/789.patch
https://github.com/ManiacalLabs/BiblioPixel/pull/789.diff

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ManiacalLabs/BiblioPixel/pull/789, or mute the thread
https://github.com/notifications/unsubscribe-auth/AA6a6hAKzCK5OmQ9jEb4Noah_60htbH2ks5tvCsOgaJpZM4Tya5D
.


Splash screen that asks if player wants to start new game or load from ID on launch

There's been a lot of times where I wanted to load a game, but had to regenerate a prompt first. This adds a load_new_story() function to story_manager.py that will initialize the story manager with a story from an ID.
It also adds a splash screen at the beginning to ask whether the user wants to load or start new story.

This is a great idea!


The tsconfig that ships with @pkg-tools/create-pkg ignores the pkg.config.ts

This means that you don't get typeahead/autocomplete in your editor. The solution direction I have in mind involves forking the config into two, 1 targeting the src/ and 1 targeting the pkg.config.ts.
This could be implemented as a tsconfig.json that includes src/, and  references a tsconfig.node.json that includes the pkg.config.ts exlusively.

In general, this is probably a cleaner (albeit slightly more verbose) pattern that can also be used in the examples.


Defaults to bottom of the page on page change

Steps to reproduce the issue:

Navigate to https://stenciljs.com/resources
Scroll to the bottom
Click the PWA link in the header

Observe the scroll position is not reset. You're at the bottom of the pwa page. FWIW, this behavior seems inconsistent to me. I cannot pinpoint exactly what the problem is, but I also haven't really tried 😆
Thanks,
Dan

Hmm, can't reproduce.  What browser are you using?

Every browser. Here is a video of the repro case.
https://drive.google.com/open?id=1l4pRCQP_BjTA9R-DlTsmJnYQlGmfoQ9L
Thanks,
Dan

This issue was fixed with PR #49.


Build Crashes when use function Import scene

I'm trying to import .ply at runtime, it is working in editor but when in build it crashes on import scene function. Can you please look into this?
Thank you.

Hi. Please provide the .ply file sample

Git don't support .ply upload so here is drive link for the file [https://drive.google.com/file/d/1xCnmKiDZ_WvttDpwUdiqS1s9pKdQELsX/view?usp=sharing]
Btw I tried with different files but it shows same result.

here  is the result.
Works without issue.
two possible reasons for your issue:
1- you provide the incorrect path
2- the path that you have needs administrator access
3- you are importing it wrong


One more

Please provide details about your commit.

This one is another example of stativ website made on HTML,CSS and JS which is a webpage.


HBASE-22231 Removed unused and '*' import

Targeted for branch-2

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
25
Docker mode activated.

_ Prechecks _

+1
hbaseanti
0
Patch does not have any anti-patterns.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 64 new or modified test files.

_ branch-2 Compile Tests _

0
mvndep
9
Maven dependency ordering for branch

+1
mvninstall
242
branch-2 passed

+1
compile
175
branch-2 passed

+1
checkstyle
169
branch-2 passed

+1
shadedjars
240
branch has no errors when building our shaded downstream artifacts.

+1
findbugs
333
branch-2 passed

+1
javadoc
126
branch-2 passed

_ Patch Compile Tests _

0
mvndep
15
Maven dependency ordering for patch

+1
mvninstall
230
the patch passed

+1
compile
175
the patch passed

+1
javac
175
the patch passed

+1
checkstyle
22
hbase-common: The patch generated 0 new + 11 unchanged - 3 fixed = 11 total (was 14)

-1
checkstyle
33
hbase-client: The patch generated 2 new + 203 unchanged - 17 fixed = 205 total (was 220)

-1
checkstyle
74
hbase-server: The patch generated 4 new + 280 unchanged - 97 fixed = 284 total (was 377)

+1
checkstyle
20
hbase-mapreduce: The patch generated 0 new + 91 unchanged - 11 fixed = 91 total (was 102)

-1
checkstyle
17
hbase-it: The patch generated 1 new + 90 unchanged - 7 fixed = 91 total (was 97)

+1
checkstyle
17
hbase-rest: The patch generated 0 new + 15 unchanged - 2 fixed = 15 total (was 17)

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedjars
235
patch has no errors when building our shaded downstream artifacts.

+1
hadoopcheck
479
Patch does not cause any errors with Hadoop 2.7.4 or 3.0.0.

+1
findbugs
375
the patch passed

+1
javadoc
116
the patch passed

_ Other Tests _

+1
unit
158
hbase-common in the patch passed.

+1
unit
195
hbase-client in the patch passed.

-1
unit
7520
hbase-server in the patch failed.

+1
unit
1525
hbase-mapreduce in the patch passed.

+1
unit
61
hbase-it in the patch passed.

+1
unit
241
hbase-rest in the patch passed.

+1
asflicense
150
The patch does not generate ASF License warnings.

13143

Reason
Tests

Failed junit tests
hadoop.hbase.client.TestAsyncTableGetMultiThreaded

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-177/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/177

Optional Tests
dupname  asflicense  javac  javadoc  unit  findbugs  shadedjars  hadoopcheck  hbaseanti  checkstyle  compile

uname
Linux 21f15b3a6b5f 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux

Build tool
maven

Personality
/testptch/patchprocess/precommit/personality/provided.sh

git revision
branch-2 / 3a1dc8ec82

maven
version: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z)

Default Java
1.8.0_181

findbugs
v3.1.11

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-177/1/artifact/out/diff-checkstyle-hbase-client.txt

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-177/1/artifact/out/diff-checkstyle-hbase-server.txt

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-177/1/artifact/out/diff-checkstyle-hbase-it.txt

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-177/1/artifact/out/patch-unit-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-177/1/testReport/

Max. process+thread count
5460 (vs. ulimit of 10000)

modules
C: hbase-common hbase-client hbase-server hbase-mapreduce hbase-it hbase-rest U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-177/1/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
26
Docker mode activated.

_ Prechecks _

+1
hbaseanti
0
Patch does not have any anti-patterns.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 64 new or modified test files.

_ branch-2 Compile Tests _

0
mvndep
20
Maven dependency ordering for branch

+1
mvninstall
222
branch-2 passed

+1
compile
174
branch-2 passed

+1
checkstyle
169
branch-2 passed

+1
shadedjars
237
branch has no errors when building our shaded downstream artifacts.

+1
findbugs
335
branch-2 passed

+1
javadoc
116
branch-2 passed

_ Patch Compile Tests _

0
mvndep
13
Maven dependency ordering for patch

+1
mvninstall
221
the patch passed

+1
compile
164
the patch passed

+1
javac
164
the patch passed

+1
checkstyle
21
hbase-common: The patch generated 0 new + 11 unchanged - 3 fixed = 11 total (was 14)

+1
checkstyle
31
hbase-client: The patch generated 0 new + 202 unchanged - 18 fixed = 202 total (was 220)

+1
checkstyle
65
hbase-server: The patch generated 0 new + 277 unchanged - 100 fixed = 277 total (was 377)

+1
checkstyle
18
hbase-mapreduce: The patch generated 0 new + 91 unchanged - 11 fixed = 91 total (was 102)

+1
checkstyle
14
hbase-it: The patch generated 0 new + 88 unchanged - 9 fixed = 88 total (was 97)

+1
checkstyle
14
hbase-rest: The patch generated 0 new + 15 unchanged - 2 fixed = 15 total (was 17)

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedjars
229
patch has no errors when building our shaded downstream artifacts.

+1
hadoopcheck
465
Patch does not cause any errors with Hadoop 2.7.4 or 3.0.0.

+1
findbugs
378
the patch passed

+1
javadoc
114
the patch passed

_ Other Tests _

+1
unit
156
hbase-common in the patch passed.

+1
unit
193
hbase-client in the patch passed.

+1
unit
7974
hbase-server in the patch passed.

+1
unit
977
hbase-mapreduce in the patch passed.

+1
unit
70
hbase-it in the patch passed.

+1
unit
252
hbase-rest in the patch passed.

+1
asflicense
203
The patch does not generate ASF License warnings.

13021

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-177/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/177

Optional Tests
dupname  asflicense  javac  javadoc  unit  findbugs  shadedjars  hadoopcheck  hbaseanti  checkstyle  compile

uname
Linux 0833eda827a7 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux

Build tool
maven

Personality
/testptch/patchprocess/precommit/personality/provided.sh

git revision
branch-2 / c9b283ec2e

maven
version: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z)

Default Java
1.8.0_181

findbugs
v3.1.11

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-177/2/testReport/

Max. process+thread count
5480 (vs. ulimit of 10000)

modules
C: hbase-common hbase-client hbase-server hbase-mapreduce hbase-it hbase-rest U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-177/2/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.


rework/fix callback handlers

The previous version of this was a little convoluted - this feels much more easy to reason about .
Tested with react-sdk demo repo

Current dependencies on/for this PR:

latest

PR #124   👈

This comment was auto-generated by Graphite.


Add ChainerX test to FunctionTestCases

This PR adds testcases to input ChainerX arrays, even if the tested functions don't have ChainerX specific implementations.  @niboshi suggested the change.

Thanks.
Jenkins, test this please.

Jenkins CI test (for commit c31ca7bec42383dc17be9488ce525253e1815498, target branch master) failed with status FAILURE.
(For contributors, please wait until the reviewer confirms the details of the error.)

LGTM


Handling of obsolete mechanical scripting APIs

When running obsolete scripting commands in the Mechanical GUI's scripting window, a warning message is printed. We may want to write out a warning in PyMechnical too (both remote and embedded)

#409 partially fixes this (embedding only, and only version 24.1+ warnings are fixed)

assigning to Sami in case this could also be done for remote

Adding @dipinknair to track this for the remote server. This isn't a high priority


build: Add code coverage and badge

We may have to merge this one "blind" because I don't think it will have any effect until it's in the upstream repo.
Closes #64 .

@fmorency Hey, that worked! I should go buy a lottery ticket. ☘


Refactor of "reductor" to "reducer"

In preparation for contributions of algorithms, I thought it best to refactor the name

I agree!  I always wondered why the name was non-standard english 😄  Though I suppose reductor is still a word, but usually associated with chemical reductions.


Back up databases to S3-compatible storage

There are many options for backing up SQLite to S3. The one that most directly focuses on continuous streaming backups and restore (as opposed to maintaining read replicas or multiple writers, etc.) is litestream, but there are others like libSQL's bottomless replication work-in-progress.
We discuss alternatives and why we aren't (yet) considering them below, but for now, we will go with simple scheduled backups.

In ayb.toml, you will configure: S3-compatible storage details, frequency of backup, and how many backups to keep before pruning old ones.
ayb will have a Snapshot data model that can store details on database snapshots (when we took it, what the hash of the database directory was, whether it was a scheduled or manual snapshot, the path to the backup on S3-compatible storage, etc.)
On the ayb server, we'll run a periodic task to check if a database has changed recently (based on the hash of its database directory, but perhaps there's a way to query for the most recent transaction ID instead) and if it has, create a new snapshot.
The backup in SQLite will happen via VACUUM, which will reduce file size and won't potentially run forever under high write loads like the online backup API.
The advantages of this approach are:

Simplest implementation path.
No dependence on other tools that are being rapidly developed and don't have documented APIs (yet).
Due to the simplicity and generality of the approach, it shouldn't be hard to apply it to other database types (e.g., DuckDB) that have less developed stream-the-WAL-to-S3 functionality.
I suspect that in practice, most databases will be write-dormant most of the time given the target use cases, so this might require less overhead than tools that constantly monitor a database log for changes.

The disadvantages of the approach are:

Potentially more work per backup: if you only update a row, we still VACUUM and export the entire database.
Periodic snapshots have a larger window for data loss on the order of tens of minutes to an hour, whereas shipping the write-ahead log typically results in only seconds of data loss.
Related to the previous reason, implementing things like multi-node ayb support with automatic failover will be difficult in the future in cases where a node becomes inaccessible and you're not sure how many minutes/tens of minutes of backups are unsynced.

The plan

[x] Ask on SQLite forum about how to find the most recent txid/modification: https://sqlite.org/forum/forumpost/2798df4be8. The approach to beat is to have each DB in its own directory (SQLite spans multiple files), and use filesystem modified times / hashes of all files in the directory to determine changes.
[x] Refactors to directory structure that support snapshots and SQLite-in-directory (https://github.com/marcua/ayb/pull/311)
[x] Refactors to allow dangerous/un-isolated requests to complete the backups (requires ATTACH) (https://github.com/marcua/ayb/pull/312)
[x] Backups to the local filesystem (https://github.com/marcua/ayb/pull/316)

[x] periodic_snapshots configuration option with details like snapshot frequency and quantity before pruning, as well as S3-compatible connection information
[x] Daemon/worker thread to periodically run backups using VACUUM approach

[x] S3 management to list/create snapshots (https://github.com/marcua/ayb/pull/317)

[x] Snapshot model with: primary key, database foreign key, creation date, snapshot ID (database hash/transaction ID, indexed), automatic/manual enum, [future not now: nullable note field for manual snapshot]

[x] end-to-end tests using localstack as stand-in for S3 (https://github.com/marcua/ayb/pull/358)
[x] Endpoints / client support for listing snapshots (#385)
[x] Endpoints / client support for restoring from snapshots (#405)
[ ] Remove old snapshots in list beyond the retention quantity/period.
[ ] Optimizations based on modified time and hashes to reduce snapshots

[ ] get fs::metadata of each file in the dir, call modified() on result, sort the times so it's stable, take max or shasum them t
ogether.

[ ] Optimizations

[ ] Consider zstd instead of gzip for compressing snapshots
[ ] Can we replace walkdir with std::fs::read_dir

[ ] Clean up: Initialize a HostedDb that has a SQLite / DuckDB implementation. Push query/backup logic into that. Consider doing this on an InstantiatedDatabase directly.
[ ] Clean up: If we don't need chrono for datetimes in hindsight, we can remove it from our snapshot models and sqlx features. (grep for chrono).
[ ] Clean up TODOs, especially in src/server/snapshots.rs

Alternatives we considered

Knowing nothing,  litestream seemed like the most obvious option. The litestream documentation isn't clear on how many databases worth of replication it's designed for, but given that adding a database requires a configuration file update and a litestream restart, it was more likely designed around single digit to tens of databases. I roughly confirmed this in a discussion in the libstream board. The discussion raised the idea of waiting on a third party to open source one of their services for solving a similar problem, or calling Go from Rust, which we can consider in the future as those projects mature.
In researching this work, the libSQL project, and in particular
libsql-server/sqld and its bottomless replication support look promising. The bottomless documentation and commit history suggests its under quite a bit of heavy development. After all of my research, my intuition is that if the team ships a stable bottomless library, we'll be able to use it directly from ayb (both are in Rust) to solve the stream-the-write-ahead-log-to-and-recover-from-S3 problem.

Both projects, but in particular the second one, seem promising for the future, but weren't directly available for import at the time of writing, so for this v1 of backup/restore, we'll fall back to a simpler solution like scheduled backups.

Asked questions to understand litestream feasibility here: https://github.com/benbjohnson/litestream/discussions/561

Based on the response to https://github.com/benbjohnson/litestream/discussions/561, it looks like there isn't yet a programmatic way to add databases to replicate or restore, and that to use litestream for now, we'd have to update a growing .yml file and restart a service. I'd like to avoid that if possible, but keep watching the project.
I'm now thinking through two different paths

Create a Snapshot data model that can store details on database snapshots (when we took it, what the hash of the database directory was, etc.) and then build scheduled backups as well as manual snapshot functionality around it. This would work well for most cases and be the simplest implementation path. The two downsides are: 1) Periodic snapshots have a larger window for data loss on the order of tens of minutes to an hour, and 2) For the first reason, implementing things like multi-node ayb support with failover will be difficult in the future.
Look into libsql-server/sqld and its bottomless replication support (the docs have a broken link, but it claims to replicate the SQLite file and WAL to S3-compatible storage)

Some updates

.sha3sum --schema is no longer the approach to beat --- it only works in the SQLite CLI. As a result, my current approach will be to have each DB in its own directory (SQLite spans multiple files), and use filesystem modified times / hashes of all files in the directory to determine changes.
I realized I was overcomplicating the model by storing snapshot data in the ayb DB when the metadata already lived in S3. For now, I moved the snapshot model into a snapshot manager module, and S3 will be the source of truth (metadata can live in file headers). The only downside I can imagine is a performance one, which we should address if we encounter performance issues.
This project has taken too long (it's a complicated one to revisit 2 hours a week, especially solo), and I plan to chunk it into tinier pieces to help with incremental progress/to reduce mental overhead

Refactors to directory structure that support snapshots and SQLite-in-directory
Refactors to allow dangerous/un-isolated requests to complete the backups (requires ATTACH)
Backups to the local filesystem
S3 management to list/create snapshots
S3 management to retrieve snapshots
Refactor to run databases in isolation as mini-services rather than isolating on each request
Bootstrapping logic to check if recovery is necessary (database not available locally when you start the mini-service)
Optimizations based on modified time and hashes to reduce snapshots

Put/list appears to be working! For next time

Clean up code (some S3 config and path syntax updates) and commit.
Start on read/recovery path
e2e testing
Clean up running to-do list in snapshots.rs (it's dated now) and introduce any optimizations listed there

Code is cleaned up!

Test force_path_style=true/false
Address problem with localstack creating zombie python3 processes
Make localstack hooks create a bucket or two for testing (or push this into integration tests)
Start on read/recovery path
e2e testing
Clean up running to-do list in snapshots.rs (it's dated now) and introduce any optimizations listed there

Localstack crash issue: the python3 ./bin/localstack status services --format=json command is zombieing an increasing number of defunct python3 processes
Will keep investigating
root       77626  0.1  0.0      0     0 ?        Z    23:45   0:00 [python3] <defunct>
root       77677  0.1  0.0      0     0 ?        Z    23:45   0:00 [python3] <defunct>
root       77721  0.1  0.0      0     0 ?        Z    23:45   0:00 [python3] <defunct>
root       77783  0.1  0.0      0     0 ?        Z    23:46   0:00 [python3] <defunct>
root       77809  0.1  0.0      0     0 ?        Z    23:46   0:00 [python3] <defunct>
root       77838  0.1  0.0      0     0 ?        Z    23:46   0:00 [python3] <defunct>
root       77862  0.1  0.0      0     0 ?        Z    23:46   0:00 [python3] <defunct>
root       77892  0.2  0.0      0     0 ?        Z    23:47   0:00 [python3] <defunct>
root       77918  0.2  0.0      0     0 ?        Z    23:47   0:00 [python3] <defunct>
root       77944  0.2  0.0      0     0 ?        Z    23:47   0:00 [python3] <defunct>
root       77970  0.2  0.0      0     0 ?        Z    23:47   0:00 [python3] <defunct>
root       77996  0.2  0.0      0     0 ?        Z    23:48   0:00 [python3] <defunct>
root       78023  0.2  0.0      0     0 ?        Z    23:48   0:00 [python3] <defunct>
root       78058  0.2  0.0      0     0 ?        Z    23:48   0:00 [python3] <defunct>
root       78087  0.3  0.0      0     0 ?        Z    23:49   0:00 [python3] <defunct>
root       78119  0.3  0.0      0     0 ?        Z    23:49   0:00 [python3] <defunct>
root       78145  0.3  0.0      0     0 ?        Z    23:49   0:00 [python3] <defunct>
root       78178  0.3  0.0      0     0 ?        Z    23:49   0:00 [python3] <defunct>
root       78207  0.4  0.0      0     0 ?        Z    23:50   0:00 [python3] <defunct>
root       78237  0.4  0.0      0     0 ?        Z    23:50   0:00 [python3] <defunct>
root       78265  0.4  0.0      0     0 ?        Z    23:50   0:00 [python3] <defunct>
root       78297  0.6  0.0      0     0 ?        Z    23:51   0:00 [python3] <defunct>
root       78330  0.7  0.0      0     0 ?        Z    23:51   0:00 [python3] <defunct>
root       78354  0.7  0.0      0     0 ?        Z    23:51   0:00 [python3] <defunct>
root       78377  1.0  0.0      0     0 ?        Z    23:51   0:00 [python3] <defunct>
root       78409  1.6  0.0      0     0 ?        Z    23:52   0:00 [python3] <defunct>
root       78439  2.6  0.0      0     0 ?        Z    23:52   0:00 [python3] <defunct>
root       78467  7.1  1.9  46100 39856 ?        S    23:52   0:00 python3 ./bin/localstack status services --format=json

Figured it out: a mere mortal would think they should start localstack with localstack start, but someone who knows The Truth will start it with localstack start -d --network ls. (The -d runs it in detached mode, less important, but the --network ls part fixed all of my Docker networking issues.)

Make localstack hooks create a bucket or two for testing (or push this into integration tests)
Test force_path_style=true/false
Start on read/recovery path
e2e testing
Clean up running to-do list in snapshots.rs (it's dated now) and introduce any optimizations listed there

Now that localstack/force_path_style is in place, the remaining list is

Read/recovery path
e2e testing
Clean up running to-do list in snapshots.rs (it's dated now) and introduce any optimizations listed there

Implementing e2e tests, leaving myself a to-do for the future

Run localstack outside of tests (script should create a sqlite-tests and postgres-tests bucket)
Configure ayb.tomls for sqlite/postgres to access their respective bucket in snapshot config
Top of test_snapshots should delete bucket/create bucket
Now that test can assume blank slate

Tests are starting to work --- we now delete the contents of the snapshot prefix we're going to be running tests on. Now it's time to validate the output (which contains timestamps, so requires regexes to match):

Make utils::testing::list_snapshots match against a regex
Finish basic "no snapshots exist, one exists after sleep, one still exists after sleep, update DB, two exist after sleep"
Start localstack in GH actions
Clean up PR
Next PR: restore from snapshot

Regex test of list_snapshots works!! Next up

More gracefully handle snapshot loop trying to access DB directory before it exists
Start localstack in GH actions
Clean up PR
Next PR: restore from snapshot

Wooohoo! Localstack now works in GH actions, and I cleaned up/standardized the e2e setup process and documented it. I submitted/reviewed/merged #385.
Next up: Functionality/endpoint for restoring from snapshot

Restore works in terms of pulling the file down from S3
But we want to be able to atomically swap the current database with the restored snapshot
using std::fs:rename doesn't work, as the destination has to be an empty directory (it's not)
it seems like the solution is a symlink

$db_path/current is a symlink to the current directory
$db_path/UUID is either a fresh database that was created, or a place where we move the restored snapshot
if current symlink exists, call std::fs::canonicalize on it to get its actual path, then create a new symlink with std::os::unix::fs::symlink / std::os::windows::fs::symlink_dir (will it overwrite the existing one like ln -snf does?), then remove the original (canonicalized) path.

#405 is shipped! Next up: hash to avoid unnecessary snapshots, and remove snapshots past the retention period!

Rough plan for hashing

Write a method to, given a directory, walk every file in it in a deterministic order (i.e., sort by filename), hash the file's contents, and hash the hashes together.
Compute a pre-hash
List available snapshots and exit early if a snapshot matches the pre-hash
Take a snapshot and compute a post-snapshot hash
Exit early if a snapshot matches the post-hash

The more I think about it, prehashing is problematic

if a database is actively being updated, the prehash might be of a different database entirely than the one in the snapshot
the prehash prevents wasted work (snapshotting and hashing) in case the database never changed, but we can accomplish that more cheaply by keeping local filesystem modification times, only snapshotting locally and hashing the first time we run the snapshotting logic in a machine/process
if we have there be only one hash (post-hash), we can change the identifier of the database to be its hash, which means we get the hash details in list_snapshots from S3 without having to issue subsequent head calls for metadata

So perhaps the path forward is

Remove pre-/post-hash headers
snapshot_id -> snapshot_hash, even in the name
Write a method to, given a directory, walk every file in it in a deterministic order (i.e., sort by filename), hash the file's contents, and hash the hashes together. Blake3 seems to be a cryptographic hash function that's faster than md5 and sha256.
Take a snapshot and compute a post-snapshot hash
Exit early if a snapshot in list_snapshots matches the hash
In a future PR, if snapshotting is really expensive, add modification metadata to the snapshotting loop so that we don't needlessly snapshot/hash

#417 introduces hash-based identification of snapshots. The current implementation is naive in that it still uploads/overwrites a snapshot if one already exists with the key. From here:

Take snapshot
Compute hash
Call list_snapshots
If hash exists in list_snapshots, don't upload new one
Update tests to ensure that modification date of that snapshot doesn't change even after waiting for another snapshot
Since we have a snapshot list, if list_snapshots.len > max_snapshots, prune the oldest snapshot
Update tests to ensure that we don't exceed the number of snapshots
In a future PR, if snapshotting is really expensive, add modification metadata to the snapshotting loop so that we don't needlessly snapshot/hash

#417 is shipped, and #418 implements the hashing/pruning optimizations above! Just have to finished up some tests for #418, and then can move on to zstd/walkdir optimizations.

gzip -> zstd is done with #424!
next up is largely code cleanups!

Wow! The code cleanups are largely complete thanks to #432 and what I think remains before the next version is to add user-facing documentation. It's been so long!

With an asynchrony issue addressed in #444 and a draft of documentation availabe in #446, all that's left is reviewing the docs, making any updates, shipping the docs, and releasing the latest ayb to complete this issue!


Fix on click edit mode states and cell suggestions

Manage the edit mode of the cell when clicking on a different cell, outside the cell and on the same cell.

Remove the contentEditable logic from the codeComponent since it was messing with the cell suggestion widget.

Fix smoke tests

This PR fixes #18299

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 48.239%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
14797

Relevant Lines:
29107

💛  - Coveralls


The underline animation when switching from File, Home, Share and Layout seems to have lost some animations compared to before while when you go in Settings the underline under Personalization, Preferences and About still is fine.

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS Version: [e.g. Windows 10 1903 18362.175]
App version: [v0.5.0]

Additional context
Add any other context about the problem here. Does this problem occur again after restarting the app?

Hello. this is a limitation with the current control for the Ribbon which we are using. I'm closing this since it's 90% likely that future enhancements to the Ribbon will address this.


Export altered images

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

7 of 15 (46.67%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-1.2%) to 66.812%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

snapbox.py
7
15
46.67%

Totals

Change from base Build<PHONE_NUMBER>:
-1.2%

Covered Lines:
153

Relevant Lines:
229

💛  - Coveralls


Cache lld binaries to avoid redundant rebuilds

Depends on https://github.com/apple/swift-sdk-generator/pull/34.
This proceeds with converting more of the generation steps to a query-based architecture. Now lld doesn't need to be rebuilt if its source directory and build options don't change. This is implemented with a new CMakeBuildQuery, which takes source directory, and build options as arguments, and returns a freshly-built binary that can be cached by the generator engine.
Also renamed DownloadQuery to DownloadArtifactQuery and moved it to a separate file.

@swift-ci test


How CCF be calculated for gene level

Hi,
Thanks for the tool. In the paper, each figure was shown as gene level evolutional signature. And the ccf was usually calculated in mutation level, how could it trans into gene level and use this tool (one gene might have multiple mutations and thus have many CCF values)?
Thanks

Dear @wt12318,
Thank you for the interest in our framework.
In the paper if I get had multiple mutations, we considered for each patient the one at higher CCF (occurring earlier).
All the best,
Daniele Ramazzotti


fix(Popper): use useLayoutEffect

This PR fixes issue with crazy scrolling when Popup is used with popupFocusTrapBehavior.
https://stardust-ui.github.io/react/components/popup#usage-on-with-focus-trap

Warnings

:warning:
There are no updates provided to CHANGELOG. Ensure there are no publicly visible changes introduced by this PR.

  Generated by :no_entry_sign: dangerJS


Adds LoadFrom to IAssemblyLoader

Makes it possible to load single assembly from path.
Make private method in AssemblyLoader public.
Rename to LoadFrom to align with LoadAllFrom

@david-driscoll @filipw Added check for null/whitespace path to avoid trowing exception.


Support of ExtensionField in proof generation

I'm trying to run the keccak_air prover example on an ExtensionField:
type Val = Goldilocks;
type Domain = BinomialExtensionField<Goldilocks, 2>;
type Challenge = BinomialExtensionField<Goldilocks, 2>;

I'm seeing a type mismatch error which is required by a bound in QuotientMmcs.

I think another blocker is TwoAdicField is not yet implemented for BinomialExtensionField.

Closed by #160


Update python version

I changed the API such that users can directly input desired frequencies to  compute the superlet transform, without first having to convert to scales. Additionally I separated out the test script, which is now also a proper test, i.e. can be run with pytest.

I still have one question though: the Scalogram trying to recapitulate Figure 3 of the NatComm publication looks as expected:

To make a proper test of the Python superlet function I checked the amplitudes of the scalogram along the harmonic frequencies (dashed lines above) against the true signal amplitudes:

The true amplitude is 10, so for the 20Hz signal there is a slight overshoot, and for the faster 60Hz signal the amplitude can only be recovered to around 80%. Is that to be expected?

Hello,
Yes, that is to be expected. The frequency neighbors are what's causing
this behavior due to constructive or destructive interferences between
the frequency components.
For example, this is what I'm getting with FASLT with o=1:30 and c1=3, on
the signal from the MATLAB variant.
[image: image.png]
Tuning the parameters more thoroughly could get you a better result, though.
If all's well, I will merge your pull request later today.
Cheers,
Harald
On Fri, 17 Feb 2023 at 13:45, tensionhead @.***> wrote:

I still have one question though: the Scalogram trying to recapitulate
Figure 3 of the NatComm publication looks as expected:
[image: test_scalogram]
https://user-images.githubusercontent.com/10595420/219645220-5e66a65c-861c-4c20-a110-ce934a1bacd0.png
To make a proper test of the Python superlet function I checked the
amplitudes of the scalogram along the harmonic frequencies (dashed lines
above) against the true signal amplitudes:
[image: scalogram_profiles]
https://user-images.githubusercontent.com/10595420/219643708-ec3abaee-d130-4a8a-a219-9fda10b5f19f.png
The true amplitude is 10, so for the 20Hz signal there is a slight
overshoot, and for the faster 60Hz signal the amplitude can only be
recovered to around 80%. Is that to be expected?
—
Reply to this email directly, view it on GitHub
https://github.com/TransylvanianInstituteOfNeuroscience/Superlets/pull/2#issuecomment-1434525022,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AR3QLFHEOFZUR47KZKPISODWX5QBTANCNFSM6AAAAAAU7KNLDE
.
You are receiving this because you are subscribed to this thread.Message
ID: <TransylvanianInstituteOfNeuroscience/Superlets/pull/2/c1434525022@
github.com>

--
Harald Bârzan, PhD
Transylvanian Institute of Neuroscience (TINS),
Department of Experimental and Theoretical Neuroscience,
Str. Ploiesti 33
https://maps.google.com/?q=Str.+Ploiesti+33&entry=gmail&source=g, 400157
Cluj-Napoca, Romania
Email: @.***

Ah ok I see, thx for the explanation! What about the 2nd (in time) localized harmonic though, it does not have a frequency neighbor yet shows very similar amplitude response? The image you posted sadly does not show here so far..

Oh, yeah! Forgot to mention that. It wholly depends on the parameters - if
you use a very large upper order (say you do 1:30), then most of your
wavelets will be larger than the target packet, thereby diminishing the
combined amplitude response. Consider it like using an FFT window - if you
fill that window completely with a sine wave, you will get the desired
amplitude. If you only fill it by 50%, you will get an amplitude that is
only 50% of the sine's amplitude. The same thing (roughly) happens with the
superlet as well.
The image I sent previously roughly looks like what you've obtained in
python. Here I'm attaching another one (this time a file, maybe it works
now, dunno what's wrong with gmail these days), where I use c1=3 and an
order range of 5:10. This doesn't solve the interference problem (because
it's a problem that's related to the signal itself, not to the estimation
technique), but the second packet does not overshoot or undershoot the
target (sine amplitude is 1 so the estimated power should read 0.5).
On Fri, 17 Feb 2023 at 15:58, tensionhead @.***> wrote:

Ah ok I see, thx for the explanation! What about the 2nd (in time)
localized harmonic though, it does not have a frequency neighbor yet shows
very similar amplitude response? The image you posted sadly does not show
here so far..
—
Reply to this email directly, view it on GitHub
https://github.com/TransylvanianInstituteOfNeuroscience/Superlets/pull/2#issuecomment-1434686921,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AR3QLFBMHHLXIBIDSRH5OCLWX57VNANCNFSM6AAAAAAU7KNLDE
.
You are receiving this because you commented.Message ID:
@.***
com>

--
Harald Bârzan, PhD
Transylvanian Institute of Neuroscience (TINS),
Department of Experimental and Theoretical Neuroscience,
Str. Ploiesti 33
https://maps.google.com/?q=Str.+Ploiesti+33&entry=gmail&source=g, 400157
Cluj-Napoca, Romania
Email: @.***

Oh yes, that makes a lot of sense! So it's the distribution of Wavelets of different cycle numbers within the superlet which determines the performance, longer wavelets won't fully cover the packet and hence penalize the weighted amplitudes .  As for FASLT we always use a linear scaling (so uniform distribution?!) of the order, tweaking the lower and upper limits of the cycle numbers does the trick as you described!
I guess sending images via email via github does not really work, I always just drag and drop them here in the github web-interface. But ok, the important thing is we have coinciding results, thx for your quick check :+1:


lens Lens Error EACCES permission denied

Describe the bug
Running lens from packages shows:
Could not load extensions: EACCES: permission denied. unlink '/home/andrey/.config/Lens/extensions/cli-lens-extension/cli-lens-extension-0.13.0.tgz'
Steps To Reproduce
Steps to reproduce the behavior:

Install lens as non-root user via configuration.nix and nixos-rebuild switch --upgrade
Run lens as non-root user
Get an error window

Expected behavior
Lens runs without an error above
Screenshots

Additional context
Notify maintainers
@dbirks
@RossComputerGuy
Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.5.6, NixOS, 23.05 (Stoat), 23.05.4172.bd1cde45c778`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.13.5`
 - channels(root): `"nixos-23.05, nixos-hardware, nixos-unstable"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

related https://github.com/NixOS/nixpkgs/issues/123688

I've ran into this issue before, not relating to nixpkgs I think, the solution was to delete the extensions directory. I think this is a bug with Lens itself and not our packaging.

Can anyone using the nixpkgs version of Lens confirm whether or not this issue is still present?
After https://github.com/NixOS/nixpkgs/pull/242911 this seems to be fixed and if others note the same I'll happily open a PR to backport this to 23.05.

Fixed in https://github.com/NixOS/nixpkgs/pull/242911 which is now available on the 23.11 release channel