The u-law encoding is badly mapping values from the [-1,1] range to [0,255]

The u-law encoding was badly mapping values from the [-1,1] range to [0,255].
The correct equation to do this is (tested):
return tf.cast(((signal + 1) * mu) / 2, tf.int32)

This modification has not passed the test_mu_law.py continuos-integration test but I think this is just because I have modified precisely the encoding u-law function. I'm sure my change is right even with this continuous integration fail.

If you make a change, you need to make sure the tests pass or fix whatever bug they may have. However, I think with your change the value 255 will never appear. Integer casting does not round but floor, that's why we add 0.5. Switching /2 and *mu around doesn't have any effect.

@lemonzi, the general equation to change from a range to another is:
NewValue = (((OldValue - OldMin) * (NewMax - NewMin)) / (OldMax - OldMin)) + NewMin
The decode u-law function do it right in this way, but the encode u-law function do it badly because the general equation becomes in:
return tf.cast(((signal + 1) * mu) / 2, tf.int32)
The one we use now is wrong and so when you decode you have an error because you are enfoding in a range different of what we use in the encoding time.

@Zeta36 If there's an error in what we are doing now, could you express it in the form of a test in test/test_mu_law.py?
I think we should probably have a "round trip" test that, for every possible quantization value (0..256) or (0...511) decodes to a float, then encodes back to an int, and asserts that the quantized value we started with is the one we got back. It would be similar to testEncodeDecode(), except it would first decode then encode, so it would be testDecodeEncode(). It would cover every possible quantized value, whereas the testEncodeDecode that we have now doesn't necessarily do that.  I think the problem you are describing would show up in that test failing.

I think to make a test case is a waste of time in this case. The question is easy:
NewValue = (((OldValue - OldMin) * (NewMax - NewMin)) / (OldMax - OldMin)) + NewMin
Decoding fase [0,255] to [-1,1]:
NewValue = (((OldValue - 0) * (1- (-1))) / (255- 0)) + (-1) = (OldValue*2)/255 - 1
And we are doing: signal = 2 * (casted / mu) - 1 and it's right in this way.
Encoding fase [-1,1] to [0,255]:
NewValue = (((OldValue - (-1)) * (255- 0)) / (1- (-1))) + 0 = ((OldValue+1)*255)/2
And we are doing (I don't know why): (signal + 1) / 2 * mu + 0.5 This is wrong.
the right way is juts: (signal + 1) * mu) / 2
If you don't want to make this clear correction don't do it but the code is encoding and decoding in different ranges.
Regards,
Samu.

@Zeta36  Well I haven't looked into the particulars yet.
If your implementation corrects the problem, and the tests are failing with your implementation, then the tests must be expecting the wrong thing. They are going to have to be fixed so that they expect the correct thing. And I think that needs to be done before we merge. We don't want a master branch with failing tests.
You probably already know, but just in case: you can run the tests locally with
python test/test_mu_law.py

@jyegerlehner , we just have to change line 17:
signal = (signal + 1) / 2 * mu + 0.5
by this:
signal = ((signal + 1) * mu) / 2
And the test will be passed. It's a silly thing.
Regards.

@Zeta36 Well in that case, it shouldn't take you long :)
While I'm not a test-driven development fanatic, when one encounters a problem like this I think it's the obvious way to go.

I have no grants to PR easily in the test case folder. Please do it you for me.
Thanks.

You can change wavenet/ops.py but you can't change test/test_mu_law.py?

But OK I'm looking at it now.

I can't believe so many troubles for this minor and clear bug correction lol.

The encoding function doesn't only change the range, but also quantizes the
value to an integer. The 0.5 is for the quantization, not for the scaling.
It has to appear either when encoding (mid-tread) or decoding (mid-riser).
We use mid-tread because it can encode 0, which makes things easier.
https://en.m.wikipedia.org/wiki/Quantization_(signal_processing)
On Tue, Oct 18, 2016, 14:28 jyegerlehner<EMAIL_ADDRESS>wrote:

But OK I'm looking at it now.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ibab/tensorflow-wavenet/pull/155#issuecomment-254596453,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADCF5ltaEAyzcXd8rsVi4xFcNpIPTibZks5q1Q-XgaJpZM4KZqmt
.

@lemonzi, other implementations of WaveNet in Github doesn't do that +0.5 and there are working better than us.
Anyway.

They may be doing other things better than us, but this is electrical
engineering 101.
On Tue, Oct 18, 2016, 14:47 Samuel Graván<EMAIL_ADDRESS>wrote:

@lemonzi https://github.com/lemonzi, other implementations of WaveNet
in Github doesn't do that +0.5 and there are working better than us.
Anyway.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ibab/tensorflow-wavenet/pull/155#issuecomment-254602216,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADCF5lXiger0ogr-pogdL9poQ8glFJZOks5q1RRBgaJpZM4KZqmt
.

Ok, but it is computer engineering that you must encode and decode in the same range, and we are not doing it right now because of that 0.5.
You cannot encode in this way:
signal = (signal + 1) / 2 * mu + 0.5
and later decode like this:
signal = 2 * (signal / mu) - 1
If you wanna stay with the +0.5 it's alright, but have to change then the decode method. The noise we are hearing in all the generation outputs until now is due to this precisely.
Anyway, I'm not gonna fight.
Good luck, you can close this PR.

@Zeta36 OK here's what I did. From the master branch without fixing or changing anything in our implementation, I added the following test to TestMuLaw in test_mu_law.py:
    QUANT_LEVELS=256
    def testDecodeEncode(self):
        # generate every possible quantized level.
        x = range(QUANT_LEVELS)

        # Encoded then decode every value.
        with self.test_session() as sess:
            # Decode into floating-point scalar.
            decoded = mu_law_decode(x, QUANT_LEVELS)
            # Encode back into an integer quantization level.
            encoded = mu_law_encode(decoded, QUANT_LEVELS)
            round_tripped = sess.run(encoded)

        # decoding then encoding every level should produce what we started
        # with.
        print("x:\n", x)
        print("round_tripped:\n", round_tripped)
        self.assertAllEqual(x, round_tripped)

If I understood you correctly, that test should fail. But in fact it passed. Which means when we take any quantized value, decode it to a floating point scalar, encode it back to a quantized value, we get back the original quantized value.
Then I made your proposed changes to mu_law_encode() and manual_mu_law_decode(). Re-running the tests, several tests fail, including the new testDecodeEncode():
AssertionError:
Arrays are not equal

(mismatch 27.734375%)
 x: array([  0,   1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12,
        13,  14,  15,  16,  17,  18,  19,  20,  21,  22,  23,  24,  25,
        26,  27,  28,  29,  30,  31,  32,  33,  34,  35,  36,  37,  38,...
 y: array([  0,   0,   1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,
        12,  14,  14,  15,  16,  17,  18,  19,  20,  21,  22,  23,  25,
        25,  27,  27,  28,  29,  31,  31,  32,  33,  34,  35,  36,  37,...

Thus if we make the change you proposed, the quantization level 1 gets mapped to 0, 2-->1, 3-->2 and so on in the round-tripped results.
So at the moment, I'm not seeing that the code is encoding and decoding in different ranges, as you put it.
Regarding the addition of 0.5: this is a standard way of rounding a float to nearest integer before a floor float-to-int op.

If you don't want to push this evident correction don't do it, but the code right now is encoding and decoding in different ranges.

Please don't be upset. Of course we want to fix a defect if there is one. We also need to make sure the implementation and tests are in sync, and the tests are passing.Tests should verify correctness of any implementation, when feasible, and we've got tests that cover the code you proposed changing. We're grateful for your efforts on this.
I can be slow and stupid and error-prone at times so maybe I'm missing something or made errors so please help me understand if I'm getting something wrong.
I think I'll go back to derive the mu-law encoding/decoding on my own and see if they match up with what we have.

@Zeta36

You cannot encode in this way:
signal = (signal + 1) / 2 * mu + 0.5

and later decode like this:
signal = 2 * (signal / mu) - 1

OK that's a slightly different point than I had understood at first, and suggests another test we could write to check if we are shifting the signal.

I wrote tests to check everything I can think of that Zeta36 might be referring to, and didn't find any defects. If we still have a problem here we need to look at please speak up. From what I can see the change in this PR would not be correct.


Queries using First() and Count()

GORM Playground Link
https://github.com/go-gorm/playground/pull/ 71
Description
Using First and Count together leeds to some unexpected behaviour (including a silent exit)

Please make the link runnable and add it back, thank you.

I'm not sure what the problem is, but I can run the tests using ./test.sh or go run (of course I got the dependencies before).


Flatpak packaging

Is your feature request related to a problem? Please describe.
Building is broken currently. Managed to get it to build by working around the node-sass problem but then the browser is just a series of errors. Also for Linux, snap is very controversial, whereas Flatpak is the mainstream accepted method.
Describe the solution you'd like
Package as a Flatpak and publish to Flathub
Describe alternatives you've considered
Building from source which is not ideal and also currently broken

Duplicate of #9894?


Extensions defining new importing controllers need to import theme.less from OpenRefine

Some extensions define new "importing controllers" (for instance the gdata and database extension). This is the expected extension point when implementing an importing mechanism which loads data from other sources.
As part of this, extensions need to define parsing pages which look like the importer preview we have in OpenRefine.
For the styling of those parsing pages, both the gdata and database extensions rely on OpenRefine-internal .less files that they explicitly import on their own side.
This looks like this:
https://github.com/OpenRefine/OpenRefine/blob/5746951ec069549345eb368ea74a77c3fec9a912/extensions/database/module/styles/theme.less#L30
While this technique avoids duplicating files from the core to the extensions, this is far from ideal because this mechanism is not available to extensions developed outside this repository.
Intuitively, it should be possible for any extension to use the standard styling of the parsing page, without having to redefine it itself. Of course, we still want extensions to be able to add some custom CSS to this page, should they need to.
Ideally, it would be great if we could fix this in a backwards-compatible way (meaning that extensions which have been built for the current situation would keep working with the new system), although this is likely to be difficult. And it is probably not so important since we can easily fix the extensions in this repository.
@antoine2711 @WaltonG this is an issue you are likely to run into (or have already run into?) for the SPARQL extension, no?

@wetneb I copied the theme.less file which contains the below line and it works fine https://github.com/OpenRefine/OpenRefine/blob/5746951ec069549345eb368ea74a77c3fec9a912/extensions/database/module/styles/theme.less#L30

It might work in your development settings but I suspect this is going to break as soon as users install it in their workspace, as documented here: https://docs.openrefine.org/manual/installing#find-the-right-place-to-install

Will definitely have to try that out


Query on interactive plot

Hi,
What software did you use to generate the 3-D plot gif

Hi,
I generated the frames manually with matplotlib (a new interpolated point is added in each frame), and used ffmpeg to compile an mp4 out of the images.

hello, I still confusing about the 3-D plots showed in the paper, which were represented as mixup/triplet mixup/Bernoulli mixup. Can you explain how to interpret these mixup methods as interpolating along line segments.
Thanks!

Hi @jingcx
If you look at the first gif, you can see 3 blue points embedded in a 3d latent space, let's call them h1, h2, and h3. What is happening is that at each frame of the video, a red point is generated by sampling some h_i and h_j (where i,j can be either of the 3) and generating interpolations between them by sampling a random alpha from a uniform(0,1) and computing h_mix = alpha*h_i + (1-alpha)*h_j. The concept also applies to higher dimensional spaces -- you're just interpolating along a line whose endpoints are defined by h_i and h_j.
In the second gif, you are interpolating along 3 points at the same time, so that would be computing something like h_mix = alpha1*h1 + alpha2*h2 + alpha3*h3, where all those alphas sum to 1 (you can sample these alphas by using a Dirichlet distribution). In this case however, we are interpolating between triplets. This corresponds to interpolating within a triangle whose vertices are determined by h1,h2,h3. If we were doing 4 points instead, you'd be interpolating within a tetrahedron, etc. (Also see this for reference: https://en.wikipedia.org/wiki/Simplex)
The Bernoulli example is a bit less intuitive, it's not doing interpolation within line segments but something else which is more akin to extrapolation.
If it's still unclear, I would suggest you write a little example and use matplotlib (with 3d scatterplots) to visualise the types of interpolations you can do.
hth

It's probably worth me putting up a Colab notebook of my own code that generated the figures. I will get to that shortly.

@christopher-beckham
Your explanation is very clear, and thank you so much for your patient!


Forgot your password link doesn't work

Is there support in LaraParse for the 'Forgot your password' link in the login screen, or do I have something configured incorrectly?
PDOException in Connector.php line 47: could not find driver
in Connector.php line 47
at PDO->__construct('mysql:host=localhost;dbname=forge', 'forge', '', array('0', '2', '0', false, false)) in Connector.php line 47
at Connector->createConnection('mysql:host=localhost;dbname=forge', array('driver' => 'mysql', 'host' => 'localhost', 'database' => 'forge', 'username' => 'forge', 'password' => '', 'charset' => 'utf8', 'collation' => 'utf8_unicode_ci', 'prefix' => '', 'strict' => false, 'name' => 'mysql'), array('0', '2', '0', false, false)) in MySqlConnector.php line 20
at MySqlConnector->connect(array('driver' => 'mysql', 'host' => 'localhost', 'database' => 'forge', 'username' => 'forge', 'password' => '', 'charset' => 'utf8', 'collation' => 'utf8_unicode_ci', 'prefix' => '', 'strict' => false, 'name' => 'mysql')) in ConnectionFactory.php line 58
at ConnectionFactory->createSingleConnection(array('driver' => 'mysql', 'host' => 'localhost', 'database' => 'forge', 'username' => 'forge', 'password' => '', 'charset' => 'utf8', 'collation' => 'utf8_unicode_ci', 'prefix' => '', 'strict' => false, 'name' => 'mysql')) in ConnectionFactory.php line 47
at ConnectionFactory->make(array('driver' => 'mysql', 'host' => 'localhost', 'database' => 'forge', 'username' => 'forge', 'password' => '', 'charset' => 'utf8', 'collation' => 'utf8_unicode_ci', 'prefix' => '', 'strict' => false), 'mysql') in DatabaseManager.php line 177
at DatabaseManager->makeConnection('mysql') in DatabaseManager.php line 65
at DatabaseManager->connection() in PasswordResetServiceProvider.php line 63
at PasswordResetServiceProvider->Illuminate\Auth\Passwords\{closure}(object(Application), array()) in Container.php line 773
at Container->build(object(Closure), array()) in Container.php line 656
at Container->make('auth.password.tokens', array()) in Application.php line 644
at Application->make('auth.password.tokens') in Container.php line 1231
at Container->offsetGet('auth.password.tokens') in PasswordResetServiceProvider.php line 39
at PasswordResetServiceProvider->Illuminate\Auth\Passwords\{closure}(object(Application), array()) in Container.php line 773
at Container->build(object(Closure), array()) in Container.php line 656
at Container->make('auth.password', array()) in Application.php line 644

it looks like you haven't changed your PasswordController that comes out of the box with Laravel. You can override as many of the methods inherited from the ResetsPasswords Trait for your needs but the key one is to override the postEmail. My implementation is pretty simple so I actually removed the ResetsPasswords trait as I didn't need it so your implementation might look slightly different. Basically my postEmail looks like this:
/**
 * Request an email password reset
 *
 * @return \Illuminate\Routing\Redirector|\Illuminate\Http\RedirectResponse
 */
public function postEmail(Request $request)
{
	$this->validate($request, ['email' => 'required|email']);

    ParseUser::requestPasswordReset(Input::get('email'));
    // Handle the redirecting of the customer any way you like or do additional logic here
    return redirect()->back()->with('status', 'Please check your email for instructions on resetting your password');
}

Note: I don't use try/catch as I have a catch all ParseException handler. It just made things a bit neater!

Thanks!


Displaying list of logs

Maybe a drop box that displays the list of logs may do

A drop box or scroll screen maybe

okay i have done a scroll screen will be pushing in a few


Symbolic links in content directory

I've got my site contents in a folder under my home directory. I have then symlinked these files to the content folder in my Pico installation under /var/www/pico
But I get a 404 for any symlinked file. When I create a file directly in the content folder Pico works fine.
I've checked file/link ownership and permission and everything looks OK to me.
I know that symlinking is working fine because I've linked an html file from my home directory to /var/www/html and that is appearing just fine.
This is running under Apache - initially on Mac OS but I'm having the same issue on Ubuntu 20.04.
Can anyone let me know what I need to do to get symbolic links resolved and rendered by Pico?

Your PHP is verly likely configured not to include arbitrary files using a open_basedir restriction. Check your php.ini. Symlinks should work just fine with Pico - as long as PHP actually has permission to read that file.

Thanks, it was permissions. When I granted the Apache user (www-data) permission on the directory with the .md files in it the site started working. Thanks very much for the pointers.


Run conformance tests for PRs, whenever that is possible

Description
Run conformance tests not requiring secrets automatically for pull requests.
Issue reference
Please reference the issue this PR will close: N/A
Checklist
Please make sure you've completed the relevant tasks for this PR, out of the following list:

[x] Code compiles correctly
[ ] Created/updated tests
[ ] Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

Sample run for PR: https://github.com/tcnghia/components-contrib/actions/runs/548821191
Sample run for Cron: https://github.com/tcnghia/components-contrib/actions/runs/548822115

/assign @mukundansundar @artursouza

No, please. This can open arbitrary code to use our credentials in Azure and other cloud providers.

No, please. This can open arbitrary code to use our credentials in Azure and other cloud providers.

Pull requests don't have access to secrets, and also, we don't run anything requiring secrets in pull requests workflow.
Plus, I can run pull requests workflow even if I am not checking this PR in. You can see that in the workflows being executed for this PR itself.

@artursouza @mukundansundar please see the workflow scheduled by this PR here https://github.com/dapr/components-contrib/actions/runs/548839442 . We can't stop PRs from running this workflow.


RV3028 driver bug, autorepeat on example buttons

Bug in getTimerCount() in driver (was ignoring top byte).
Add auto-repeat on the +/- timer buttons in the example.
Allow PicoExplorer/PicoDisplay selection in the example with #define USE_PICO_EXPLORER 1.
Note that CMakeLists.txt includes the display and explorer libraries, but only one is used.

Good grief, you're fast!
I think we should build repeat into the libraries somehow. Right now I just throw my AutoRepeat class at everything:
https://github.com/Gadgetoid/pico-rgb-keypad-debounce/blob/b609f525b067941152c657d2f98355bb33d2fbe8/main.cpp#L11-L57
AutoRepeat autorepeat_a;

while(true) {
    auto a_pressed = autorepeat_a.next(pico_display.is_pressed(pico_display.A);
}

Which dramatically simplifies the "main" loop.

I was helped by having commented the Display->Explorer transform in the code, so just had to follow my own instructions!
(That, and I have been doing this for 40 years, so have had a bit of practice!).  I'd also just finished a very boring work Zoom meeting, so needed some mental stimulation :-)
For the autorepeat, I wanted to leave the keys on the left as single-shot (to select what the clock was going to do), and the ones on the right mostly repeaters, but with a varying repeat speed (gets faster after the first 10 increment/decrements). I did start writing code to use the alarm setting functions, but got bogged down in which key did what, and which bits of information you'd have to select (there seem to be 8 different alarm modes). I was also getting to the point where I needed short and long presses, separately from autorepeat, so it was all getting a bit complicated. Probably need to leave that for another day.

 Got a piezo on there too :thinking:
Thank you!


Add a splashscreen

Add a splash screen for when you visit the application

I would like to work on this too mr @ArcadyRob, master.


Run out of GPU memory when using Launchpad

Hi I am running a distributed agent in Acme (/acme/examples/control/lp_local_d4pg.py) and I get the following error (run of out GPU memory)
I0522 20:03:38.349611<PHONE_NUMBER>75040 node.py:61] Reverb client connecting to: localhost:18200
Traceback (most recent call last):
  File "/home/lorenzo/acme/lib/python3.8/site-packages/launchpad/nodes/python/process_entry.py", line 80, in <module>
    app.run(main)
  File "/home/lorenzo/acme/lib/python3.8/site-packages/absl/app.py", line 303, in run
    _run_main(main, args)
  File "/home/lorenzo/acme/lib/python3.8/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "/home/lorenzo/acme/lib/python3.8/site-packages/launchpad/nodes/python/process_entry.py", line 75, in main
    functions[task_id]()
  File "/home/lorenzo/acme/lib/python3.8/site-packages/launchpad/nodes/python/node.py", line 71, in _construct_function
    return functools.partial(self._function, *args, **kwargs)()
  File "/home/lorenzo/acme/lib/python3.8/site-packages/launchpad/nodes/courier/node.py", line 106, in run
    instance = self._construct_instance()  # pytype:disable=wrong-arg-types
  File "/home/lorenzo/acme/lib/python3.8/site-packages/launchpad/nodes/python/node.py", line 164, in _construct_instance
    return self._constructor(*args, **kwargs)
  File "/home/lorenzo/git/acme/acme/agents/tf/d4pg/agent_distributed.py", line 149, in actor
    networks = self._network_factory(self._environment_spec.actions)
  File "/home/lorenzo/git/acme/acme/agents/tf/d4pg/agent_distributed.py", line 67, in wrapped_network_factory
    networks_dict = network_factory(action_spec)
  File "/home/lorenzo/git/planning_2d/test_speed.py", line 118, in make_networks
    networks.DiscreteValuedHead(vmin, vmax, num_atoms),
  File "/home/lorenzo/acme/lib/python3.8/site-packages/sonnet/src/base.py", line 126, in __call__
    module.__init__(*args, **kwargs)
  File "/home/lorenzo/git/acme/acme/tf/networks/distributional.py", line 63, in __init__
    vmin = tf.convert_to_tensor(vmin)
  File "/home/lorenzo/acme/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py", line 206, in wrapper
    return target(*args, **kwargs)
  File "/home/lorenzo/acme/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 1430, in convert_to_tensor_v2_with_dispatch
    return convert_to_tensor_v2(
  File "/home/lorenzo/acme/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 1436, in convert_to_tensor_v2
    return convert_to_tensor(
  File "/home/lorenzo/acme/lib/python3.8/site-packages/tensorflow/python/profiler/trace.py", line 163, in wrapped
    return func(*args, **kwargs)
  File "/home/lorenzo/acme/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 1566, in convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/lorenzo/acme/lib/python3.8/site-packages/tensorflow/python/framework/tensor_conversion_registry.py", line 52, in _default_conversion_function
    return constant_op.constant(value, dtype, name=name)
  File "/home/lorenzo/acme/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 264, in constant
    return _constant_impl(value, dtype, shape, name, verify_shape=False,
  File "/home/lorenzo/acme/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 276, in _constant_impl
    return _constant_eager_impl(ctx, value, dtype, shape, verify_shape)
  File "/home/lorenzo/acme/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 301, in _constant_eager_impl
    t = convert_to_eager_tensor(value, ctx, dtype)
  File "/home/lorenzo/acme/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 97, in convert_to_eager_tensor
    ctx.ensure_initialized()
  File "/home/lorenzo/acme/lib/python3.8/site-packages/tensorflow/python/eager/context.py", line 554, in ensure_initialized
    context_handle = pywrap_tfe.TFE_NewContext(opts)
tensorflow.python.framework.errors_impl.InternalError: CUDA runtime implicit initialization on GPU:0 failed. Status: out of memory

I tried on different hardware (rtx 1080 and rtx 1650) and I still get the same issue.
If I disable the GPU and run only CPU any number of  programs can be spawned successfully instead. Obviously it's much slower though.
Any idea on what caused the error?
I am posting it here instead of acme because the issue seems related to Launchpad

The issue has been solved, check https://github.com/deepmind/acme/issues/121.
The problem is caused by the way TF manages GPU memory.


Integration test summary

Problem:
Integration tests currently provide a summary that looks like
PASSED test_client_authentication.py::test_client_auth_with_s2n_server[P-256-TLS_CHACHA20_POLY1305_SHA256]
PASSED test_client_authentication.py::test_client_auth_with_s2n_server[P-256-TLS_AES_128_GCM_256]
PASSED test_client_authentication.py::test_client_auth_with_s2n_server[P-256-TLS_AES_256_GCM_384]
PASSED test_client_authentication.py::test_client_auth_with_s2n_server[P-384-TLS_CHACHA20_POLY1305_SHA256]

This shows that we can display the important information about an integ test, but we aren't displaying it in the most useful way.
Proposed Solution:

[ ] Update the way Pytest creates and displays test summaries. Work with stakeholders to figure out what people care about and show whatever that may be.
[ ] Find a way to store this summary somewhere useful (release notes? in repo?). If this isn't straightforward, create a new issue detailing what to do and why.

There are scaling concerns here. After some initial tests with pytest-json-report, the report file is getting very large (100mb for just the happy path).
Consider why this report will be used and the most accessible way to use it.
A condensed json file w/ a script that allows developers to pull test results by id?
A single HTML file (possibly gigabytes in size) but easy browser access?
Maybe we can restrict some of the stdout data?


WIP: DS-3533 Use correct json property name for embedded collections

Updated Description 2020/01/07:
This PR introduces an issue with the items/uuid/edit/mapper page when an item is not mapped. I have put it back in WIP state...waiting for some adjustments to the associated REST PR. I will update the description when ready.
Original Description
This is a minor adjustment that should be merged when the corresponding REST PR is merged:
https://github.com/DSpace/DSpace/pull/2625
This property name changed in order to be consistent with other embedded list property names.

This PR is no longer needed; closing. The name change was not necessary, and I have tested angular master with the PR, and confirmed mapped collections edit page works fine whether the item is currently mapped or not.


Update index.md

Removed .value from overlay, its undefined and does not exist on the Overlay object.

It's also worth noting that using :position makes it impossible to use .setPosition (the former overrides the latter). It should be noted in the docs!


Originality update/improvement

Based on the suggested improvements in the pdf. I am now pull the tournament_data to allow the user submission data and other submissions to be broken down into the validation, test, and live data sets respectively. We can then perform the various independence tests on all 3 data sets and determine if the users submission is "original".
The update procedure is as follows:

Once the user submission is pulled to have the originality calculated we gather all the validation, test, and live probabilities into their own data sets.
We then go through the same procedure of pull all other submissions and comparing the user submission with those. But we will be also be gathering the other submissions probabilities into a validation, test  and live set to compare against the users data sets.
Calculate a new independence score on the validation, test, and live data sets and if one is below a certain threshold it will fail originality.
Repeat the above for all other submissions and if it does not fail originality we can conclude that the submission is indeed "original"

Independence Test
The KS two sample test has been removed since it has proven itself to be vulnerable to ad-hoc modifications which allows a user to pass the originality test without necessarily being "original".
In place of this was going to be a Kullback-Leibler divergence entropy test. Since it should determine if two Probability Distribution Functions should behave similarly if the result approaches 0 and conversely if the result approaches 1. When testing this with some predictions from multiple models (NN, Linear Regression, GBTs) the results almost were always on the scale of 1.0e-7. Close to 0 so this measure was scraped due to almost all models having the same entropy.
In place of the entropy test I decided that perhaps a more simple measure may be better for an independence test. The test applies sorting both data sets to get an ascending curve and determining the normalized residual error between the curves as a measure of how similar they are. This seems to make a bit more sense as it applies how the mean difference relative to the maximum curve.
Also benchmarking the originality with this test takes on average 3.88ms against other submissions as opposed to the 8.86ms with the most recent version of master.
I am open to suggestions to either another/improving the independence test or to adding another. This one made the most sense to me after some research and testing.
Reward
Since this is PR does not cover all of the improvements laid out in the doc I think that if this is merged is worth 30-40 NMR since the only other improvements are to Concordance and the Staking tournament. Since it seems like the staking is more on the side of Numerai of generating new data with their keys for the encryption. It does not seems like that is worth much to those working on this project outside the company.
Numerai username: cpurta

I'm a bit confused by this test statistics (1.0 / n1) * (np.sum(np.absolute(data1 - data2)) / max(np.sum(data1), np.sum(data2))) * 10**math.floor(math.log10(n1)).
It seems like (1/n)*10**floor(log10(n))  should always be about 1, but it does oscillate depending on what n happens to be.
Since sum(data1) approx sum(data2), then it looks like (np.sum(np.absolute(data1 - data2)) / max(np.sum(data1), np.sum(data2))) is related to |cdf1 - cdf2|.
Would doing something like mean(|cdf1-cdf2|) make more sense? The KS test is essentially doing max(|cdf1-cdf2|). Using max might be easier to game, since you only need to change one of your predictions to change the max a lot.
I do like the idea of using an information theory type metric, but it is a bummer that KL divergence doesn't work. Maybe a difference type of divergence would work better? https://en.wikipedia.org/wiki/Bregman_divergence Has a pretty decent list of different types.

Yeah I ended up adding the 10**math.log(n1) since the score ended up being really small (1.0e-07) so multiplying by that ended up just being a bit more comparable to the thresholds being used. I think that you may be right in using the mean(|cdf1 - cdf2|) to get a better score. But I was doing some more research in KL-Divergence and I was not applying the divergence to a cdf, but rather just the sorted probabilities.
I have tested the KL-Divergence on the cdfs and have gotten better results against different models (xgboost vs regression) and have had the entropy measure show confirm that the models will behave differently (i.e. score -> inf as opposed to approaching 0) also I have tested the cheating mentioned in the improvements pdf and with the entropy measure shows that the models are extremely similar.
I have also updated the thresholds used for KL-Divergence testing since it quantifiably easier to determine if two models are going to behave similarly (i.e. the score is close to 0)

Per request in the slack, I'm leaving this feedback on this pull request as well.
In my opinion, any sort of distributional check isn't actually comparing how similar two sets of predictions are in any meaningful way.
If I generate a set of predictions, then shuffle them, a divergence test on the distributions of the original vs shuffled is going to say my predictions are identical, which is clearly false.
A check on originality should be required to maintain the same sorting between whatever vectors it's checking (e.g. correlation, and other distance/similarity metrics).

I agree with using another correlation and currently implementing the spearman correlation coefficient as another test with the pearson correlation that way we can determine if there exists any monotonic function between submissions.
Currently looking into some distance metrics to implement as another originality measure and thinking about using the mean canberra distance. I think that this could be useful since it should be robust against cheating by ad hoc modifications shown in the pdf. Any thoughts on whether this metric is useful or not are welcome.

@zoso95 I used your originality quality benchmark on my branch using the mean canberra distance and got the following results:
LogisticRegression 0.3900000000000002
NaiveBayes 0.4100000000000002
RandomForest 0.4000000000000002
QDA 0.4200000000000002
KNN 0.4200000000000002
SVC 0.3900000000000002
MLP 0.3900000000000002
AdaBoosting 0.3900000000000002
DecisionTree NAN

If I am following your code correctly those numbers are a measure of how much noise one would have to add on top of those baseline predictions in order to have them be original. If the threshold to be original is greater than 0.5 then it may as well be just random noise.
So the greater the threshold on those models the more robust the metrics is against gaming, correct?

Do we know what a reasonable threshold for spearman correlation is?

It seems like the threshold should be pretty high or on par with the pearson correlation. Since it is a measure of how two data sets can be described using a monotonic function.


python310Packages.smart-open: 6.2.0 -> 6.3.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python310Packages.smart-open is: "Library for efficient streaming of very large file"
meta.homepage for python310Packages.smart-open is: "https://github.com/RaRe-Technologies/smart_open"
meta.changelog for python310Packages.smart-open is: ""
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 6.2.0 with grep in /nix/store/zfqcgrhfj1iii7w55cls7wj37rkvb20s-python3.10-smart-open-6.2.0
found 6.2.0 in filename of file in /nix/store/zfqcgrhfj1iii7w55cls7wj37rkvb20s-python3.10-smart-open-6.2.0

Rebuild report (if merged into master) (click to expand)

13 total rebuild path(s)

13 package rebuild(s)

First fifty rebuilds by attrpath
python310Packages.gensim
python310Packages.pathy
python310Packages.smart-open
python310Packages.spacy
python310Packages.spacy-transformers
python310Packages.textacy
python39Packages.gensim
python39Packages.pathy
python39Packages.smart-open
python39Packages.spacy
python39Packages.spacy-transformers
python39Packages.textacy
snakemake

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/zfqcgrhfj1iii7w55cls7wj37rkvb20s-python3.10-smart-open-6.2.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python310Packages.smart-open https://github.com/r-ryantm/nixpkgs/archive/a4dfeeec250d7ab4545af35bbf3032fdccfd4c8a.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/zfqcgrhfj1iii7w55cls7wj37rkvb20s-python3.10-smart-open-6.2.0
ls -la /nix/store/zfqcgrhfj1iii7w55cls7wj37rkvb20s-python3.10-smart-open-6.2.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

6 packages failed to build:

python310Packages.spacy
python310Packages.spacy-transformers
python310Packages.textacy
python39Packages.spacy
python39Packages.spacy-transformers
python39Packages.textacy

7 packages built:

python310Packages.gensim
python310Packages.pathy
python310Packages.smart-open (python310Packages.smart_open)
python39Packages.gensim
python39Packages.pathy
python39Packages.smart-open (python39Packages.smart_open)
snakemake

Maintainer pings
cc @jyp for testing.

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python310Packages.smart-open is: "Library for efficient streaming of very large file"
meta.homepage for python310Packages.smart-open is: "https://github.com/RaRe-Technologies/smart_open"
meta.changelog for python310Packages.smart-open is: ""
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 6.3.0 with grep in /nix/store/y8zq7rqa18n2z47my5l6pbhmsw8wjnav-python3.10-smart-open-6.3.0
found 6.3.0 in filename of file in /nix/store/y8zq7rqa18n2z47my5l6pbhmsw8wjnav-python3.10-smart-open-6.3.0

Rebuild report (if merged into master) (click to expand)

15 total rebuild path(s)

15 package rebuild(s)

First fifty rebuilds by attrpath
python310Packages.gensim
python310Packages.pathy
python310Packages.ray
python310Packages.smart-open
python310Packages.spacy
python310Packages.spacy-transformers
python310Packages.textacy
python39Packages.gensim
python39Packages.pathy
python39Packages.ray
python39Packages.smart-open
python39Packages.spacy
python39Packages.spacy-transformers
python39Packages.textacy
snakemake

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/y8zq7rqa18n2z47my5l6pbhmsw8wjnav-python3.10-smart-open-6.3.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python310Packages.smart-open https://github.com/r-ryantm/nixpkgs/archive/261e8aba0576c9be59625aeecb96c86081f344ca.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/y8zq7rqa18n2z47my5l6pbhmsw8wjnav-python3.10-smart-open-6.3.0
ls -la /nix/store/y8zq7rqa18n2z47my5l6pbhmsw8wjnav-python3.10-smart-open-6.3.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

15 packages built:

python310Packages.gensim
python310Packages.pathy
python310Packages.ray
python310Packages.smart-open (python310Packages.smart_open)
python310Packages.spacy
python310Packages.spacy-transformers
python310Packages.textacy
python39Packages.gensim
python39Packages.pathy
python39Packages.ray
python39Packages.smart-open (python39Packages.smart_open)
python39Packages.spacy
python39Packages.spacy-transformers
python39Packages.textacy
snakemake

Maintainer pings
cc @jyp for testing.


Manufacturing the hardware boards - PCB / Firmware Programming questions

I'm looking at quotes for producing the controller and display boards on a small scale (~50).

Is there a solder-type requirement for making these boards or do the components care what solder is used?

It's unclear to me how to "set the correct memory layout for each device in the linker script."

Where is the referenced linker script and do I adjust it before uploading firmware for each board and MCU (controller and display)?
2.a Do I need to send this instruction to the PCB manufacturer, will they understand what that means?
2.b. Are there any instructions needed for properly uploading the provided firmware to the boards or do I just compile and upload everything as a single hex file as is?

What is the SMT oven profile for these boards?

Is conformal coating needed?

I received a bit more info from the manufacturer on these  questions.

ROHS - No Clean Pb Free solder is standard

Oven profile is for heat-sensitive components. Im wondering are there sensitive components on these boards we need to be aware of?

Conformal coating is really for outdoor / high-moisture environments. So its not necessary.

Lingering questions about testing and setting up system:
What is the procedure for testing operation of these boards?
There is a protocols section for testing the PCBs over a 16-pin header, https://github.com/googlecreativelab/anypixel/blob/master/firmware/PROTOCOLS.md , but it is missing the descriptions of several things (dot correction, rgb raw, calibration lookup table request, etc) . It is unclear how one should compile a header/test sequence. A simple diagram showing the google wall with referenced variables used would help out a lot.
Some step by step instructions with example of stacking the bytes together would be awesome.
The unit address section has (20 + 10* row + column) How are these referenced on existing google wall?
What is the sequence number?
What is the port number?
What is the proper format of ip address?
How does one set the memory layout of each device?
Where is the referenced linker script for setting memory layout?
Do I need to worry about setting the memory layout in the firmware or can it be done after the upload?

Hey thanks for your interest in AnyPixel and for your super pointed questions. We're running around right now on a couple other projects but we'll get back to you soon. Thanks for your patience.

@HedronUser did you make any progress with your project or did you stop it? Would be awesome to hear any feedback!

Hey Dirk,
Our client went another direction so we stopped moving forward with it. It was unfortunate that the BOM for the hardware was so difficult to track down and that the hardware design is so expensive and complicated for anyone, even a professional electronics engineer, to replicate. For example, the RGB LEDs specified are out of production and without a direct replacement available on the market a redesign of the board would be necessary to make the design work. Why didn’t google go the route of the on-die addressable LED like the ws2812 or similar with dedicated data and power injection circuits?  Developing that software stack would have been more applicable IMO. On the other hand, I understand wanting to use dedicated processing at the submatrix array level but it is a lot of extra chip programming, extra expense, and a lot of potential debug.
Thanks for reaching out,
-Jesse

On Nov 13, 2018, at 9:08 AM, Dirk Dresch<EMAIL_ADDRESS>wrote:
@HedronUser did you make any progress with your project or did you stop it? Would be awesome to hear any feedback!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.


Autopause not resuming

Running Docker with docker-minecraft through Docker-compose. Running with portforwarding on port 25565. MC version 15.5.2
I've enabled autopause on my server with Watchdog disabled.
The server pause starts as it should when a user has been disconnected for 15 minutes.
However when entering the game (and letting it ping) the server doesn't wake up. When trying to connect to the server this doesn't work.
The only thing that made my server awake again was by manually starting the RCON connection.
[16:06:43] [Server thread/INFO]: com.mojang.authlib.GameProfile@xxxxx[id=<null>,name=Username,properties={},legacy=false] (/XX.XX.XX.XX) lost connection: Disconnected,
[init] Running as uid=1000 gid=1000 with /data as 'drwxrwxr-x    5 1000     1000          4096 Jun 10 15:02 /data',
[init] Resolved version given LATEST into 1.15.2,
[init] Autopause functionality enabled,
[init] server.properties already created, skipping,
[init] Resolving type given VANILLA,
[init] log4j2.xml already created, skipping,
[init] Checking for JSON files.,
[init] Setting initial memory to 1G and max to 1G,
[init] Starting the Minecraft server...,
[15:42:57] [main/WARN]: Ambiguity between arguments [teleport, destination] and [teleport, targets] with inputs: [Player, 0123, @e, ID],
[15:42:57] [main/WARN]: Ambiguity between arguments [teleport, location] and [teleport, destination] with inputs: [0.1 -0.5 .9, 0 0 0],
[15:42:57] [main/WARN]: Ambiguity between arguments [teleport, location] and [teleport, targets] with inputs: [0.1 -0.5 .9, 0 0 0],
[15:42:57] [main/WARN]: Ambiguity between arguments [teleport, targets] and [teleport, destination] with inputs: [Player, 0123, ID],
[15:42:57] [main/WARN]: Ambiguity between arguments [teleport, targets, location] and [teleport, targets, destination] with inputs: [0.1 -0.5 .9, 0 0 0],
[15:42:57] [Server thread/INFO]: Starting minecraft server version 1.15.2,
[15:42:57] [Server thread/INFO]: Loading properties,
[15:42:57] [Server thread/INFO]: Default game type: SURVIVAL,
[15:42:57] [Server thread/INFO]: Generating keypair,
[15:42:58] [Server thread/INFO]: Starting Minecraft server on *:25565,
[15:42:58] [Server thread/INFO]: Using epoll channel type,
[15:42:59] [Server thread/INFO]: Preparing level "world",
[15:42:59] [Server thread/INFO]: Reloading ResourceManager: Default,
[Autopause loop] MC Server listening for connections - stopping in 600 seconds,
[15:43:42] [Server thread/INFO]: Loaded 6 recipes,
[15:43:42] [Server thread/INFO]: Loaded 825 advancements,
[15:43:42] [Server thread/INFO]: Preparing start region for dimension minecraft:overworld,
[15:43:44] [Server thread/INFO]: Preparing spawn area: 0%,
[15:43:45] [Server thread/INFO]: Preparing spawn area: 20%,
[15:43:45] [Server thread/INFO]: Time elapsed: 2983 ms,
[15:43:45] [Server thread/INFO]: Done (46.793s)! For help, type "help",
[15:43:45] [RCON Listener #1/INFO]: RCON running on <IP_ADDRESS>:25575,
[15:43:45] [Server thread/INFO]: Starting remote control listener,
[15:44:42] [User Authenticator #1/INFO]: UUID of player User is UUID,
[15:44:42] [Server thread/INFO]: User[/XX.XX.XX.XX] logged in with entity id 155 at (69.90212833930714, 63.0,<PHONE_NUMBER>440748),
[15:44:42] [Server thread/INFO]: User joined the game,
[Autopause loop] Client connected - waiting for disconnect,
[15:44:54] [Server thread/INFO]: User lost connection: Disconnected,
[15:44:54] [Server thread/INFO]: User left the game,
[Autopause loop] All clients disconnected - stopping in 600 seconds,
[Autopause loop] No client reconnected - stopping,
[15:54:55] [RCON Listener #1/INFO]: Rcon connection from: /<IP_ADDRESS>,
[15:54:56] [Server thread/INFO]: [Rcon: Saved the game],
[2020-06-10T15:54:56+0200] [Autopause] Pausing Java process,
[2020-06-10T16:06:43+0200] [Autopause] Knocked, resuming Java process,
[16:06:43] [RCON Listener #1/INFO]: Rcon connection from: /<IP_ADDRESS>,
[16:06:43] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 707737ms or 14154 ticks behind,
[16:06:43] [Server thread/INFO]: com.mojang.authlib.GameProfile@id[id=<null>,name=User,properties={},legacy=false] (/XX.XX.XX.XX) lost connection: Disconnected,
[16:06:43] [Server thread/INFO]: com.mojang.authlib.GameProfile@id[id=<null>,name=User,properties={},legacy=false] (/XX.XX.XX.XX) lost connection: Disconnected

What's going wrong?

How does Knocking work?
Tried to knock through /bin/bash through the container of mc itself first:
bash-4.4# knock <IP_ADDRESS> 25575:tcp
bash-4.4#

Also tried service knockd stop but whatever I try, it doesn't do anything.
I'm using the console of portainer to access the container.

knocking doesn't work from inside the container, because, as far as I understood, knockd latches itself to the interface, it doesn't listen on ports. The rcon port should be accessible from the traefik network, even when it is not directed from the host. knockd isn't started as service, it's started in https://github.com/itzg/docker-minecraft-server/blob/b273bf29478ebe6f390d8d914addf4c2cb96e487/files/autopause/autopause-daemon.sh#L7 (this is running as minecraft user, so sudo is neccessary) as daemon. Therefore I suggested killing the running process from bash with something along killall -SIGTERM knockd and running the above line (you're root in bash) without daemon. This should (a) print the interface it is attaching to in this configuration and (b) print some line when it is successfully knocked.
Running the rcon client from within the container does not knock the interface. Before it was merged we had the docker health command spawning healthchecks that tried to interact with the paused process and didn't time out (same scenario, accessing the MC port from within).
Also host_network was declared unsupported because knockd couldn't attach to the host eth0.
Can you provide the container's ifconfig output please? As far as I can see it doesn't contain sensitive data, no promises though.

Hmm, maybe we should spawn a daemon for each interface

I have the same issue with bridge network.
Startring docker:
docker run -d -it -e EULA=TRUE -e VERSION=1.15.2 -e MEMORY=2G -e USE_AIKAR_FLAGS=TRUE -e TYPE=SPIGOT -e ENABLE_AUTOPAUSE=true -e AUTOPAUSE_TIMEOUT_EST=600 -e AUTOPAUSE_TIMEOUT_INIT=180 -e AUTOPAUSE_TIMEOUT_KN=120 -e AUTOPAUSE_PERIOD=10 -p 25565:25565 -v /home/mc/mc-spigot:/data --name mc-spigot itzg/minecraft-server:1.3.0-multiarch --noconsole`

After 2 minutes server goes to sleep:
[Autopause loop] No client connected since startup / knocked - stopping
[14:11:59] [RCON Listener #1/INFO]: Rcon connection from: /<IP_ADDRESS>
[14:11:59] [Server thread/INFO]: [Rcon: Saved the game]
[2020-06-21T14:11:59+03:00] [Autopause] Pausing Java process

docker inspect mc-spigot:
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "23932da9e94d98ec12aa6386759e328caf2d197dece5edb875ec7b0f82970cb4",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "25565/tcp": [
                    {
                        "HostIp": "<IP_ADDRESS>",
                        "HostPort": "25565"
                    }
                ],
                "25575/tcp": null
            },
            "SandboxKey": "/var/run/docker/netns/23932da9e94d",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "c61cf21c4c579c929c9414412ba9ad4d1fc0039405bc0ddbb7084f11c34f7127",
            "Gateway": "<IP_ADDRESS>",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "<IP_ADDRESS>",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:02",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "2293ffa88cb251cf20a6f2804e2c470465535df7825c4f82af8255f98b75c64f",
                    "EndpointID": "c61cf21c4c579c929c9414412ba9ad4d1fc0039405bc0ddbb7084f11c34f7127",
                    "Gateway": "<IP_ADDRESS>",
                    "IPAddress": "<IP_ADDRESS>",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:02",
                    "DriverOpts": null
                }
            }
        }

FYI, latest image (and tagged at 1.8.0) includes knock 0.8.1-r0

i mean its the save thing with my server also, and now my minecraft altogether is autopausing

@dnib1 can you please elaborate? What is your setup (which tag), how are you running the container? Keep in mind, that since the comments above I've added the ability to select the interface to listen to, so the information above might be outdated.

I'm having the same issue, here is my docker-compose.yml
`version: "3"
services:
vanilla-jei:
image: itzg/minecraft-server:java8
ports:
- 25565:25565
environment:
EULA: "TRUE"
ENABLE_AUTOPAUSE: "TRUE"
RCON_PASSWORD: "********"
TYPE: "FORGE"
VERSION: "1.16.5"
FORGEVERSION: "36.0.44"
volumes:
- ./vanilla-jei-data:/data`
My server crashes everyday after some hours of pausing.
Heres the crash log:
2021-03-05-1.log
It seems that the minecraft watchdog kills the server, cause the ticks took way to long.

I believe I'm experiencing the same issue. The Java process autopauses, but it never wakes up. Is there something specific that I have to do to wake it up other than launch Minecraft and have it ping the server?

@tekook from the readme in autopause:

From the server's point of view, the pausing causes a single tick to take as long as the process is stopped, so the server watchdog might intervene after the process is continued, possibly forcing a container restart. To prevent this, ensure that the max-tick-time in the server.properties file is set correctly. Non-vanilla versions might have their own configuration file, you might have to disable their watchdogs separately (e.g. PAPER Servers).

I've just created a Forge server with your versions (and current java8 tag, one commit after 2021.6.0-java8) and noticed, that the initial properties file is created without the disabled watchdog. After restarting with -1 set in the server.properties file it's working as expected. When you grep for autopause in the logs you should see that a warning was printed at server start. It can easily be missed because forge stuff clutters the logs. Forge doesn't have a separate watchdogs (with reference to the last quoted sentence).
@mcblum same as above (my previous comment), I need more information on this. What is your setup (which tag), how are you running the container? Can you provide a compose file and the relevant log entries?

@Oekn5w thanks for the info.
I set the max-tick-time to -1 and will reenable autopause and check this afternoon.

appears to be the oldest open issue for this (https://github.com/itzg/docker-minecraft-server/issues/555, https://github.com/itzg/docker-minecraft-server/issues/651, and https://github.com/itzg/docker-minecraft-server/issues/744 that I have found).
The short version seems to be that when AutoPause is enabled, the server eventually exits (gracefully) as the tick time exceeds the maximum allowed.  per the linked issues, it does not seem that any other config setting affects it, just autopause.
this is a server log file from a Paper minecraft server that was launched and no one ever logged on before it crashed.
2021-03-12-2.log
launch command:
docker run -d -e INIT_MEMORY=600M -e MAX_MEMORY=3G -e ICON=/mnt/minecraft-2/earth/icon.png -e USE_AIKAR_FLAGS=true -e MOTD="A Minecraft Survival World" -e EULA=TRUE -e ONLINE_MODE=FALSE -e ENFORCE_WHITELIST=TRUE -e MAX_WORLD_SIZE=90000 -e FORCE_GAMEMODE=true -e SERVER_NAME=earth -e TYPE=PAPER -e VERSION=1.16.5 -p 8123:8123 -p 25565:25565 -v /mnt/minecraft-2/minecraft/plugins:/plugins -v /mnt/minecraft-2/earth:/data -v /etc/timezone:/etc/timezone:ro --name mc-earth itzg/minecraft-server:adopt11

@Oekn5w thanks for the info.
I set the max-tick-time to -1 and will reenable autopause and check this afternoon.

FYI, max-tick-time to -1 worked like a charm.

@finndo77 it doesn't look like you disabled the watchdog by setting
https://github.com/itzg/docker-minecraft-server#max-tick-time
which is mentioned here
https://github.com/itzg/docker-minecraft-server#autopause
however, I do see now that it could be more explicit in how to disable the watchdog in the common case.

...I have added more emphasis and details to
https://github.com/itzg/docker-minecraft-server#autopause

I'm seeing a similar issue to @finndo77. When the server starts from being paused it throws the same errors in his logs and causes the container to restart. I have max-tick-time=-1 and running the latest image.
docker-compose.yaml
version: "3"
services:
  minecraft:
    image: itzg/minecraft-server
    container_name: minecraft
    ports:
      - "25565:25565"
    volumes:
      - /media/ssd1/minecraft:/data
    environment:
      EULA: "TRUE"
      OVERRIDE_SERVER_PROPERTIES: "TRUE"
      TYPE: "SPIGOT"
      ENABLE_RCON: "true"
      RCON_PASSWORD: "xxxxxx"
      RCON_PORT: 28016
      # enable env variable replacement
      REPLACE_ENV_VARIABLES: "TRUE"
      # define an optional prefix for your env variables you want to replace
      ENV_VARIABLE_PREFIX: "CFG_"
      # and here are the actual variables
      CFG_DB_HOST: "http://localhost:3306"
      CFG_DB_NAME: "xxxxxx"
      CFG_DB_PASSWORD: "xxxxxx"
      WHITELIST: "xxxxxx"
      INIT_MEMORY: "1G"
      MAX_MEMORY: "8G"
      ENABLE_AUTOPAUSE: "TRUE"
      AUTOPAUSE_TIMEOUT_EST: "600"
      MAX_TICK_TIME: "-1"
      OPS: "xxxxxx"
      OVERRIDE_WHITELIST: "true"
    restart: unless-stopped
  rcon:
    image: itzg/rcon
    container_name: rcon
    ports:
      - "4326:4326"
      - "4327:4327"
    environment:
      RWA_USERNAME: xxxxxx
      RWA_PASSWORD: xxxxxx
      RWA_ADMIN: "TRUE"
      RWA_RCON_HOST: minecraft
      RWA_RCON_PASSWORD: xxxxxx
      RWA_RCON_PORT: 28016
    volumes:
      - /media/ssd1/minecraft:/opt/rcon-web-admin/db
    restart: unless-stopped

server.properties
#Minecraft server properties
#Thu Mar 18 17:45:47 GMT 2021
spawn-protection=16
max-tick-time=-1
query.port=25565
server-name=Dedicated Server
generator-settings=
sync-chunk-writes=true
force-gamemode=false
allow-nether=true
enforce-whitelist=false
gamemode=survival
broadcast-console-to-ops=true
enable-query=false
player-idle-timeout=0
text-filtering-config=
difficulty=easy
spawn-monsters=true
broadcast-rcon-to-ops=true
op-permission-level=4
pvp=true
entity-broadcast-range-percentage=100
snooper-enabled=true
level-type=default
hardcore=false
enable-status=true
enable-command-block=true
max-players=20
network-compression-threshold=256
resource-pack-sha1=
max-world-size=29999984
function-permission-level=2
rcon.port=28016
server-port=25565
debug=false
texture-pack=
server-ip=
spawn-npcs=true
allow-flight=false
level-name=world
view-distance=10
resource-pack=
spawn-animals=true
white-list=true
rcon.password=xxxxxx
generate-structures=true
max-build-height=256
online-mode=true
level-seed=
use-native-transport=true
prevent-proxy-connections=false
enable-jmx-monitoring=false
enable-rcon=true
motd=A Spigot Minecraft Server powered by Docker
rate-limit=0

resumeFromPauseCrash.log

Hmm, that's weird it's trying a ./start.sh
[17:45:34] [Spigot Watchdog Thread/INFO]: Startup script './start.sh' does not exist! Stopping server.

In this image there's /start but not a start.sh and I don't see any references to that.
@mafiaboy01 did you make sure disable any watchdog Spigot is doing? You only showed the vanilla properties.

It seems to be related to Spigot because I switched to Forge (I'm new to Minecraft so figuring out what type of server to run) which doesn't have this issue.
From what I read seems like the max-tick-time paremeter in the spigot.yml needs to be set to 1000 do disable it.
    max-tick-time:
      tile: 1000
      entity: 1000

But I haven't tested this myself.

I have the same issue with a fabric server. Set max-tick-time to -1 and it never unpauses. I don't get anything in the log after [Autopause] Pausing Java process

For me the issue I think this line in /autopause/knockd-config.cfg:
sudo /sbin/su-exec minecraft:minecraft /autopause/resume.sh
I am not running as UID/GID 1000 .. if I manually run this with the correct user and group it resumes but not if I manually run the line above.

I'm seeing something similar. I'm curious if it has to do with https://github.com/itzg/docker-minecraft-server/blob/master/files/autopause/autopause-daemon.sh#L98
The only way resume.sh is called is if there's an rcon client listed in the ps output. Maybe I just don't know what all starts an rcon client connect, but it doesn't appear to be happening on the Multiplayer page ping or attempt to connect.
The only way I've been able to get autopause to resume is to manually exec an rcon-cli connection like podman exec -it pause rcon-cli. That kind of defeats the purpose if that's required, so I'm guessing something is wrong.
Snippet of output:
[Autopause loop] No client reconnected - stopping
[21:45:55] [RCON Listener #1/INFO]: Thread RCON Client /0:0:0:0:0:0:0:1 started
[21:45:55] [Server thread/INFO]: [Rcon: Saved the game]
[21:45:55] [RCON Client /0:0:0:0:0:0:0:1 #4/INFO]: Thread RCON Client /0:0:0:0:0:0:0:1 shutting down
[2021-06-05T21:45:55+00:00] [Autopause] Pausing Java process
[2021-06-05T22:05:41+00:00] [Autopause] Knocked, resuming Java process
[22:05:41] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 1185651ms or 23713 ticks behind
[22:05:41] [RCON Listener #1/INFO]: Thread RCON Client /0:0:0:0:0:0:0:1 started
[Autopause loop] Server was knocked - waiting for clients or timeout
[22:05:41] [Server thread/INFO]: com.mojang.authlib.GameProfile@4bd2d1f5[id=<null>,name=GodsSyntaxError,properties={},legacy=false] (/<IP_ADDRESS>:41914) lost connection: Disconnected
[22:05:41] [Server thread/INFO]: com.mojang.authlib.GameProfile@595c6c8e[id=<null>,name=GodsSyntaxError,properties={},legacy=false] (/<IP_ADDRESS>:41924) lost connection: Disconnected
[22:05:41] [Server thread/INFO]: com.mojang.authlib.GameProfile@771cdee8[id=<null>,name=GodsSyntaxError,properties={},legacy=false] (/<IP_ADDRESS>:41922) lost connection: Disconnected
[22:05:41] [Server thread/INFO]: com.mojang.authlib.GameProfile@3b349526[id=<null>,name=GodsSyntaxError,properties={},legacy=false] (/<IP_ADDRESS>:41906) lost connection: Disconnected
[22:05:42] [RCON Client /0:0:0:0:0:0:0:1 #5/INFO]: Thread RCON Client /0:0:0:0:0:0:0:1 shutting down
[Autopause loop] No client connected since startup / knocked - stopping
[22:06:11] [RCON Listener #1/INFO]: Thread RCON Client /0:0:0:0:0:0:0:1 started
[22:06:11] [Server thread/INFO]: [Rcon: Saved the game]
[22:06:11] [RCON Client /0:0:0:0:0:0:0:1 #6/INFO]: Thread RCON Client /0:0:0:0:0:0:0:1 shutting down
[2021-06-05T22:06:11+00:00] [Autopause] Pausing Java process

Command:
podman run --privileged -d -p 25566:25565 -v /home/jmighion/minecraft:/data:Z --name pause -e EULA=TRUE -e OPS=GodsSyntaxError -e INIT_MEMORY=1G -e MAX_MEMORY=4G -e WHITELIST=GodsSyntaxError -e ENFORCE_WHITELIST=true -e RCON_PORT=25576 -e RCON_PASSWORD=pauser -e ENABLE_AUTOPAUSE=TRUE -e MAX_TICK_TIME=-1 -e AUTOPAUSE_KNOCK_INTERFACE=tap0 -e AUTOPAUSE_TIMEOUT_EST=60 -e AUTOPAUSE_TIMEOUT_INIT=30 -e AUTOPAUSE_TIMEOUT_KN=30 -e OVERRIDE_SERVER_PROPERTIES=true itzg/minecraft-server
I've disabled the max tick, overriding the server properties, and set a non-standard rcon port and password. Things look to work if an rcon-cli connection is made, but that's the only way.

rcon is the only way to resume from inside the container. knockd handles the resuming independent from the autopause daemon. Are you sure that the tap0 interface is the correct one? Which other interfaces do you see in the container ( ifconfig -s inside)

$ podman exec -it mc /bin/bash
root@1fb642264605:/data# ifconfig -s
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
lo       65536 386954954      0      0 0      386954954      0      0      0 LRU
tap0     65520     4690      0      0 0          4502      0      0      0 BRU

I went with tap0 because the docs said Using the loopback interface (lo) does likely not yield the desired results.
I tried again with lo and still couldn't get the server to resume without exec-ing rcon-cli on the container. So I'm not sure what I'm missing for knockd or autopause to resume the server without manually starting it via rcon-cli.

Might there be a problem with me running the container with podman as a user vs docker as root? Podman with --privileged allowed autopause and knockd to start, but I'm wondering if there's more that necessary for listening on the network interface from a user vs root perspective.

You could try --cap-add=NET_ADMIN instead of priveledged

It needs privileged or to be ran as root. I might have some more time tomorrow to try root. Without privileged :
[init] Starting the Minecraft server...
could not open tap0: tap0: You don't have permission to capture on that device (socket: Operation not permitted)
[Autopause loop] Failed to start knockd daemon.
[Autopause loop] Probable cause: Unable to attach to interface "tap0".
[Autopause loop] Available interfaces within the docker container:
[Autopause loop]   lo tap0

podmon runs the containers as non-root, correct? In that case, docker really might solve the issue, or something to have access to the interfaces

It can and I am running it as a separate user just for security because the
server will be exposed to the internet. Tomorrow I might get a chance to
test a local one as root and will report back.
On Sat, Jun 12, 2021 at 3:47 PM Michael Kirsch @.***>
wrote:

podmon runs the containers as non-root, correct? In that case, docker
really might solve the issue, or something to have access to the interfaces
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/itzg/docker-minecraft-server/issues/555#issuecomment-860122509,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADANPQLD34A55D5UDDMJLTTSPPZXANCNFSM4N2NTRRQ
.

FYI, latest image (and tagged at 1.8.0) includes knock 0.8.1-r0

I noticed that knockd is on version 0.7 in the current "latest" image. Can you update that to 0.8 as well? 0.7 lacks IPv6 support and i had to mount knockd 0.8 myself to resume it via IPv6

Forgot about this thread! Thanks for the bump. I should maybe just create a separate issue, but updating here because there's already history here.
Tested out with podman as root like was suggested above. That works for autopause. I noticed as a rootfull container, it had eth0 for it's interface. All settings for the container that I tried as root worked with autopause! Still required some capabilities for autopause to work, but I didn't have time to narrow that down and just ran with --privileged.
I tried updating the network for the rootless container to have eth0 as well, but that did not change anything. I did this with --net bridge and the container then had eth0 like the rootfull one did. Still required me to run rcon-cli though to get it to wake up again.
FYI --privileged and --cap-add are mutually exclusive now for podman. --privileged add all the capabilities that are available. One note from the docs that caught my attention was
Rootless containers cannot have more privileges than the account that launched them.

My guess is the rootless user is missing some permissions that are required for knockd to work properly.

I noticed that knockd is on version 0.7 in the current "latest" image. Can you update that to 0.8 as well? 0.7 lacks IPv6 support and i had to mount knockd 0.8 myself to resume it via IPv6

I am pulling knockd via the base image distro release, so it is constrained to 0.7.0:
https://github.com/itzg/docker-minecraft-server/blob/f97cea4b06ee1c236b5199ec848df9005d3f53a1/Dockerfile#L22

I'm having this problem with only one of my servers.
I have 2 containers running at the same time, 1 vanilla and 1 modded.
Both server.properties are the same, both have environment "ENABLE_AUTOPAUSE=TRUE".
The modded server never comes back from pause.
I've noticed the vanilla server gets "knocked" every 5 minutes or so while the modded server does not.
How do I configure the modded server to get knocked (I would suppose that would take care of the problem) ?

I've been trying to figure out how to make this work all week and finally got it.
I tested on vanilla servers on a docker swarm and on 2 modded servers with docker-compose, the only ones that do not get out of autopause are the servers that do not have the default minecraft port.
Compose file important parts :
    ports:
      - 25566:25565
    environment:
      - OVERRIDE_SERVER_PROPERTIES=TRUE
      - MAX_TICK_TIME=-1
      - ENABLE_AUTOPAUSE=TRUE
      - AUTOPAUSE_TIMEOUT_INIT=30
      - AUTOPAUSE_TIMEOUT_EST=10

Autopause logs :
[init] Running as uid=1000 gid=1000 with /data as 'drwxrwxr-x. 13 65534 65534 29 Nov 18 20:35 /data'
[init] Resolved version given 1.17.1 into 1.17.1
[init] Autopause functionality enabled
[Autopause loop] MC Server listening for connections - stopping in 30 seconds
[Autopause loop] No client connected since startup / knocked - stopping
[22:15:28] [RCON Listener #1/INFO]: Thread RCON Client /<IP_ADDRESS> started
[22:15:29] [Server thread/INFO]: [Rcon: Saved the game]
[22:15:29] [RCON Client /<IP_ADDRESS> #2/INFO]: Thread RCON Client /<IP_ADDRESS> shutting down
[2021-11-18T22:15:30-05:00] [Autopause] Pausing Java process

Doing ifconfig inside the container, I get eth0, eth1 and lo.
Installed tcpdump ('cus I don't know what else to use) I figured out eth1 is the active interface for the server.
Doing tcpdump -ani eth1  port 25565 I do get a "ping" when I try to connect to the server.
I looked what was in knockd-config.cfg and manually sending /usr/sbin/gosu minecraft:minecraft /autopause/resume.sh does what it should do.
I added :
    environment:
      - AUTOPAUSE_KNOCK_INTERFACE=eth1

And voila!
The server now finally resumes as I was expecting it to!
I was not thinking I was using any special server configuration (ubuntu cli), so I did not expect to have to set the knock interface.
As an FYI : vanilla minecraft works A+ on a docker swarm.
You "just" need a shared storage between all workers which is a bit of work to get going but really nice.
I have not been able to make it work on a modded server yet.

@NotFluffy that's great you figured out your scenario! I'm not sure what would lead to a container having two interfaces other than maybe the networks config. Can you provide the rest of your compose file?

Here are the 2 vanilla server compose files.
I tried to keep them as close as possible to each other to make sure nothing else could explain the problem.
version: '3.9'

services:
  minecraft_test:
    image: itzg/minecraft-server:java16
    networks:
      - mc
    ports:
      - 25565:25565
    volumes:
      - data:/data
    environment:
      - EULA=TRUE
      - VERSION=1.17.1
      - TYPE=VANILLA
      - OVERRIDE_SERVER_PROPERTIES=TRUE
      - MOTD=A test
      - DIFFICULTY=easy
      - ENABLE_WHITELIST=true
      - OVERRIDE_OPS=true
      - OPS=me
      - ANNOUNCE_PLAYER_ACHIEVEMENTS=true
      - SNOOPER_ENABLED=false
      - MAX_TICK_TIME=-1
      - VIEW_DISTANCE=10
      - INIT_MEMORY=750m
      - MEMORY=4G
      - TZ=America/Toronto
      - ENABLE_AUTOPAUSE=TRUE
      - AUTOPAUSE_TIMEOUT_INIT=30
      - AUTOPAUSE_TIMEOUT_EST=10
      - DEBUG=true

volumes:
  data:
    name: mc_test_data
    external: true

networks:
  mc:
    name: mc_test
    external: true

version: '3.9'

services:
  minecraft_vanilla:
    image: itzg/minecraft-server:java16
    networks:
      - mc
    ports:
      - 25566:25565
    volumes:
      - data:/data
    environment:
      - EULA=TRUE
      - VERSION=1.17.1
      - TYPE=VANILLA
      - OVERRIDE_SERVER_PROPERTIES=TRUE
      - MOTD=The server with a big circle house.
      - DIFFICULTY=hard
      - ENABLE_WHITELIST=true
      - OVERRIDE_OPS=true
      - OPS=someone,me
      - ANNOUNCE_PLAYER_ACHIEVEMENTS=true
      - SNOOPER_ENABLED=false
      - MAX_TICK_TIME=-1
      - VIEW_DISTANCE=10
      - INIT_MEMORY=750m
      - MEMORY=4G
      - TZ=America/Toronto
      - ENABLE_AUTOPAUSE=TRUE
      - AUTOPAUSE_TIMEOUT_INIT=30
      - AUTOPAUSE_TIMEOUT_EST=10
      - AUTOPAUSE_KNOCK_INTERFACE=eth1
      - DEBUG=true

volumes:
  data:
    name: mc_vanilla_data
    external: true

networks:
  mc:
    name: mc_vanilla
    external: true

Creation of networks :
docker network create -d overlay --attachable mc_vanilla
docker network create -d overlay --attachable mc_test
And just for anyone who wants to make this on a docker swarm, you have to create the volumes from the shared storage on each workers (do not prune volumes, it will delete these and you don't want that) :
docker volume create --driver local \
-o o=bind \
-o type=none \
-o device=/shared_storage/vanilla_mc/data \
mc_vanilla_data

what is the correct syntax to add JVM_DD_OPTS=disable.watchdog:true to a docker-compose.yml file?

Additional related info: Using the alpine image does not have gosu installed by default, and thus the resume script cannot run.

what is the correct syntax to add JVM_DD_OPTS=disable.watchdog:true to a docker-compose.yml file?

Since I came across my OWN question 2 years later while setting this up, the correct format for a paper docker-compose.yml is
 JVM_DD_OPTS: disable.watchdog=true


[FIX] Add a periodic task to cleanup old data

Task added
Setting added to define how long data should be kept in the DB
README updated

Fixes #1

as noted on discord, accepting but am gonna update some bits.


Deleted code samples

Why did code samples for C#, Java, and Node were deleted by Pull Request #12?

Customers should be using the supported Azure Functions language workers for those languages, not custom handlers. We found that having them here sends led to confusions.

I understand, but there are edge cases on Node, for example, that custom handlers saves a lot of time and work.

Can you share what these edge cases are? Would like to address them in the Node worker if possible.

Customers should be using the supported Azure Functions language workers for those languages, not custom handlers.

@anthonychu  Then C# with .NET 5.0 supported?

Also README.md still includes those samples. Those links cause 404.

The following samples demonstrate how to implement a custom handler in the following languages:

C#
Go
Java
JavaScript
R
Rust

Could someone from the team elaborate on why they were removed? For Java, for instance, using a custom handler means:

I can choose the version I want
I can have DI
I can use model binding / all the other Spring stuff
I can use well established test frameworks

...why wouldn't I want that?

@anthonychu for Node, there is one edge case that I can share.
If I have a big legacy project in Express.js with lots of endpoints and I want to deploy it to Azure Functions without rearchitecting my code, then Custom Handler can do the job.


Improve Bitcoin Properties tab

Prerequisites

[X] I have searched for existing issues that already suggest this feature.

Describe the Feature Request
I have modified the main.rs to improve Properties tab for the bitcoin core wrapper and submitted a PR about a week ago and would be grateful if someone could get around to checking it. I have already built and tested it to ensure it works properly and have included a screenshot of what the updated code looks like.
Link to PR: https://github.com/Start9Labs/bitcoind-startos/pull/154
Describe the Use Case
Properties added for added convenience:

REAL Max Mempool Size in MB given core's overhead and safety buffer
Current Mempool Usage in MB and as a percentage
Mempool Transaction Count
Progress to next halving added (counts in years, days, hours, mins)
Node uptime since service was started
Current total supply of Bitcoin based on issuance schedule (may be very slightly inaccuate compared to gettxoutsetinfo but is great for monitoring at a glance without needing to wait)

I also consolidated the blockchain sync summary to one line to save screen real estate.
Here is an image of what it looks like:

Describe Preferred Solution
Link to PR: https://github.com/Start9Labs/bitcoind-startos/pull/154
Describe Alternatives
Reasons 'total bitcoin supply' may not be exactly the same as gettxoutinfo:

Varied Inputs and Outputs: The total amount shown represents all unspent coins across all outputs, which includes not only block rewards but also many user transactions, each with its unique input and output values. These transactions may not align with exact multiples of the block reward.
Fee Distribution: Transaction fees also contribute to the total amount but are not typically set to neat multiples. They are instead dynamic values determined by users and miners, so they add irregular values to the total supply.
Decay in Reward Structure: If you're working within a system where block rewards decrease over time, previous block rewards might have been higher or lower than 3.125, meaning historical outputs would still affect the total.
Fractional Accumulation: With every transaction and UTXO creation, there could be small rounding effects or dust amounts, which, over thousands of transactions, contribute to a value that doesn't perfectly align with any specific multiple.

However overall this is a great at a glance metric imo to keep a check on things (with a fraction of a percent difference), especially since gettxoutsetinfo is resource and time intensive.
Anything else?
No response

@copy2018 thank you for the request. Please re-create this ticket in the Bitcoin service repository, as it is not a StartOS issue.


Timing/order issue with multiple clients connecting to different ADS ports

Observed a surprising behavior when using AdsRouterConsole as a local proxy to connect multiple clients to different target instances on the same IP/AmsNetID.
Example minimal reproduction:
using TwinCAT.Ads;

var ports = new List<int>
{
    851,
    852,
    853,
    854
};

var clients = new List<AdsClient>();
foreach (var port in ports)
{
    var client = new AdsClient();
    clients.Add(client);
    client.Connect("<IP_ADDRESS>.1.1", port);
}

var tasks = clients.Select(client => client.ReadStateAsync(default));

await Task.WhenAll(tasks);

var i = 0;
foreach (var task in tasks)
{
    i++;
    Console.WriteLine($"Task {i} - Success: {task.Result.Succeeded} - Result: {task.Result.State.AdsState.ToString()} - Error: {task.Result.ErrorCode}");
}

foreach (var client in clients) client.Disconnect();

Running this, 3 of the 4 tasks fail:
Task 1 - Success: True - Result: Run - Error: Succeeded
Task 2 - Success: False - Result: Invalid - Error: ClientSyncTimeOut
Task 3 - Success: False - Result: Invalid - Error: ClientSyncTimeOut
Task 4 - Success: False - Result: Invalid - Error: ClientSyncTimeOut

AdsRouterConsole outputs:
warn: TwinCAT.Ads.AdsRouterService.RouterService[0]
      Connection reset by peer

When limited to a single client, it works:
Task 1 - Success: True - Result: Run - Error: Succeeded

The strange thing is: After the first run with a single client, subsequent attempts to connect 4 clients works:
Task 1 - Success: True - Result: Run - Error: Succeeded
Task 2 - Success: True - Result: Run - Error: Succeeded
Task 3 - Success: True - Result: Run - Error: Succeeded
Task 4 - Success: True - Result: Run - Error: Succeeded

It seems surprising that "warming up" AdsRouterConsole with an initial singular connection has an effect.

Tried this out, and I can commit the issue also for the latest versions of the AdsRouterConsole. The issue solved in 6.1.304 is an AdsClient issue that is independent of this problem here. Thanks for reporting.
As you already found out, the issue only appears, when more than one parallel connection is established to a target system initially    the first time. Subsequent connection requests will run without issues ...

I released a fix in Version 6.1.312. Hope that helps.

Thanks Ralf!
May I ask you to push the updated source code to this repo, and/or check that the package Beckhoff.TwinCAT.Ads.AdsRouterConsole is published on NuGet?
I get Unable to find package using nuget install Beckhoff.TwinCAT.Ads.AdsRouterConsole (followed instructions on https://www.nuget.org/packages/Beckhoff.TwinCAT.Ads under "Systems without TwinCAT Installation").

@crishoj install the latest Beckhoff.TwinCAT.Ads.TcpRouter package to your router project with
dotnet add package Beckhoff.TwinCAT.Ads.TcpRouter --version 6.1.312
After that it seems to work as should, great! Same thing with Node.js version.
Task 1 - Success: True - Result: Run - Error: NoError
Task 2 - Success: True - Result: Run - Error: NoError
Task 3 - Success: True - Result: Run - Error: NoError
Task 4 - Success: True - Result: Run - Error: NoError

As @jisotalo stated. The implementation of the Router is in the Beckhkoff.TwinCAT.Ads.TcpRouter package. The AdsRouterConsole is only the host sample application demonstrating the use that package. As starting point use the samples from here:
https://github.com/Beckhoff/TF6000_ADS_DOTNET_V5_Samples/tree/main/Sources/RouterSamples
The AdsRouterConsole package on Nuget is obsolete. It is unlisted already and it shouldn't be found by standard nuget search anymore ...

@RalfHeitmann it seems that the AdsRouterConsole  is mentioned multiple times at https://www.nuget.org/packages/Beckhoff.TwinCAT.Ads
There is even an install command for it. That might confuse alot.

The implementation of the Router is in the Beckhkoff.TwinCAT.Ads.TcpRouter package. The AdsRouterConsole is only the host sample application demonstrating the use that package.

Right, I had forgotten about this distinction.
After recompiling with the latest Beckhoff.TwinCAT.Ads.TcpRouter, the problem is gone. Thanks again for the rapid fix, Ralf!


Can You Please Elaborate Block All Access Through Public Endpoint?

Hi I have question for block all access through public endpoint.
Document says:
"
You can also use the firewall to block all access through the public endpoint when using private endpoints.
"
So is below configuration the right setting to block all access through public endpoint?

[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 2632af75-7664-b79b-ac5b-c7c17e6a609a
Version Independent ID: 6ecf71b6-77c4-3ab6-8ec9-353ea407d70f
Content: Configure Azure Storage firewalls and virtual networks
Content Source: articles/storage/common/storage-network-security.md
Service: storage
Sub-service: common
GitHub Login: @santoshc
Microsoft Alias: normesta

@RonnieQuan Thanks for the feedback. We're investigating this and will get back to you shortly.

@RonnieQuan Apologies for the delayed response here. That is correct, you can see this information in the section here. Also noted at the top of the page in the Important section: Turning on firewall rules for your storage account blocks incoming requests for data by default, unless the requests originate from a service operating within an Azure Virtual Network (VNet) or from allowed public IP addresses. Requests that are blocked include those from other Azure services, from the Azure portal, from logging and metrics services, and so on. I am closing out this issue but please tag me and let us know if you have further questions and we can reopen if necessary.


feat: added new sidebar component - simple sidebar

Added a new subsection in Navigation called Sidebar
Made a sidebar component - SImple SIdebar

Added new component sidebar with a header on main area

@hauptrolle is there any way I can control the transition speed ? I feel like close transition is happening but it's faster. Nothing in docs about that.

@hauptrolle is there any way I can control the transition speed ? I feel like close transition is happening but it's faster. Nothing in docs about that.

Damn did not know that. I will merge it this way. :)

@all-contributors please add @sudoDeznit  for code


error

-severity low,medium,high not work in nuclei

What version of Nuclei are you running? Could you please paste the error you are getting here?

nuclei v2.1

It should be working in Nuclei v2.1.1 which is installed by the script. Could you please install the latest version of Nuclei and try again? (It's merged here).

working sorry pro Thank you
you can add some tools like cookieless hinject

Glad to hear it :)
Nuclei should be able to detect host header injection vulnerabilities. I'm not aware of Cookieless, but for XSS I was thinking of implementing Dalfox. Other suggestions are of course very welcome! To be continued!


Rotate list

Related Issue
Closes: #776
Describe the changes you've made
Algorithm for rotating the linked list
Language used for coding
Java

Have you read Contribution Guidelines
Your folder name should be same as Problem name on platform which is "Rotate List"


" keymap intersects normal workflow

I wanted to yank and paste from the system clipboard to neovim with "+y and "+p, it works fine until I enter tmux. So I dug in and found this key bind from the plugin intersects it.
https://github.com/aserowy/tmux.nvim/blob/57220071739c723c3a318e9d529d3e5045f503b8/lua/tmux/copy.lua#L95
Can you change this or let me customize the key bindings?
For now, I will use vim.api.nvim_del_keymap('n', '"') as a workaround.

Heho,
normally this uses tmux 'registers' as middleware. Thus, system copies should end up in tmux and nvim uses tmux which includes your systems clipboard.
Can you paste your systems clipboard into tmux at all?
Regards
Alexander

Yeah, I can paste from the system clipboard to tmux with cmd+v on my Mac, but can't copy from tmux neovim to the system clipboard and I still rely on "+y that I bind to y on my config

I just realized that what the " and + registers don't synchronize with the system clipboard. So deleting the keymap still doesn't work for me though.

cmd+v will not use tmux buffers. maybe your integration of system clipboard in tmux is not working as well.
you can try to copy into system clipboard and let tmux show his buffers to control if the copied stuff gets shown in tmux. if not, the plugin is working as expected :)

Yes, the system clipboard content is not in tmux buffer
from your comment before

normally this uses tmux 'registers' as middleware. Thus, system copies should end up in tmux and nvim uses tmux which includes your systems clipboard.
I don't think it is the default behavior of tmux, system copies don't show up in tmux buffer at all.

Back to the main point, I can use sync_clipboard = false to make nvim handle system clipboard yank and paste now, but the " keymap prevents normal behavior. I suggest changing it, or give option for users to change it.

If you do not intend to use tmux clipboard, disable it completly. Then, no bindings get set.
copy_sync = {
-- enables copy sync. by default, all registers are synchronized.
-- to control which registers are synced, see the sync_* options.
enable = false,

I do use the tmux clipboard for syncing clipboard between tmux panes, just not the system clipboard.

I guess the request is out of scope of this plugin. You should get system clipboard sync under tmux working. Thus, you can facilitate the plugin to work the intended way.
We should not add further setup customizations to enable a not fully functional setup i guess.
Im closing this issue. If you need help to get system -> tmux working, feel free to chat here. A good start into this topic would be e.g. https://www.freecodecamp.org/news/tmux-in-practice-integration-with-system-clipboard-bcd72c62ff7b/


Update UG DG HelpGuide

Codecov Report

Merging #175 (b84563b) into master (d01ed5f) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #175   +/-   ##
=========================================
  Coverage     72.31%   72.31%
  Complexity      504      504
=========================================
  Files            79       79
  Lines          1557     1557
  Branches        185      185
=========================================
  Hits           1126     1126
  Misses          375      375
  Partials         56       56

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d01ed5f...b84563b. Read the comment docs.

LGTM


[IOS13][flutter run] ios-display issue / module importing failed

I'm using flutter run on Mac OS Mojave, Xcode 11, and a ios 13.1 beta device on usb.
I tried on Xcode 10.3, on stable, beta, dev channels, and with ios-display 1.9.4 and 1.10.0-beta.3.
On flutter stable channel :
2019-09-16 01:04:44.142 ios-deploy[21322:293363] [ !! ] Error 0xe800007f: Device doesn't support wireless sync. AMDeviceStartService(device, CFSTR("com.apple.debugserver"), &gdbfd, NULL)
Could not install build/ios/iphoneos/Runner.app on b54286e56fea33387fc501ba1c20f90ee4b7be96.
Try launching Xcode and selecting "Product > Run" to fix the problem:

Product>Run on Xcode  works great, but flutter run not (meaning no hot reloading).
It seems to be partially fixed on master channel thanks to #40398 , however on master i get:
Xcode build done.                                           20,4s
error: module importing failed: invalid pathname
Installing and launching...

Only the launch screen shows up on the device.

Hello,
Exactly same issue. And build work directly from Xcode. The problem happened with flutter command.

Thanks for filing this! I am currently working on a fix for this. I am closing this issue as a duplicate of #40448, please follow that issue for updates.


Error printing sys.path

Environment data
VS Code version: 1.19.2
Python Extension version: 0.9.1
Python Version: 2.7.14 (Anaconda)
OS and version: Windows 10 , version 1709, build 16299.192
Actual behavior
Sometimes i get a No such file or directory error pointing at the line where print sys.path is placed. This does not happen every time the code is run and does not happen outside of Visual Studio Code (command prompt, powershell, ipython and jupyter are fine).
Expected  behavior
I expected the code to always print correctly with no errors.
Steps to reproduce:
Create a new file in Visual Studio Code and add the following lines
import sys
print sys.path

Right click inside window and select Run python file in Terminal.
Logs
Output from Python output panel
C:\Users\aless\Documents\code\PythonProjects\Test>python c:/Users/aless/Documents/code/PythonProjects/Test/test.py
['c:\\Users\\aless\\Documents\\code\\PythonProjects\\Test', 'C:\\Users\\aless\\Miniconda2\\python27.zip', 'C:\\Users\\aless\\Miniconda2\\DLLs', 'C:\\Users\\aless\\Miniconda2\\lib', 'C:\\Users\\aless\\Miniconda2\\lib\\plat-win', 'C:\\Users\\aless\\Miniconda2\\lib\\lib-tk', 'C:\\Users\\aless\\Miniconda2', 'C:\\Users\\aless\\Miniconda2\\lib\\site-packages', 'C:\\Users\\aless\\Miniconda2\\lib\\site-packages\\win32'Traceback (most recent call last):
  File "c:/Users/aless/Documents/code/PythonProjects/Test/test.py", line 2, in <module>
    print sys.path
IOError: [Errno 2] No such file or directory

Output from Console window (Help->Developer Tools menu)
/C:/Program Files/Microsoft VS Code/resources/app/out/vs/workbench/workbench.main.js:4054 [C:\Users\aless\.vscode\extensions\ms-vscode.cpptools-0.14.6]: Il comando `workbench.action.gotoSymbol` è presente più volte nella sezione `commands`.
/C:/Program Files/Microsoft VS Code/resources/app/out/vs/workbench/workbench.main.js:4054 [C:\Users\aless\.vscode\extensions\ms-vscode.cpptools-0.14.6]: Il comando `workbench.action.showAllSymbols` è presente più volte nella sezione `commands`.
/C:/Program Files/Microsoft VS Code/resources/app/out/vs/workbench/workbench.main.js:2571 [Extension Host] debugger listening on port 9333
/C:/Program Files/Microsoft VS Code/resources/app/out/vs/workbench/workbench.main.js:2569 [Extension Host] debugger inspector at %cDebugger listening on port 9333.
Warning: This is an experimental feature and could change at any time.
To start debugging, open the following URL in Chrome:
    chrome-devtools://devtools/bundled/inspector.html?experiments=true&v8only=true&ws=<IP_ADDRESS>:9333/6fc63825-c520-4878-abb9-a26e74f6b157

This is my fist time pointing out an issue so if I made some mistakes i'm sorry, i'm no expert, blame my lack of experience.

+1
Also experiencing a similar problem - it looks like print occasionally fails with an IOError.  It's very hard to reproduce consistently.

In vscode do:

Python
in the python terminal type for x in range(0,1000): print x

This will fail under powershell or cmd (INSIDE VSCODE ONLY)
If you run it in a separate CMD or Powershell window (not under vscode) it succeeds

I believe this is the same as a Windows bug that has been reported upstream on VS Code and should get fixed at some point (I have no insight on Windows bugfix release schedules so I can't give an ETA).


Copypaste fail

Copy and paste inside the Atom works. When you insert one Atom to another application it hangs up for a second and nothing sticks. Then, the clipboard is cleared. When you paste from another application in the Atom, the Atom sometimes falling, sometimes inserts. Change the frequency, once every hour or two, once every few inserts.

duplicate of https://github.com/atom/atom/issues/6221

Yeah, I'm guessing this is the same problem as in #6221. Closing this in favor of that issue, but let us know if you can reproduce this after that issue is fixed.


chore: release v0.6.13

:robot: I have created a release beep boop
0.6.13 (2024-03-14)
What's Changed

fix: correctly set default version in install.sh by @coffeebeats in https://github.com/coffeebeats/gdenv/pull/205
fix(ci): correctly update PATH in setup action by @coffeebeats in https://github.com/coffeebeats/gdenv/pull/207
fix(ci): correctly export environment variable in setup action by @coffeebeats in https://github.com/coffeebeats/gdenv/pull/208
fix(ci): correctly reference home directory in action; use correct gdenv flag by @coffeebeats in https://github.com/coffeebeats/gdenv/pull/209
fix(ci): correctly use environment variable in cache path by @coffeebeats in https://github.com/coffeebeats/gdenv/pull/210
fix(ci): use correct path in cache key by @coffeebeats in https://github.com/coffeebeats/gdenv/pull/211
fix(ci): conditionally check for executable on path during setup by @coffeebeats in https://github.com/coffeebeats/gdenv/pull/212

Full Changelog: https://github.com/coffeebeats/gdenv/compare/v0.6.12...v0.6.13

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/coffeebeats/gdenv/releases/tag/v0.6.13 :sunflower:


docs: add aria-label for actionbar close button (CSS-464)

Description
Adds aria-label="Clear selection" to the Close Buttons found within the Action Bar component.
CSS-464
How and where has this been tested?

How this was tested:
Browser(s) and OS(s) this was tested with:

Screenshots

To-do list

[ ] If my change impacts other components, I have tested to make sure they don't break.
[ ] If my change impacts documentation, I have updated the documentation accordingly.
[ ] I have read the CONTRIBUTING document.
[ ] I have tested these changes in Windows High Contrast mode.
[ ] I have updated any relevant storybook stories and templates.

[ ] This pull request is ready to merge.

These look good! My only thought is that while you're adding aria labels to the close button, you could fix them for the other buttons in this component as well. I can't comment on the code since you didn't change it but there are unneeded aria labels on the SVGs that have an aria-hidden="true" attribute. These should be removed if the button has text, but added to the <button> element if they only contain the SVG. Currently they aren't read at all.
<svg class="spectrum-Icon spectrum-Icon--sizeM spectrum-ActionButton-icon" focusable="false" aria-hidden="true" aria-label="Delete">

Great suggestion. I just pushed up a commit to take care of those.


THC 1068 optimize dev hub search function

https://adjustcom.atlassian.net/browse/THC-1068

@Sporiff preview is working now with environment variables connected to the test Typesense cloud