Memory keeps increasing as we viewer.layers.pop(0) and viewer.add_image()

🐛 Bug
I am using napari (thanks for this great tool!) to browse a large folder of images and using callback functions to move files to different folders.
For instance, the callback for the space key is as follow:
    @viewer.bind_key("Space")
    def space_callback(viewer):
        global i, all_files
        viewer.layers.pop(0)
        shutil.move(all_files[i], os.path.join(FOLDER, os.path.basename(all_files[i])))
        all_files.pop(i)
        i = i % len(all_files)
        img = imread(all_files[i])
        viewer.add_image(img, name=str(i))

The issue is that the memory usage keeps increasing, while I would expect it to be constant since, in theory, I always only have one image in memory at the same time.
I thought viewer.layers.pop(0) would release memory but this is not the case.

Hi @kevin-keraudren, that's no good! We've been making some improvements to adding/ removing layers that will be in the upcoming 0.4.3 release, but might not be in the current 0.4.2 release (See #1955 and #1984). Are you testing with 0.4.2 or are you on the current version of master? If you're on 0.4.2 can you let me know if you see this on master, I'm hoping it will be fixed already!!

Just adding a note that we should make sure this is fixed before 0.4.3. I havn't checked yet, but I'm hoping it is fixed on master

@tlambert03 do you mind taking a quick look at this one. I think it was fixed by those PRs above, but want to be sure before we close!

@tlambert03 do you mind taking a quick look at this one. I think it was fixed by those PRs above, but want to be sure before we close!

Nope, sadly this is still there:
import napari
import numpy as np
import psutil
import os

v = napari.Viewer()

def get_process_memory():
    return psutil.Process(os.getpid()).memory_info().rss / 1e6

for i in range(1000):
    m = get_process_memory()
    v.add_image(np.random.rand(1024, 1024))
    v.layers.pop(0)
    print(f"Mem used after {i:3} layers: {get_process_memory():0.2f} (MB)")

Nope, sadly this is still there:
import napari
import numpy as np
import psutil
import os

v = napari.Viewer()

def get_process_memory():
    return psutil.Process(os.getpid()).memory_info().rss / 1e6

for i in range(1000):
    m = get_process_memory()
    v.add_image(np.random.rand(1024, 1024))
    v.layers.pop(0)
    print(f"Mem used after {i:3} layers: {get_process_memory():0.2f} (MB)")

Thanks for that great script @tlambert03. I can confirm this is not fixed on master, and it looks like it has always beens an issue (at least for over a year). I've bumped the milestone on this as we won't have it fixed for 0.4.3, but thanks for adding the high priority label - we definitely want to get to the bottom of this!!

Thanks for that great script @tlambert03. I can confirm this is not fixed on master, and it looks like it has always beens an issue (at least for over a year). I've bumped the milestone on this as we won't have it fixed for 0.4.3, but thanks for adding the high priority label - we definitely want to get to the bottom of this!!

was looking at this some more.  It appears that we have ~7 remaining references left to clean up when we append and pop a layer.  Running this script:
import gc
import weakref
import sys
import napari

viewer = napari.Viewer()
r = weakref.ref(viewer.add_points())

viewer.layers.pop()
print(gc.collect(), "unreachable objects after collection")

print("referenced layer is:", repr(r()))

print(f"\n{sys.getrefcount(r()) - 1} remaining references:")
for q in enumerate(gc.get_referrers(r())):
    print(q[0], q[1])

gives this output:
0 unreachable objects after collection
referenced layer is: <Points layer 'Points' at 0x7fe62169dbe0>

7 remaining references:
0 {<Points layer 'Points' at 0x7fe62169dbe0>: <napari._vispy.vispy_points_layer.VispyPointsLayer object at 0x7fe62169c520>}
1 {'layer': <Points layer 'Points' at 0x7fe62169dbe0>, 'grid_layout': <PyQt5.QtWidgets.QGridLayout object at 0x7fe5f151f310>, 'opacitySlider': <PyQt5.QtWidgets.QSlider object at 0x7fe5f151f9d0>, 'blendComboBox': <PyQt5.QtWidgets.QComboBox object at 0x7fe5f151f8b0>, 'sizeSlider': <PyQt5.QtWidgets.QSlider object at 0x7fe5f151f820>, 'faceColorEdit': <napari._qt.widgets.qt_color_swatch.QColorSwatchEdit object at 0x7fe5f14e4280>, 'edgeColorEdit': <napari._qt.widgets.qt_color_swatch.QColorSwatchEdit object at 0x7fe5f14e4790>, 'symbolComboBox': <PyQt5.QtWidgets.QComboBox object at 0x7fe5f14e4700>, 'ndimCheckBox': <PyQt5.QtWidgets.QCheckBox object at 0x7fe5f14e4b80>, 'select_button': <napari._qt.widgets.qt_mode_buttons.QtModeRadioButton object at 0x7fe5f14e4a60>, 'addition_button': <napari._qt.widgets.qt_mode_buttons.QtModeRadioButton object at 0x7fe5f14e49d0>, 'panzoom_button': <napari._qt.widgets.qt_mode_buttons.QtModeRadioButton object at 0x7fe5f14e4040>, 'delete_button': <napari._qt.widgets.qt_mode_buttons.QtModePushButton object at 0x7fe5f14e43a0>, 'textDispCheckBox': <PyQt5.QtWidgets.QCheckBox object at 0x7fe5f14e4310>, 'button_group': <PyQt5.QtWidgets.QButtonGroup object at 0x7fe5f14e4160>}
2 {'events': None, 'layer': <Points layer 'Points' at 0x7fe62169dbe0>, '_array_like': False, 'node': <Compound at 0x7fe5f15344f0>, 'MAX_TEXTURE_SIZE_2D': 16384, 'MAX_TEXTURE_SIZE_3D': 2048}
3 {'layer': <Points layer 'Points' at 0x7fe62169dbe0>, 'mode': <Mode.SELECT: 'select'>}
4 {'layer': <Points layer 'Points' at 0x7fe62169dbe0>, 'mode': <Mode.ADD: 'add'>}
5 {'layer': <Points layer 'Points' at 0x7fe62169dbe0>, 'mode': <Mode.PAN_ZOOM: 'pan_zoom'>}
6 {'layer': <Points layer 'Points' at 0x7fe62169dbe0>}

The major culprits appear to be all the places where the layer object is used as a key in dicts in other objects, as well as the self.layer = layer assignment on all of the vispy and qt_controls.  For instance, if I just change self.layer = weakref.ref(layer) (and then dereference as needed) in the vispy layers and qt controls, I can drop the remaining references by two.  Will try to get it down to zero

was looking at this some more.  It appears that we have ~7 remaining references left to clean up when we append and pop a layer.  Running this script:
import gc
import weakref
import sys
import napari

viewer = napari.Viewer()
r = weakref.ref(viewer.add_points())

viewer.layers.pop()
print(gc.collect(), "unreachable objects after collection")

print("referenced layer is:", repr(r()))

print(f"\n{sys.getrefcount(r()) - 1} remaining references:")
for q in enumerate(gc.get_referrers(r())):
    print(q[0], q[1])

gives this output:
0 unreachable objects after collection
referenced layer is: <Points layer 'Points' at 0x7fe62169dbe0>

7 remaining references:
0 {<Points layer 'Points' at 0x7fe62169dbe0>: <napari._vispy.vispy_points_layer.VispyPointsLayer object at 0x7fe62169c520>}
1 {'layer': <Points layer 'Points' at 0x7fe62169dbe0>, 'grid_layout': <PyQt5.QtWidgets.QGridLayout object at 0x7fe5f151f310>, 'opacitySlider': <PyQt5.QtWidgets.QSlider object at 0x7fe5f151f9d0>, 'blendComboBox': <PyQt5.QtWidgets.QComboBox object at 0x7fe5f151f8b0>, 'sizeSlider': <PyQt5.QtWidgets.QSlider object at 0x7fe5f151f820>, 'faceColorEdit': <napari._qt.widgets.qt_color_swatch.QColorSwatchEdit object at 0x7fe5f14e4280>, 'edgeColorEdit': <napari._qt.widgets.qt_color_swatch.QColorSwatchEdit object at 0x7fe5f14e4790>, 'symbolComboBox': <PyQt5.QtWidgets.QComboBox object at 0x7fe5f14e4700>, 'ndimCheckBox': <PyQt5.QtWidgets.QCheckBox object at 0x7fe5f14e4b80>, 'select_button': <napari._qt.widgets.qt_mode_buttons.QtModeRadioButton object at 0x7fe5f14e4a60>, 'addition_button': <napari._qt.widgets.qt_mode_buttons.QtModeRadioButton object at 0x7fe5f14e49d0>, 'panzoom_button': <napari._qt.widgets.qt_mode_buttons.QtModeRadioButton object at 0x7fe5f14e4040>, 'delete_button': <napari._qt.widgets.qt_mode_buttons.QtModePushButton object at 0x7fe5f14e43a0>, 'textDispCheckBox': <PyQt5.QtWidgets.QCheckBox object at 0x7fe5f14e4310>, 'button_group': <PyQt5.QtWidgets.QButtonGroup object at 0x7fe5f14e4160>}
2 {'events': None, 'layer': <Points layer 'Points' at 0x7fe62169dbe0>, '_array_like': False, 'node': <Compound at 0x7fe5f15344f0>, 'MAX_TEXTURE_SIZE_2D': 16384, 'MAX_TEXTURE_SIZE_3D': 2048}
3 {'layer': <Points layer 'Points' at 0x7fe62169dbe0>, 'mode': <Mode.SELECT: 'select'>}
4 {'layer': <Points layer 'Points' at 0x7fe62169dbe0>, 'mode': <Mode.ADD: 'add'>}
5 {'layer': <Points layer 'Points' at 0x7fe62169dbe0>, 'mode': <Mode.PAN_ZOOM: 'pan_zoom'>}
6 {'layer': <Points layer 'Points' at 0x7fe62169dbe0>}

The major culprits appear to be all the places where the layer object is used as a key in dicts in other objects, as well as the self.layer = layer assignment on all of the vispy and qt_controls.  For instance, if I just change self.layer = weakref.ref(layer) (and then dereference as needed) in the vispy layers and qt controls, I can drop the remaining references by two.  Will try to get it down to zero

more progress! was able to get the layer references down to zero after popping the layer. It seems to work fine for everything but the image layer, where something in the ImageSliceData chain of objects is holding on the numpy array

more progress! was able to get the layer references down to zero after popping the layer. It seems to work fine for everything but the image layer, where something in the ImageSliceData chain of objects is holding on the numpy array

i was eventually able to prevent the layer.data from being retained (memory used around 300 MB after adding/popping 100+ layers, down from ~1200MB after 100 layers), but I basically had to break the whole ImageSliceData, ImageSlice, ImageView, ImageLoader, ImageMassager, ImageWhisperer chain...  The main culprit seems to be the ImageView class which is hanging on to data even after layer deletion.  But my first attempts to use weakref on that class broke everything.  I'll open a PR with what I've got, and we can try to figure out what's going on in there

i was eventually able to prevent the layer.data from being retained (memory used around 300 MB after adding/popping 100+ layers, down from ~1200MB after 100 layers), but I basically had to break the whole ImageSliceData, ImageSlice, ImageView, ImageLoader, ImageMassager, ImageWhisperer chain...  The main culprit seems to be the ImageView class which is hanging on to data even after layer deletion.  But my first attempts to use weakref on that class broke everything.  I'll open a PR with what I've got, and we can try to figure out what's going on in there

Maybe we should revisit this after #3217 and see the current behavior. Things should be improved now, but might not be totally fixed yet

Hi, did you already find a way to circumvent memory increase when scrolling through a list of images? I am using napari 0.4.14 for scrolling through a list of 2D ome-zarr files and I have similar problems as @kevin-keraudren . It works very nice in the beginning, but after a while the memory is running full even though only one layer should be in memory at a time. Or is my code not optimal?

`
viewer = napari.Viewer()
viewer.open(layers[0]['Path'], contrast_limits = [0,65553], affine = layers[0]['Affine'], visible = True)
current_layer = 0
@viewer.bind_key('k')
def next_layer(viewer):
global current_layer
if current_layer < len(layers) -1:
next_layer = current_layer +1
viewer.open(layers[next_layer]['Path'], contrast_limits = [0,65553], affine = layers[next_layer]['Affine'], visible = True)
viewer.layers.pop(0)
current_layer = next_layer

Hi, did you already find a way to circumvent memory increase when scrolling through a list of images? I am using napari 0.4.14 for scrolling through a list of 2D ome-zarr files and I have similar problems as @kevin-keraudren . It works very nice in the beginning, but after a while the memory is running full even though only one layer should be in memory at a time. Or is my code not optimal?

viewer = napari.Viewer()
viewer.open(layers[0]['Path'], contrast_limits = [0,65553], affine = layers[0]['Affine'], visible = True)
current_layer = 0
@viewer.bind_key('k')
def next_layer(viewer):
    global current_layer
    if current_layer < len(layers) -1:
        next_layer = current_layer +1
        viewer.open(layers[next_layer]['Path'], contrast_limits = [0,65553], affine = layers[next_layer]['Affine'], visible = True)
        viewer.layers.pop(0)
        current_layer = next_layer

I'm still not sure about where the memory is leaking, but I would personally do this by setting the layer data, rather than removing and adding a new layer.
viewer = napari.Viewer()
viewer.open(layers[0]['Path'], contrast_limits = [0,65553], affine = layers[0]['Affine'], visible = True)
current_layer = 0

@viewer.bind_key('k')
def next_layer(viewer):
    global current_layer
    current_layer += 1
    if current_layer < len(layers):
        viewer.layers[0].data = read(layers[current_layer]['Path'])
        viewer.layers[0].affine = layers[current_layer]['Affine']

Of course, you need to find what reader function napari is using when you do viewer.open. You can do that by running your script and checking viewer.layers[0].source, but if you're doing ome-zarr then I suspect you will be fine using ome_zarr.io.parse_url(path).load().
I hope this will solve your memory issues @jjsirn, even if we still have a bug lying around!

@jni ,thank you for your suggestions!! Yet, I am probably not understanding ome_zarr good enough for getting it running... I do not manage to update the layer data.
Checking with 'viewer.layers[0].source' returns: reader_plugin='napari', but this did not help me to find a respective reader function.
Trying
viewer.layers[0].data = ome_zarr.io.parse_url(layers[current_layer]['Path'], mode='r').load()
resulted in the error message "array not found at path %r ''
I checked the description from Constantin Pape on how to read ome.zarr data (https://gist.github.com/constantinpape/69e3cb8e0401621365d814b4d6fda0bc) and modified the code to:
'loc = ome_zarr.io.parse_url(layers[current_layer]['Path'], mode="r")
zarr_reader = ome.zarr.reader.Reader(loc).zarr
viewer.layers[0].data = zarr_reader.load()'
I got the same error message as above.
Do you have an idea what is wrong?

Yes, if the reader plugin is napari, you are not in fact reading the ome-zarr data, only the array data, with dask.array.from_zarr. So, on the plus side, that's the function you need! 😃 The only downside is you miss metadata like scale, but you were not using that anyway so 🤷 I say go with it!

Currently using @tlambert03's script: https://github.com/napari/napari/issues/2027#issuecomment-766182182
there is 1 remaining reference:
11120 unreachable objects after collection
referenced layer is: <Points layer 'Points' at 0x28760bcd0>

1 remaining references:
0 {'layer': <Points layer 'Points' at 0x28760bcd0>}

@kephale that's cool! Is the expected number 1 or 0? 😅 Does this mean we can close this? 🥳

This cannot be closed. There is definitely still a leak. @tlambert03 did some amazing work pushing this to 0. This is one of the cases that depresses me about being a few months too late...

The last reference is I believe
        self.delete_button = QtModePushButton(
            layer, #this
            'delete_shape',
        )

from qt_points_control.py.

Ok, I got it in the bind_button a closure kept a ref to the button.
Now for tests.

Possibly, but I doubt it, the one I'm tracking right now is more the opposite. Objects living too long.
It's not impossible that the cleanup/fixes I'm doing right now may help trigger or narrow down above error.


Stub implementation of NetworKCurl

Adjusted enum declaration of RequestId type.
Stub implementation of NetworkCurl added.
Relates-to: OLPEDGE-369
Signed-off-by: Sergii Vostrikov<EMAIL_ADDRESS>

That's a draft changes, should not be treated as ready for merge.


Is it possible to password protect root directory but not it's subfolders?

I would like to password protect my root folder so user can't see all my files.

Have you tried to set protectedRoutes to ["/"] and add a .password file in the root directory?

Have you tried to set protectedRoutes to ["/"] and add a .password file in the root directory?

Yes but it password protects the folders inside it too.

I think that's the expected behaviour, setting / as protected will protect all your files with a password, what do you expect to do?

I think that's the expected behaviour, setting / as protected will protect all your files with a password, what do you expect to do?

I want to share folders inside the root but not let the users see the whole index.

If users can't see the index, how can they know what directories are there and access the files? only via direct links?
This feature haven't been implemented yet.
Or maybe I didn't get you correct? Could you please describe your use case by giving a specific example?

I want to share Supernatural.Academy.S01.1080p.AMZN.WEB-DL.DD+5.1.H.264-playWEB folder with users but I don't want to give them the permission to see the whole index.
They should be able to see only this folder and root folder should be password protected. This way they can't view/download all the files from my index.

Yes... This feature isn't implemented and probably not planned since it may cause confusion and difficulty in setup. Maybe you can reorganize your folder structure?

I want to share XXX.XXX.S01.1080p.AMZN.WEB-DL.DD+5.1.H.264-XXX folder with users but I don't want to give them the permission to see the whole index.
They should be able to see only this folder and root folder should be password protected. This way they can't view/download all the files from my index.
I don't know if you can use it now. Another onedrive index project onemanager is to turn on the no password to download files function, Turn on this function and know the link to directly download files other than password files. (Please ignore the machine translation error)

I want to share XXX.XXX.S01.1080p.AMZN.WEB-DL.DD+5.1.H.264-XXX folder with users but I don't want to give them the permission to see the whole index.
They should be able to see only this folder and root folder should be password protected. This way they can't view/download all the files from my index.

I don't know if you can use it now. Another onedrive index project onemanager is to turn on the no password to download files function, Turn on this function and know the link to directly download files other than password files. (Please ignore the machine translation error)

As I mentioned in https://github.com/lyc8503/onedrive-cf-index-ng/issues/8#issuecomment-1872424009, you should probably not rely on URL itself to keep your files private. Also, the issue itself is trying to share a subfolder without password protection, which is different from the no password to download files feature you're talking about.


Add access control to service level

Add access control lists to the service level, so both the dataNFT and service access control lists are checked at consumption time.

[ ] define the access policy to both NFT and service level.

We already have credentials at the DDO level  (NFT).  (see https://docs.oceanprotocol.com/developers/ddo-specification#credentials)
We need to add it to service level as well(so, each service will also have a credentials object), then have a combination of access:

if denied on DDO or service level -> deny access
if allowed on DDO, but denied on service level -> deny access
if allowed on both -> allow


Misleading Line In Getting Started

Non-modular application
Since Java 9, applications should be modular, and distributed with tools like jlink.

I would suggest that this entire section be reworked to deemphasize trying to make the end user application modular. I tried to make this work and ran into nothing but problems.
I spent a long, long time trying to make my app[s] fully modular and it was essentially impossible for anything but a trivial Hello World app. As soon as I brought in any dependencies it turned into a nightmare.
Here is a very simple, clean Maven template for building a JavaFX app, including GitHub Actions for creating the installers.
https://github.com/wiverson/maven-jpackage-template
Feel free to grab any/all of it for the JavaFX documentation.

I agree with this, I spent long hours trying to make a modular app to work. With no success. I wasn't lucky enough to see this issue at the first try.

This part of the documentation should get an update, especially now that jpackage is out of incubation and a part of jdk tools.
I will have a look at it, unless someone beats me to it :)

I would be happy to send in some updates for docs. Is a PR the preferred process?

@wiverson A PR would be great!


Styles: Fix issues created by using global theme variables

#1030 introduced a potentially breaking change that still isn't versioned in that:

If you're using the compiled CSS in packages/styles/css/index.css, you don't have the theme variables
If you're using the individual component CSS, you don't have the theme variables.

How to fix this

[ ] We need to add the following line to packages/styles/scss/index.scss

@use "theme";

[ ] We need to include documentation that, if you're using individual component CSS, you have to import packages/styles/css/global.css first

[ ] We need to release a minor version of the Twig library

cc @ademarco

Hey @GGKapanadze, if you can agree this is the right approach, I can take care of it as I think it's blocking @ademarco on a few things.

For the current bundling system its correct

Fixed by #1092


Paths with special characters in the filename are not processed correctly

Describe the bug
Note: This issue was opened in vitest first but was said to be a vite issue here https://github.com/vitest-dev/vitest/issues/2731
I have a file in a directory called __({[ % & $ # @  ~ ,)}]__/special-characters.spec.ts`.
The full path to the test is
/Users/jonlucadecaro/Documents/Other/json-schema-ref-parser/test/specs/__({[ % & $ # @ ` ~ ,)}]__/special-characters.spec.ts

The error is
 FAIL  test/specs/__({[ % & $ # @ ` ~ ,)}]__/special-characters.spec.ts [ test/specs/__({[ % & $ # @ ` ~ ,)}]__/special-characters.spec.ts ]
Error: Failed to load url /Users/jonlucadecaro/Documents/Other/json-schema-ref-parser/test/specs/__({[ % & $ # @ ` ~ ,)}]__/special-characters.spec.ts (resolved id: /Users/jonlucadecaro/Documents/Other/json-schema-ref-parser/test/specs/__({[ % & $ # @ ` ~ ,)}]__/special-characters.spec.ts). Does the file exist?
 ❯ loadAndTransform node_modules/vite/dist/node/chunks/dep-5e7f419b.js:39304:21

Reproduction
https://stackblitz.com/edit/vitejs-vite-kksrdc?file=package.json
Steps to reproduce
Run yarn install followed by yarn test
System Info
System:
    OS: macOS 13.1
    CPU: (10) arm64 Apple M1 Max
    Memory: 366.42 MB / 64.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 19.4.0 - /opt/homebrew/bin/node
    Yarn: 1.22.19 - ~/.yarn/bin/yarn
    npm: 9.2.0 - /opt/homebrew/bin/npm
  Browsers:
    Brave Browser: <IP_ADDRESS>
    Chrome: 109.0.5414.87
    Firefox: 108.0.2
    Safari: 16.2
  npmPackages:
    vitest: ^0.27.2 => 0.27.2

Used Package Manager
yarn
Logs
No response
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to vuejs/core instead.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

Removing # from the path fixes the issue. Vite only supports # for dependencies (because there exist some popular packages using them, like es5-ext/string/#/contains), but it doesn't support it in user source code. Both # and ? are reserved for compatibility with browser URLs.


Refactoring to remove duplicate code (by Frank/Nuage)

The https://github.com/apache/cloudstack/pull/3537 would revert the merged PR #3152 as we're close to 4.13 RC1 and have low confidence on what other regressions it might add. This simply re-creates PR #3152 using the same source fork/branch. We can work on this PR and attempt again to merge towards 4.14 after fixing discovered regressions and a more thorough review and testing.
Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[x] Cleanup (Code refactoring and cleanup, that may add test cases)

@fmaximus please see https://github.com/apache/cloudstack/pull/3537, we'll need to revert your PR #3152 for this purpose I've reopened your PR from the original branch. Kindly help test and fix regressions discovered in your branch, esp around registration and upload of archived templates and iso (templates/isos that are extractable liike *.bz2, *gz etc).

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-230

@blueorangutan package

@fmaximus thanks for self assigning, can you help fix the conflict? I'll try and help you to get this merge now that master is not frozen anymore.

@rhtyd Conflict is resolved.
Jenkins build has failed due to some build node issues:
FATAL: Couldn't find any executable in /home/jenkins/tools/maven/apache-maven-3.3.3

Thanks @fmaximus I'll kick a build
@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian.  I'll keep you posted as I make progress.

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-283

@blueorangutan test matrix

@rhtyd a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware65, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

Trillian test result (tid-373)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 32648 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3538-t373-xenserver-71.zip
Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
Smoke tests completed. 76 look OK, 1 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

test_01_scale_vm
Failure
20.86
test_scale_vm.py

Trillian test result (tid-375)
Environment: vmware-65u2 (x2), Advanced Networking with Mgmt server 7
Total time taken: 44053 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3538-t375-vmware-65u2.zip
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 76 look OK, 1 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

test_05_rvpc_multi_tiers
Failure
553.46
test_vpc_redundant.py

test_05_rvpc_multi_tiers
Error
595.79
test_vpc_redundant.py

@fmaximus @rhtyd does this also remove this global settings?

@svenvogel No, that's not part of it. This was a change I had been working on,
which I still wanted to offer the community.

@fmaximus thanks for answer. will this removed from you in a seperated PR or should the community track this?

@svenvogel That should go in a separate PR for sure, as it's a bugfix which should be fixed in the same release as Remove Nuage Plugin.

CentOS7 job failed, rekicking
@blueorangutan test

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-377)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 32381 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3538-t377-kvm-centos7.zip
Smoke tests completed. 77 look OK, 0 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

LGTM, tests LGTM as well. Using lgtms from the original PR, merging this.


Add attribute to ignore auto_now fields?

We've noticed that sometimes the only fields that have changed are auto_now fields and they're creating lots of event records we don't really care about.
To deal with this, we made a custom tracker that has an extra parameter, ignore_auto_now. When set to true, it ignores changes to models that only affect auto_now fields.
In usage, it looks like this:
@pghistory.track(CustomSnapshot(ignore_auto_now_fields=True))
class Note(models.Model):
    ...

We'd be pleased to do a small PR with this new feature, if the maintainers were interested. Any interest?

I currently use pgtrigger to revert any auto_now field updates, and to update the auto_now fields only when other fields are actually updated. Your feature would be a welcome addition to pghistory. Maybe even expand it to ignore a list of specified fields?

One other thing, @mrvisiol, are you a maintainer of this project?

Great!

Maybe even expand it to ignore a list of specified fields?

So perhaps like this:
@pghistory.track(CustomSnapshot(ignore_fields=["date_modified", ...]))

And the docs could explain that ignored fields show up in the event models but do not trigger new events if they're the only changed field?

That's exactly what I was thinking.
(And no, I'm not a maintainer of this project.)

@mlissner @mrvisiol
To clarify, you want these fields to show up in the snapshots, right? If it wasn't already known, you can ignore the fields entirely with exclude (see the section here towards the end about this particular argument). I need to double check, but excluding these fields would also no longer trigger snapshots. Since snapshot models already include pgh_created_at, you may not really need auto updated fields in your snapshot models in the first place. Would this solve your issue?
If not, you'll need to use the condition argument for this (see conditional tracking for an example ) . Your condition will need to OR together any changes of the fields you care about. Check out the pghistory code itself to see how it makes this condition for when exclude is used.
It's not pretty code.. This is one of the more awkward conditions to make, and it's pretty common too.
How about this - Let me make a custom Condition object that you can use to more easily express conditions where you want to ignore fields. If this solves your problem, I can post some code here with the intent of getting it into django-pgtrigger so that everyone writing trigger conditions can use it.
Hope I understood your problem correctly. Let me know if I didn't

I think @quevon24 has some pretty simple code for this that he put over in #85 before I asked him to move it to a separate PR. The general idea though is that we want to store the values of a field in the history table, but changes to the field shouldn't trigger new history events. Some fields change too frequently to trigger history events each time they change, but I'd like to know their last value when something else in the model changes.
Kevin, could you please post in your snippet for doing this that you had?
This isn't the most important feature for us, if we need to, we'll either build it into the other PR or we'll just exclude the field entirely, but it'd definitely be nice to be able to do this.

As discussed in https://github.com/Opus10/django-pghistory/pull/85, users are able to create manual trackers a bit more easily now using the pghistory.Changed condition. I have an example tracker in the test suite here and an example test that validates it
It's not the prettiest solution, but I think long term it might be better to break the API and consolidate the Snapshot tracker and all of the other utility trackers since they have subtle differences in what they do.
I will keep this issue open for now until I add an example to the docs. The PR was deployed in v2.7!

Follow up - Pghistory includes aliases for some of pgtrigger's conditional utils, and these include the ability to exclude specific fields or ignore auto* fields.
There's a brief example of using pghistory.AnyChange in the pghistory docs.
In other words, say you have this model:
class MyModel(models.Model):
    int_field = models.IntegerField()
    char_field = models.CharField()
    auto_now_field = models.DateTimeField(auto_now=True)

You can snapshot updates to everything but when the non-auto fields change:
@pghistory.track(
    pghistory.pghistory.UpdateEvent(
        row=pghistory.Old,
        condition=pghistory.AnyChange(exclude_auto=True)
    )
)
class MyModel...


Handling stale token

Hi,
I'm was wondering if anyone knows how to gracefully handle stale tokens by refreshing them and re-invoking the upload process?
Adrian

Hi @adrian-mitangi, thats part of your component. Inject a AuthService, that handles also token refreshes.


total balance matches transaction stake report does not it's too high

only 5 tx now error was seen when there was 4
Method	Address	Account	Amount	Confirmations	Time	Txid	Comment
Orphan	NMvZJtMDoDjZY2uuDf1jyomdvWbssU4GWp
0.31328767
0
Sat Apr 1 17 2:45 pm
623e2c53ab4b1eb42863f05db4310596d50426d11657ccb4af24b4508802a6df
Generate	NMvZJtMDoDjZY2uuDf1jyomdvWbssU4GWp
1.91150684
274
Sat Apr 1 17 12:33 pm
a477e8465dec1457af5dec4b90c74c890097c52929455ecf18b47d41a82dc1f6
Generate	NMvZJtMDoDjZY2uuDf1jyomdvWbssU4GWp
0.99972602
658
Sat Apr 1 17 9:20 am
d42175f98e787b6d0f3047e6bf4dcef602b7485b8ce047d970353a892cac532e
Receive	NMvZJtMDoDjZY2uuDf1jyomdvWbssU4GWp
50000
992
Sat Apr 1 17 5:51 am
14ad091efe793be13a2b74e35d505a1e11a548fe421ebd0371bc12aebf52ba6e
Receive	NMvZJtMDoDjZY2uuDf1jyomdvWbssU4GWp
50000
1082
Sat Apr 1 17 5:50 am
4d56cda949e41d9b782887528ceefcc8b69115cac45ae869b2ea48a5aa2658bb
Available Balance:<PHONE_NUMBER>3286 NavCoin
Stake report
Last 24h: 3.71684929 NAV
2 50k receive 2 valid stakes one orphan
orphan is not issue I saw this before the orphan happened
looks 1st stake counted twice on report.
i

4.01.1 fixed it


Generates enum that is listed in types.json

I added a custom enum type to types.json and use it in my graphql spec.
Expected Behaviour
gqlgen would use the custom type instead of generating enum into models_gen.go
Actual Behavior
enum code is generated into models_gen.go and a mismatch error is reported
type mismatch on Foo.state, expected <package>/foo.FooState got <package>/foo/custom.FooState

Minimal graphql.schema and models to reproduce
graph.graphql:
type Query {
    foo(): Foo!
}

type Foo {
    state: FooState!
}

enum FooState {
    BIN
    BAR
    BAZ
}

types.json:
{
  "FooState": "<package>/foo/custom.FooState"
}

custom/foo.go:
package custom

import (
	"fmt"
	"io"
	"strconv"
)

type FooState string

const (
	FooStateBin FooState = "BIN"
	FooStateBar FooState = "BAR"
	FooStateBaz FooState = "BAZ"
)

func (e FooState) IsValid() bool {
	switch e {
	case FooStateBin, FooStateBar, FooStateBaz:
		return true
	}
	return false
}

func (e FooState) String() string {
	return string(e)
}

func (e *FooState) UnmarshalGQL(v interface{}) error {
	str, ok := v.(string)
	if !ok {
		return fmt.Errorf("enums must be strings")
	}

	*e = FooState(str)
	if !e.IsValid() {
		return fmt.Errorf("%s is not a valid FooState", str)
	}
	return nil
}

func (e FooState) MarshalGQL(w io.Writer) {
	fmt.Fprint(w, strconv.Quote(e.String()))
}

I have managed to work around this issue using the newly added -modelpackage flag so that it generates the enum into the same folder as my custom models

Thanks for the bug report, should be fixed now.
I updated model to take directory names so enums could live with the other models, otherwise you end up with import loops pretty quickly.
What was the reason you needed a custom enum? Is it common enough to be worth folding into the generated enum?

I was attempting to avoid import loops and keep my models together. The new flag for module dir is exactly what I needed.
It looks like the module code is still using -package


Where did Powerpack's Metadata namespace go?

It used to be in this project...right? It seems most of the types are still there, but I'm having a hard time getting my hands on an FSharpAssembly instance (it used to have a Load static method).
Or what's the story for parsing metadata in compiled assemblies? The Powerpack page refers to here.
Thanks!

I thinking of ways to support units of measure in FSharp.Data.SqlClient. Seems like I need logic that check if a type is UoM.  Something along these lines
http://stackoverflow.com/questions/4005474/reflection-for-f-units-of-measure/4005908#4005908
Do I have to go thru all complexity shown in @7sharp9 snippet? // @fsgit @vladima

@dmitry-a-morozov I have come across this limitation as well and created https://github.com/matthid/Yaaf.FSharp.Scripting/ which adds the FromAssembly methods (from your stackoverflow link) as extension methods. But basically I'm doing exactly what @7sharp9 has shown above (behind the scenes).
I planned to discuss this here on FSharp.Compiler.Service once I'm happy with the API.

Cool. I'll keep watching progress on this issue.

It's really heavy weight to get the information that way, we could do with a direct method of doing this.  Latency of the current method can be 5 or more seconds. :-(

In my tests it's only slow the first time, if you re-use the checker instance the it's kind of OK. But you are right the initial delay is a pain.
One more Note: Before starting my scripting project from above I tried to implement a more direct approach in FCS, but I failed as it seems to be a lot more difficult as one might assume, so it may take some time until we get that functionality back (if ever). I hope I'm wrong on this one though...

@dmitry-a-morozov -
For the Freebase type provider prototype, we managed to support UoM in limited forms of queries pretty reasonably - we had information of the units of columns, and propagated that up through both type system and query translation.  But it was a thorny, and relied on the specialized shapes of operations permitted in queries, would have been much easier with full reflective information.
I do wonder if it should be possible in F# to ask for normalized reflective info of UoM parameters which are "ground".  This would mean that FSharp.Core would have something like this:
val inline unitof< [<Measure>] ^u > : System.Type

Which would allow
unitof<kg*kg>
unitof<m*m/m>  // same as unitof<m>

That in itself would be useful. Of course you still don't unit information in quotations for queries or attached to System.Type for, say, float<kg>.  But it is a start and may be enough that you could build an entirely new set of "unitized" query builder and query operations.
This line of thinking should probably be added to http://fslang.uservoice.com (I haven't done that, but feel free to do so).

@matthid It would be nice to get a sane way of retrieving a compiled F# dlls metadata, Im currently doing the same as we discussed about with my soothsayer tool which displays the signatures from an F# dll for tooling etc.

Closing old feature request


Implement a permanent storage

Here are some ideas of a structure we could use, please review and let's discuss.

Some early thoughts:

I don't think we have the need for redis graph, we won't have a lot of relations (I don't think we will have relations at all), I think a bare redis will do the work and be easier we could switch later if needed
Loading a player from database should not use the classic observer/reducer struct instead it should provide the initial_state of the player

let state = initial_state
if (USE_PERSISTANT_STORAGE)
  state = load_from_database(initial_state, client.uuid)

I'm afraid redis is a bit limited as a database, basic stuff like secondary key indexing needs us to do more work, redisearch is good but still limited and more overhead to learn that redisgraph (and still not a database), look at the doc for Disk :/ the choice for me is either mongodb or redisgraph, while mongodb force us to express relations with foreign keys

I'm afraid redis is a bit limited as a database, basic stuff like secondary key indexing needs us to do more work, redisearch is good but still limited and more overhead to learn that redisgraph (and still not a database), look at the doc for Disk :/ the choice for me is either mongodb or redisgraph, while mongodb force us to express relations with foreign keys

The thing is that for me we will not need to use the complex parts of redisearch, for the game data we only need a Map<UUID, JSON>, so no secondary key, we don't even need a structure since we will only access the data as a whole, I think we should just store the json as a string first in redis, then if we need to acces part of the data use redisjson (maybe the website will need that)

Forgot the existence of redisjson 👍 that's indeed the best and simplest choice for us

PR is ready :+1: yet there is an infinite amount of typing error, deal with it


Comma separated Okta groups are considered as Single group by Paralus

Expected vs actual behavior
Comma separated Okta groups should be considered as different group each by Paralus and if granted access to one group in which the IDP users is part of, user should be able to access the projects based on permissions given
All the comma separated groups populated in paralus are cosidered as single group and user is not able to access projects even if group is granted access to project.
Steps to reproduce the bug

Integrate Okta with Paralus.
Enable group mapping to paralus from okta
Login to paralus with user ID who is part of multiple groups from Okta end.

Are you using the latest version of the project?
v0.2.3
You can check your version by running helm ls|grep '^<deployment-name>' or using pctl, pctl version, and provide the output.
v0.2.3
What is your environment setup? Please tell us your cloud provider, operating system, and include the output of kubectl version --output=yaml and helm version. Any other information that you have, eg. logs and custom values, is highly appreciated!
AWS EKS.
Using custom jsonnet.
local claims = std.extVar('claims');
local fName = if "name" in claims && claims.name!=null && std.length(std.findSubstr(" ", claims.name)) > 0 then std.splitLimit(claims.name, " ", 1)[0] else "Paralus";
local lName = if "name" in claims && claims.name!=null && std.length(std.findSubstr(" ", claims.name)) > 0 then std.splitLimit(claims.name, " ", 1)[1] else "User";
{
identity: {
traits: {
email: claims.email,
first_name: fName,
last_name: lName,
[if "groups" in claims.raw_claims then "idp_groups" else null]: [ claims.raw_claims.groups ],
},
},
(optional) If you have ideas on why the bug happens or how it can be solved, please provide it here

[ ] I've described the bug, included steps to reproduce it, and included my environment setup with all customizations.
[ ] I'm using the latest version of the project.

@divyanjali1 Can you try this following line instead
[if "groups" in claims.raw_claims then "idp_groups" else null]: std.split(claims.raw_claims.groups, ','),

By default we would expect OIDC groups as a list of items instead of string. But one can change Jsonnet script for user data mapping as per their OIDC setup and need.
Discussed in Slack thread here.

Thank you Akshay , changing okta jsonnet resolved the issue.

Glad it helped. Closing this issue, feel free to reopen if you've further queries.


OrgCSS: adopt build procedure

OrgCSS is the CSS being applied to Cuboids Manifesto. Currently a single built org.css file has been added to the repo. But to tweak the styling will need to build from OrgCSS's source. So, time for a docs build process, which might as well CICD publish as well.

Looks like it's Gulp based:
https://github.com/gongzhitaao/orgcss/blob/master/gulpfile.js#L54

Decided to come up with a standard CSS for all Reconstrue docs produced from *.org files, not simply this repo (where the Cuboids Manifesto is to be the first published file exported from an org file. So, instead of bringing OrgCSS production into this repo, forked the OrgCSS repo to https://github.com/reconstrue/orgcss.
So, tweak and build in that repo and just import the generated *.css into this repo.
Moving forward, the tweaking is being tracked in the presentations repo. This issue will just be a TODO reminder to fold in the built *.css file.


fix: allow customizing cluster domain

Q
A

Bug fix?
yes

New feature?
no

API breaks?
no

Deprecations?
no

Related tickets
fixes #1021

License
Apache 2.0

What's in this PR?

Enable users to customize cluster domain
Checklist

[x] Code meets the Developer Guide
[x] Related Helm chart(s) updated (if needed)

To Do

[x] If the PR is not complete but you want to discuss the approach, list what remains to be done here

This fix work for fluentBit Secret-TLS.
But logging operator generate secret with FluentBit Config File, where in OutPut block:
[OUTPUT]
    Name          forward
    Match         *
    Host          logging-operator-logging-fluentd.logging.svc.**cluster.local**
    Port          24240

Need change this cluster.local

I'm only vaguely familiar with how DNS resolution works when the cluster domain is not specified. I know it works in most cases, but I'm sure there are some edge cases e.g. maybe with a service mesh (e.g. Istio) or other multi-cluster setup.
Also, the trailing dot that we leave in the generated domain name if cluster domain is defined as "" might be problematic.

@siliconbrain I think I addresses all the comments you left. Please have another check.


feat: use "prettier-tslint" and "prettier-eslint" when they exist

When the scm root has ./node_modules/prettier-tslint and/or ./node_modules/prettier-eslint, we use them on any applicable modules.
We prefer prettier-eslint for .js, .jsx, and .mjs modules. If prettier-eslint does not exist, we try prettier-tslint and then prettier alone.
We prefer prettier-tslint for .ts and .tsx. If prettier-tslint does not exist, we try prettier-eslint and then prettier alone.
All other file types use prettier only.

PS: You can add me as a maintainer here, too. ;)

Thinking I'd favour a more explicit API, like pretty-quick --tslint, pretty-quick --eslint. It will save a small amount of IO for users who don't use those tools, and users who have e.g. prettier-eslint still in node_modules (maybe unused) will be less confused when the behaviour changes after upgrading this package.
We can also make it work with PnP that way.

We can also make it work with PnP that way.

Why would replacing implicit loading with an explicit flag make PnP work? We already wrap the require.resolve call in a try..catch block, so PnP shouldn't choke on it.

We can just do if (opts.eslint) require('prettier-tslint'), shouldn't need to hardcode node_modules.

Can you explain why someone would want pretty-quick to not use prettier-eslint and/or prettier-tslint when they are installed?

I've improved the implicit behavior by ensuring the resolved prettier-tslint and/or prettier-eslint paths always ignore globally-installed versions. Basically, they must be in your package.json to signal to pretty-quick that you're using them.

I've added shebang support (using prettier.getFileInfo), which requires Prettier v1.13.0 or higher.
I've updated the peer/dev dependencies on prettier to match.
Like I said, we can add Yarn PnP support in the future. I think we would take this approach.
LGTM! :)

Hmm.. Looks like Travis is using a version of Prettier that doesn't include getFileInfo.
Even though I updated yarn.lock. 🤔
@azz Could you take a look?

I deleted the build caches and restarted the build. Hopefully that helps.

I think we can merge this.
Things to fix later:

add Yarn PnP support
get prettier-eslint to work with shebang modules

Hey guys, is this feature abandoned? :(

Any news on this one? 🤔
Ping @aleclarson @azz

No thanks, but you can looking forward to https://github.com/prettier-eslint/prettier-plugin.


Added the regulation term, still awaiting PMID ref

cc
@thomaspd
@RLovering
https://github.com/geneontology/synapse/issues/186

Thanks @ukemi, I'll update the equivalence axiom as you advise.
@vanaukenk could you please update the GO Editors' Guide to reflect this change?
The current instructions say 'biological regulation', not 'biological process':
http://go-ontology.readthedocs.io/en/latest/NewRegTerm.html
Thank you,
Barbara

Argh. You are right. I just checked the pattern.

equivalentTo:
text: "'biological regulation' and 'regulates' some %s"
vars:

entity

So should I leave it as it is?

Yup.

ok. I'll merge then. Thank you!

Sorry about that. I was pretty sure we had decided on biological process.

No worries. Thank you for checking.


Wrong $ref relative path generated for properties when Schema is injected manually

I've created a demo project to illustrate the problem: https://github.com/potocnika/swagger-tester
.NET Core 2.0 / Kestrel / VS2017
What we're attempting to do:

Have a /schema endpoint overload which serves the schema for given endpoint - this is working fine with NJsonSchema
Using NSwag, generate a swagger with Swagger3 UI that flattens parametrised endpoints - problem

In the demo project, this is signified by the following endpoints:

/animals/{type}
/animals/{type}/schema

When generating swagger documentation, we're using hooks to flatten {type} into concrete types:

/animals/cat
/animals/dog
/animals/horse
/animals/cat/schema
/animals/dog/schema
/animals/horse/schema

All concrete types for {type} inherit from a base abstract class (Animal)
Everything is working fine with one complication - the UI shows errors for $ref properties and looking into the generated swagger.json (/swagger/swagger.json) shows that the $ref properties are rendered as for example "#/paths//animals/dog/Get/responses/200/schema/definitions/Dog" instead of "#/definitions/Dog"
Testing this on the equivalent /animals/dog/schema - this is generated correctly using exact same json contract and settings. Yet the paths are relative to the whole document when that same schema is manually injected into the swagger specification within hooks. I've tested the problem by using NSwag Studio and replacing all $ref properties so that the paths are relative to the schema and not to the swagger document. When this is done, the swagger document validates correctly.
Is this a problem with swagger specification? Should $ref paths relative to the swagger document be resolved or should they only be relative to individual schema?
I looked through the NSwag to see if rendering of the references can be modified to no avail. When debugging, the .Reference of the Json node exists and is set correctly (to expected Type). The problem seem to be when rendering the node, the value is somehow change to point to document root.

Hi @RSuter , any update on this? It's still an issue.

I think it might work if whatever is creating the JSON Pointer ref escaped the slashes in the path key, and also lower cased the http verb (Get)
#/paths//animals/dog/Get/responses/200/schema/definitions/Dog
becomes
#/paths/~1animals~1dog/get/responses/200/schema/definitions/Dog
Escaping slashes as per https://tools.ietf.org/html/rfc6901
This clears the errors (cant comment on whether the pointer should be to another section of the JSON).

All this “low-level” schema/path handling stuff is handled in NJS: https://github.com/RSuter/NJsonSchema/blob/master/src/NJsonSchema/JsonPathUtilities.cs

So we need to fix this there, e.g. by first scanning the definitions instead of paths

... ah i think you just need to add the injected schema also to the root definitions and remove it from the other definitions. This way the path is correctly resolved...
You could also use the JsonSchemaVisitor to remove all inner schemas from the definitions and insert it into the definitions of the swagger document:
https://github.com/RSuter/NJsonSchema/blob/master/src/NJsonSchema/Visitors/JsonSchemaVisitorBase.cs

Thanks @RSuter - thats a lot of help!
I think the problem occurs when switching out the response schema eg
 // previously the abstract Animal type - want to replace with the concrete Dog type response.Schema = JsonSchema4.FromTypeAsync(typeof(Dog), settings).Result;
this would cause the generation of an inner definitions section, and the JSON Pointer generated to refer to it is malformed as detailed above.
As you mentioned, a work around could be to populate the root definitions and get the $ref to point to that instead. That would sidestep the parameter escaping problem.
Is there a better way to replace the return type? Or could you point me to an example of using the JsonSchemaVisitor?

In scenarios like that its important to not use JsonSchema4.FromTypeAsync but the more controllable generator and resolver class so that definitions are added to the correct location and types not generated multiple times..

... this clearly needs more documentation.

Where do you assign response.Schema in your code? In an operation processor?

That makes sense - I was looking for an earlier hook in  because modifying the schema after its generated didnt feel right.
That schema was being reassigned in a Document Processor - hence the schema wrestling.
Im a little confused about the generator/validator naming conventions - are you suggesting jumping right up to implementing a AspNetCoreToSwaggerGenerator type generator or using SwaggerJsonSchemaGenerator and SwaggerSchemaResolver (or something else) from within an Operation Processor?
Appreciate the pointers here - once I'm done perhaps I could contribute a write-up to the docs?

In the document processor, you get the current context and there you get the current generator and resolver...


Bug fix/do not reclaim shard ownership.

During a move-shard job which moves the leader there is a
situation in which the old owner of a shard can reclaim ownership
(after having resigned already), with a small race where it allows to
write documents only locally, but then continue the move-shard to a
server without those documents. An additional bug in the MoveShard
Supervision job would then leave the shard in a bad configuration
with a resigned leader permanently in charge.
Scope & Purpose
(Can you describe what functional change your PR is trying to effect?)

[ ] Bug-Fix for devel-branch (i.e. no need for backports?)
[X] Bug-Fix for a released version (did you remember to port this to all relevant release branches?)
[ ] Strictly new functionality (i.e. a new feature / new option, no need for porting)
[ ] The behavior in this PR can be (and was) manually tested (support / qa / customers can test it)
[ ] The behavior change can be verified via automatic tests

Related Information
(Please reference tickets / specification etc )

[ ] There is a GitHub Issue reported by a Community User:
[ ] There is an internal planning ticket:
[ ] There is a JIRA Ticket number (In case a customer was affected / involved):
[X] There is a design document: https://github.com/arangodb/documents/blob/master/DesignDocuments/99_UNSORTED/Supervision.md

Testing & Verification
This PR adds tests that were used to verify all changes:

[ ] Added Regression Tests (Only for bug-fixes)
[X] Added new C++ Unit Tests (Either GoogleTest or Catch-Test)

Did you add tests for a new RestHandler subclass ?
Did you add new mocks of underlying code layers to be able to verify your functionality ?
...

[ ] Added new integration tests (i.e. in shell_server / shell_server_aql)
[ ] Added new resilience tests (only if the feature is impacted by failovers)

Additionally:

[X] There are tests in an external testing repository (i.e. node-resilience tests, chaos tests): arangosync
[ ] I ensured this code runs with ASan / TSan or other static verification tools

http://jenkins.arangodb.biz:8080/view/PR/job/arangodb-matrix-pr/11539/

http://jenkins.arangodb.biz:8080/view/PR/job/arangodb-matrix-pr/11546/

This run should fix the failures of the previous one.

http://jenkins01.arangodb.biz:8080/view/PR/job/arangodb-matrix-pr/11552/

http://jenkins01.arangodb.biz:8080/view/PR/job/arangodb-matrix-pr/11553/

https://jenkins01.arangodb.biz/job/arangodb-matrix-pr-linux-nonmaintainercompile/10928/
(rerun because of Jenkins problems)

http://jenkins01.arangodb.biz:8080/job/arangodb-matrix-pr-linux/13208/

https://jenkins01.arangodb.biz/job/arangodb-matrix-pr/11559/

Tests blue.


fix(format): Don't leak bufnr into opts tables

Since Util.merge performs an in-place merge, bufnr is being leaked into the opts tables before being sent to the formatter.
This causes, in rare cases, the bufnr to be replaced by the wrong bufnr (i.e, not the current buffer), causing the formatting to apply to the wrong file.
This bug caused random text to be copied and inserted into unrelated files on save, which eventually snuck into our QA environment. Luckily, it got caught and I started investigating the issue. I found that the issue happened after (afca0b7), and found the root cause.
To demonstrate why this is a problem, do as follows:

Open an arbitrary file and save it, triggering conform
Note which bufnr this has via :echo bufnr('%')
Open a different file (do not save)
Check that bufnr is now a different bufnr as expected via :echo bufnr('%')
Run the following :lua vim.print(require("lazyvim.util").opts("conform.nvim").format or {})
Observe that bufnr is present in the conform.nvim options table, and that it points to the old buffer

Because of a recent change (afca0b7), the conform.nvim opts.format table is applied into the opts table before formatting, overwriting the proper bufnr value in the opts sent to the formatter, because conform.nvim would now contain it's own, stale bufnr.
This made me look at other uses of Util.merge and see that this indeed happens in several places.
The fix is to make sure we make a clone of the opts table before we trigger format, avoiding leaking the bufnr value into the plugin opts.
After this fix, observe that :lua vim.print(require("lazyvim.util").opts("conform.nvim").format or {}) does not contain bufnr in the conform.nvim table.

Hi @AlexanderArvidsson
I'm the one who made the commit you're referring to.
You're right that bufnr is present in opts.format table and with your change it's not present any more. However, I checked with LazyVim 10.12.1 by binding the LazyVim to commit = "e3bfcb9" in its spec (my commit was included in release 10.13) and it had the same behavior, that is bufnr is still present in opts.format table (following exactly the same steps that you described).
My commit however does introduce this line and I can't tell with certainty that it doesn't cause the problem you're observing. But the steps you described are reproducible even in 10.12.1 before my commit.

Looks good to me. Thanks!

@folke Would you mind explaining if my change introduced this and why it didn't happen before my change (for example in 10.12.1) where bufnr was still leaked to opts.format table?
I'd appreciate it if you could take the time to answer my question, as I really want to understand.
Thanks in advance.

I imagine it's because opts is a shared object used in different buffers/windows so if it gets changed in one to point to its buffer it will also affect other buffers. Which is why it's best to create a different object for each merge in this case?

So, if I understand correctly what you're saying, bufnr was also present in conform's opts.format table prior to this change, it's just that it wasn't introduced in the lsp.format.opts before so there wasn't a problem. Is that correct @briandipalma ?

I'm not sure your change actually introduced the bug. It might have been happening before.
(I didn't have this issue at all, even with your changes)
The reason is indeed that deep_copy (and Util.merge) will merge into the first table, so definitely better to merge in a new empty table instead.

Tnx @folke for replying. I did test with 10.12.1 and bufnr was even then present in conform's opts.format table, but the difference is that it maybe wasn't introduced in lsp.format.opts before.
I understand the logic for merging into a new table and why bufnr is not present any more. Appreciate your time explaining.

@dpetka2001 You're correct, the bug as described by my PR was present before your change, but the practical effects of the bug was only surfaced after the change. Just to make things clear; I did not link the commit you made to put blame, I believe your change is great because it allows me to override conform formatting settings! So good job identifying that! ❤
When I was researching the bug, I found the effects of the bug did not happen before your commit, but that's because the conform opts table was not merged into the formatting before the change. The changes this PR makes fixes the underlying bug which was surfaced at a much later stage (I believe the underlying bug was introduced by @folke over 5 months ago, but at the time it may not have been a concern and as such there's nothing wrong with that).
Again, I'm glad I was able to find the cause and the fix is in now, great job all of your and especially @folke for this great setup!
Thanks for the merge, now I can continue using the main branch! 😄

@AlexanderArvidsson No problem. I was just genuinely trying to understand the reason why this happened even though the bufnr was leaked before my commit as well. I never encountered this problem, but that may be because i usually use a specific formatter via conform and don't rely on LSP formatting.
I'm still new at programming and still don't really understand so much about what's going under the hood, so my main concern in my previous reply was to try and understand why it happened and I never encountered it. That's all.
Thanks for taking time explaining.


Inline annotation & Hover & Direct actions for i18next

Inline annotation & Hover & Direct actions for i18next doesn't seem to be working.

This is because I do not really know what's the usage syntax like for i18next. Thus I make the annotation detecting regex empty temporary.
If you can provide some usage examples, I would love to support them. Thanks!

I have read some docs about i18next and make it work for some syntax I have seen. Please update to v1.3.1 and check it out. If there is any uncovered use case, feel free to let me know. Thanks.

Works perfectly for me. Thanks for your fast response.


NO QR code for Wemo Bridge

What issue do you have? Please be as thorough and explicit as possible.
I was having issues with the latest version of your Wemo Plugin due to my Hoobs version being 4.0. (I was getting an error saying that the plugin may be unstable due to the version of Hoobs I was running.)
I updated to Hoobs v. 4.1.21 (Homebridge Version 1.3.9), uninstalled the Wemo Bridge and Plugin, cleared the cache of the Hoobs, rebooted my Hoobs and reinstalled the Wemo Bridge and Plug-in. However, when I went to the bridge, I did not have a QR code to scan for Homekit under pairing. (Prior to this, I had a QR code for this bridge.) All of my other bridges have QR codes in the same location.
Furthermore, as soon as the Wemo Bridge was installed (even though I could not link it to my Homekit), I started getting the following error for all of my other bridges:
WARNING unhandled rejection: SyntaxError: Unexpected token u in JSON at position 0
I have since uninstalled the Wemo Bridge and Plug-in due to this error and the instability they were creating with my other bridges.
I would love to fix this, as I was very happy using the Wemo bridge until recently.
Details of your setup.

Do you use (1) Homebridge UI-X (2) Homebridge CLI or (3) HOOBS?

HOOBS

Which version of Homebridge/HOOBS do you have?

Hoobs 4.1.21

Which platform do you run Homebridge/HOOBS on (e.g. Raspberry Pi/Windows/HOOBS Box)? Please also mention your version of Node.js/NPM if known.

Hoobs Box
Node Version 16.13.1

Which version of this plugin (homebridge-wemo) do you have? Has the issue started since upgrading from a previous version?

The latest version (v. 4.5.3) // Yes.

Which Wemo devices do you have that are causing issues? Please include product models if applicable.

Two switches, two outlets
Please paste any relevant logs below.

1/6/2022, 1:33:37 PMPlatform Orbit BridgeWARNINGunhandled rejection: SyntaxError: Unexpected token u in JSON at position 0
1/6/2022, 1:33:37 PMAugust Smart Locks BridgeWARNINGunhandled rejection: SyntaxError: Unexpected token u in JSON at position 0
1/6/2022, 1:33:37 PMRing BridgeWARNINGunhandled rejection: SyntaxError: Unexpected token u in JSON at position 0

Hi @jvs2010
Can you show me your wemo config? At the bottom of the wemo config page there is a button called "Advanced". Can you click on that and show me the text config?

I would need to reinstall it. I uninstalled due to the issue it was creating. Give me a moment and I will do that.

I installed the Plugin from the Plugin page. It cycled for a while, created a Wemo Bridge, but when I go to the bridge it says "No plugins installed. Install Plugin."

When I clicked on "Install Plugin" it took me back to the same page. It then gave me the option to install by creating a new bridge or by selecting the "Wemo Bridge" it installed as mentioned in #1 above.

Sorry, not sure what the "cannot read config file" is in the last comment. I meant in "Number 1" above.

I selected to install it using the Wemo Bridge it created in Number 1 above. I did that and it gave me the uninstall/install/configuration option.

As soon as I installed the Bridge, I again got these errors for each of my other bridges. I did not even configure the Wemo bridge yet:
1/6/2022, 4:24:00 PMPlatform Orbit BridgeWARNINGunhandled rejection: SyntaxError: Unexpected token u in JSON at position 0
1/6/2022, 4:24:00 PMAugust Smart Locks BridgeWARNINGunhandled rejection: SyntaxError: Unexpected token u in JSON at position 0
1/6/2022, 4:24:00 PMRing BridgeWARNINGunhandled rejection: SyntaxError: Unexpected token u in JSON at position 0

This is what the config page shows:
{
"platform": "Wemo"
}

I configured one light switch:
{
"platform": "Wemo",
"name": "Wemo",
"mode": "semi",
"wemoLights": [
{
"label": "Front Lights",
"serialNumber": "221533K1300D9B",
"listenerType": "http"
}
]
}

But I still do not have the QR code for pairing when I go to the Bridge.

Uninstall the wemo plugin again for now, I think I will ask the HOOBS support staff about this. I will get back to you...

Thanks.

That error comes when something is wrong in the Advanced config. Can you please delete that bridge and recreate it with the wemo plugin only? Paste your config from the Advanced section here to make sure the syntax is correct

I'm not sure I understand what you mean by "delete that bridge and recreate it with the wemo plugin only"? I already deleted the wemo bridge and plugin, reinstalled it (as mentioned above) and pasted what was in the advanced config section. I got the error a second time, so I once again deleted the bridge and plugin. Do you want me to go through all of the same steps above again?

Yes except do not use the advanced config section. Use the plugin configuration page to configure the plugin. It has measures in place to prevent invalid json. Make sure only the wemo plugin is running on a fresh bridge.

I used the plugin configuration page to configure the plugin last time. (I just copied and pasted the text from the advanced config for Ben so that he could see it. I'm not a coder, so I don't configure the Hoobs using code -- only the web forms.) And it was the only plugin running on the Wemo Bridge it created. (There was nothing else on that bridge.)

Each time I try to install the Wemo plug-in from the "featured plugins" it creates a bridge, but then tells me there is no plug-in installed. If I click on the hyperlink where it says no plugin is installed, it takes me back to the Wemo plug-in page and "Install" is once again the only choice. It then asks me if I want to create a new bridge or use the Wemo bridge I just created (or another bridge).

Thanks for your help with this. Would this json error also create the issue where there is no QR code?

The QR code only appears when the bridge is running. If the bridge is stopped or in an error state it will not display because it won't be able to communicate with Homekit.
Not sure what's causing that plugin installation issue for you but there's a fix command you can run from the HOOBS terminal that will check and fix all the issues it can find:
curl -sk https://dl.hoobs.org/fix | sudo bash -

Running that now.

So I ran the fix command you provided. I then installed the Wemo plugin. This time I had no issues with the plugin creating a bridge and then installing. Also, I am no longer getting the json error for my other bridges in the log. HOWEVER, I still don't have a QR code in the plugin section and the bridge only has Start and Cache as options.

Click start then see if the code appears, otherwise check the logs. You can download the log file from the top of the log page and post it here

This is all that happens in the log when I click "start"
1/6/2022, 8:20:01 PMwemobridge stopping
1/6/2022, 8:20:06 PMWemo Bridge starting

Nothing else happens. (And it still has "Start" and "Cache" as the only options for the bridge.
Is there some reason why there are no spaces for "wemobridge" when it is stopping, but it is spaced out as "Wemo Bridge" when it is starting?

I just installed it myself and it runs fine. Is this what you have in the advanced config for wemo?

Here is what I have. I configured this using the configuration page. (I have removed the actual serial numbers and replaced them with Xs in this text file since I am posting it online.)
{
"accessories": [],
"platforms": [
{
"platform": "Wemo",
"name": "Wemo",
"mode": "semi",
"wemoLights": [
{
"label": "Front Door",
"serialNumber": "XXXXX",
"listenerType": "http"
}
],
"wemoOutlets": [
{
"label": "Den Lamp",
"serialNumber": "XXXXXXX",
"showAs": "default",
"listenerType": "http"
}
]
}
]
}

When I go to Homekit, if I choose add an accessory and click "more options", I see "Wemo Bridge FA26" as a choice.

that json looks valid. you see it but you can't add it cause you dont have the code. Have you tried rebooting the HOOBS Box? top right menu in hub settings. Also are you using the HOOBS Desktop app?

I rebooted the HOOBS box earlier, but i can try again. I didn't know about a Desktop app. I've been accessing HOOBS via hoobs.local

After rebooting the HOOBS box, the QR code appeared for the Wemo Bridge. I was able to add it to my Homekit. But... no Wemo accessories were added to my homekit. When I checked the log, it said:
No plugins installed.
1/6/2022, 8:49:59 PMWemo BridgeLoading 1 platforms...
1/6/2022, 8:49:59 PMWemo BridgeBridge is running on port 51856.
So then I had to reinstall the Wemo Plugin.
I restarted both the plugin and the bridge.
Once I did that, it added one of the lights I had configured -- but gave it the wrong name! And it didn't add the switch I had configured. I am going to try uninstalling and reinstalling again.

After clearing all of the caches (for the Hub and the plugin), deleting the bridge, rebooting the HOOBS again and reinstalling the Wemo plugin again... I finally got it to add my other switches and outlets, but it still insists on giving two switches with different serial numbers the same name (an old name I used in a previous iteration, but not the name I gave either switch in the configuration). I went ahead and added the bridge to Homekit using the QR code and renamed the lights in Homekit. Everything (fingers crossed) now appears to be working. Many thanks for your help, @askovi. I really appreciate it.


Handling Schema Discrepancies

A way of solving discrepancies would be to commit the migrations. First, this would allow for tracking the history of changes to the schema without having to revert to older commits.
More importantly, it allows keeping the local database in sync with the EF model while also preserving data. It would become the responsibility of the commit's author to modify the migration accordingly to copy data over when:

Dropping tables
Dropping columns
Adding required columns to existing tables
Renaming
Et cetera

Will the "Migration" then handle updating/creating the database as needed if not exists?

Yes. When you create the migration, it generates a file with a class that inherits Microsoft.EntityFrameworkCore.Migrations.Migration. You can go over it and add any changes as needed to preserve data (the generated migration won't do this part for you). Once changes are made, it can be applied to the local database.
Read this for more info. There's also a section on use in team environments which I recommend looking over.

Started using migrations in ab7d46f.


Cassidy flashbang projectile orb is too large

Created by: MaxwellJung
The size of the orb used to create Cassidy flashbang is large enough to prevent Cassidy from flashing enemies whe aiming close to a corner/wall.
In OW1, flashbang had a width of 1 pixel.

moved from ow1_emulator_backup#708

Created by: MaxwellJung
Projectile orb is already at its smallest size. Impossible to fix


:four_leaf_clover: Proposal: Use embed to read the default config

What Would You Like to Add? Why is This Needed?
In internal/pkg/show/config, the default config is all written in text. Why not use embed to import template files?
This way you can have highlighting and the separate files are easy to maintain.
Design
All the default config vars are placed in one file(internal/pkg/show/config/config.go), and the default config yaml files are placed in the configs dir.
Take githubactions-* as an example:
var (
	//go:embed configs/githubactions-golang.yaml
	GithubActionsGolangDefaultConfig string

	//go:embed configs/githubactions-nodejs.yaml
	GithubActionsNodejsDefaultConfig string

	//go:embed configs/githubactions-python.yaml
	GithubActionsPythonDefaultConfig string
)

Anything else
Similarly, all code that involves templates can be done like this。
Issues that may be affected:
#596 #597 #598

It's so cool! This idea is especially awesome!
Let's discuss a simple design that I think would look like as below:

add a plugins directory in the examples directory, which puts all the template configuration of all the plugins, similar to: argocd.yaml file contain the default configuration of argocd.
delete all the files in the internal/pkg/show/config/plugin directory, because all the content in it is put under examples/plugins
update the implementation of the internal/pkg/show/config package to include ideas that you can use according to your "talent"
internal/pkg/show/config/defaultconfig.go can probably be left untouched, because this piece of logic will be refactored soon, I'm working on it. #597

@daniel-hutao
I'm currently having a problem with go:embed not working . or .. which means that the parent directory cannot be referenced.
My idea is to leave a configs package in examples that reads all these yaml files and provides them to the rest of the program.
Related Issues:
https://github.com/golang/go/issues/46056
https://github.com/golang/go/issues/41191#issuecomment-686621090

How about to put the yaml files to internal/pkg/show/config? @iyear

@daniel-hutao The latest commit has moved the plugin yaml default config to that dir, but quickstart.yaml will not be read..
I still think the https://github.com/golang/go/issues/41191#issuecomment-686621090 approach is appropriate, as all examples can be exposed to other packages.

Or I'll start by making the current DefaultConfig standalone as a file. #597 I'll do it later and now I fix this issue.


ClusterMap::PartialEq is incredibly slow

While doing #856 I added a benchmark for the ClusterMap::PartialEq implementation, and it shows it's very slow. This is because it does a full equality check for every key and all endpoints. I suspect it could be made dramatically faster by computing a single hash for the endpoints that can be compared in the implementation instead since in many cases most of the endpoints for each key will not be different.
Another option would be just a more clever way of doing change detection, because AFAICT the only reason for PartialEq is to satisfy Watch<ClusterMap>, and the current method of change detection is incredibly brute force, which is especially bad since eg. handle_discovery_responses will do this brute force change detection n times where n is how many Cluster resources there are in the protobuf. That should probably? be moved out to a single .write() regardless, but at least making change detection faster/better would improve that and other cases as well.

benchmark for the ClusterMap::PartialEq implementation, and it shows it's very slow.

Yep, 100% it is that was another thing I saw in the flame graphs.

the current method of change detection is incredibly brute force

Absolutely it is was just the easiest way to allow for thread safe watchers of a type that interior mutability. If there's a better way to do it, we should pursue it.

Only used by tests after #876


Issues with evaling a new ns form in clojurescript

Hello, I am super excited about conjure. It looks like a pretty sweet plugin for integrating prepl (especially clojurescript) into neovim. Everything seems to be working like a charm except for this weird edge case I ran into.
The issue happens when I try to eval a ns form in clojurescript that hasn't been eval'd before. I found eval-str and noticed that conjure unconditionally wraps all code in a do form.
This probably works in almost all use cases except the one I mentioned above. The clojurescript compiler seems to be emitting different code if ns is wrapped in a do form. This causes issues resolving the ns dependencies unlinke if it were a top level form.
I would be open to providing a PR but wanted to bring up the issue for discussion first, thanks.

Thank you for opening an issue first! I'm off to sleep now but I'll think on this a little and get back to you with my thoughts ASAP. Maybe we can avoid the do without breaking anything else 🤔

Check out the develop branch, I've pushed a commit that'll only wrap ClojureScript evals if it thinks there's more than one form or it can't work out how many there are. So if you send a single form it'll remove the (do ...) wrapped. Maybe that'll fix it? If not, it'll be the in-ns switching.

@Olical That seems to fix my issue. Thanks for the quick fix!

:shipit: :tada:
Also worth noting that CLJS exceptions will be ugly data that just gets returned as a result to you until someone merges and releases my prepl patch: https://clojure.atlassian.net/browse/CLJS-3096
As soon as that's in you'll get nicely formatted errors, just like Clojure.


Guide compatibility elements should change background color based on percentage match between user and the guide

Background remains gray if user is not logged in or has not given us data to compare to

This looks like it broke in the recent guide shuffle. Going to reopen, but I reckon that @warpling can pretty easily fix this in the morn.

Think I fixed it.
On Fri, Jan 16, 2015 at 2:49 AM, Simon<EMAIL_ADDRESS>wrote:

This looks like it broke in the recent guide shuffle. Going to reopen, but I reckon that @warpling can pretty easily fix this in the morn.
Reply to this email directly or view it on GitHub:
https://github.com/openfarmcc/OpenFarm/issues/56#issuecomment-70237382


Support org scoped projects

I don't speak TypeScript very well, but if I follow correctly this Action currently supports only repository scoped projects. It would be ideal if it supported organization scoped project as well.

I looked into this today, and it seems like it's unlikely to work well. I think we should be able to optionally iterate over and update a user or an org's projects when we're triggered. The problem is that we'd never get any events when things happened to user or org scoped projects; only issues. So for example dragging a card from one column to another in an org scoped project would not cause the estimates to be updated.


Is the project attempting a fork on Github of Apache Cloudstack?

Why in the world is Apache Cloudstack creating an organization externally on Github? At a minimum, you can't use the Apache Cloudstack logo?

Sorry was talking about this: https://github.com/cloudstack/cloudstack which I saw pop up.

We are actively discussing the role it will play on the dev list.  We are currently exploring our options for potentially using it for CI and distributed integration testing since apache is not giving us any way to do this with the current repository.
You can get more information on the types of problems we are trying to solve here:  https://issues.apache.org/jira/browse/INFRA-11429

@chrismattmann how about you send email to private@ before sending some dummy PR.
thank you

Which ironically we can't close, so I hope @chrismattmann  you will close it, thank you

You could close it by pushing a dummy commit. But yes, I have to agree that sending a PR is some weird communication form!

private@ is not the right place for a discussion about community in an Apache project (though the PR approach is somewhat unusual)

Hi, I will go ahead and jump on the dev list and discuss it. I tried to open up an issue in the forked project, but it took me here, sorry.

@chrismattmann would you mind closing this PR since we can't and we don't need extra clutter in here.  Thx.

closed.


fix: Backwards compatibility in 4.x.x series in iam-user submodule

Description
Revert AWS provider version requirement for recent release, in order to prevent something like the following error out of a terraform init:
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider hashicorp/aws:
│ no available releases match the given constraints >= 2.45.0, ~> 3.0, < 4.0.0,
│ >= 4.1.0

Motivation and Context
Not break compatibility in workspaces that still couldn't upgrade their AWS provider -- but probably should eventually.
Breaking Changes
The opposite, actually.
How Has This Been Tested?
terraform init working again.

Thanks for this PR. Yes, there is a small mess with the versions (my apologies!). I think your fix is good enough for now and we will bump the version of the whole module at some point later.

This PR is included in version 4.17.1 :tada:

Looks like password attribute does not exist regarding to the oficial documentation in provider version 3.X
https://registry.terraform.io/providers/erindatkinson/aws-test/latest/docs/resources/iam_user_login_profile
  on .terraform/modules/iam/modules/iam-user/outputs.tf line 33, in output "iam_user_login_profile_password":
  33:   value       = try(aws_iam_user_login_profile.this[0].password, "")
This object has no argument, nested block, or exported attribute named
"password".


test(combineAll, combineLatest operators): execute tests with TestScheduler#run

Use TestScheduler#run for the combineAll and combineLatest operator tests

ah, sorry - I should have checked my work more carefully. I'll make these fixes now

All set I think. I'll look out for these issues in the other operator specs as I get them

YES! So good!


Jseldess/faq

I think our faq is close-to-final for beta, but I want final opinions and feedback. I addressed @vivekmenezes' and most of @tschottdorf's feedback in issue #8. Some open questions:

Under Does CockroachDB have ACID semantics?, Toby gave the following feedback:

Consistency: I think this point should focus on our optimistic concurrency model. Raft is really more about keeping consistent replicas (which is another meaning of "consistent")
Durability: again there is a mixup of Raft and Replication which I think could be clarified.

@tschottdorf, can you suggest some rewrites here?
@mberhault, please take a look at my answer for What is CockroachDB’s security model?.
cc: @bdarnell, @petermattis, @jess-edwards, @mrtracy

LGTM, specifically for the security model section.

Thanks, @mberhault. Changed that bit of wording in the security faq.

LGTM

LGTM

LGTM!


Feature tuya shutter motor control

Hi,
is it possible to control shutter motors? And if yes, which mqtt topic do i have to use? I already have all keys that I need to send the message to the correct topic. The only thing I´m missing is the right topic for this type of device.
Thanks

Hey,
just try to use it like an socket. If the motor is also controlled via the DPS value 1, it's worth a try.
This might work. There is no topic explicitly for shutter motors yet.
In the future I will probably simplify the Topics anyway a little bit.

Hey,
thanks for the fast response. I already tried but got some exceptions. But I´ll give it another shot.
Gonna post the results here afterwards.

I checked out the tuyapi project from codetheweb and managed to connect to the device. I can also receive data from it. But it seems to be encrypted. I´ve got the right localKey for sure. Output is this: Data from device: n\P®è¹I;;ã _Kgãª
uoÂ& ö¡.dåp'm?`
Do you know if there are any other decrypt meassures needed?

Unfortunately I can't help, because all data is decrypted by the TuyAPI. I think an update of the TuyAPI will solve the problem.
I am currently updating the project to the latest TuyAPI version 4.0.4.


No MTU handling

Can ipstack handle mtu issues? If it is larger than mtu, it will be sent in subpackages.
The attachment is packet capture information that failed due to an MTU error.
mtu_error.txt

Hello,
To investigate this issue I need rust logs alongside of a pcap file.


feature request: dashboard provisioning via chart

Hello,
It would be convenient to be able to provision a configmap containing https://grafana.com/grafana/dashboards/7424 as done in fluent-bit chart here.
The caveat is that the dashboard configmap in this chart and the dashboard hosted on grafana.com will be decoupled and have to stay in sync manually afaik when updates occur (or they can drift 🤷 ).
Does that sound good? If so I can open a PR.

As an opt-in yes, this would be welcome thank you! Please feel free to submit a PR :bow:


Reimplement anchor/unanchor delay

About the PR
The anchorable system now properly adds any delay added after raising AnchorAttemptEvent or UnanchorAttemptEvent.
AtmosUnsafeUnanchor delay slightly increased from 1.5 -> 2. I felt this let me actually read and comprehend the message with enough time to act if I wanted to cancel. Before it was just a touch too fast. I also moved the message to popup over the entity instead of the cursor. For whatever reason the cursor popup would appear in the top left of the screen sometimes. This also felt more consistent since the "Anchored" and "Unanchored" popups already appear on the entity.
Also did a light renaming of systems in the two touched files to remove "system" from their names since I was in there.
Fixes https://github.com/space-wizards/space-station-14/issues/10911
Screenshots

https://user-images.githubusercontent.com/112137107/189247413-47acc22a-10fb-47b3-96e1-d5de3c85b393.mp4
Changelog

:cl: ashtronaut

fix: Pipes with high pressure will warn you when unachoring them and take slightly longer.

A request: If the pressure is over 4MPa can you make it use ThrowingSystem to toss the offender into a nearby wall. This should work with mobs now, and you can subtract the map position of the pipe from the map position of the mob then normalize to get your direction vector.

A request: If the pressure is over 4MPa can you make it use ThrowingSystem to toss the offender into a nearby wall. This should work with mobs now, and you can subtract the map position of the pipe from the map position of the mob then normalize to get your direction vector.

If spacewind was enabled (seems to be off by default), isn't that what would end up happening after the pipe dumps a bunch of high pressure gas on the tile? Or was spacewind for something else


HTTP Response 429 (Too Many Requests) if I select Open Street Maps

...
<IP_ADDRESS> - - [29/Apr/2020 00:15:04] "POST /download-tile HTTP/1.1" 200 -
HIT: https://a.tile.openstreetmap.org/{z}/{x}/{y}.png
RETURN: 429
...

You need find a way for add header at request 'urllib.request.urlretrieve', like this: {"User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0"} ?
Your code should seem like a Webbrowser for OSM servers, not a script.

I'm not sure if it goes against OSM terms and conditions

I made a small example. I try to download the OSM tile in two ways:

wget (without any headers)
requests.get (with header)

Only the second method works with OSM.
See code and result below.
Code:
import wget
import requests

url = "https://a.tile.openstreetmap.org/1/0/0.png"

print("Try download OSM tile with 'wget'...")
try:
	wget.download(url) # saves '0.png'
except Exception as e:
	print(repr(e))

print("\nTry download OSM tile with 'requests'...")
r = requests.get(url, headers={"User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0"})
print(r)
if r.status_code == 200:
	with open("0_.png", "wb") as f:  # saves '0_.png'
		f.write(r.content)

Result:
Try download OSM tile with 'wget'...
<HTTPError 429: 'Too Many Requests'>

Try download OSM tile with 'requests'...
<Response [200]>

OSM server does not allow to download the tile if no headers is specified...
If you change url to another tile server, for example: https://maps.wikimedia.org/osm-intl/1/0/0.png, both download methods are works properly:
Try download OSM tile with 'wget'...
100% [..............................................................................] 11622 / 11622

Try download OSM tile with 'requests'...
<Response [200]>

Note that it is against OSM tiles server rules to bulk download tiles like this.


Handling multiple disjoint tags on a single sync via CI

I'd like to ask about the proper, sanctioned way of handling multiple disjoint configurations, probably belonging to different service owners, for which tags and select tags seem to be appropriate.
Our scenario is:

We'd like to break the configuration file into multiple services, each with their own tag select_tags, e.g. service1 and service2, and multiple files for each service, probably one for the services/routes and another for the consumers e.g. service1/services.yaml and service1/consumers.yaml
Whenever full CI passes and something is merged to master, deploy to the running kong
Do this via sync of the whole directory of configuration files, making sure that what is deployed is exactly what is in the code
Have the ability to either deploy everything or just a subset of the config, such as only the files / directory relevant to a service

Since deck's strategy is to merge all the tags of the different files in the configuration directory and apply them to all involved entities, this does not work. It requires writing some script that goes through the different directories and apply them in chunks based on the service name (e.g. deck sync -s kong-conf/service1/ and deck sync -s kong-conf/service2/).
Encountering this behaviour of the tags was highly unintuitive for me. If the behaviour is intended (as https://github.com/hbagdi/deck/issues/116 suggests), it should probably be more clearly documented, otherwise reconsider this behaviour. In the lattter case, some rationale would be really welcome 👍
Thanks!

Whenever full CI passes and something is merged to master, deploy to the running kong

Have the ability to either deploy everything or just a subset of the config, such as only the files / directory relevant to a service

The reason you are running into this problem is because the way decK is designed, these two are at odds with each other.
The reason select-tags feature exists is to allow for distributed configuration. Now, in your case, you want to do both, incremental updates for specific services and sync the whole thing as well.
Although this is technically possible, my position so far has been that you should script this as you would like because that allows for maximum flexibility and low maintenance.
I'm seeing increasing reports of users wanting to do this but I've not yet figured out a way to add this feature and keep  most users happy.

probably be more clearly documented

We sure can do this part. Any suggestions?
To implement this feature, we will have to figure out if tags should only be merged on a per-file or per-directory basis and see how it also interacts with the workspaces feature in Kong Enterprise.

@hbagdi Thanks for the really quick answer and explanation.
This is not a blocker for us, since as you mention one can always script over this. It's just cumbersome and a bit weird, at least for me 😅
I'll create a PR later this week with some extra docs about the current behaviour. It's the least we can do, this is an awesome project and I'm hoping we can use it in production 👍
/cc @bufferoverflow

I forgot, I can't really comment on the Kong Enterprise implications since I haven't got a license and haven't tested it.

@hbagdi Would it be possible to add an option to render files with select-tags similar to how the docker-compose config command works? Our configuration is split into 2 parts; one file managed by operations and another used to synchronize configurations managed from a self-service UI for customers to manage a subset of consumers.

@hbagdi Would it be possible to add an option to render files with select-tags similar to how the docker-compose config command works?

Interesting. Can you please elaborate a little more? Bonus points for a small example to demonstrate the inputs and outputs.
I've been thinking a little bit around this but I want to know more about your use case before I start rambling on.

@hbagdi Any progress on this issue? Thoughts or feedback on the docker-compose solution?

@kcentralsquare For your use-case, distributed configuration is not a workaround, it is the actual solution and the feature was designed with your use-case in mind.
In your example, you start with the assumption that some resources, tagged as external will not be managed by decK. Later in your example, you switch to using decK to manage those resources. You either want to manage a resource using decK or not using a decK. There is no middle ground here.
Hope that helps!

@hbagdi My use case was a bad example, I'll admit. I still think the ability to pre-render the YAML before applying would give developers more insight into the resulting configuration applied by decK.
In the absence of that functionality, documentation should mention the way decK applies all select_tags to all resources when multiple --state arguments are supplied and an error/warning when attempting to do so via the CLI:

Tags in the select_tags directive will be combined and applied to ALL resources

@hbagdi My use case was a bad example, I'll admit. I still think the ability to pre-render the YAML before applying would give developers more insight into the resulting configuration applied by decK.

decK merges the YAML files you give it. Use any YAML tooling to generate it. I think pre-rendering makes sense but there are not enough use-cases to support it (yet). Another place where it could be useful is #191.

In the absence of that functionality, documentation should mention the way decK applies all select_tags to all resources when multiple --state arguments are supplied and an error/warning when attempting to do so via the CLI:

Do you mind sending us a PR for it on the https://github.com/kong/docs.konghq.com site?

Closing this due to lack of activity. Please re-open if needed.