[help wanted]ffmpeg._run.Error: ffmpeg error (see stderr output for detail)

What do these errors represent, stderr is of no help.
How can I solve this?
OS: Windows 10
CPU: AMD Ryzen7 3700X
Memory: 32GB
GPU: NVIDIA GeForce RTX 2060
Traceback (most recent call last):
  File "F:\RVC-beta-v2-0528\my_utils.py", line 14, in load_audio
    ffmpeg.input(file, threads=0)
  File "F:\RVC-beta-v2-0528\runtime\lib\site-packages\ffmpeg\_run.py", line 325, in run
    raise Error('ffmpeg', out, err)
ffmpeg._run.Error: ffmpeg error (see stderr output for detail)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "F:\RVC-beta-v2-0528\infer-web.py", line 161, in vc_single
    audio = load_audio(input_audio_path, 16000)
  File "F:\RVC-beta-v2-0528\my_utils.py", line 19, in load_audio
    raise RuntimeError(f"Failed to load audio: {e}")
RuntimeError: Failed to load audio: ffmpeg error (see stderr output for detail)

Traceback (most recent call last):
  File "F:\RVC-beta-v2-0528\runtime\lib\site-packages\gradio\routes.py", line 321, in run_predict
    output = await app.blocks.process_api(
  File "F:\RVC-beta-v2-0528\runtime\lib\site-packages\gradio\blocks.py", line 1007, in process_api
    data = self.postprocess_data(fn_index, result["prediction"], state)
  File "F:\RVC-beta-v2-0528\runtime\lib\site-packages\gradio\blocks.py", line 953, in postprocess_data
    prediction_value = block.postprocess(prediction_value)
  File "F:\RVC-beta-v2-0528\runtime\lib\site-packages\gradio\components.py", line 2076, in postprocess
    processing_utils.audio_to_file(sample_rate, data, file.name)
  File "F:\RVC-beta-v2-0528\runtime\lib\site-packages\gradio\processing_utils.py", line 206, in audio_to_file
    data = convert_to_16_bit_wav(data)
  File "F:\RVC-beta-v2-0528\runtime\lib\site-packages\gradio\processing_utils.py", line 219, in convert_to_16_bit_wav
    if data.dtype in [np.float64, np.float32, np.float16]:
AttributeError: 'NoneType' object has no attribute 'dtype'

i get this exact issue and my path to the audio to convert is C:\Users\lucky\Desktop\Audio so it doesnt seem to be a path issue? exact same error message

after getting the stderr i get this in an admin terminal i literally cant give it more permission than it has. even in the security tab i gave full control and modify to the folder
`C:\Users\lucky\Desktop\RVC-beta-v2-0528>runtime\python.exe infer-web.py --pycmd runtime\python.exe --port 7897
Use Language: en_US
Running on local URL:  http://<IP_ADDRESS>:7897
loading weights/CSGOSAS_e10.pth
gin_channels: 256 self.spk_embed_dim: 109

stdout:
stderr: ffmpeg version n4.3.2-160-gfbb9368226 Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 9.3-win32 (GCC) 20200320
configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-w64-mingw32- --arch=x86_64 --target-os=mingw32 --enable-version3 --disable-debug --disable-w32threads --enable-pthreads --enable-iconv --enable-zlib --enable-libxml2 --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-opencl --enable-libvmaf --disable-vulkan --enable-libvorbis --enable-amf --enable-libaom --disable-avisynth --enable-libdav1d --disable-libdavs2 --enable-ffnvcodec --enable-cuda-llvm --disable-libglslang --enable-libass --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvpx --enable-libwebp --disable-lv2 --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librav1e --disable-librubberband --enable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libtwolame --disable-libvidstab --disable-libx264 --disable-libx265 --disable-libxavs2 --disable-libxvid --enable-libzimg --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-libs=-lgomp
libavutil      56. 51.100 / 56. 51.100
libavcodec     58. 91.100 / 58. 91.100
libavformat    58. 45.100 / 58. 45.100
libavdevice    58. 10.100 / 58. 10.100
libavfilter     7. 85.100 /  7. 85.100
libswscale      5.  7.100 /  5.  7.100
libswresample   3.  7.100 /  3.  7.100
C:\Audio: Permission denied
Traceback (most recent call last):
File "C:\Users\lucky\Desktop\RVC-beta-v2-0528\infer-web.py", line 161, in vc_single
audio = load_audio(input_audio_path, 16000)
File "C:\Users\lucky\Desktop\RVC-beta-v2-0528\my_utils.py", line 22, in load_audio
raise e
File "C:\Users\lucky\Desktop\RVC-beta-v2-0528\my_utils.py", line 14, in load_audio
ffmpeg.input(file, threads=0)
File "C:\Users\lucky\Desktop\RVC-beta-v2-0528\runtime\lib\site-packages\ffmpeg_run.py", line 325, in run
raise Error('ffmpeg', out, err)
ffmpeg._run.Error: ffmpeg error (see stderr output for detail)
Traceback (most recent call last):
File "C:\Users\lucky\Desktop\RVC-beta-v2-0528\runtime\lib\site-packages\gradio\routes.py", line 321, in run_predict
output = await app.blocks.process_api(
File "C:\Users\lucky\Desktop\RVC-beta-v2-0528\runtime\lib\site-packages\gradio\blocks.py", line 1007, in process_api
data = self.postprocess_data(fn_index, result["prediction"], state)
File "C:\Users\lucky\Desktop\RVC-beta-v2-0528\runtime\lib\site-packages\gradio\blocks.py", line 953, in postprocess_data
prediction_value = block.postprocess(prediction_value)
File "C:\Users\lucky\Desktop\RVC-beta-v2-0528\runtime\lib\site-packages\gradio\components.py", line 2076, in postprocess
processing_utils.audio_to_file(sample_rate, data, file.name)
File "C:\Users\lucky\Desktop\RVC-beta-v2-0528\runtime\lib\site-packages\gradio\processing_utils.py", line 206, in audio_to_file
data = convert_to_16_bit_wav(data)
File "C:\Users\lucky\Desktop\RVC-beta-v2-0528\runtime\lib\site-packages\gradio\processing_utils.py", line 219, in convert_to_16_bit_wav
if data.dtype in [np.float64, np.float32, np.float16]:
AttributeError: 'NoneType' object has no attribute 'dtype'`

alright gotta apoloigze it seems the actual issue is i gave it the file path without acctualy supplying the filename.
i did this C:\Audio
wheen it shouldve been  C:\Audio\bucklemyshoe.wav

alright gotta apoloigze it seems the actual issue is i gave it the file path without acctualy supplying the filename. i did this C:\Audio wheen it shouldve been C:\Audio\bucklemyshoe.wav

I tried but could not solve the problem. Is there anything else you have changed?

Start the webui with an admin console I guess?

I will add the details of the error.
This occurred when converting an audio file.
In "Inferencing voice" I specified the pth file I downloaded and in "Enter the path of the audio file to be processed" I entered "F:\output.wav".
Then I specified the wav file and clicked "Convert", and it took about 5 seconds before an error message appeared.

I had the same problem, but it worked fine for me before. I don't know what happened to cause the error

I printed the err and the key information is as follows:
Codec AVOption threads (set the number of threads) specified for input file # 0 (F: AI RVC beta train new/desktop. ini) have not been used for any stream. The most likely reason is either wrong type (e.g. a video option with no video streams) or that it is a private option of some decoder which was not actually used for any stream
You can see that ffmpeg mistakenly treated audio files as lrc files. But I didn't find the desktop. ini file, it's a strange error.
Although I don't understand why there is this bug, creating a new folder and copying audio files to the new folder can solve this problem.

same error, moving file sometimes fixes it and sometimes doesn't. it's strange.
Traceback (most recent call last):
  File "L:\RCV_voiceclone\my_utils.py", line 14, in load_audio
    ffmpeg.input(file, threads=0)
  File "L:\RCV_voiceclone\runtime\lib\site-packages\ffmpeg\_run.py", line 325, in run
    raise Error('ffmpeg', out, err)
ffmpeg._run.Error: ffmpeg error (see stderr output for detail)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "L:\RCV_voiceclone\infer-web.py", line 161, in vc_single
    audio = load_audio(input_audio_path, 16000)
  File "L:\RCV_voiceclone\my_utils.py", line 19, in load_audio
    raise RuntimeError(f"Failed to load audio: {e}")
RuntimeError: Failed to load audio: ffmpeg error (see stderr output for detail)

Traceback (most recent call last):
  File "L:\RCV_voiceclone\runtime\lib\site-packages\gradio\routes.py", line 321, in run_predict
    output = await app.blocks.process_api(
  File "L:\RCV_voiceclone\runtime\lib\site-packages\gradio\blocks.py", line 1007, in process_api
    data = self.postprocess_data(fn_index, result["prediction"], state)
  File "L:\RCV_voiceclone\runtime\lib\site-packages\gradio\blocks.py", line 953, in postprocess_data
    prediction_value = block.postprocess(prediction_value)
  File "L:\RCV_voiceclone\runtime\lib\site-packages\gradio\components.py", line 2076, in postprocess
    processing_utils.audio_to_file(sample_rate, data, file.name)
  File "L:\RCV_voiceclone\runtime\lib\site-packages\gradio\processing_utils.py", line 206, in audio_to_file
    data = convert_to_16_bit_wav(data)
  File "L:\RCV_voiceclone\runtime\lib\site-packages\gradio\processing_utils.py", line 219, in convert_to_16_bit_wav
    if data.dtype in [np.float64, np.float32, np.float16]:
AttributeError: 'NoneType' object has no attribute 'dtype'
Traceback (most recent call last):
  File "L:\RCV_voiceclone\my_utils.py", line 14, in load_audio
    ffmpeg.input(file, threads=0)
  File "L:\RCV_voiceclone\runtime\lib\site-packages\ffmpeg\_run.py", line 325, in run
    raise Error('ffmpeg', out, err)
ffmpeg._run.Error: ffmpeg error (see stderr output for detail)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "L:\RCV_voiceclone\infer-web.py", line 161, in vc_single
    audio = load_audio(input_audio_path, 16000)
  File "L:\RCV_voiceclone\my_utils.py", line 19, in load_audio
    raise RuntimeError(f"Failed to load audio: {e}")
RuntimeError: Failed to load audio: ffmpeg error (see stderr output for detail)

Traceback (most recent call last):
  File "L:\RCV_voiceclone\runtime\lib\site-packages\gradio\routes.py", line 321, in run_predict
    output = await app.blocks.process_api(
  File "L:\RCV_voiceclone\runtime\lib\site-packages\gradio\blocks.py", line 1007, in process_api
    data = self.postprocess_data(fn_index, result["prediction"], state)
  File "L:\RCV_voiceclone\runtime\lib\site-packages\gradio\blocks.py", line 953, in postprocess_data
    prediction_value = block.postprocess(prediction_value)
  File "L:\RCV_voiceclone\runtime\lib\site-packages\gradio\components.py", line 2076, in postprocess
    processing_utils.audio_to_file(sample_rate, data, file.name)
  File "L:\RCV_voiceclone\runtime\lib\site-packages\gradio\processing_utils.py", line 206, in audio_to_file
    data = convert_to_16_bit_wav(data)
  File "L:\RCV_voiceclone\runtime\lib\site-packages\gradio\processing_utils.py", line 219, in convert_to_16_bit_wav
    if data.dtype in [np.float64, np.float32, np.float16]:
AttributeError: 'NoneType' object has no attribute 'dtype'

I got a similar issue on the preprocess:
OS: Windows 11
CPU: AMD Ryzen7 5800X
Memory: 64GB
GPU: NVIDIA GeForce RTX 4090
start preprocess
['trainset_preprocess_pipeline_print.py', 'F:\\RVC-beta-v2-0618\\audio\\us', '40000', '16', 'F:\\RVC-beta-v2-0618/logs/us', 'False']
F:\RVC-beta-v2-0618\audio\us/test_us.wav->Traceback (most recent call last):
  File "F:\RVC-beta-v2-0618\my_utils.py", line 14, in load_audio
    ffmpeg.input(file, threads=0)
  File "F:\RVC-beta-v2-0618\runtime\lib\site-packages\ffmpeg\_run.py", line 325, in run
    raise Error('ffmpeg', out, err)
ffmpeg._run.Error: ffmpeg error (see stderr output for detail)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "F:\RVC-beta-v2-0618\trainset_preprocess_pipeline_print.py", line 79, in pipeline
    audio = load_audio(path, self.sr)
  File "F:\RVC-beta-v2-0618\my_utils.py", line 19, in load_audio
    raise RuntimeError(f"Failed to load audio: {e}")
RuntimeError: Failed to load audio: ffmpeg error (see stderr output for detail)

I've trained my model with and without pitch guidance all in model v2 and v1. I Tried 0 protection 0.33 and 0.5, harvest crepe and pm give the same error.
The path is good no spaces no () only - _
Anyone had luck in solving this?
Traceback (most recent call last):
  File "V:\AI\Retrieval-based-Voice-Conversion-WebUI\my_utils.py", line 14, in load_audio
    ffmpeg.input(file, threads=0)
  File "C:\tools\miniconda3\envs\rvc\lib\site-packages\ffmpeg\_run.py", line 325, in run
    raise Error('ffmpeg', out, err)
ffmpeg._run.Error: ffmpeg error (see stderr output for detail)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "V:\AI\Retrieval-based-Voice-Conversion-WebUI\infer-web.py", line 178, in vc_single
    audio = load_audio(input_audio_path, 16000)
  File "V:\AI\Retrieval-based-Voice-Conversion-WebUI\my_utils.py", line 19, in load_audio
    raise RuntimeError(f"Failed to load audio: {e}")
RuntimeError: Failed to load audio: ffmpeg error (see stderr output for detail)

Traceback (most recent call last):
  File "C:\tools\miniconda3\envs\rvc\lib\site-packages\gradio\routes.py", line 321, in run_predict
    output = await app.blocks.process_api(
  File "C:\tools\miniconda3\envs\rvc\lib\site-packages\gradio\blocks.py", line 1007, in process_api
    data = self.postprocess_data(fn_index, result["prediction"], state)
  File "C:\tools\miniconda3\envs\rvc\lib\site-packages\gradio\blocks.py", line 953, in postprocess_data
    prediction_value = block.postprocess(prediction_value)
  File "C:\tools\miniconda3\envs\rvc\lib\site-packages\gradio\components.py", line 2076, in postprocess
    processing_utils.audio_to_file(sample_rate, data, file.name)
  File "C:\tools\miniconda3\envs\rvc\lib\site-packages\gradio\processing_utils.py", line 206, in audio_to_file
    data = convert_to_16_bit_wav(data)
  File "C:\tools\miniconda3\envs\rvc\lib\site-packages\gradio\processing_utils.py", line 219, in convert_to_16_bit_wav
    if data.dtype in [np.float64, np.float32, np.float16]:
AttributeError: 'NoneType' object has no attribute 'dtype'


Write failures when persisting to MarkLogic are not communicated properly

When attempting to persist a document that isn't currently supported by a MarkLogic XML mount, the write fails but the error isn't properly reported. It isn't clear yet whether this is related to the MarkLogic connector or is a more general issue with reporting write errors from any connector.

Assigning this the may iteration, since it seems that #2124 (which is already in May) depends on this. @wemrysi - assigning this to you for now, but please reassign, if it is something deeper across all connectors.

@wemrysi I can probably work on this while altering the embedding of EJSON in XML. I'll assign it to myself.

@jsantos17 Thanks, but I actually already fixed the problem, just need to add a regression test and PR.


Zoom in &out in invert color mode results in zathura to "blink" when browsing PDF

On GitLab by @lyhokia on Jun 12, 2021, 07:57

Way to reproduce:

open some pdf
turn on invert color mode
zoom in/out

Expect behavour:

Zathura can smoothly zoom in&out

Actual behavour:

The page blinks once (i.e. turn to normal mode, then back to invert color mode)

On GitLab by @lyhokia on Jun 12, 2021, 08:03

BTW, I'm using Sway on Arch Linux, maybe XWayland should be blame for this? But I don't see any blur from zathura in HiDPI, so I assume it's not running XWayland


Finished texts of Feldspar and Gabbro & Fix some typo in Chinese translation

@solitary-sails Thank you!!!
I was wondering about these missing Chinese texts for a long time, so your translation is very helpful!!!

Your translation is released in v1.1.2.


SME: Is it ok if the signature overlaps the words a little?

To give the user more room for their signature

Kate says that yes it is totally fine

Kate says that yes it is totally fine


Exclude OE_API_VERSION=2 from CMake package

The samples do not expect this to be defined, and we did not intend to define it for all users of the SDK (yet). However, it should be internally public because our own build system expects it to be defined.
So we constrain it to our build system but not our package using a generator expression to exclude it from the install interface.

This was spun out of #1472, and is needed despite #1493.

@CodeMonkeyLeet We should not expose the V2 define in package (which we're accidentally doing here) until we "flip the switch" and drop support for V1. A requirement of doing that is #1493, which opts the samples into V2.

Ah, I see what you're saying. I had the understanding that we're moving everything to v2 within a week or two, @paulcallen can you chime in on that?
Thanks!
bors r+

bors r+

bors r+


Fix menu entry instantiation

Tileman Mode plugin is crashing and thus disappears from the Plugin hub. This is due to MenuEntry being changed to an interface and no longer being instantiatable
This MR updates menu entry code to use new MenuEntry creation API, which should fix the crash and restore the plugin to Runelite. Also updates the runeliteVersion

LGTM, cleans up the code a bit :)


Moving the grids on the example page causes flickering in Chrome

When I start dragging the last element in the grid example the grid just a few pixels it starts flickering a little bit by appearing on a different position. This seeems to happen only with the last element (using Chrome Version 74.0.3729.157). Hard to explain, see the video:
https://youtu.be/GCbqq8OAwp8

Seems to be the same issue as #1450.
Does it happen only on the grid demo? Or the other lists as well?
If you edit the code and set forceFallback: true in the options, does it still happen?

On the example page it seems to happen only on the grid demo. But in my implementation it's the same issue. I'm using the latest Bootstrap grid system and made the rows sortable there.
Maybe it's the display-Type that's causing this issue? The bootstrap grid uses flex.
However, when adding forceFallback: true the issue is gone in my case (didn't try it for the grid demo), so thanks for this workaround, though it's not that nice because obviously the items can be dragged anywhere and are not staying within their parent container.

@sattes-faction What browser are you using, and what version?

See my first post, it's Chrome Version 74.0.3729.157 on Windows 10. The issue does not appear in Firefox.

As with the other reports of this bug, I still cannot reproduce this :/

@sattes-faction I have opened an issue in the Chrome bug tracker.
If you have any relevant information to help them track this down I would greatly appreciate it. At the very least, if you could comment that you are also experiencing the issue (assuming you are)
https://bugs.chromium.org/p/chromium/issues/detail?id=990572

@kevin-j-morse ok I posted a comment there with the youtube link from the first post. Hope it helps. I did no further investigation since then...


Should be scrollable if chart does not fit in the view

A huge thank you for this amazing component. I really love it.
Checklist:

[* ] I updated to the latest version available
[* ] I cleared the cache of my browser

Release with the issue:
v1.9.3
Last working release (if known):
Browser and Operating System:
Windows 10 and Chrome 109 with large monitors -> OK
Android 11 and Chrome with small screen and portrait orientation -> not so good
Description of problem:
Some days ago I added another section so I get energy consumption per room in my house also. That's when it started to become more difficult to see sensors in my mobile.
When sankey charts have 4 or more sections, then the width will grow beyond my mobile phones capabilities in portrait orientation. This makes all other components in that view wider and difficult to look at.
Javascript errors shown in the web inspector (if applicable):
Additional information:
Perhaps not a bug per see, but I believe all components should handle these basic things so filing it as a bug. Maybe related to #86. I have fiddled with the wide option without success.
So better would be that the entire component becomes scrollable when it does not fit in the view.
Also, I'm using the view type grid everywhere.

I am aware of this and am thinking about it but am not sure if scrolling is the solution yet. I've also been thinking about a vertical layout for mobile but that would be a huge change.
For now I just turn my phone horizontally :/

You can now set minimum width per section. If the screen width is less than the sum of min width of all sections. The card will extend beyond the screen and be scrollable

Ah, got it. Thanks, it works.


[BUG] Missing autocompletion for request.app

Describe the bug
When I am using the request instance in an endpoint, my IDE does not provide me with autocompletion for request.app.
Is it maybe due to missing type hints? Or because the request instance is from Starlette and cannot be annotated in FastAPI?
To Reproduce
Steps to reproduce the behavior:

Create a file with

from fastapi import FastAPI
from starlette.requests import Request

app = FastAPI()

@app.get('/')
def get(request: Request):
    request.app

Try to run autocompletion for request.app

Expected behavior
I can use the auto-completion provided by me IDE (e.g. PyCharm)
Screenshots
n/a
Environment:

OS: macOS

FastAPI Version: 0.42

Python version: 3.7.4

Additional context

It isn't working because have typehint typing.Any:
https://github.com/encode/starlette/blob/master/starlette/requests.py#L50

So would it be possible to have this Request object as a subclassed object in FastAPI where we can change the type annotation?

I think more correct will create issue in starlette

@tiangolo @dmontagu Would you be willing to accept a PR where we subclass the Starlette objects so that we can benefit from the type annotations in the FastAPI class?
A straightforward example would be:
from starlette.testclient import TestClient as tc

class TestClient(tc):
    def __init__(self, app: FastAPI):
        super().__init__(app)
        self.app: FastAPI = app

Another candidates apart from TestClient would be Request.
I am asking because this would also involve rewriting a couple of the documentation pages.
And: Am I missing some downsides of doing that?

I think it would make sense for starlette to add the type hint of Starlette on request.app, that would go a long way toward getting useful autocompletion.

I believe it would work if you create your own custom subclass

class FastAPIRequest(Request):
    app: FastAPI

Then use FastAPIRequest in your endpoints -- fastapi should still identify it as a Request and inject the appropriate value (though it will be a vanilla starlette request that gets injected).
If this didn't work you could do something like the following which definitely would work (maybe requiring some minor tweaks to get mypy to like it):
if typing.TYPE_CHECKING:
    class FastAPIRequest(Request):
        app: FastAPI
else:
    FastAPIRequest = Request

The above approaches should work for TestClient too.

I'm not sure whether it makes sense to add this level of nuanced typing to FastAPI or just to fix it in Starlette as best as possible. It will add a bit of a maintenance burden if/when starlette changes its API. I'd be more in favor if presented with more of a practical problem that better type hints would address (i.e., what completion do you wish you had that you currently don't?). I think in some (many?) cases fixing the annotations in starlette might solve it; if so that's probably preferable.

I am fine with only making these changes in my own project, @dmontagu. I just have trouble to understand how a better annotation in Starlette would help FastAPI. My assumption is that properties or methods that are only available in FastAPI cannot be made available via type annotations in Starlette (?).

That's right, but what properties or methods do you want to use that are only available in FastAPI? If there are important ones to support then I think it would make sense to try to address this. Usually the base Starlette ones are enough though, at least in my experience.

Is there a reason you couldn't access them on the app directly though? For example, adapting your initial example, let' say you wanted to return the docs_url:
from fastapi import FastAPI
from starlette.requests import Request

app = FastAPI()

@app.get('/')
def get(request: Request):
    return request.app.docs_url

I agree it's annoying you don't get completion, but it seems to me you should just be using the app directly:
from fastapi import FastAPI
from starlette.requests import Request

app = FastAPI()

@app.get('/')
def get(request: Request):
    return app.docs_url

That would give you access to the autocompletion you want.

I think it could make sense to make it easier to get autocompletion for request.state, request.app, and request.app.state, and I actually do this in my own codebases now by subclassing (at least in if TYPE_CHECKING: blocks). But all of the approaches I've seen to this so far (including my own) seem pretty opinionated, enough that I'm not sure it makes sense to integrate directly into FastAPI.

Also, just to give an example, this is what I mean by subclassing to get autocompletion:
class MyServerState:
    database: Database
    cache: Optional[FanoutCache]
    orm: MyORM
    search: SearchEngine

    def __init__(self, database: Database, cache: FanoutCache = None) -> None:
        self.database = database
        self.cache = cache
        self.orm = MyORM(self.database, self.cache)
        self.search = SearchEngine(self.orm)

class MyServer(FastAPI):
    state: MyServerState  # type: ignore[assignment]

    def setup_state(self, database: Database, cache: FanoutCache = None) -> None:
        self.state = MyServerState(database, cache)

class MyRequest(Request):
    app: MyServer

I understood the subclassing approach and will also follow it. I thought it might be a good idea to suggest this auto-completion improvement to have it for other users, as well. If that's not the case, feel free to close the issue.

It seems like the starlette people are not changing their implementation cf. https://github.com/encode/starlette/issues/688.

The first thing would be, why do you want to access the app through the request?
Maybe if we understand the use case better we can find the right solution for it.
I think I can imagine a use case where it would be better to access the app through the request rather than directly as @dmontagu says.
About accessing docs_url and openapi_url, if you need to use that in other parts of your code, maybe it would be a better idea to put them in a Pydantic's BaseSettings that you can use everywhere.
About dependency_overrides, why would you want to access it in a path operation?
About types for the TestClient, I think it's better to wait until the TestClient is based on HTTPX, to not lose that effort. Also, that itself might solve the problem.

I had two use cases for accessing fields of the app instance in a path operation

Redirecting based user agent

if any(browser in user_agent for browser in BROWSERS):
            return RedirectResponse(url=request.app.docs_url)
        return RedirectResponse(url=request.app.openapi_url)

This would be solved by your suggestion to extract these values into a settings object.

Include app version into a path endpoint

I guess your same suggestion would apply here.
I am fine with that and I missed the proposed options when creating the issue.

Great! Thanks for replying back @chbndrhnns . Then I think we can close this issue, right?


Code challenge error when running the test app

[Enter feedback here]
Hello, I followed this page, I don't think I made any error (but I can't be sure).
When I test the sample app, I get an error upon signin in : the server expects a code challenge that the sample app does not provide.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 866b331e-e1ac-cd02-9aa3-f7334fa59546
Version Independent ID: 5f2b1f45-f873-ad9a-17ed-efc1e1bdb5ae
Content: Configure authentication in a sample Node.js web application by using Azure Active Directory B2C (Azure AD B2C) - Azure AD B2C
Content Source: articles/active-directory-b2c/configure-a-sample-node-web-app.md
Service: active-directory
Sub-service: b2c
GitHub Login: @kengaderdus
Microsoft Alias: kengaderdus

@joelbloch Thanks for your feedback. We will investigate and update the thread.

@joelbloch You are getting this error because you have chosen "SPA" under redirect URI which registering the application which enabled PKCE flow and looking for code_challenge parameter which is required to get the access token in PKCE flow. Please update you redirect URI with "Web" as mentioned in the document and you will be able to run the application as specified in the document.

Closing the thread as this is not documentation issue. Any functionality related issues are investigated at  All Questions - Microsoft Q&A


Typerror with primus

I am trying to run this on Ubuntu 14.04 running node v0.10.25 -- I am getting quite a few errors:
a) with the lambda functions <(a,b,c)  => { }> not being recognized ( I got past them by replacing '=>' with 'function')
b)

TypeError:Object function Object() { [native code] } has no method 'assign'

which I got past with Object.assign = require('object-assign') at the beginning of the corresponding files.
c) I also got

SyntaxError:Use of const in strict mode

which I got past with node --harmony app.js
d) My latest error is

/data/binb/node_modules/primus/transformers/faye/server.js:54
const spark = new this.Spark(
^
TypeError: undefined is not a function

I'm not sure how to get past this. Any hints would be useful. Thanks!

Hi, use a newer version of Node.js, version 0.10 is no longer supported. See https://github.com/nodesource/distributions

I had a feeling it might have to do with versions :-) Thats fixed now! Thanks.


Where is remove-calico-policy.sh ? is it deprecated？

not in https://github.com/tigera/docs, and is removed in https://github.com/projectcalico/calico/commit/5826e2ff33fd336040d599f7263404cdc43856ff

Expected Behavior

a up to date clean up script
Current Behavior

can't find effective scripts for cleanup.
Possible Solution

Steps to Reproduce (for bugs)

Context

Your Environment

Calico version master branch
Orchestrator version (e.g. kubernetes, mesos, rkt):
Operating System and version:
Link to your project (optional):

This may have been an accidental casualty of the docs move. @ctauchen might know better if there are plans to add that back, but guessing it was just an accident.
@joeyleeeeeee97 That said, that script is not really maintained and is probably long out of date - where did you see a reference to it?

I see that the hack directory was included in the docs site files. But it wasn't part of the published documentation. The content there seems well out of date.
@caseydavenport Perhaps we could consider restoring this in projectcalico? If we wanted to publish any of this on the docs site, we'd need to make revisions first.

@caseydavenport , I am deploying calico in my EKS, after uninstall it, I saw left calico iptable ruls on the machine.
In that case the network is broken, can't access any.
After manually run the script,  this issue is fixed.

@ctauchen @caseydavenport Hi, I have few questions about calico's left iptable rules:

what it does? I am only using it to enable wireguard, I am not applying any network policy.
Is this ever cleaned? if yes when and how could we ensure clean up happened.

Thanks!

yeah, it was never meant to be part of the published documentation - it's really meant for developers rather than end-users.

Action here is still to remove this from the documentation and move it back to the projectcalico/calico repository.

@caseydavenport we should maintain this script, sometimes uninstalling calico will leave uncleaned iptables, very tricky.

The script isn't supported or maintained, so should be removed. It was never intended as a supported way of uninstalling Calico, even when it was first introduced.
When uninstalling Calico (or any CNI plugin really), you should reboot your nodes to ensure transient state is cleaned (including iptables, routes, ipsets, interface state and configuration, etc).


Add docker compose files to appservers

Related issue #4414 because the names should match

Please put in juiste map


Fixed the file searching code for linux also

Sorry @the-ethan-hunt , while trying to resolve conflicts I accidently closed the previous PR.
So, started a new one.

Don't repeat this action. Please merge your conflicts in the same PR itself.

I performed a hard reset on my last commit. That's why the PR got closed automatically.
I wrote the code again and create a new PR.
I didn't deliberately close it.


Fix css class collision in widget scroll indicators

Styles meant for the main scroll indicator applied to the scroll
indicators inside the navigation and overview widgets.
Prefix classes with widget name to prevent collision.

Coverage remained the same at 99.053% when pulling 8104ac9faba9ac6781d65f8bcb5a476d3c604a9e on tf:widget-scroll-indicators-fix into 008a2457e3c57d74947e5329afa96f36ea208e43 on codevise:master.

Coverage remained the same at 99.053% when pulling 8104ac9faba9ac6781d65f8bcb5a476d3c604a9e on tf:widget-scroll-indicators-fix into 008a2457e3c57d74947e5329afa96f36ea208e43 on codevise:master.


[DO NOT MERGE] Uses the updated version for bls12377js with the new greatest bit location

Description
https://github.com/celo-org/bls-zexe/pull/140 changes the location of the greatest bit, so we must update bls12377js as well.

This is the relevant change in the underlying library: https://github.com/celo-org/bls12377js/commit/ca0092230efcc5e6c190689d564d8c61b5ba710e.

close?

Closing as this hasn't had movement for several months. Please reopen if this is still relevant!


Not supported CPU AMD Ryzen 5 5600X 6-Core Procesor

Hi, I'd be really interested in your app, that I would use inside an arcade cabinet and not an handle console, but my CPU isn't supported by your app. If you might give it a shot once, I'd be really happy of it! But if this kind of things comes out of your project, I'll totally understand.
Have a nice day and thanks for your work!

this software is not being supported anymore. find my latest here which likely supports this cpu:
https://github.com/project-sbc/Power-Control-Panel-v2

Hey! Working on v2 overhaul which will support 2nd gen onward, should be out in a week or so
Get Outlook for iOShttps://aka.ms/o0ukef

From: TheBloodstomper @.>
Sent: Wednesday, July 6, 2022 5:32:19 PM
To: project-sbc/Power-Control-Panel @.>
Cc: Subscribed @.***>
Subject: [project-sbc/Power-Control-Panel] Not supported CPU AMD Ryzen 5 5600X 6-Core Procesor (Issue #31)
Hi, I'd be really interested in your app, that I would use inside an arcade cabinet and not an handle console, but my CPU isn't supported by your app. If you might give it a shot once, I'd be really happy of it! But if this kind of things comes out of your project, I'll totally understand.
Have a nice day and thanks for your work!
—
Reply to this email directly, view it on GitHubhttps://github.com/project-sbc/Power-Control-Panel/issues/31, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASZ4DJOVAZY6V3JDOVSAE2LVSX3OHANCNFSM523GXREQ.
You are receiving this because you are subscribed to this thread.Message ID: @.***>


[WIP] Use Azure ML Environments and add code upload option

First attempt at PR to resolve #14. Tests to be updated after first feedback.

So it basically uploads all of the files in your current directory and executes the code in the provided environment, right? If it works, then it's fine to me as long as we preserve the standard flow with the docker.
@em-pe what are your thoughts on this?

So it basically uploads all of the files in your current directory and executes the code in the provided environment, right? If it works, then it's fine to me as long as we preserve the standard flow with the docker.

Indeed. It does preserve the current Docker workflow, although the default would be to have code upload. I think this makes sense as this is how Azure ML Jobs are used in the documentation and examples.

As for environment creation verification - maybe we can check whether the checksum of local Dockerfile matches the checksum of the environment's Dockerfile (if it exists), if not, we can trigger the build via AML SDK API.

That would indeed work if the Environment is created just using a Dockerfile. However, Azure ML Environments can be created in several ways (curated, Docker images in registry, Dockerfile, Docker build context, conda YAML file, and possibly others). Supporting a diff checking for all these methods would be hard to impossible. Just as an example: for the Docker build context you would need to check whether any of the files used by the Dockerfile changed since the last time you created the Environment.

OK, we're fine with the code-upload workflow, but let's keep it as an alternative, leaving the docker workflow as default (which would be coherent across all of our plugins).
Please proceed with this PR, we will then test it and merge it. It seems like some of the unit tests are failing in the CI:
  ==================================== ERRORS ====================================
  ______________________ ERROR collecting tests/test_cli.py ______________________
  ImportError while importing test module '/home/runner/work/kedro-azureml/kedro-azureml/tests/test_cli.py'.
  Hint: make sure your test modules/packages have valid Python names.
  Traceback:
  /opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/importlib/__init__.py:126: in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
  tests/test_cli.py:14: in <module>
      from kedro_azureml.constants import FILL_IN_DOCKER_IMAGE
  E   ImportError: cannot import name 'FILL_IN_DOCKER_IMAGE' from 'kedro_azureml.constants' (/home/runner/work/kedro-azureml/kedro-azureml/kedro_azureml/constants.py)

  ---------- coverage: platform linux, python 3.10.6-final-0 -----------
  Name                             Stmts   Miss Branch BrPart  Cover   Missing
  ----------------------------------------------------------------------------
  kedro_azureml/__init__.py            3      0      0      0   100%
  kedro_azureml/cli.py                85     31     18      2    54%   29, 35->exit, 46-47, 72-88, 100->exit, 143-177, 214-217, 250-259
  kedro_azureml/cli_functions.py      21     12      6      0    33%   13-32, 36-42
  kedro_azureml/client.py             43     27     10      0    34%   19-39, 44-45, 53-80
  kedro_azureml/config.py             23      0      8      0   100%
  kedro_azureml/constants.py           3      0      0      0   100%
  kedro_azureml/datasets.py           32     15     10      0    45%   22-27, 31, 35, 41-45, 48-52, 55
  kedro_azureml/generator.py          73     50     20      0    27%   29-34, 37-66, 69-72, 75, 78, 87, 113-125, 136-155, 158-166
  kedro_azureml/runner.py             22      9      4      0    58%   16-18, 29-34, 39
  kedro_azureml/utils.py              27     10      4      0    68%   21-25, 30, 33-37, 40
  ----------------------------------------------------------------------------
  TOTAL                              332    154     80      2    49%
  Coverage XML written to file coverage.xml

  =========================== short test summary info ============================
  ERROR tests/test_cli.py
  !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
  =============================== 1 error in 2.43s ===============================
  Error: ERROR: InvocationError for command /opt/hostedtoolcache/Python/3.10.6/x64/bin/poetry run pytest --cov kedro_azureml --cov-report xml --cov-report term-missing --ignore=venv --verbose -W ignore::marshmallow.warnings.RemovedInMarshmallow4Warning (exited with code 2)

@tomasvanpottelbergh any updates? :)

Sorry, it's been pretty busy. I hope to work on this on Friday.

Sorry for the long wait... The unit tests now pass locally. Would be great if you could test and review!

@marrrcin could it be that there's something wrong with the SonarCloud config?

Thanks for your contribution @tomasvanpottelbergh, we will test it this week!
SonarCloud will not pass on PRs due to missing key in the configuration.

@tomasvanpottelbergh FYI: I had to restore some of the functionality, because your PR broke the backward compatibility.
Also, I've went through the quickstart with code upload you introduced and there were a few uncovered issues, especially w.r.t environments and the file exclusions. I've added some functionality and meaningful log messages to guide the users better through this process, because without them, the quickstart with Spaceflights start was impossible to run.

Thanks for accepting the PR and cleaning it up! It was indeed not intended to be backwards compatible, since I thought breaking the API was preferable over complicating the code at this point, but that's of course up to you to decide.
I noticed you mention that code upload is the default in the updated quickstart, but I actually reverted this as requested in https://github.com/getindata/kedro-azureml/pull/15/commits/65e78411377f6c80a8b7d34a5b38512e9fb7b53d. So you would need to set code_directory: "." in the config template if you want it to be the default.

I've created https://github.com/Azure/azure-cli-extensions/issues/5560 for that.


Implement occurrence download

[x] Download button on occurrence page pops up form / open new page
[x] If required user is prompted to log in
[x] Let user select download format
[x]  Show user summary of issues, suggesting e.g. hasGeospatialIssue=false etc

Blocked by #62

I'd like to propose trying to dissuade users from downloading the entire index. If we detect they are about the download the entire index, ask them to confirm this is intended and that they understand a suite of facets and filters are available for use.
I'd also like to propose adding a disclaimer making the user aware that:

GBIF assigns the download a DOI with the intention it makes citing the download easier
GBIF assumes the responsibility of ensuring this DOI resolves indefinitely
We encourage the user to either

associate the download with a journal article or to indicate their intended purpose for the download  (see issue #64)
delete this download if they know it will be unused, is a duplicate, etc (see issue #64)

We explain that the download may be deleted/cleaned up after a specified time

In some repositories they give the downloader the opportunity to share the reason of the download, in a simple text box. Wouldn't it be nice to have this for GBIF downloads. Not mandatory, but just the opportunity to share the 'story-reason' of downloading data. This might give us more overall insight on downloading events.

Currently the user has to enter username and password every time as there isn't any login. But it is possible to download and there is added a basic report of issues in the data and a few other warnings (living specimens and fossils).


repository-hdfs no work with Kerberos authenticate hdfs

Issue description
No authenticate hdfs seem to be OK，but Kerberos authenticate hdfs is faild
Steps to reproduce

kinit every data node

 klist -e
Ticket cache: FILE:/tmp/krb5cc_1001
Default principal<EMAIL_ADDRESS>
Valid starting    Expires           Service principal
28/12/2016 15:32  30/12/2016 15:32  krbtgt/xxx@xxxM
	renew until 04/01/2017 15:32, Etype (skey, tkt): des3-cbc-sha1, des3-cbc-sha1

add  repository

{
  "type": "hdfs",
  "settings": {
    "path": "/es_snapshot",
    "max_restore_bytes_per_sec": "500mb",
    "compress": "true",
    "uri": "hdfs://10x0.6x0:9000",
    "max_snapshot_bytes_per_sec": "500mb",
    "conf.hdfs_config": "/home/elk/etc/hadoop/hdfs-site.xml",
    "conf.hadoop_config": "/home/elk/etc/hadoop/core-site.xml",
    "user_keytab": "/home/elk/elk_sa.keytab",
    "user_principal": "elk_sa",
    "user_principal_hostname": "scheduler"
  }
}

Strack trace:
{
  "error": {
    "root_cause": [
      {
        "type": "repository_exception",
        "reason": "[dt_backup] failed to create repository"
      }
    ],
    "type": "repository_exception",
    "reason": "[dt_backup] failed to create repository",
    "caused_by": {
      "type": "creation_exception",
      "reason": "Guice creation errors:

1) Error injecting constructor, org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
  at org.elasticsearch.repositories.hdfs.HdfsRepository.<init>(Unknown Source)
  while locating org.elasticsearch.repositories.hdfs.HdfsRepository
  while locating org.elasticsearch.repositories.Repository

1 error",
      "caused_by": {
        "type": "access_control_exception",
        "reason": "SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]",
        "caused_by": {
          "type": "remote_exception",
          "reason": "SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]"
        }
      }
    }
  },
  "status": 500
}

I want to use KERBEROS，why notice SIMPLE authentication is not enabled
Version Info
OS:         :  debian 8
JVM         : orc java 8
Hadoop/Spark:  hadoop5.6.0-cdh2.6.0
ES-Hadoop   : elasticsearch-repository-hdfs-2.3.4.BUILD-SNAPSHOT.jar Yarn
ES          :  2.3.4

Feature description

this works for us using repository-hdfs version 2.4.3
also i saw in your config this:
    "user_principal": "elk_sa",
    "user_principal_hostname": "scheduler"

user_principal_hostname only does something if you have a _HOST placeholder in your user_principal. e.g.
"elk_principal": "elk_sa/_HOST"

@koertkuipers how to get user_principal_hostname variable?

if you use the string "_HOST" in your user_principal it will automatically get replaced by the hostname. the hostname is automatically inferred for each machine.
you can also set explicitly the hostname that replaces _HOST in the user_principal using the variable user_principal_hostname. however i don't really see the purpose of that: if you want to set the hostname explicitly don't use _HOST but instead use the correct hostname in user_principal.
so i dont really see a reason someone would ever want to set user_principal_hostname. i think its an unnecessary and confusing variable.

@koertkuipers
{
  "type": "hdfs",
  "settings": {
    "path": "/es_snapshot",
    "max_restore_bytes_per_sec": "500mb",
    "compress": "true",
    "uri": "hdfs://10.1x0.x.100:9000",
    "max_snapshot_bytes_per_sec": "500mb",
    "conf.hdfs_config": "/home/elk/etc/hadoop/hdfs-site.xml",
    "conf.hadoop_config": "/home/elk/etc/hadoop/core-site.xml",
    "user_keytab": "/home/elk/elk_sa.keytab",
    "user_principal": "elk_sa/_HOST"
  }
}

same error

it could be the version of the plugin (i am using 2.4.3)
also you can kinit on the hosts from the keytab?
e.g. kinit -t -t /home/elk/elk_sa.keytab elk_sa/scribe

hi @zcola , have you resolved the problem? i encountered the same problem on 5.1.1.

@xy-xin noop

"SIMPLE authentication is not enabled" is a confusing error. it means earlier your client failed to get a kerberos token and therefore is trying to log in without kerberos (which is called SIMPLE authentication).
so it shows your client never tried to connect to hdfs with kerberos permissions.
in the core-site.xml that you provided does it say:
  <property>
    <name>hadoop.security.authentication</name>
    <value>kerberos</value>
  </property>

i upgrade es 2.4.4 in testing env, open trace level log,same error
[2017-02-23 11:22:59,799][TRACE][cluster.service          ] [elk-edata01-102_hot] will process [put_repository [my_backup6]]
[2017-02-23 11:22:59,799][DEBUG][cluster.service          ] [elk-edata01-102_hot] processing [put_repository [my_backup6]]: execute
[2017-02-23 11:22:59,800][DEBUG][repositories             ] [elk-edata01-102_hot] creating repository [hdfs][my_backup6]
[2017-02-23 11:22:59,879][DEBUG][repositories.hdfs        ] [elk-edata01-102_hot] Login using keytab [/home/elk/elk_sa.keytab] and principal [elk_sa] (with hostname [null])
[2017-02-23 11:22:59,880][DEBUG][org.apache.hadoop.hdfs.BlockReaderLocal] dfs.client.use.legacy.blockreader.local = false
[2017-02-23 11:22:59,880][DEBUG][org.apache.hadoop.hdfs.BlockReaderLocal] dfs.client.read.shortcircuit = false
[2017-02-23 11:22:59,881][DEBUG][org.apache.hadoop.hdfs.BlockReaderLocal] dfs.client.domain.socket.data.traffic = false
[2017-02-23 11:22:59,881][DEBUG][org.apache.hadoop.hdfs.BlockReaderLocal] dfs.domain.socket.path =
[2017-02-23 11:22:59,881][DEBUG][org.apache.hadoop.io.retry.RetryUtils] multipleLinearRandomRetry = null
[2017-02-23 11:22:59,882][DEBUG][org.apache.hadoop.ipc.Client] getting client out of cache: org.apache.hadoop.ipc.Client@67509721
[2017-02-23 11:22:59,882][DEBUG][org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil] DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection
[2017-02-23 11:22:59,882][DEBUG][repositories.hdfs        ] [elk-edata01-102_hot] Using file-system [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1976723552_29, ugi=elk (auth:SIMPLE)]]] for URI [hdfs://<IP_ADDRESS>:9000], path [hdfs://<IP_ADDRESS>:9000/es_snapshot]
[2017-02-23 11:22:59,927][TRACE][org.apache.hadoop.ipc.ProtobufRpcEngine] 29: Call -> /<IP_ADDRESS>:9000: getFileInfo {src: "/user/elk"}
[2017-02-23 11:22:59,928][DEBUG][org.apache.hadoop.ipc.Client] The ping interval is 60000 ms.
[2017-02-23 11:22:59,929][DEBUG][org.apache.hadoop.ipc.Client] Connecting to /<IP_ADDRESS>:9000
[2017-02-23 11:22:59,934][DEBUG][org.apache.hadoop.ipc.Client] IPC Client (830895739) connection to /<IP_ADDRESS>:9000 from elk: starting, having connections 1
[2017-02-23 11:22:59,934][DEBUG][org.apache.hadoop.ipc.Client] IPC Client (830895739) connection to /<IP_ADDRESS>:9000 from elk sending #4
[2017-02-23 11:22:59,936][DEBUG][org.apache.hadoop.ipc.Client] IPC Client (830895739) connection to /<IP_ADDRESS>:9000 from elk got value #-1
[2017-02-23 11:22:59,936][DEBUG][org.apache.hadoop.ipc.Client] closing ipc connection to /<IP_ADDRESS>:9000: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
	at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1126)
	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:974)
[2017-02-23 11:22:59,937][DEBUG][org.apache.hadoop.ipc.Client] IPC Client (830895739) connection to /<IP_ADDRESS>:9000 from elk: closed
[2017-02-23 11:22:59,937][DEBUG][org.apache.hadoop.ipc.Client] IPC Client (830895739) connection to /<IP_ADDRESS>:9000 from elk: stopped, remaining connections 0
[2017-02-23 11:22:59,937][TRACE][org.apache.hadoop.ipc.ProtobufRpcEngine] 29: Exception <- /<IP_ADDRESS>:9000: getFileInfo {org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]}

does the file /home/elk/etc/hadoop/core-site.xml have the correct settings for secure hadoop?

@koertkuipers yes
  <!-- Web Interface Configuration -->
  <property>
    <name>hadoop.http.filter.initializers</name>
    <value>org.apache.hadoop.http.lib.StaticUserWebFilter,org.apache.hadoop.security.AuthenticationFilterInitializer</value>
  </property>
  <property>
    <name>hadoop.http.authentication.type</name>
    <value>kerberos</value>
  </property>
  <property>
    <name>hadoop.http.authentication.simple.anonymous.allowed</name>
    <value>true</value>
  </property>

  <!-- security setting -->
  <property>
    <name>hadoop.security.authentication</name>
    <value>kerberos</value>
  </property>
  <property>
    <name>hadoop.security.authorization</name>
    <value>true</value>
  </property>
  <property>
    <name>hadoop.security.groups.cache.secs</name>
    <value>14400</value>
  </property>
  <property>
  <name>hadoop.kerberos.kinit.command</name>
  <value>/usr/bin/kinit</val

very strange. it is clear it wont work when this is being logged:
[2017-02-23 11:22:59,882][DEBUG][org.apache.hadoop.ipc.Client] getting client out of cache: org.apache.hadoop.ipc.Client@67509721
[2017-02-23 11:22:59,882][DEBUG][org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil] DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection
[2017-02-23 11:22:59,882][DEBUG][repositories.hdfs        ] [elk-edata01-102_hot] Using file-system [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1976723552_29, ugi=elk (auth:SIMPLE)]]] for URI [hdfs://10.1x0.xx9.100:9000], path [hdfs://<IP_ADDRESS>:9000/es_snapshot]
[2017-02-23 11:22:59,927][TRACE][org.apache.hadoop.ipc.ProtobufRpcEngine] 29: Call -> /<IP_ADDRESS>:9000: getFileInfo {src: "/user/elk"}

the issue is the auth:SIMPLE
checking if we have any other settings... we have:
load_defaults: "false" # optional - whether to load the default Hadoop configuration (default) or not
and in elasticsearch.yml we have:
security.manager.enabled: false

@zcola I have run into the same issue while testing some changes to the HDFS Repository Plugin in 5.2.
In your settings you are trying to pass configuration files to the plugin :
    "conf.hdfs_config": "/home/elk/etc/hadoop/hdfs-site.xml",
    "conf.hadoop_config": "/home/elk/etc/hadoop/core-site.xml"

but I do not believe these are affecting the underlying configuration in they way you think it might be. Instead of adding those files as sources for the configuration, these lines are just setting two properties to point to the given strings. You may be confusing those settings with the conf_location option.
@koertkuipers is correct in their explanation of the host property. It is best to leave this off. Its original inclusion was most likely set as an option for completeness, but is rather confusing.
Setting the keytab information is not enough when trying to authenticate with Kerberos, and as you've seen, if the authentication mode is not changed from SIMPLE, then no log in will actually be performed in the code (even if we do call login with the keytab info). In this case I would try setting conf.hadoop.http.authentication.type to kerberos in the plugin configuration request. This should add the property correctly to the Hadoop configuration. I'd steer clear of the conf_location option as it will not be available in later versions of the plugin (The plugin was re-written in 5.0).

security.manager.enabled: false
Why should the security manager be set to false?
because you added kerberos to manage the security instead, no need to let java security manager do the job?
Up to now, I have not seen a successful case for creating hdfs repository with security manager enabled. Is it possible?

@mikej1688 Authenticating to a remote service such as Kerberos is not the same as code level security. Code level security dictates that the author of the code executing may not be you, and thus, may need to be kept from performing certain actions as you. This is different from proving you are who you say you are to an external service. The HDFS client executes quite a bit of code that requires a large number of elevated privileges to function. Running the plugin below 5.0 with the SecurityManager enabled is unlikely to succeed. In Elasticsearch 5.0, the plugin was re-written to be compliant with the installed SecurityManager. If you require the manager to be enabled, please upgrade to Elasticsearch 5.0 or higher.

Thanks for the explanation.
Currently we cannot upgrade to 5.x since there is an outstanding snapshot
creation issue even in the latest 5.2.2 version. It was said by ES
consultant that the issue might be fixed in the incoming 5.3. but we cannot
rely upon the uncertainty.
Thus my question is: in 2.x version, is it a security hole when we have
security.manager.enabled :  false?
If yes, how can we avoid it?
Another question : is the parameter, x-pack security manager in 5.x, not
related to the security.manager.enabled in 2.x?
Thus in 5.x, no any extra setting for code level security is needed in
elasticsearch.yml?
On Mar 17, 2017 5:21 PM, "James Baiera"<EMAIL_ADDRESS>wrote:
@mikej1688 https://github.com/mikej1688 Authenticating to a remote
service such as Kerberos is not the same as code level security. Code level
security dictates that the author of the code executing may not be you, and
thus, may need to be kept from performing certain actions as you. This is
different from proving you are who you say you are to an external service.
The HDFS client executes quite a bit of code that requires a large number
of elevated privileges to function. Running the plugin below 5.0 with the
SecurityManager enabled is unlikely to succeed. In Elasticsearch 5.0, the
plugin was re-written to be compliant with the installed SecurityManager.
If you require the manager to be enabled, please upgrade to Elasticsearch
5.0 or higher.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/elastic/elasticsearch-hadoop/issues/921#issuecomment-287474329,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHjJHQ6Ms1P5_j-4uQskD-bivhqPNMfiks5rmvlZgaJpZM4Ln2ih
.

@mikej1688 Strictly speaking, if the security manager is disabled, then yes, that could pose a potential security risk should you be loading plugins or running scripts that are not trusted and are looking to cause damage.

I'm closing this issue for the time being as the discussion has gotten off course and the original author has not responded for a while.


Set to snapshot

Signed-off-by: Matthew de Detrich<EMAIL_ADDRESS>

Codecov Report

Merging #61 into master will increase coverage by 0.2%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master      #61     +/-   ##
=========================================
+ Coverage   74.34%   74.54%   +0.2%
=========================================
  Files          22       22
  Lines         499      499
  Branches        8        8
=========================================
+ Hits          371      372      +1
+ Misses        128      127      -1

Impacted Files
Coverage Δ

...n/scala/org/zalando/kanadi/api/Subscriptions.scala
69.6% <0%> (+0.44%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b725a49...4a3d89b. Read the comment docs.


Adds API local auth validation

This PR adds the following global settings for API local auth password webhook and description:
api.auth.local.validate.url and api.auth.local.validate.description
The password entered while creating admin user or changing password while user has logged in, is passed to the validation url, checked and the status is returned. Depending on the validation, either the user is created or error is displayed

@prachidamle can you please review this?

@prachidamle I've made the changes

@prachidamle The TFs are because of 409 errors, I don't think the other two are related

https://github.com/rancher/rancher/issues/1983

LGTM

The TFs seem unrelated, test_snapshot_lifecycle and test_root_volume_restore failed with 409 error and test_validate_service_scaleup_scaledown is a known TF.

https://github.com/rancher/rancher/issues/5973


Authorization endpoint error when using cloud foundry UAA locally

I'm playing around with Cloud Foundry UAA as a stand alone server for my applications and I'm trying to set it up locally on my machine to test it out. I've followed the instructions as set out
here...
However, after running ./gradlew run and opening another terminal to retrieve a token for the default test user "marissa" I get the following error:
$ uaac token get marissa koala
CF::UAA::BadResponse: status 400
attempt to get token failed

The uaa logs show this error:
[2017-05-28 17:37:44.680] cloudfoundry-identity-server - ???? [http-nio-8080-exec-9] .... DEBUG --- AuthzAuthenticationFilter: Located credentials in request, with keys: [password, username]
[2017-05-28 17:37:44.683] cloudfoundry-identity-server - ???? [http-nio-8080-exec-9] .... DEBUG --- ChainedAuthenticationManager: Attempting chained authentication of org.springframework.security.authentication.UsernamePasswordAuthenticationToken@a3913f17: Principal: marissa; Credentials: [PROTECTED]; Authenticated: true; Details: remoteAddress=0:0:0:0:0:0:0:1, clientId=cf; Not granted any authorities with manager:org.cloudfoundry.identity.uaa.authentication.manager.CheckIdpEnabledAuthenticationManager@1c964b71 required:null
[2017-05-28 17:37:44.685] cloudfoundry-identity-server - ???? [http-nio-8080-exec-9] .... DEBUG --- AuthzAuthenticationManager: Processing authentication request for marissa
[2017-05-28 17:37:44.797] cloudfoundry-identity-server - ???? [http-nio-8080-exec-9] .... DEBUG --- AuthzAuthenticationManager: Password successfully matched for userId[marissa]:0b2ef36d-ead3-459d-971c-45d4df8b7a53
[2017-05-28 17:37:44.801] cloudfoundry-identity-server - ???? [http-nio-8080-exec-9] ....  INFO --- Audit: UserAuthenticationSuccess ('marissa'): principal=0b2ef36d-ead3-459d-971c-45d4df8b7a53, origin=[remoteAddress=0:0:0:0:0:0:0:1, clientId=cf], identityZoneId=[uaa]
[2017-05-28 17:37:44.804] cloudfoundry-identity-server - ???? [http-nio-8080-exec-9] .... DEBUG --- ChainedAuthenticationManager: Chained Authentication status of org.springframework.security.authentication.UsernamePasswordAuthenticationToken@a3913f17: Principal: marissa; Credentials: [PROTECTED]; Authenticated: true; Details: remoteAddress=0:0:0:0:0:0:0:1, clientId=cf; Not granted any authorities with manager:org.cloudfoundry.identity.uaa.authentication.manager.ChainedAuthenticationManager$AuthenticationManagerConfiguration@4a6e0cf9; Authenticated:true
[2017-05-28 17:37:44.819] cloudfoundry-identity-server - ???? [http-nio-8080-exec-9] ....  INFO --- UaaAuthorizationEndpoint: Handling OAuth2 error: error="invalid_grant", error_description="Invalid redirect https://uaa.cloudfoundry.com/redirect/cf did not match one of the registered values"

I see its trying to make a call to uaa.cloudfoundry.com/redirect/cf which must be set somewhere that I can't figure out.
I can't seem to find any steps in the docs to get this to work or any properties/values I can overrride.
Can any one help me out with this?
Thanks.

UAAC documentaation has been updated to address this. Please refer to the latest

Hello @sreetummidi, could you precise what you meant here because, as far as I know, the redirect URI still refers to "https://uaa.cloudfoundry.com/redirect/cf" which is not good I guess for run.pivotal.

@sreetummidi, The instructions here still seem to have the same omission:
https://docs.cloudfoundry.org/concepts/architecture/uaa.html
Specifically, step 5 states:

Run uaac token get USERNAME PASSWORD to log in. Replace USERNAME with your user name, and PASSWORD with your password. For example:
 $ uaac token get marissa koala

which still fails as the OP describes.

That documentation is incorrect. The command you have provided here uaac token get marissa koala seems to represent a misunderstanding about what the uaac is doing. When you use the uaac you are exercising the UAA's OAuth functionality using a specific OAuth client and grant type. It is impossible to get a token without a client. The (incorrect) command you found in that documentation is attempting to get a token using the implicit grant type using a client called marissa. This isn't going to work for you since marissa is a user and not a client.
To perform administrative actions during local development, the thing to do is
uaac token client get admin -s adminsecret

This fetches a token using the client_credentials grant type using the admin client which was configured here. If you're not using local deployment, but in fact are using a CF deployment, you will need to find the actual admin client secret before you can run this command. In cf-deployment, the relevant variable is called uaa_admin_client_secret.
If you want a token with marissa's permissions via the password grant type (like what the CF cli does), the uaac can use the password grant to get tokens, e.g. uaac token owner get cf marissa -s "" -p koala. The cf client configuration is here. If you look through that xml file you will find some other "showcase" clients you can play with.
I hope that helps you out. I will reach out to the docs team to fix that documentation. @jzheaux

@jhamon Thanks for the thorough explanation! That's clear now, and I can now see how the command in the docs is incorrect.
Also, I can confirm that both the commands you just shared work for me. Thank you for your help!

To follow up on the questions asked above, @jhamon how do I get an authorization grant for the admin admin client which was configured . I can't seem to know what to do after getting the token for the admin client as all PI calls return an error.
P.S i am also developing locally.


Make rec.log_file  able to overwrite entity_path to make it easier to handle them without being dependant on the filename which can be ugly

Make rec.log_file  able to overwrite entity_path to make it easier to handle them without being dependant on the filename which can be ugly.
Describe the annoyance
Let say we want to create an app that let users log their urdf file. let's say the file name is going to be piper.urdf.
Currently, the only thing we can do on the entity_path is prefix it with something like production_piper.urdf
Now the thing is that this entity is refering to something i personally call bob, well I can't rename this entity_path bob. I can only rename the file. But maybe I want to use this file for another entity call mary.
Expected behavior
Add a function to log_file but that can actually overwrite the entity_path.
Maybe I didn't find it.

All rec.log_file does is transfer control to a dataloader. That dataloader is then free to log whatever it wants, to whatever entity path it wants -- Rerun itself has no say in any of this.
I assume you're referring to our URDF example in this specific case. This examples uses the filename as its entity path because it explicitly chooses to:
https://github.com/rerun-io/rerun-loader-python-example-urdf/blob/539252acc50771829f7e2aacef27d6cb2cbd7928/rerun_loader_urdf.py#L313-L316
if args.entity_path_prefix is not None:
    prefix = args.entity_path_prefix
else:
    prefix = os.path.basename(args.filepath)

You should fork/copy-paste this dataloader and modify its code in order to make it behave however you'd like it to, you'll have full control from there.
Dataloaders are just cheap, arbitrary, external scripts -- they are designed that way so that they can be easily fork to accommodate to anyone's needs.
See also: https://rerun.io/docs/reference/data-loaders/overview

On a related note: even if we were to add an --entity-path flag, you would still need to fork and modify the dataloader itself in order for it to actually support that flag.
What I would like us to have in the future is server-side rewrite rules, so that you could pass e.g. regex-based remappings in log_file(), and have the Rerun database monkey-patch the dataloaded data so that it follows these rules,e.g.:
# entity path `xxx` will by remapped to `yyy` by monkey patching the data directly on the server-side.
rr.log_file("model.urdf", [EntityPathRemapRule("s/xxx/yyy/g")])

But that's a much, much bigger can of worms -- for now everything is handled client-side, by the dataloader itself.

Got it. Thanks for the detailed explanation!
Will patch rerun urdf loader in that case.


multi vlan support networking

For my current project I need the following features:

A network port can be part of several vlans.
Applications should be able to specify, which vlan they wish to work with, as in linux, where a virtual network device is created and the application can choose to work with e.g.
eth0     -- without vlan
eth0.7  -- with vlan7
eth0.8  -- with vlan8

A network port can be part of several vlans.

By "port" do you mean network interface?

Applications should be able to specify, which vlan they wish to work with, as in linux, where a virtual network device is created and the application can choose to work with e.g.
eth0     -- without vlan
eth0.7  -- with vlan7
eth0.8  -- with vlan8

VLANs are configured similar way in zephyr too. After starting the application, it needs to call net_eth_vlan_enable(iface, vlan_tag); to attach VLAN tag to the interface and enable VLAN.

Yes, this is exactly how zephyr works too. One bigger difference is that the amount of VLANs need to be known in advance as the virtual VLAN interfaces are allocated during compile time. At runtime application will need to enable the actual VLAN functionality. See how to do that for example in samples/net/sockets/echo_server/src/vlan.c

Thanks for your reply. I had a look at the sample you mention, but it seems to me, that it only adds 1 vlan per interface. For me the problem manifests itself when I call the shell commands:
net vlan add 7 1
net vlan add 8 1
Here I get an error for the second command. Currently our net_if interfaces reflect our physical network ports, so their number is the same as the number of physical network ports. I understand your explanation such, that we should create more interfaces than our physical ports and do the vlan add on these 'virtual interfaces'.

it only adds 1 vlan per interface

Yes, but that interface is the "VLAN" virtual interface. So in Zephyr, there is the master interface (like eth0 in Linux), then there are virtual VLAN interfaces (like eth0.10 etc in Linux). All of these interfaces are created in advance during compilation. The virtual interfaces are not usable themselves for VLAN, but require setting of VLAN tag and IP addresses etc. See also documentation at https://docs.zephyrproject.org/latest/reference/networking/vlan.html
So in the net-shell example, you should not try to add VLAN to the master interface but to the VLAN interfaces so something like
net vlan add 7 2
net vlan add 8 3

Thanks for the explanation. Now it is much clearer.

@jukkar
Thanks for you answer.
I will try to ask specific question.
Currently, we are creating 1 interface per physical port by using:
NET_DEVICE_INIT_INSTANCE(...)
If I would like to create in addition virtual interface, I will create more instance by using?
NET_DEVICE_INIT_INSTANCE
if yes, so now I have 2 interface per one physical port,
for example RX side on this port, I am  receiving packets by using
if (net_recv_data(iface, pkt) < 0) {
should I have one iface for both of the interface? or I need to parsing it by vlan id and select the right interface?
If they have the same iface, what is the MACRO do it?
Thanks a lot,
Ehud
Thanks,
Ehud

Currently, we are creating 1 interface per physical port by using:
NET_DEVICE_INIT_INSTANCE(...)

You need to use ETH_NET_DEVICE_INIT() to create all the virtual VLAN interfaces, the NET_DEVICE_INIT() does not support VLANs.

Hi @jukkar
Thank you for your help,
another question, I found  an issue:

I have 2 interface on the same physical port.
I would like that both of them have the same mac address -> and also the same ipv6 local address.
I configure the second interface to be with vlan, while the first interface without it.
while ping to the vlan interface the packet is dropped in tx.

It is drop in  ethernet_send->set_vlan_tag
tatic enum net_verdict set_vlan_tag(struct ethernet_context *ctx,
   			     struct net_if *iface,
   			     struct net_pkt *pkt)
{
   int i;

   if (net_pkt_vlan_tag(pkt) != NET_VLAN_TAG_UNSPEC) {
   	return NET_OK;
   }

#if defined(CONFIG_NET_IPV6)
   if (net_pkt_family(pkt) == AF_INET6) {
   	struct net_if *target;

   	if (net_if_ipv6_addr_lookup(&NET_IPV6_HDR(pkt)->src,
   				    &target)) {
   		if (target != iface) {
   			NET_DBG("Iface %p should be %p", iface,
   				target);

   			iface = target;
   		}
   	}
   }

the net_if_ipv6_addr_lookup will return the first interface ptr instead the second, as result afterwards it wont find the vlan(becasue it isn't exist on the first iface) and the packet will be drop.
I am suggesting to replace the

if (net_if_ipv6_addr_lookup(&NET_IPV6_HDR(pkt)->src,
					    &target)) {
			if (target != iface) {
				NET_DBG("Iface %p should be %p", iface,
					target);

				iface = target;
			}
		}

with
               if (!net_if_ipv6_addr_lookup_by_iface(iface, &NET_IPV6_HDR(pkt)->src)) {
                      return NET_DROP;
                }

of course, similar thing for Ipv4 too, first I would like to understand what do you think.
Thanks,
Ehud

Hi @ehudmarvell, sorry for late reply, I just got back from vacation.

have 2 interface on the same physical port.

I would like that both of them have the same mac address -> and also the same ipv6 local address.

The MAC address will be the same for VLAN ports as there is only one physical network interface involved. Each port will have unique IPv6 link local address, the reason being that the neighbor cache implementation in zephyr uses IPv6 address to find out the neighbor, so cache could return wrong network interface in that case.
I am not sure what exactly is the issue. Could you prepare separate issue instead of adding new stuff to this one. If you have patches, just them as PRs, it is easier to review them that way and also try and test them.

I am closing this one as the original question was answered.


Make the whole button a link

Hello,
To open the docs, we need to click on the little button read docs but if the whole item was a button, it could help to open docs more easily.

Hey there @Barbapapazes , this enhancement will be featured in the next release 😀.


standardize.py does not have column loaded_on converted to timestamp type, load fails in 02_standardize notebook

The line t_parkingbay_sdf, t_parkingbay_malformed_sdf = s.standardize_parking_bay(parkingbay_sdf, load_id, loaded_on) fails with the following during the append into interim.parking_bay table:

Error:
AnalysisException: Cannot write incompatible data to table '`interim`.`parking_bay`':
- Cannot safely cast 'loaded_on': string to timestamp;

I noticed in standardize.py  > standardize_parking_bay and standardize_sensordata functions:

lit(loaded_on.isoformat()).alias("loaded_on") is generating a string, which is probably why the append into the Parquet table fails, since it's expecting a timestamp:

Not sure how this ever worked before, I see the standardize.py file hasn't changed since December 5, 2019 😕

Possible fix - FYI
I changed both instances to lit(loaded_on.isoformat()).cast("timestamp").alias("loaded_on") in my own environment to fix the issue:

Also, along the way, had to change .withColumn("last_edit", to_timestamp("last_edit", "YYYYMMddHHmmss")) to .withColumn("last_edit", col("last_edit").cast("timestamp")), it seems Spark 3.0 no longer supports YYYYMMddHHmmss pattern for typecasting to_timestamp (according to the error message at least):

I didn't want to dig too much into the syntax, cast("timestamp") did the trick.
Then finally, the job succeeded:

This is fixed by this PR: https://github.com/Azure-Samples/modern-data-warehouse-dataops/pull/322


this.$recaptcha.reset is not a function

I'm trying to reset the captcha when the user clicks on a button by running the this.$recaptcha.reset();
In my local project it works as expected but once I test it in deployment server it gives me the following error:

@hmsk @dotneet @aldarund can anyone help me please it's an urgent issue that I need to solve

same problem here

I am facing the same issue.


Minimal ios support

Thanks for the PR.
I don't know anything about Mac and iOS, so what is the difference between DefaultOutput and RemoteIO? If you want, adding a code comment that explains that would be nice.

The DefaultOutput unit isn't available in iOS unfortunately. RemoteIO is a sensible replacement.
Here is what the iOS doc says:
iOS provides three I/O (input/output) units. The vast majority of audio-unit applications use the Remote I/O unit, which connects to input and output audio hardware and provides low-latency access to individual incoming and outgoing audio sample values. For VoIP apps, the Voice-Processing I/O unit extends the Remote I/O unit by adding acoustic echo cancelation and other features. To send audio back to your application rather than to output audio hardware, use the Generic Output unit.
See https://developer.apple.com/library/content/documentation/MusicAudio/Conceptual/AudioUnitHostingGuide_iOS/UsingSpecificAudioUnits/UsingSpecificAudioUnits.html


[fix] (compaction) fix compaction score in time series policy (#40242)

Proposed changes
pick from master #40242

run buildall


style(readme): fix some typo in documentation

Description
Hello! Congrats for the release. 🎉
This pull request fixes some typo errors in the README file.
Related Issue
I haven't created any issue for this task. 🙈
Motivation and Context
I don't like typos, they make me sad. 😿
How Has This Been Tested?
I read it twice! 😸
Types of changes

Chore (non-breaking change which refactors / improves the existing code base)
~Bug fix (non-breaking change which fixes an issue)~
~New feature (non-breaking change which adds functionality)~
~Breaking change (fix or feature that would cause existing functionality to
change)~

Checklist:

✅ My change requires a change to the documentation.
✅ I have updated the documentation accordingly.
🤷♂ I have added tests to cover my changes.
✅ All new and existing tests passed.
✅ I have read the CONTRIBUTING document.

Thank you for the PR @arnaudmorisset 😘


feat!: remove offset from decodeLog and decodeFunctionResult methods

Interface.decodeLog and Interface.decodeFunctionResult are top-level functions that don't need to return a data offset because you pass in all the data and the returned offset is always the length of that data. If somebody wants to get the offset value after this change, they can just call data.length.
const theInterface = new Interface(...);

// previously
const [decodedLog, offset] = theInterface.decodeLog(data, logId);
// now
const decodedLog = theInterface.decodeLog(data, logId);

// previously
const [decodedFnResult, offset] = theInterface.decodeFunctionResult(fnName, data);
// now
const decodedFnResult = theInterface.decodeFunctionResult(fnName, data);

I tagged it as a feat because it improves the DX of the methods.
Supersedes #2308

✨ A PR has been created under the rc-2514 tag on fuels-wallet repo.
https://github.com/FuelLabs/fuels-wallet/pull/1374

@arboleya the offset value is equal to data.length so people can always get it that way. I updated the description to be more explicit about this.

Pattern
Could we be breaking the patterns already in use?
https://github.com/FuelLabs/fuels-ts/blob/e165e37307d65deda99f675723ea26470bf807d8/packages/transactions/src/coders/transaction.ts#L104-L133
Wether this is a good or a bad thing, I'll let other chime in.
let decoded;
let o = offset;

[decoded, o] = new BigNumberCoder('u64').decode(data, o);
const scriptGasLimit = decoded;

[decoded, o] = new B256Coder().decode(data, o);
const receiptsRoot = decoded;

// ..

Exceptions
Also, it seems some Coders have non-obvious ways of computing the offset:
https://github.com/FuelLabs/fuels-ts/blob/e165e37307d65deda99f675723ea26470bf807d8/packages/abi-coder/src/encoding/coders/B256Coder.ts#L43
https://github.com/FuelLabs/fuels-ts/blob/e165e37307d65deda99f675723ea26470bf807d8/packages/abi-coder/src/encoding/coders/VecCoder.ts#L63-L69
API Preferences
Regarding personal preference, I'd go with Objects instead of Arrays.
But only if we'd change the pattern everywhere, which is not the case.
const { decodedLog, offset } = theInterface.decodeLog(data, logId);
// vs
const [decodedLog, offset] = theInterface.decodeLog(data, logId);

Conclusion
I'm not sure if the breaking change is worth it.
The change is subtle, but the latter hides information.
const [decodedLog] = theInterface.decodeLog(data, logId);
// vs
const decodedLog = theInterface.decodeLog(data, logId);

As for if this is a good or a bad thing, I'll let others chime in.

@arboleya you're right, introducing this would break the pattern present in other coders; I didn't think of that. If I remember correctly, there was talk on the offsite around not needing the offset value across the board on all our coders and that this would be considered in #1795.
The argument for merging it right now is that this is a high level API relative to the coders and we can get the DX benefits of it right now because the offset value is never needed when using these methods. The counter-argument is that API consistency is more important and that it should be done in tandem with #1795.
In the end, I agree with your uncertainty

I'm not sure if the breaking change is worth it.

and I'll close this PR. I've added a comment on #1795 linking to this PR.


Add support for multiple cache instances

Only one instance is possible at a time which causes shared cache between opposing players and other inconveniences.

Done in fbeaad1b6c1acb511403a2d24db81a798c844f01


How to do this in Pad?

Thank you very much. Just today I installed this wonderful plugin and to be honest, I have a lot of questions.

How did you put pictures on group chats?
I noticed that the contact filter sometimes does not display users, but shows the group - "bots" where one user is located and that's me! I click the "refresh" arrow button and all users show up. Is this some kind of bug?
The same thing if you put the status "Online". Sometimes correctly shows sometimes empty. It is solved when you put the filter by name and all at once appear.
I created a common room for the company. And I have to give everyone an invitation. In open air, I put a user group on the room that they are members of this room (200 users), but they themselves do not appear in the group in Pade. How do people do it right?


Silence spam output from noisy tests

A recently added test is spewing out extraneous debug output.  This shouldn't happen -- silence this test!
$ nosetests -v tests/datadriven/
...
tests.datadriven.test_variants.VariantSetTest.testVariantSetMetadata:1kgPhase1 ... ok
[W::_vcf_parse_format] FORMAT 'GT' is not defined in the header, assuming Type=String
[W::_vcf_parse_format] FORMAT 'GT' is not defined in the header, assuming Type=String
[W::_vcf_parse_format] FORMAT 'GT' is not defined in the header, assuming Type=String
tests.datadriven.VariantSetTest.testProtocolElementValid:1kgPhase3 ... ok
...

One way to do this might be creating a @SuppressOutput decorator in tests.utils that redirects stdout and stderr to /dev/null for the running duration of the test and then restores the file handles at the end.

Alternatively, fix the data file (looks like one of the 1kg VCF files has a malformed header line)

These warnings are coming from htslib, which would be nice to suppress in general; is there a way to silence it through pysam?

The error looks like it comes from this line in htslib:
https://github.com/samtools/htslib/blob/f3e1602196bbf03f426dfb363a4841932a042194/vcf.c#L1522
It looks like it's writing it directly to stderr without going through any kind of logging subsystem.  So there are probably no hooks in pysam or htslib to redirect / silence.

@jeromekelleher do you know what line in our code is accessing the htslib code that is emitting the error?

Nevermind, seems to be DataDrivenTest.__init__ where self._gaObject is initialized

Redirecting stdout and stderr doesn't seem to do it... this diff still emits the output...
     def __init__(self, setId, baseDir):
         self._setId = setId
         self._dataDir = os.path.join(baseDir, setId)
+        origStdout = sys.stdout
+        origStderr = sys.stderr
+        devnull = open(os.devnull, 'w')
+        sys.stdout = devnull
+        sys.stderr = devnull
         self._gaObject = self.getDataModelClass()(
             self._setId, self._dataDir)
+        sys.stdout = origStdout
+        sys.stderr = origStderr

Not sure what the deal is here.  Maybe stderr gets lots in the C/python binding?

Aha, looks like python's stderr object is a wrapper around something but C accesses the file descriptors directly.
http://stackoverflow.com/questions/5081657/how-do-i-prevent-a-c-shared-library-to-print-on-stdout-in-python

Fixed in #261


Untitled files with associated path behave very weird

Steps to Reproduce:

run from the command line code <folder> <non existing file>
notice how in the backups folder you see a folder hierarchy being created for the untitled file instead of a single file with the path of the untitled as hash

restart

=> the files get restored in a weird way:

This actually also needed a fix from my end in the way how the untitled file is serialized.


Remote Tab Disappears after Atom closes After latest Atom update.

Prerequisites

#[X] Put an X between the brackets on this line if you have done all of the following:

Checked that your issue isn't already filed: https://github.com/icetee/remote-ftp/issues

Description
When using the latest atom (1.33.0) the remote tab doesn't stay open after closing and re-opening atom. The only way to get it backis to remove the project folder, add it back and then the project and remote tab appears. Otherwise you can connect but the file tree for the remote never appears
Steps to Reproduce

Launch Atom
Open fresh project by using a new folder
Toggle on remote tab and connect and all appears to work fine.
Close Atom
Open Atom (mine opens back onto the project)
There will be no tree view for the remote tab. Toggling does nothing. You can still connect but the tree view for the remote server doesn't show. Making it impossible to select files to modify.

Expected behavior: Atom to open back up with the remote tab there like it used to do.
Actual behavior: Atom opens up with no remote tab and has no way of getting it back apart from removing project folder and adding it back which brings back the project bit at the top of the tree which then allows the remote tab to be seen.
Reproduces how often: 100% Everytime I open Atom, tested on 2 devices with latest release of atom
Versions
Atom    : 1.33.0
Electron: 2.0.11
Chrome  : 61.0.3163.100
Node    : 8.9.3
Additional Information
nope

I have the same problem.
Atom: 1.32.2 x64
Electron: 2.0.9
OS: Ubuntu 18.04.1


layout is not changing.

Target device: (Note: simulators are not supported for running the UIKit)

Device: [e.g. iPhone6]
OS:

[X] IOS
[x] Android

OS Version: [e.g. 22, 14.1]

App Info
"agora-react-native-rtm": "^1.5.1",
"agora-rn-uikit": "^5.0.1",
"react": "17.0.2",
"react-native": "0.67.5",
"react-native-agora": "^4.2.0"
Describe the issue
Trying to change layout to grid
To Reproduce
<AgoraUIKit
connectionData={connectionData}
rtcCallbacks={rtcCallbacks}
rtcProps={{layout: Layout.Grid}}
/>
Screenshots
If applicable, add screenshots to help explain your problem.

Working -
<AgoraUIKit
connectionData={connectionData}
rtcCallbacks={rtcCallbacks}
settings={{
layout: Layout.Grid,
activeSpeaker: true,
disableRtm: false,
displayUsername: true,
dual: true,
enableMediaOnHost: true,
mode: ChannelProfileType.ChannelProfileCommunication1v1,
profile: '720p_6',
role: ClientRoleType.ClientRoleBroadcaster,
showPopUpBeforeRemoteMute: true,
}}
/>


Update insync to <IP_ADDRESS>131

After making all changes to the cask:

[x] brew cask audit --download {{cask_file}} is error-free.
[x] brew cask style --fix {{cask_file}} left no offenses.
[x] The commit message includes the cask’s name and version.

It seems the original sha256 was correct.


Clippy cleanup:

redundant_guards
unnecessary_cast
clone_on_copy
bool_assert_comparison
non_snake_case
missing_safety_doc

ping @andoriyu review please.

thank you so much


selectable key-binding for console

I am unable to use the console as my laptop does not have an End key.  Would it be possible to add a feature to allow a user to define their own key-binding or a secondary console key?

It has it already, after running with CET once the config.json will have a console_key entry added to it, AFAIK the value for that needs to be a VK number, http://www.kbdedit.com/manual/low_level_vk_list.html has a list of them, you might need to convert the hex number to decimal first though.


fix: fix numeration after 10

Description

Related to #303
Fixed numeration after 10

/deploy-review


Change Keli scoped query function name

Query specific code for Keli was added. The preferred function has changed in the Keli platform from q() to query().
The new function will always return an array, never a single object.

Thank you Nathanael!

@beaudesigns - Not a problem Beau; I've pushed v1.2.3 to NPM with your PR.


Show SRPM url

[ ] Add a new column to the SRPM Builds tab here. (Or find a better place where we can put this info.)

Requires: https://github.com/packit/packit-service/issues/958

could be probably included in packit/packit-service#912

@mfocko I don't so -- it's about date-time and result page, not a dashboard.

@mfocko I don't so -- it's about date-time and result page, not a dashboard.

yeah I know, but we could include more info in the SRPM logs page apart from logs, e.g. commit sha, said date-time or also the URL
from commit status we can get to build result and from there to srpm logs and COPR or SRPM URL
however from dashboard we can get only directly to COPR or SRPM logs, so adding more info to either the page with SRPM logs or dashboard would be beneficial

Since we want to replace result-page by the dashboard, I would leave this issue as is (here).
But you have a good point, there is other information missing on the dashboard site. Although I expected this to be done when working on the internal SPIKE card, we can create a new issue that will define some dashboard alternative to the current result page(s). (The table for all results will not be enough for all the information. I have some detail single-result view in mind.)
(Once that other issue is implemented, this one won't be needed probably. But this is much quicker to do in the meantime.)

Since we want to replace result-page by the dashboard, I would leave this issue as is (here).

i don't have the local dev environment for packit service setup right now but if you can make that information available via the API, I'll add it to the dashboard

@IceWreck That would be really nice!
Sure, we can. I'll describe at least some basics in a new issue so you can know what to expect.

I'll describe at least some basics in a new issue so you can know what to expect.

Here it is: #71

The API already contains the URL value. (example: https://stg.packit.dev/api/srpm-builds/3000)

Resolved by 5c0bb706214e34c660db28f0b703f9d0f3d7ec9a, it is kept in result page
@lachmanfrantisek do we still want the column in the table or are we fine with the current state?

@mfocko You are right, we can maybe improve formatting, but good enough for now:


How to add path specific express middleware?

I'm not seeing any documentation on how I can add middleware on a path by path basis.

I'm working on a more integrated solution, but there are two options currently:

Execute the middleware in your controller: middleware(req, res, next);
Add the middleware in app.js: app.use();

Scott

Option 1 feels dirty, as you'd have to use async
Option 2 only applies to the global app correct?

You don't have to use async. You can just incorporate it like this:
var middleware = require('otherCoolMiddleware');
function myController(req, res, next) {
  middleware(req, res, function(err) {
    if (err) { return next(err); }
    // do whatever I need to do here...
  }
}

I've had upwards of 5 pieces of middleware in the past, which would lead to calllback hell without async.

Hey, I'm just throwing out ideas... :)
And, yes, you can do it in app.js non-globally... but I don't believe you can use that format with use, I think you have to do it per-operation...
app.get('/path', middleware);
app.post('/path', middleware);

And I appreciate it!
Ok, adding the middleware like you've just described would likely work.  I was thinking it would be nice to define it in the .yml, but that would proly be too express centric.

That's an interesting idea. I'll definitely give it some thought. Thanks!

Looks like krakenjs/swaggerize-express follows the array approach: https://github.com/krakenjs/swaggerize-express#handler-middleware

Can you take a look at the new beta and let me know what you think? https://github.com/swagger-api/swagger-node/issues/294. This will allow you to run middleware in your pipeline you want. If you still think we need another way of doing what you'd like, let's discuss.

Looks like object literals, is that correct?  If so, then I think it's relying on a behavior not guaranteed by the es standard (I.E. object literal properties should be returned in declaration order).
I've begun using this library with success and prefer it's approach a lot more: krakenjs/swaggerize-express.

Not at all. All pipes, and therefore the ordering of the modules, are defined as an array.

@theganyo please provide an example how to use these bagpipes for a specific path. Thank you!

I'm a little late here, but I ended up doing something along these lines:
    var util = require('util');

    module.exports = {
      hello: stackMiddleware[middleware1, middleware2, hello]
    };

    function hello(req, res) {
      var name = req.swagger.params.name.value;
      var hello = name ? util.format('Hello, %s', name) : 'Hello, stranger!';
      res.json(hello);
    }

function stackMiddleware(middlewares){

  return (req, res, next)=>{
    const stack = [];
    for (let i=middlewares.length; i--; i>0){
      const middleware = middlewares[i];
      if (i === 0){
        middleware(req, res, stack[stack.length - i - 1]);
      } else if (i === middlewares.length-1) {
        stack.push(
          ()=>{ middleware(req, res, next); }
        );
      } else {
        stack.push(
          ()=>{ middleware(req, res, stack[stack.length - i-1]); }
        );
      }
    }
  }
}


Student schedules

Since the semester will be starting soon I would like to get an idea of what everyone's availability schedule is.  Please set your likely work schedule up in the google calendar.  If you need to give any other information about your schedule let me know in slack or a comment here.  Once you've entered your schedule check the box next to your name here.

[x] @lmulhall-phet
[x] @JRomero0613
[x] @megan-phet
[x] @EthanWJohnson
[x] @kellywurtz

Thanks everyone


[Data Virtualization]: Basic offline capabilities

Design doc contained in this task AB#7243
AB#7244
AB#7245
Description
Build out happy path for data virtualization offline capabilities. When a datastore gets virtualized via loadingGroupId we still want offline to work as expected. When going offline need to download all the snapshots for datastores that were already downloaded via groupId. This PR figures out where it goes.
The next piece is to load the data from the snapshot. This PR also figures out how to parse that data and make sure that the offline ops can be applied as expected.
Testing

DataStore with loadingGroupId that is locally created can be modified offline
DataStore with loadingGroupId that is remotely created can be modified offline

Design
After we realize a datastore - we store its snapshot in baseSnapshot. When we go offline, we look at all our dataStoreContexts, check if they have a groupId, and download their baseSnapshots and store it in the runtime offline blob.
When we load a new container from that blob, we pass those snapshots to the datastores class and realize it all.

Closing as this isn't the right direction. We want to do it in the loader layer as that would consolidate all the code in one place


[Doc]: Sequence as a Backend for REST APIs

Purpose
Provide custom mediation logic for REST APIs which can act as a backend or provide configurations for connectors.
Approach

Provide a new Endpoint Type "Sequence Backend" for REST APIs under Endpoint Tab
Allow Sequences to be added for Sandbox and Production Key types separately,.
Perform download, delete, upload operations for Sequence Backends.
Add support for APICTL import and export scenarios for REST APIs
Introduced a new table to store Sequence Backend. Following is the schema for the table.

User stories

Provide custom mediation logic to integrate REST APIs with connectors and work as a backend for REST APIs.

Related PRs

https://github.com/wso2/apim-apps/pull/743
https://github.com/wso2/carbon-apimgt/pull/12524

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Mixed data example

May I know if there are any examples of causal-learn usage on mixed data (categorical and continuous/ordinal)? This is supported in tetrad. Many thanks!

Hi @sailinz-- as you probably know we have two methods in Java Tetrad for the mixed discrete/Gaussian case:
(1) Conditional Gaussian
(2) Degenerate Gaussian
Here are the papers for them, respectively:
Andrews, B., Ramsey, J., & Cooper, G. F. (2018). Scoring Bayesian networks of mixed variables. International journal of data science and analytics, 6(1), 3-18.
Andrews, B., Ramsey, J., & Cooper, G. F. (2019, July). Learning high-dimensional directed acyclic graphs with mixed data-types. In The 2019 ACM SIGKDD Workshop on Causal Discovery (pp. 4-21). PMLR.
Our colleague Bryan Andrews (first author on these papers) is a great Python programmer and has the degenerate Gaussian score (I think also test?) coded up in Python, which he is willing to contribute. If all goes well we'll have that in causal-learn in the near future. He doesn't have a Python version of conditional Gaussian, but perhaps we could code that up separately.
Anyway we'll start these conversations with the causal-learn programmers soon.

Hi @jdramsey and @bja43, after discussed with Kun, we think this is really needed and a great feature we should support. I am wondering whether @bja43 can contribute your code to our causal-learn library to support mixed data? :)

Hi, I sent Bryan's code to Yujia and Wei, but I neglected to cc you--maybe you can get it from them? They were going to adapt it to causal-learn. It's working code. Bryan is a bit overworked at the moment, as am I, unfortunately. :( It's the code for degenerate Gaussian.

Maybe you can send me an email :)

Related to: https://github.com/py-why/causal-learn/issues/139
and I would also mention this paper:
Raghu VK, Poon A, Benos PV. Evaluation of Causal Structure Learning Methods on Mixed Data Types. Proc Mach Learn Res. 2018 Aug;92:48-65. PMID: 31080946; PMCID: PMC6510516.
From: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6510516/