Incorrect sample for New-AzLogicApp and Set-AzLogicApp cmdlets

The documentation I am referencing: New-AzLogicApp and Set-AzLogicApp
The definition for  New-AzLogicApp, there is no a parameter called AppServicePlan, however, the examples given in the docs have that.
New-AzLogicApp
   -ResourceGroupName <String>
   -Name <String>
   -Location <String>
   [-State <String>]
   -Definition <Object>
   [-IntegrationAccountId <String>]
   [-Parameters <Object>]
   [-ParameterFilePath <String>]
   [-DefaultProfile <IAzureContextContainer>]
   [-WhatIf]
   [-Confirm]
   [<CommonParameters>]

New-AzLogicApp
   -ResourceGroupName <String>
   -Name <String>
   -Location <String>
   [-State <String>]
   -DefinitionFilePath <String>
   [-IntegrationAccountId <String>]
   [-Parameters <Object>]
   [-ParameterFilePath <String>]
   [-DefaultProfile <IAzureContextContainer>]
   [-WhatIf]
   [-Confirm]
   [<CommonParameters>]

regarding the definition below for Set-AzLogicApp, I don't know how to use the -AppServicePlan  parameter, can't find it nowhere in the Azure Logic Apps Documentation and also in the Azure portal of a logic app instance
Set-AzLogicApp
   -ResourceGroupName <String>
   -Name <String>
   [-AppServicePlan <String>]
   [-State <String>]
   [-Definition <Object>]
   [-DefinitionFilePath <String>]
   [-IntegrationAccountId <String>]
   [-Parameters <Object>]
   [-ParameterFilePath <String>]
   [-Force]
   [-DefaultProfile <IAzureContextContainer>]
   [-WhatIf]
   [-Confirm]
   [<CommonParameters>]

@ToMakeSense Thanks for the feedback! We are currently investigating and will update you shortly.

Hi @ToMakeSense Thanks for pointing this out and helping us drive improvements towards MS Docs! I have created PRs that remove the use of -AppServicePlan parameter (in all samples in both docs) and other references as well. We will now proceed to close this thread. If there are further questions regarding this matter, please reopen it and we will gladly continue the discussion.

@mike-urnun-msft  Thanks for your updates, you are so productive to moving forward 👍 💯

@mike-urnun-msft any updates on this, I found it's still not changed.

@ToMakeSense Thanks for following up. Looks like the PRs were closed without merging. I've reached out to our content team to clarify. I will update you here once I receive more info. Could you clarify how you've located and arrived at the concerning documentations?

@mike-urnun-msft  I found more issues regarding logic app cmdlets during my work, so I have a look at this issue, found these documents New-AzLogicApp and Set-AzLogicApp is still the same.

MicrosoftDocs/azure-docs-powershell/issues/1178
MicrosoftDocs/azure-docs-powershell/issues/1179
MicrosoftDocs/azure-docs-powershell/issues/1181

@mike-urnun-msft

Could you clarify how you've located and arrived at the concerning documentations?

I knew the AzureRM and Az powershell official documentation very well, I could google or use Get-Help New-AzLogicApp -Online in powershell to get it.
By the way, the documentation below are still the same.

New-AzLogicApp
Set-AzLogicApp


[FLINK-24667][runtime] Fix error handling in ChannelStateCheckpointWriter

What is the purpose of the change
Fix error handling in ChannelStateCheckpointWriter.
The change is split into 4 commits to ease reviewing;
before merging they should be squashed.
Verifying this change
Added UnalignedCheckpointFailureHandlingITCase.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? no

@Myasuka, @dawidwys, I've updated the PR to address your comments (or replied to them).
Do you have any further feedback?
(the build failed because of FLINK-24662 which is fixed in master).

@flinkbot run azure

@flinkbot run azure

Thanks for the review, I'll rebase the PR to pull the fix from master.
@Myasuka please let me know if you have any further concerns.

@rkhachatryan Sorry for late reply as I am on business trip for a two days' on-call.
I think this PR looks good in general. The only concern is that we could avoid to introduce the MockOperatorStateBackend if we have a job without operator state. DefaultOperatorStateBackend would not call closeAndGetHandle if no states registered (see here)


Improve GitHub Actions for pull requests

See https://github.com/Senzing/community-roadmap/issues/71

Pull requests:

https://github.com/Senzing/postgresql-client/pull/39


Point shader improvements

Ticks a few boxes in  #48

[x] Move point shader into wgsl file.
[x] PointMaterial.size_mode replaces vertes_sizes to be consistent with color_mode.
[x]  PointMaterial.size_space, similar to line.
[x] PointMaterial.aa to control anti-aliasing behavior.
[x] Good nan-behavior.
[x] Validation examples.

Markers -> new pr.

By creating the validation examples, I found a bug related to tiny points (smaller than a pixel), that also affected lines. Fixed it in both. 🚀


Add 5GCoreNetSDK project

The SDK is more like a tools. Should we move it to the tools part?

Sounds good to me, I'll do the modification today!

Updated ! Letting you check before squashing :+1:


[Feature] New suggestions and problems.

Experience Pylon is even good, but still lack possibility to drain exp from player for later storage.
Additionally XP Bottlers pack(block) crafted from 9 exp bottles would be a better solution for xp storage. :)

Interdiction plate work too weak imho. Its repel power is too weak so is't effective against speed monsters like a spiders or mounted monsters(like a zombie on zombie horse). It's doesn't repel zombie/skeleton horses too. (see "rough mob" mod).
Moreover -  repel in itself is weak effective if area isn't ideally flat or there is present a obstacle like wall or shallow holes. So this plate isn't suitable for protection NPC, etc.
I propose, make this plate to does kill instantly all monsters or teleport them away from range. Or simply stronger repel power plus Wither X(10) effect for all monsters in range.

As known a furnane minecart is bugged(loses all power after cross corner). Would can you managed to create a new, separate one whose don't have this bug, and eventually pushed minecarts with little more stronger and speed? Maybe some GUI with control panel and slot for coal?  ;)

A autocrafing table/machine, connected with hoppers?

In general to, thank you for creating and sill deleveping this mod, same as for conside our ideas. :)

Yes, a way to take exp in from the player would be perfect. i will add that.  (i had just been using it a mob farm to collect exp on the ground.)

of course, i should have thought about obstacles.  if you put one of these in a house with a villager. it should keep the villager safe regardless of if theres a wall in the way. Killing seems too strong, but adding maybe weakness and slowness would be good.  Teleport seems like a bit of a different style of thing, so for that i should add a different new item, like an upgraded version.  and with that i should reduce the cost of the original.  Another idea i had, if its possible i could try to see if the updated version could stop things spawning, so it wouldnt need to kill just stop the spawn event, and also TP those that wander close.

I was actually thinking about adding a faster normal minecart anyway like a gold one, so maybe a gold furnace minecart.  This wont be easy but i can look into it eventually.

I want to add autocrafting as well.  If you look here, this is where i add my own ideas/plans for future things https://github.com/PrinceOfAmber/Cyclic/milestones  .  Autocrafting is on there as an item listed https://github.com/PrinceOfAmber/Cyclic/milestones

Do you play normally in Minecraft 1.10 or 1.11?

Do you play normally in Minecraft 1.10 or 1.11?

MC 1.11.2.

Thanks for the link and the info

Latest release has a bunch of these features:   https://minecraft.curseforge.com/projects/cyclic/files/2411176
Interdiction plate upgrade is in latest release.  It seems to work well enough against the rough mob creepers and such.
Also the EXP Pylon has new features, including a deposit from player, and i made an autocrafter.
The minecart stuff i moved to my backlog. ill look at it later https://github.com/PrinceOfAmber/Cyclic/milestones

@GrzechuGrek  FYI ive started work on new minecarts. a plain one that is just faster than normal, and then a new furnace minecart that loses almost no momentum as long as it still has fuel, screenshot here https://twitter.com/lothrazar/status/862526147514347522


[FLINK-14375][runtime] Refactor task state updating to only notify scheduler about state changes that really happened

What is the purpose of the change
This PR is to avoid fake state update. More details see FLINK-14375.
Besides that, it also has a few other benefits:

we can get rid of the hack code in Execution#processFail
all states are possible to be notified to SchedulingStrategy, i.e. it solves FLINK-14233

Brief change log

Change DefaultScheduler to not handle the state update in SchedulerBase#updateTaskExecutionState
Change ExecutionGraph to notify all execution state changes to scheduler

Verifying this change
This change is already covered by existing tests in DefaultSchedulerTest and IT cases.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

This PR can be closed?

This PR can be closed?

Yes. Closing now.


Allow single-line templates

Using input instead of textarea
Like GitHub search bar, for use in similar contexts

This is not core right now so will be implemented later.


Error logs while cleanup

Describe the bug
Whenever I destroy the mercury instance, I am seeing a lot of error messages of type:
Error in /home/ngx/qa/scale-out-nvme/mercury/src/mchecksum/src/mchecksum.c:93 (mchecksum_destroy): Checksum not initialized.
Error in /home/ngx/qa/scale-out-nvme/mercury/src/mchecksum/src/mchecksum.c:93 (mchecksum_destroy): Checksum not initialized.
Error in /home/ngx/qa/scale-out-nvme/mercury/src/mchecksum/src/mchecksum.c:93 (mchecksum_destroy): Checksum not initialized.
Error in /home/ngx/qa/scale-out-nvme/mercury/src/mchecksum/src/mchecksum.c:93 (mchecksum_destroy): Checksum not initialized.

On the server side, there are a lot of messages, but on the client side, I see only 4-5 messages.
I have recently updated to version 2.2.0rc1 and have started seeing these logs.
I have tried build mercury with and without "MERCURY_USE_SYSTEM_MCHECKSUM" but it hasn't solved the problem.
To Reproduce
Steps to reproduce the behavior:

Initialize mercury instance.
Run RPC
Destroy mercury instance

Expected behavior
Error logs should not be seen
Screenshots
If applicable, add screenshots to help explain your problem.
Platform (please complete the following information):
Mercury version: v2.2.0rc1
Additional context
Add any other context about the problem here.

ok thanks for letting me know I will take a look. You can turn off mchecksum debug output by setting MCHECKSUM_ENABLE_DEBUG to OFF. Also we upgraded the checksum library, you might want to make sure you're using the latest one. It's a defect that we're not checking for the right mchecksum version then.

@soumagne I do not see the option mentioned by you when I run ccmake .. under build/
I only see the following options related to checksum
 MCHECKSUM_ENABLE_VERBOSE_ERROR   OFF
 MCHECKSUM_USE_ISAL               OFF
 MCHECKSUM_USE_SSE4_2             ON
MERCURY_USE_SYSTEM_MCHECKSUM     ON

I have tried switching off the first option but still see the log

right that's because you are using a system installed mchecksum, you'll have to either to either turn off MERCURY_USE_SYSTEM_MCHECKSUM or update the one you had previously installed.

I do not see the log now by switching off MERCURY_USE_SYSTEM_MCHECKSUM.
What is the repercussion of this in the prod environment?

I'm not sure I understand your question. This option only controls whether or not you use the mchecksum library that comes within mercury as a submodule or the one that you installed as an external dependency. It depends on your environment but I don't think there's any best recommendation. You can also completely remove checksums and use of that library by turning off MERCURY_USE_CHECKSUMS.


Tooltip not working on IOS(Ipad Pro)

Hello,
I am using :
bootstrap": "4.4.1",
It seems that the tooltip is not being triggered on Ipad Pro
Any idea how to go around this please?
Thank you.

Bug reports must include a live demo of the issue. Per our contributing guidelines, please create a reduced test case via CodePen or JS Bin and report back with your link, Bootstrap version, and specific browser and Operating System details.
This is an automated reply

Closing due to lack of response. You can comment here and we can re-open the PR, if you decide to resume working on the requested changes.
This is an automated reply


*539.exe always errors out

Wget working correctly, as per a suggestion in a similar issue. (Win11 for workstations, 32GB ram, RX6800XT)
Command prompt output below:

C:\SD>shark_sd_20230216_539.exe
shark_tank local cache is located at C:\Users\consolation.local/shark_tank/ . You may change this by setting the --local_tank_cache= flag
vulkan devices are available.
cuda devices are not available.
Running on local URL:  http://<IP_ADDRESS>:8080
To create a public link, set share=True in launch().
Found device AMD Radeon RX 6800 XT. Using target triple rdna2-unknown-windows.
Using tuned models for Linaqruf/anything-v3.0/fp16/vulkan://00000000-2d00-0000-0000-000000000000.
Downloading (…)cheduler_config.json: 100%|█████████████████████| 341/341 [00:00<00:00, 341kB/s]
huggingface_hub\file_download.py:129: UserWarning: huggingface_hub cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\Users\consolation.cache\huggingface\diffusers. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the HF_HUB_DISABLE_SYMLINKS_WARNING environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.
To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to see activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development
torch\jit_check.py:172: UserWarning: The TorchScript type system doesn't support instance-level annotations on empty non-base types in __init__. Instead, either 1) use a type annotation in the class body, or 2) wrap the type in torch.jit.Attribute.
warnings.warn("The TorchScript type system doesn't support "
No vmfb found. Compiling and saving to C:\SD\euler_scale_model_input_1_512_512fp16.vmfb
Using target triple -iree-vulkan-target-triple=rdna2-unknown-windows from command line args
Saved vmfb in C:\SD\euler_scale_model_input_1_512_512fp16.vmfb.
WARNING: [Loader Message] Code 0 : Layer name GalaxyOverlayVkLayer does not conform to naming standard (Policy #LLP_LAYER_3)
WARNING: [Loader Message] Code 0 : Layer name GalaxyOverlayVkLayer_VERBOSE does not conform to naming standard (Policy #LLP_LAYER_3)
WARNING: [Loader Message] Code 0 : Layer name GalaxyOverlayVkLayer_DEBUG does not conform to naming standard (Policy #LLP_LAYER_3)
WARNING: [Loader Message] Code 0 : windows_read_data_files_in_registry: Registry lookup failed to get layer manifest files.
No vmfb found. Compiling and saving to C:\SD\euler_step_1_512_512fp16.vmfb
Using target triple -iree-vulkan-target-triple=rdna2-unknown-windows from command line args
Saved vmfb in C:\SD\euler_step_1_512_512fp16.vmfb.
WARNING: [Loader Message] Code 0 : Layer name GalaxyOverlayVkLayer does not conform to naming standard (Policy #LLP_LAYER_3)
WARNING: [Loader Message] Code 0 : Layer name GalaxyOverlayVkLayer_VERBOSE does not conform to naming standard (Policy #LLP_LAYER_3)
WARNING: [Loader Message] Code 0 : Layer name GalaxyOverlayVkLayer_DEBUG does not conform to naming standard (Policy #LLP_LAYER_3)
WARNING: [Loader Message] Code 0 : windows_read_data_files_in_registry: Registry lookup failed to get layer manifest files.
Inferring base model configuration.
Cannot initialize model with low cpu memory usage because accelerate was not found in the environment. Defaulting to low_cpu_mem_usage=False. It is strongly recommended to install accelerate for faster and less memory-intense model loading. You can do so with:
pip install accelerate

.
Downloading (…)_pytorch_model.bin";: 100%|████████████████| 3.44G/3.44G [01:26<00:00, 39.9MB/s]
Downloading (…)ain/unet/config.json: 100%|█████████████████████| 901/901 [00:00<00:00, 901kB/s]
Retrying with a different base model configuration
Cannot initialize model with low cpu memory usage because accelerate was not found in the environment. Defaulting to low_cpu_mem_usage=False. It is strongly recommended to install accelerate for faster and less memory-intense model loading. You can do so with:
pip install accelerate

.
torch\fx\node.py:250: UserWarning: Trying to prepend a node to itself. This behavior has no effect on the graph.
warnings.warn("Trying to prepend a node to itself. This behavior has no effect on the graph.")
Loading Winograd config file from  C:\Users\consolation.local/shark_tank/configs/unet_winograd_vulkan.json
Retrying with a different base model configuration
Cannot initialize model with low cpu memory usage because accelerate was not found in the environment. Defaulting to low_cpu_mem_usage=False. It is strongly recommended to install accelerate for faster and less memory-intense model loading. You can do so with:
pip install accelerate

.
Retrying with a different base model configuration
Cannot initialize model with low cpu memory usage because accelerate was not found in the environment. Defaulting to low_cpu_mem_usage=False. It is strongly recommended to install accelerate for faster and less memory-intense model loading. You can do so with:
pip install accelerate

.
Retrying with a different base model configuration
Traceback (most recent call last):
File "gradio\routes.py", line 374, in run_predict
File "gradio\blocks.py", line 1017, in process_api
File "gradio\blocks.py", line 835, in call_function
File "anyio\to_thread.py", line 31, in run_sync
File "anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread
File "anyio_backends_asyncio.py", line 867, in run
File "apps\stable_diffusion\scripts\txt2img.py", line 116, in txt2img_inf
File "apps\stable_diffusion\src\pipelines\pipeline_shark_stable_diffusion_utils.py", line 220, in from_pretrained
File "apps\stable_diffusion\src\models\model_wrappers.py", line 383, in call
SystemExit: Cannot compile the model. Please create an issue with the detailed log at https://github.com/nod-ai/SHARK/issues
Keyboard interruption in main thread... closing server.

Happy to try any suggestions,
TIA

Can you clear all the generated files in the folder? Or simply add flag --clear_all

It has no effect, other than run the initial downloads again.

Can you try with --no-use_tuned flag?

Can you try with --no-use_tuned flag?

Same error, I think - looking at the board, there seems to be a bunch of issue threads that are the same problem?

C:\SD>shark_sd_20230216_539.exe --no-use_tuned
shark_tank local cache is located at C:\Users\consolation.local/shark_tank/ . You may change this by setting the --local_tank_cache= flag
vulkan devices are available.
cuda devices are not available.
Running on local URL:  http://<IP_ADDRESS>:8080
To create a public link, set share=True in launch().
Found device AMD Radeon RX 6800 XT. Using target triple rdna2-unknown-windows.
Using tuned models for Linaqruf/anything-v3.0/fp16/vulkan://00000000-2d00-0000-0000-000000000000.
torch\jit_check.py:172: UserWarning: The TorchScript type system doesn't support instance-level annotations on empty non-base types in __init__. Instead, either 1) use a type annotation in the class body, or 2) wrap the type in torch.jit.Attribute.
warnings.warn("The TorchScript type system doesn't support "
loading existing vmfb from: C:\SD\euler_scale_model_input_1_512_512fp16.vmfb
WARNING: [Loader Message] Code 0 : Layer name GalaxyOverlayVkLayer does not conform to naming standard (Policy #LLP_LAYER_3)
WARNING: [Loader Message] Code 0 : Layer name GalaxyOverlayVkLayer_VERBOSE does not conform to naming standard (Policy #LLP_LAYER_3)
WARNING: [Loader Message] Code 0 : Layer name GalaxyOverlayVkLayer_DEBUG does not conform to naming standard (Policy #LLP_LAYER_3)
WARNING: [Loader Message] Code 0 : windows_read_data_files_in_registry: Registry lookup failed to get layer manifest files.
loading existing vmfb from: C:\SD\euler_step_1_512_512fp16.vmfb
WARNING: [Loader Message] Code 0 : Layer name GalaxyOverlayVkLayer does not conform to naming standard (Policy #LLP_LAYER_3)
WARNING: [Loader Message] Code 0 : Layer name GalaxyOverlayVkLayer_VERBOSE does not conform to naming standard (Policy #LLP_LAYER_3)
WARNING: [Loader Message] Code 0 : Layer name GalaxyOverlayVkLayer_DEBUG does not conform to naming standard (Policy #LLP_LAYER_3)
WARNING: [Loader Message] Code 0 : windows_read_data_files_in_registry: Registry lookup failed to get layer manifest files.
Inferring base model configuration.
Cannot initialize model with low cpu memory usage because accelerate was not found in the environment. Defaulting to low_cpu_mem_usage=False. It is strongly recommended to install accelerate for faster and less memory-intense model loading. You can do so with:
pip install accelerate

.
Retrying with a different base model configuration
Cannot initialize model with low cpu memory usage because accelerate was not found in the environment. Defaulting to low_cpu_mem_usage=False. It is strongly recommended to install accelerate for faster and less memory-intense model loading. You can do so with:
pip install accelerate

.
torch\fx\node.py:250: UserWarning: Trying to prepend a node to itself. This behavior has no effect on the graph.
warnings.warn("Trying to prepend a node to itself. This behavior has no effect on the graph.")
Loading Winograd config file from  C:\Users\consolation.local/shark_tank/configs/unet_winograd_vulkan.json
Retrying with a different base model configuration
Cannot initialize model with low cpu memory usage because accelerate was not found in the environment. Defaulting to low_cpu_mem_usage=False. It is strongly recommended to install accelerate for faster and less memory-intense model loading. You can do so with:
pip install accelerate

.
Retrying with a different base model configuration
Cannot initialize model with low cpu memory usage because accelerate was not found in the environment. Defaulting to low_cpu_mem_usage=False. It is strongly recommended to install accelerate for faster and less memory-intense model loading. You can do so with:
pip install accelerate

.
Retrying with a different base model configuration
Traceback (most recent call last):
File "gradio\routes.py", line 374, in run_predict
File "gradio\blocks.py", line 1017, in process_api
File "gradio\blocks.py", line 835, in call_function
File "anyio\to_thread.py", line 31, in run_sync
File "anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread
File "anyio_backends_asyncio.py", line 867, in run
File "apps\stable_diffusion\scripts\txt2img.py", line 116, in txt2img_inf
File "apps\stable_diffusion\src\pipelines\pipeline_shark_stable_diffusion_utils.py", line 220, in from_pretrained
File "apps\stable_diffusion\src\models\model_wrappers.py", line 383, in call
SystemExit: Cannot compile the model. Please create an issue with the detailed log at https://github.com/nod-ai/SHARK/issues

Attached output with both flags running
clearallnotune.txt

Have you ever had any of the previous releases working? Might be a driver issue @powderluv?

I'm using the currently linked driver, which is the new AMD one. I had it working previously on Debian, but the latest git sync broke it - although there's a chance Debian Testing hosed it. Hence the attempted switch to a windows version, while I try to unravel that mess...
I tried using a the 1.4 SD model from HugginFace in custom models directory: got a somewhat different output:

C:\SD>shark_sd_20230216_539.exe --no-use_tuned --clear_all
shark_tank local cache is located at C:\Users\consolation.local/shark_tank/ . You may change this by setting the --local_tank_cache= flag
CLEARING ALL, EXPECT SEVERAL MINUTES TO RECOMPILE
vulkan devices are available.
cuda devices are not available.
Running on local URL:  http://<IP_ADDRESS>:8080
To create a public link, set share=True in launch().
Found device AMD Radeon RX 6800 XT. Using target triple rdna2-unknown-windows.
Tuned models are currently not supported for this setting.
torch\jit_check.py:172: UserWarning: The TorchScript type system doesn't support instance-level annotations on empty non-base types in __init__. Instead, either 1) use a type annotation in the class body, or 2) wrap the type in torch.jit.Attribute.
warnings.warn("The TorchScript type system doesn't support "
No vmfb found. Compiling and saving to C:\SD\euler_scale_model_input_1_512_512fp16.vmfb
Using target triple -iree-vulkan-target-triple=rdna2-unknown-windows from command line args
Saved vmfb in C:\SD\euler_scale_model_input_1_512_512fp16.vmfb.
WARNING: [Loader Message] Code 0 : Layer name GalaxyOverlayVkLayer does not conform to naming standard (Policy #LLP_LAYER_3)
WARNING: [Loader Message] Code 0 : Layer name GalaxyOverlayVkLayer_VERBOSE does not conform to naming standard (Policy #LLP_LAYER_3)
WARNING: [Loader Message] Code 0 : Layer name GalaxyOverlayVkLayer_DEBUG does not conform to naming standard (Policy #LLP_LAYER_3)
WARNING: [Loader Message] Code 0 : windows_read_data_files_in_registry: Registry lookup failed to get layer manifest files.
No vmfb found. Compiling and saving to C:\SD\euler_step_1_512_512fp16.vmfb
Using target triple -iree-vulkan-target-triple=rdna2-unknown-windows from command line args
Saved vmfb in C:\SD\euler_step_1_512_512fp16.vmfb.
WARNING: [Loader Message] Code 0 : Layer name GalaxyOverlayVkLayer does not conform to naming standard (Policy #LLP_LAYER_3)
WARNING: [Loader Message] Code 0 : Layer name GalaxyOverlayVkLayer_VERBOSE does not conform to naming standard (Policy #LLP_LAYER_3)
WARNING: [Loader Message] Code 0 : Layer name GalaxyOverlayVkLayer_DEBUG does not conform to naming standard (Policy #LLP_LAYER_3)
WARNING: [Loader Message] Code 0 : windows_read_data_files_in_registry: Registry lookup failed to get layer manifest files.
Diffusers' checkpoint will be identified here :  C:/SD/models/model
Loading diffusers' pipeline from original stable diffusion checkpoint
Traceback (most recent call last):
File "gradio\routes.py", line 374, in run_predict
File "gradio\blocks.py", line 1017, in process_api
File "gradio\blocks.py", line 835, in call_function
File "anyio\to_thread.py", line 31, in run_sync
File "anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread
File "anyio_backends_asyncio.py", line 867, in run
File "apps\stable_diffusion\scripts\txt2img.py", line 116, in txt2img_inf
File "apps\stable_diffusion\src\pipelines\pipeline_shark_stable_diffusion_utils.py", line 220, in from_pretrained
File "apps\stable_diffusion\src\models\model_wrappers.py", line 340, in call
File "apps\stable_diffusion\src\utils\utils.py", line 427, in preprocessCKPT
File "diffusers\pipelines\stable_diffusion\convert_from_ckpt.py", line 976, in load_pipeline_from_original_stable_diffusion_ckpt
File "torch\serialization.py", line 810, in load
return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "torch\serialization.py", line 1173, in _load
result = unpickler.load()
^^^^^^^^^^^^^^^^
File "torch\serialization.py", line 1166, in find_class
return super().find_class(mod_name, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'pytorch_lightning'
Keyboard interruption in main thread... closing serv

The AMD driver works with A1111's sd build fwiw

Did you try to select model from the drop-down box? Can you try if "stabilityai/stable-diffusion-2-1-base" works?

Did you try to select model from the drop-down box? Can you try if "stabilityai/stable-diffusion-2-1-base" works?

Yeah, I tried all of them, same error.

Could it be the venv environment, in the exe, conflicting with the installed Python? Or, am I way off base here?

Hmm, can you check if you have this folder created C:\Users\consolation.local/shark_tank/ and if there's anything saved in the folder? Another way, are you able to checkout out our github repo and try the latest with command lines, there are instructions in README.

Could it be the venv environment, in the exe, conflicting with the installed Python? Or, am I way off base here?

Also are you using python 3.11?

Did you meant C:\Users\consolation.local\shark_tank\  or consolation.local ? Either way, no, neither has been created.
I initially tried it with no Python installed, as I don't really use it in windows and it is not listed as a prereq' for the exe - I figured it's contained in the exe. When I hit this error I did try installing latest 3.11, just in case - it didn't help. At that stage I tried A1111's build, but that required a 10.3.6 version, so I removed .11 and switched to .6
I can spin up a clean windows install and try with 3.11 via github install, but that will have to wait till this evening.
The main reason I'm trying to get shark running, via exe, is that I need to set it up for a couple of "not super tech savvy" users - and the self contained nature is very appealing.

So maybe it's the local shark_tank folder not appropriately created? Can you try to set it to another directory with --local_tank_cache= flag?

You genius - you did it! :-)

Here is the fist console from the first run. It took a hot minute as expected.
setcache1strun.txt

Awesome! Thanks for your interest and patience!

All models and custom ones work correctly btw. Should I mark this as closed? I'm guessing that a bunch of the threads can be fixed with this.
Also, glad I could help - if you ever need a crash test dummy for builds, let me know.


Fixed typo in the link to the style guide

The style guide link was uppercase which caused a 404 error.

Thanks!


Observer PositionObserver

Observer of object's position change.
class PositionObserver(ABC):
    @abstractmethod
    def on_position_change(self, origin: Vector2d, destination: Vector2d) -> None:

Finished.


MenuAnchor animations broken after 3.19

Steps to reproduce
Making a simple animation for a menu made with MenuAnchor using widgets like AnimatedOpacity() no longer works after 3.19. Downgrading back to 3.16.9 fixes it. Not sure if this is the best way to animate a menu in the first place, but it had been working well for me. I think maybe it has to do with this new change #130534, though I really have no idea. That's just the only thing I noticed explicitly changed about MenuAnchor.

Make a MenuAnchor menu that pops up a menu on clicking a button
wrap the menuChildren in an animated widget like AnimatedOpacity
Make this widget animate a change like opacity based on whether the menu is open/visible or not.
The animations do not work, regardless of duration, curve, or type of animation (AnimatedContainer, opacity, crossfade, etc are all broken)

Expected results
The menu that pops up should be animated. In this case, opacity should fade in.
Actual results
There is no animation.
Code sample
import 'package:flutter/material.dart';

void main() {
  runApp(const AnimatedMenuAnchor());
}

class AnimatedMenuAnchor extends StatefulWidget {
  const AnimatedMenuAnchor({super.key});

  @override
  State<AnimatedMenuAnchor> createState() => _AnimatedMenuAnchorState();
}

class _AnimatedMenuAnchorState extends State<AnimatedMenuAnchor> {
  bool _visible = false;

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: Scaffold(
        body: Center(
          child: MenuAnchor(
            onClose: () => setState(() {
              _visible = false;
            }),
            onOpen: () => setState(() {
              _visible = true;
            }),
            builder: (BuildContext context, MenuController controller,
                Widget? child) {
              return TextButton(
                onPressed: () {
                  if (controller.isOpen) {
                    controller.close();
                  } else {
                    controller.open();
                  }
                },
                child: const Text('OPEN MENU'),
              );
            },
            menuChildren: <Widget>[
              AnimatedOpacity(
                opacity: _visible ? 1.0 : 0.0,
                duration: const Duration(milliseconds: 500),
                child: Column(
                  children: [
                    MenuItemButton(
                      child: const Text('Option 1'),
                      onPressed: () {},
                    ),
                    MenuItemButton(
                      child: const Text('Option 2'),
                      onPressed: () {},
                    ),
                    MenuItemButton(
                      child: const Text('Option 3'),
                      onPressed: () {},
                    ),
                  ],
                ),
              ),
            ],
          ),
        ),
      ),
    );
  }
}

Screenshots or Video

Screenshots / Video demonstration
This is how it looked in 3.16.9: In 3.19 there is no animation.
https://github.com/flutter/flutter/assets/35315935/eee04aa9-ae03-4482-a1df-ad54d066c04e
To see that it no longer works, you can copy the code into dartpad.

Logs
Logs
[Paste your logs here]

Flutter Doctor output
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.19.0, on Ubuntu 23.10 6.5.0-17-generic, locale en_US.UTF-8)
[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
[✓] Chrome - develop for the web
[✓] Linux toolchain - develop for Linux desktop
[✓] Android Studio (version 2023.1)
[✓] VS Code (version 1.86.1)
[✓] Connected device (2 available)
[✓] Network resources

• No issues found!

Thanks for the report. I also can see the issue. Bisected this to PR https://github.com/flutter/flutter/pull/130534 (Convert menus to use OverlayPortal) as mentioned.

flutter doctor -v (stable and master)
[✓] Flutter (Channel stable, 3.19.0, on macOS 14.1 23B74 darwin-x64, locale en-VN)
    • Flutter version 3.19.0 on channel stable at /Users/huynq/Documents/GitHub/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision bae5e49bc2 (2 days ago), 2024-02-13 17:46:18 -0800
    • Engine revision 04817c99c9
    • Dart version 3.3.0
    • DevTools version 2.31.1

[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at /Users/huynq/Library/Android/sdk
    • Platform android-34, build-tools 34.0.0
    • ANDROID_HOME = /Users/huynq/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 15.0.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 15A507
    • CocoaPods version 1.14.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2023.1)
    • Android Studio at /Applications/Android Studio Hedgehog.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.7+0-17.0.7b1000.6-10550314)

[✓] Android Studio (version 2022.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • android-studio-dir = /Applications/Android Studio.app/
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)

[✓] VS Code (version 1.86.0)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.82.0

[✓] Connected device (3 available)
    • RMX2001 (mobile) • EUYTFEUSQSRGDA6D • android-arm64  • Android 11 (API 30)
    • macOS (desktop)  • macos            • darwin-x64     • macOS 14.1 23B74 darwin-x64
    • Chrome (web)     • chrome           • web-javascript • Google Chrome 121.0.6167.160
    ! Error: Browsing on the local area network for iPad. Ensure the device is unlocked and attached with a cable or associated with the same local area network as this Mac.
      The device must be opted into Developer Mode to connect wirelessly. (code -27)

[✓] Network resources
    • All expected network resources are available.

• No issues found!

[!] Flutter (Channel master, 3.20.0-10.0.pre.79, on macOS 14.1 23B74 darwin-x64, locale en-VN)
    • Flutter version 3.20.0-10.0.pre.79 on channel master at /Users/huynq/Documents/GitHub/flutter_master
    ! Warning: `flutter` on your path resolves to /Users/huynq/Documents/GitHub/flutter/bin/flutter, which is not inside your current Flutter SDK checkout at /Users/huynq/Documents/GitHub/flutter_master. Consider adding /Users/huynq/Documents/GitHub/flutter_master/bin to the front of your path.
    ! Warning: `dart` on your path resolves to /Users/huynq/Documents/GitHub/flutter/bin/dart, which is not inside your current Flutter SDK checkout at /Users/huynq/Documents/GitHub/flutter_master. Consider adding /Users/huynq/Documents/GitHub/flutter_master/bin to the front of your path.
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 35bdc94dda (43 minutes ago), 2024-02-20 21:29:23 -0500
    • Engine revision 3557277c57
    • Dart version 3.4.0 (build 3.4.0-157.0.dev)
    • DevTools version 2.33.0-dev.6
    • If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly to perform update checks and upgrades.

[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at /Users/huynq/Library/Android/sdk
    • Platform android-34, build-tools 34.0.0
    • ANDROID_HOME = /Users/huynq/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 15.0.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 15A507
    • CocoaPods version 1.14.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2023.1)
    • Android Studio at /Applications/Android Studio Hedgehog.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.7+0-17.0.7b1000.6-10550314)

[✓] Android Studio (version 2022.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • android-studio-dir = /Applications/Android Studio.app/
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)

[✓] VS Code (version 1.86.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.82.0

[✓] Connected device (2 available)
    • macOS (desktop) • macos  • darwin-x64     • macOS 14.1 23B74 darwin-x64
    • Chrome (web)    • chrome • web-javascript • Google Chrome 121.0.6167.184

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 1 category.

cc: @gspencergoog

@beefchalupa Not an expert and/or a team member, but I believe this occurs because onOpen is called before the widget is first built, and layout is deferred under-the-hood:
_OverlayPortal(
    overlayLocation: _getLocation(zOrderIndex, widget._targetRootOverlay),
    overlayChild: _DeferredLayout(child: Builder(builder: widget.overlayChildBuilder)),
    child: widget.child,
);

https://github.com/flutter/flutter/assets/59215665/340e5ad9-1407-4afd-b685-1594df21fb8b
A not-so-optimal solution would be to call setState((){}) in a post frame callback so that the widget gets built twice: once with _visible set to false, and again with _visible set to true (check the video for an example). But building two times in a row is rarely necessary.
A technique that may be more performant (or, at a minimum, easier to control) would be to use an AnimationController. This would also give you the ability to animate the menu closed.
import 'package:flutter/material.dart';

void main() {
  runApp(const AnimatedMenuAnchor());
}

class AnimatedMenuAnchor extends StatefulWidget {
  const AnimatedMenuAnchor({super.key});

  @override
  State<AnimatedMenuAnchor> createState() => _AnimatedMenuAnchorState();
}

class _AnimatedMenuAnchorState extends State<AnimatedMenuAnchor>
    with SingleTickerProviderStateMixin {
  late final AnimationController _animationController;
  final MenuController _menuController = MenuController();

  @override
  void initState() {
    super.initState();
    _animationController = AnimationController(
      vsync: this,
      duration: const Duration(milliseconds: 200),
    )..addStatusListener((AnimationStatus status) {
        if (status == AnimationStatus.dismissed) {
           // The moment the menu is closed, it will no longer be on the screen or animatable.
           // To allow for a closing animation, we wait until our closing animation is finished before we close the menu anchor.
          _menuController.close();
        } else if (!_menuController.isOpen) {
         // The menu should be open while the animation status is forward, completed, or reverse
          _menuController.open();
        }
      });
  }

  @override
  void dispose() {
    _animationController.dispose();
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: Scaffold(
        body: Center(
          child: MenuAnchor(
            controller: _menuController,
            onOpen: _animationController.forward,
            builder: (
              BuildContext context,
              MenuController controller,
              Widget? child,
            ) {
              return TextButton(
                onPressed: () {
                  if (_animationController.status
                      case AnimationStatus.forward ||
                          AnimationStatus.completed) {
                    _animationController.reverse();
                  } else {
                    _animationController.forward();
                  }
                },
                child: const Text('OPEN MENU'),
              );
            },
            menuChildren: <Widget>[
               // In the example video, I'm wrapping the FadeTransition with a SizeTransition.
              FadeTransition(
                  opacity: _animationController,
                  child: Column(
                    children: <Widget>[
                      MenuItemButton(
                        child: const Text('Option 1'),
                        onPressed: () {},
                      ),
                      MenuItemButton(
                        child: const Text('Option 2'),
                        onPressed: () {},
                      ),
                      MenuItemButton(
                        child: const Text('Option 3'),
                        onPressed: () {},
                      ),
                    ],
                  ),
              )
            ],
          ),
        ),
      ),
    );
  }
}

https://github.com/flutter/flutter/assets/59215665/42b91798-42fa-4723-9d82-2aaf1d506096
All of that said, none of what addresses a fix to the MenuAnchor (if one is needed). Also, while I usually wouldn't use a Column as a MenuAnchor child, I also don't know of a better solution for animating menu children (I'm not sure if the overhead of wrapping each child individually is any better).

Will continue to work on PR #143416 ASAP.


[Design Proposal] Add Remote Storage Options for Improved Durability

This design proposal is WIP. We will continue to add details to Sections that are marked with ToDo.

Goal
This doc proposes a high level and low level design to provide durability guarantees for the indexed data using remote store. For more info on overview and challenges, please refer: Feature Proposal
Requirements
Functional

Store indexed data to the configured remote store
Provide a mechanism to restore data from the configured remote store
Provide durability guarantees based on configured remote store

Non-Functional

Correctness - data should not be missing or duplicated.
Performance impact should be minimal. Different knobs should be provided to user to tune the performance with remote store.
Scale - Enabling remote store should not impact the scale of the existing OpenSearch cluster.
Security - Data stored in the remote store should adhere to the security model of OpenSearch.
Integration with existing features - Even though it is not in the scope of durability, remote store can be extended to work with existing OpenSearch constructs like replication, peer recovery, snapshot etc.

Out of Scope
This design doc focuses on durability aspect of using the remote store. Even though the design will be extensible to integrate remote store with other constructs, those details will not be discussed here.
Approach
Hot data storage in OpenSearch is divided into two parts:

Translog
Segments

Successful indexing operation writes data to Translog. A periodic job consumes translog and creates segment files. Translog is purged once Lucene commit is triggered. In order to provide durability guarantees of the indexed data, we need to provide remote store support for translog as well as segments. In the following sections, we use remote translog and remote segment store to store translog and segments respectively.
Invariant
At any point in time, remote translog and remote segment store together contain all the indexed data.
Storage

Each index operation will write to remote translog. This will be a sync operation in the indexing request path.

Will add the async mode with buffer for lower latency and tolerance to RPO worth of data

Each refresh/commit will upload new segments to remote segment store.

Segments will only be uploaded from primary node of the shard.
Segment files are identified based on checksum and the diff is incrementally uploaded.

Remote store will keep metadata of all the segment files with checksum

Segment data can be duplicated with segment merge, failover etc. but this will not lead to data duplication similar to how it is handled in the local disk.
A periodic job will delete segments from remote store that are not part of the live data (merged away segments, segments from old primary).
We can consider segment age before deleting it. This will enable point-in-time restore in case of accidental deletes.

Remote translog will be purged based on the sequence number of latest segment in the remote segment store
In some scenario (will be discussed below), remote translog can grow continuously. We will apply write block if the remote translog size breaches X GB threshold.

Restore

Manual, API based restore

It will replace all the segments on local with the segments in remote
Replace translog with remote translog and replay the translog
Optional timestamp parameter for point-in-time restore

Index level restore

As translog and segment files will be stored at shard level in the remote store, index level restore can also be supported.

Once we setup restore flow, the next step would be to add support for automated restore. If index state turns red and no replicas are configured, the system should be able to pull data from remote store to restore the red index.

High Level Design
Architecture Diagram

Remote Translog
Data Model
ToDo
Requirements
ToDo
Remote Segment Store
Data Model

Remote segment store will contain segment files which are immutable. So, segment store can be seen as store of immutable blobs.
Operations performed on these blobs (segment files): upload, download, delete
In OpenSearch, Segments are created on a shard level. Two segment files of two different shards can have the same name. This requires to maintain index_UUID/shard_id hierarchy (the exact hierarchy may have more elements) in the remote store while storing the segment files.

Requirements

Support for uploading large files: As Lucene segment files can grow big (forcemerge can create segment with size > 5GB), the segment store should be capable of storing large objects with low latency
Data Integrity: When dealing with upload and download of files, it becomes important to check the correctness using checksum and remote store should provide the support for the same
Scale: As OpenSearch cluster can support petabyte of data, remote segment store is expected to scale
Availability and Durability: It is important to choose highly available and durable remote segment store as it will directly impact the durability guarantees.

API Design
ToDo: Add verb, payload, success and error response against each API

Enable/disable remote storage
Configure remote storage
Backfill existing data to remote store
Restore data from remote storage for a given index and shard
Get status of the restore operation
Get status of remote store sync

Low Level Design
Remote Translog
ToDo
Remote Segment Store
Class Diagram

The interface for remote segment storage (RemoteDirectory) will be extended from Lucene’s Directory.  Using APIs provided by Directory will ensure that remote segment store is able to provide the same functionality that local disk does. Using the same APIs for local disk and remote segment store will also make it extensible where local disk can be replaced by the remote segment store (this is not part of durability but should avoid re-architecture of the APIs/system later)

Note: RemoteDirectory may not support all the APIs of Directory from the first release. Plan is to open up more and more Directory APIs based on the milestone targeted for the upcoming release.

Today, with Snapshots, OpenSearch already supports writing segment files to remote blob store. It also supports reading these segment files while restoring data from segments. BlobContainer interface is defined to read and write segments (blobs) from a blob store (HDFS, Azure Blob Storage, S3).

RemoteDirectory will have referece to BlobContainer and will delegate Directory API implementation to BlobContainer with the required pre and post processing. This will avoid duplicating the same code to read/write segment files from/to the remote store.

Sequence Diagram

Following sequence diagrams explain the flow and components involved. Actual code flow will take care of optimizations (if any).

Segment Upload

Segment Restore
ToDo
Integration with existing constructs/features
Failover
ToDo: Evaluating different options here
Peer Recovery
ToDo
Replication Strategy
Document Based Replication
ToDo
Segment Based Replication
ToDo
Cross Cluster Replication
ToDo
Snapshot
ToDo
Metrics/Stats
ToDo
Open Questions

Would it be a plugin based implementation?
What would be the durability guarantees?
Will the existing data be backfilled or durability starts from the point of enabling durability for the cluster?

Can the existing snapshot data be used for backfilling?

Will it be built on the top of existing snapshot implementation?
What metrics/stats would be required around durability?

Started adding low level design details for Remote Segment Store.

I'll offer some opinions on a few open questions :)

Would it be a plugin based implementation?

As long as it does not introduce new dependencies, then I think it should probably not be a plugin. The concrete implementation for interfacing with a remote store should be a plugin (like the current repository plugins), but the core logic of this feature can probably live directly inside the core and need not be a separate plugin.

What would be the durability guarantees?

I think the key guarantee needs to be that all acknowledged writes are persisted to the remote store. The specific durability guarantee will differ depending on the remote store, but I think OpenSearch's guarantee should be that the document is successfully persisted to the remote store before the write is acknowledged as successful to the user.

Will the existing data be backfilled or durability starts from the point of enabling durability for the cluster?

Is this new "durability" feature a property of the cluster or of an index? Assuming it is a property of an index, then I would assume you can enable for a new index at index creation time. Given that, a user should be able to use the reindex mechanism to effectively enable remote durability for an existing index by reindexing it to a new index with durability enabled. There may be future optimizations to make this process more lightweight, but I'd strongly consider leaving that out of scope for the initial version. (If durability is intended to be enabled at a cluster level and not per-index, then my comment doesn't really make sense)

Can the existing snapshot data be used for backfilling?

As stated above, I'd try to leave backfilling out of scope for the initial version if the reindex mechanism can work.

Will it be built on the top of existing snapshot implementation?

What exactly do you mean by "snapshot implementation"? I'd hope the existing repository interface can be used/extended to meet the needs of copying segment files to/from the remote store. Not sure about the translog as it will likely have a very different access pattern.

Nice proposal, few comments/questions below.

How does it relate to “searchable remote index” https://github.com/opensearch-project/OpenSearch/issues/2900?
What are the Performance knobs that we are planning to offer?
Doesn’t this assume remote store itself is durable? Might want to clarify that.
Any overlap with segment replication https://github.com/opensearch-project/OpenSearch/issues/1694? Isn’t segment replication a pre-requisite?
Is RemoteDirectory in addition to LocalDirectory or a replacement?
As you are re-using the repository abstraction, does it unecessarily force remote store implementer to implement snapshot related interfaces also (and vice-versa)?
Do we really need separate apis for restoring from remote store. Shouldn’t this automatically happen in case of primary loss similar to how peer recovers from primary in today’s world. Any shard irrespective of primary or replica should recover from remote storage automatically. If that is a requirement, is there a use case for explicit api to restore? I can think of some feature like clone index. Also, if we need an api for some reason, can it reuse snapshot restore api?

How does it relate to “searchable remote index” https://github.com/opensearch-project/OpenSearch/issues/2900?

Searchable remote index is currently focused on supporting remote indices in snapshot format. As @sohami replied to one of the comments https://github.com/opensearch-project/OpenSearch/issues/2900#issuecomment-1105638839, the plan is to support remote indices that will be stored as a part of this proposal. We need to work closely to understand the overlap and possible impact on the design.

What are the Performance knobs that we are planning to offer?

As of now, we are thinking of providing sync/async way to syncing data to the remote store. Sync mechanism will impact performance (for remote translog) and async mechanism will impact durability guarantees. We need to make the trade-off clear and user needs to choose based on the requirement. Having said this, the exact impact will be clear once we carry out performance tests.

Doesn’t this assume remote store itself is durable? Might want to clarify that.

It does but durability guarantees of different remote stores will be different. We called it out in feature proposal as one of the considerations. I will add the same in this doc as well.

Any overlap with segment replication https://github.com/opensearch-project/OpenSearch/issues/1694? Isn’t segment replication a pre-requisite?

In V1, we want to support only Segment Replication. Due to the same set of segments in primary and replica, Segment Replication makes it easy for the initial implementation of the remote store (more details here). We will support document replication in the subsequent releases.

Is RemoteDirectory in addition to LocalDirectory or a replacement?

It would be in addition to the LocalDirectory. Class diagram does not make it clear. I will update it to reflect both.

As you are re-using the repository abstraction, does it unecessarily force remote store implementer to implement snapshot related interfaces also (and vice-versa)?

Good point! It definitely adds an overhead of implementing corresponding Repository and RepositoryPlugin. But the amount of code requires to implement these is minimal. Example: AzureRepository and AzureRepositoryPlugin.
On the other hand, the BlobStore and BlobContainer interfaces that actually deal with reading/writing from/to the remote store will remain same and contain majority of changes in the plugin implementation. The advantage we get is re-use and we can avoid the duplicate implementation.

Do we really need separate apis for restoring from remote store. Shouldn’t this automatically happen in case of primary loss similar to how peer recovers from primary in today’s world. Any shard irrespective of primary or replica should recover from remote storage automatically. If that is a requirement, is there a use case for explicit api to restore?

Agree. My approach was to provide API in V1 and to provide automated approach in V2. But we can have automated restore as a part of V1 as well.

I can think of some feature like clone index. Also, if we need an api for some reason, can it reuse snapshot restore api?

Yes, snapshot API can be changed to take a query parameter where it understands to fetch segment and translog data from the remote store.

How does it relate to “searchable remote index” https://github.com/opensearch-project/OpenSearch/issues/2900?

Searchable remote index is currently focused on supporting remote indices in snapshot format. As @sohami replied to one of the comments https://github.com/opensearch-project/OpenSearch/issues/2900#issuecomment-1105638839, the plan is to support remote indices that will be stored as a part of this proposal. We need to work closely to understand the overlap and possible impact on the design.

What are the Performance knobs that we are planning to offer?

As of now, we are thinking of providing sync/async way to syncing data to the remote store. Sync mechanism will impact performance (for remote translog) and async mechanism will impact durability guarantees. We need to make the trade-off clear and user needs to choose based on the requirement. Having said this, the exact impact will be clear once we carry out performance tests.

Doesn’t this assume remote store itself is durable? Might want to clarify that.

It does but durability guarantees of different remote stores will be different. We called it out in feature proposal as one of the considerations. I will add the same in this doc as well.

Any overlap with segment replication https://github.com/opensearch-project/OpenSearch/issues/1694? Isn’t segment replication a pre-requisite?

In V1, we want to support only Segment Replication. Due to the same set of segments in primary and replica, Segment Replication makes it easy for the initial implementation of the remote store (more details here). We will support document replication in the subsequent releases.

Is RemoteDirectory in addition to LocalDirectory or a replacement?

It would be in addition to the LocalDirectory. Class diagram does not make it clear. I will update it to reflect both.

As you are re-using the repository abstraction, does it unecessarily force remote store implementer to implement snapshot related interfaces also (and vice-versa)?

Good point! It definitely adds an overhead of implementing corresponding Repository and RepositoryPlugin. But the amount of code requires to implement these is minimal. Example: AzureRepository and AzureRepositoryPlugin.
On the other hand, the BlobStore and BlobContainer interfaces that actually deal with reading/writing from/to the remote store will remain same and contain majority of changes in the plugin implementation. The advantage we get is re-use and we can avoid the duplicate implementation.

Do we really need separate apis for restoring from remote store. Shouldn’t this automatically happen in case of primary loss similar to how peer recovers from primary in today’s world. Any shard irrespective of primary or replica should recover from remote storage automatically. If that is a requirement, is there a use case for explicit api to restore?

Agree. My approach was to provide API in V1 and to provide automated approach in V2. But we can have automated restore as a part of V1 as well.

I can think of some feature like clone index. Also, if we need an api for some reason, can it reuse snapshot restore api?

Yes, snapshot API can be changed to take a query parameter where it understands to fetch segment and translog data from the remote store.

Adding scope for V1 (Refer Meta Issue for tasks break-down)
Scope
In V1, remote store will only be supported for newly created indices for which segment replication is also enabled.
Setup

Install and configure repository plugin corresponding to the remote store.

If repository plugin is not available for a given remote store, user needs to write the plugin first.
Existing plugin implementations: repository-hdfs, repository-azure, repository-s3, repository-gcs

Create a snapshot repository with name specific to remote store. It can be provided by the user.
Create an index with segment_replication and remote_store enabled.

Remote store is supported only for the new indices with segment replication enabled.
remote_store setting is non-dynamic. Once index is created, this setting can not be changed.

Storage
For an index with setting segment_replication = true and remote_store = true:

Translog is written to remote translog after each successful indexing operation.
Segments are written to remote store after each refresh. Segment upload will happen only from primary.
Segments deleted on the disk will be deleted from remote store.

Restore

Automated restore on data loss.
API to get status of restore operation.
API to get remote store sync status:

in-sync with local disk
out-of-sync with local disk

Number of segment files not in remote store
Number of segment files still not deleted from remote store

Thanks @sachinpkale for the detailed plan for V1 scope. Is there any reason why you ste segment replication as prerequisite for remote storage? My understanding you should be able to copy segments to remote storage regardless of the replication method.

Is there any reason why you set segment replication as prerequisite for remote storage? My understanding you should be able to copy segments to remote storage regardless of the replication method.

Yes @anasalkouz, you are right. The segments will be copied to remote storage irrespective of the replication method.
But there is a fundamental difference wrt segment creation in document and segment replication. In segment replication, the set of segments in primary and replica would be exactly same. It is not true in document replication as primary and replica creates segments locally.
When failover happens (primary goes down and one of the replicas becomes new primary), we need to sync new primary with remote store but we can't do it incrementally as segment files of new primary can be completely different than old primary. This requires either copying all segments from remote store to new primary or copying all segments from new primary to remote store. As this flow is not straightforward, we have decided to support document based replication in subsequent versions. We have covered this in one of the POCs: https://github.com/opensearch-project/OpenSearch/issues/2481

Thanks @sachinpkale for the proposal, overall looks great.
Few points/clarifications

Maybe we can keep an in-memory checkpoint for remote store and avoid list calls
Would we be able to concurrently write segments to local and remote store, we at least save the fsync time which might be low for a majority of times but would help with slow/degraded/NFS backed devices
Strongly agree all RED indices should be auto-recovered after an initial delay and auto-cancelled if the node that transiently dropped joins back quickly
Refresh interval can be at minimum set to 1s, would that result in remote store uploads lagging if there were significant number of actively written indices on a node. Do we plan to queue up uploads
Do we want to support soft-deletes on remote store to prevent accidental index deletions. Should actual deletions happen asynchronously periodically as such garbage collection task
If we plan on supporting snapshots concurrently with the remote store repository, would it mean more storage costs and most importantly contained network bandwidth on primaries at the snapshot hour/min. Do we want to also think about snapshotting index metadata and performing the actual data copy between two remote repositories instead of double uploads

Hi @sachinpkale.
There are a few other requirements that are worth calling out for this to work with segment replication.  While we don't have the first wave of segrep merged into main yet from our feature branch, we have a general idea of the files that need to be copied to/from a store.
Primaries will need to include a byte[] of its current SegmentInfos object in addition to the segments.  The SegmentInfos is required so that we can copy out after refresh and not wait for the next commit point.  I don't think anything in the directory API exists that would push this, it would be an addition.
Second, we'll need a way to fetch this SegmentInfos on replicas in addition to the segment files.
The general flow I'm thinking of is similar to what we have in the feature branch & in the proposal, with an interface (SegmentReplicationSource) extracted so we can implement for both node->node replication or a remote store using the same flow.

Primaries refresh and push all segments and SegmentInfos to a remote store.  Primaries publish a checkpoint for replicas.
Replicas receive the new checkpoint and start a replication event.
First the replica calls SegmentReplicationSource.getCheckpointMetadata to fetch the checkpoint metadata, which means the StoreFileMetadata associated with the checkpoint and the SegmentInfos bytes[].
Replica computes a diff based on the metadata and makes a list of the files it needs.
Replica calls SegmentReplicationSource.getFiles and fetches the files.
Replica performs validation of all new files & updates its reader with the new SegmentInfos.

I'm thinking this SegmentReplicationSource interface would be something like.
interface SegmentReplicationSource {

       public void getCheckpointMetadata(
        long replicationId,
        ReplicationCheckpoint checkpoint,
        StepListener<CheckpointMetadataResponse> listener);

    public void getFiles(
        long replicationId,
        ReplicationCheckpoint checkpoint,
        Store store,
        List<StoreFileMetadata> filesToFetch,
        StepListener<GetFilesResponse> listener)
}

Where getFiles does not complete the listener until all requested files have been copied.

Thoughts?

An alternative to pushing the SegmentInfos bytes in my previous comment is that with remote store enabled we change the behavior of a refresh to instead perform a Lucene commit.  This way we continuously push an updated commit point / segments_N file to the remote store.  We can disable the fsync on the local directory implementation to make commits less expensive because segments will be durably stored remotely.  This would remove any need for any change to the directory implementation to push the SegmentInfos bytes.  I'm not sure of the perf trade-off here.
Either way I think we can include the SegmentInfos object itself in CheckpointMetadataResponse above instead of the byte array and leave it up to the remote store implementation how it is constructed.

@mch2 Thanks for the heads up.
I am inclined towards the first approach. In remote store, we can have another file (with suitable prefix/suffix to segments_N) which is re-uploaded at each refresh.
I would like to understand one thing though: is segments_N updated in memory at each refresh and not fsynced or it is only updated at commit? I will definitely check it from my side but if you already know this, it would help.

An alternative to pushing the SegmentInfos bytes in my previous comment is that with remote store enabled we change the behavior of a refresh to instead perform a Lucene commit.

Lucene commit is expensive and OpenSearch internally performs various operations (like purging translog) when flush is called. As commit makes things durable on local store, changing the behaviour would need to understand the complete commit flow (which I don't currently).

@mch2 based on the alternative proposal wouldn't it create bi-modal code paths which would couple itself up with the storage layer? I wouldn't recommend this. Lucene commit as @sachinpkale pointed out is certainly expensive and might need modifications in the OpenSearch flush flow as well.

I would like to understand one thing though: is segments_N updated in memory at each refresh and not fsynced or it is only updated at commit?

It's only updated & flushed to disk on a commit.
@Bukhtawar.  Sorry, I meant conceptually to perform more frequent commits instead of the existing refresh behavior, not splitting the code path of a refresh.  I'd be curious how much more expensive commits are with fsyncs turned off.  This would mean we could always write & re-read SegmentInfos from a stable _N file.
If we don't have a function on the Directory to push the byte[] as a stream writing the file is not a problem.  This would use similar logic to what SegmentInfos uses already to write for commits - SegmentInfos.write, we are already using the public write function to write to a byte[].  I wrote a little unit test to see that it works, the name of the file would need to be different to not conflict with the latest commit point.
    public void writeSegmentInfosTest() throws Exception {
        List<Engine.Operation> operations = generateHistoryOnReplica(
            between(1, 500),
            randomBoolean(),
            randomBoolean(),
            randomBoolean()
        );
        for (Engine.Operation op : operations) {
            applyOperation(engine, op);
        }
        final SegmentInfos latestSegmentInfos = engine.getLatestSegmentInfos();
        long nextGeneration = latestSegmentInfos.version;
        final String FILE_PREFIX = "latest_infos";
        String segmentFileName =
            IndexFileNames.fileNameFromGeneration(FILE_PREFIX, "", nextGeneration);

        try (IndexOutput output = store.directory().createOutput(segmentFileName, IOContext.DEFAULT)) {
            latestSegmentInfos.write(output);
        }

        try (final ChecksumIndexInput checksumIndexInput = store.directory().openChecksumInput(segmentFileName, IOContext.DEFAULT)) {
            final SegmentInfos segmentInfos = SegmentInfos.readCommit(store.directory(), checksumIndexInput, latestSegmentInfos.getGeneration());
            assertEquals(segmentInfos.files(true), latestSegmentInfos.files(true));
        }
    }

@mch2 If we take remote store out of the picture, where are we planning to store SegmentInfos bytes[]? Will it be in memory of primary? Can we create a new file which is appended with new data (or if we want this file immutable, we can replace the file) each time refresh happens? This way, nothing changes once remote store comes into picture.

@mch2 If we take remote store out of the picture, where are we planning to store SegmentInfos bytes[]? Will it be in memory of primary?

@sachinpkale For node-node replication the byte[] isn't written to disk, we pull the primary's latest SegmentInfos in memory, ensure all segments it references are incRef'd for the duration of the copy event and then serialize it over the wire.

Can we create a new file which is appended with new data (or if we want this file immutable, we can replace the file) each time refresh happens? This way, nothing changes once remote store comes into picture.

We could, this is what the snippet above does.  I would prefer we write the object directly to the remote store and avoid the extra flush to the local disk if we can.  I think this is possible with what you've implemented in #3460 although theres an extra step of converting the IndexOutput back to Input.  Something like...
        ByteBuffersDataOutput buffer = new ByteBuffersDataOutput();
        try (ByteBuffersIndexOutput tmpIndexOutput = new ByteBuffersIndexOutput(buffer, "temporary", "temporary")) {
            segmentInfos.write(tmpIndexOutput);
        }
        final BufferedChecksumIndexInput indexInput = new BufferedChecksumIndexInput(
            new ByteBuffersIndexInput(buffer.toDataInput(), "SegmentInfos")
        );
        copyBytes(indexInput...)

@mch2 Please take a look at commit: https://github.com/sachinpkale/OpenSearch/commit/b195a2a3daf6267c1aca53c675311d65935b2872#diff-45c1ec7b5c456e144972500c9883018f4e74f4e7d054c448bf44ef308dbbb8e6R116-R123. In this commit, we are uploading intermediate segments_N file after each refresh. Let me know if this works.

@sachinpkale I'd like to help. Are there some small tasks that I can get started with? Thanks

Sure @mashah . We are keeping track of all the issues in this meta issue: https://github.com/opensearch-project/OpenSearch/issues/2992


1e crashes with java.lang.NumberFormatException: For input string: ""

I'd expect 1e to print 2.71828182846 instead I'm getting an exception:
java.lang.NumberFormatException: For input string: ""
    at java.lang.Integer.parseInt(Integer.java:627)
    at java.lang.Integer.parseInt(Integer.java:650)
    at java.math.BigDecimal.<init>(BigDecimal.java:336)
    at java.math.BigDecimal.<init>(BigDecimal.java:444)
    at com.udojava.evalex.Expression$59.eval(Expression.java:1363)
    at com.udojava.evalex.Expression.eval(Expression.java:1398)
    at com.udojava.evalex.Expression.eval(Expression.java:1273)

There is no implicit multiplication, you have to specify a multiplication. See #64.

My problem is similar to the one from @FlashTek. The input is coming from a user and he'd expect it to be multiplied.

E.g. What about A5BSIN(2) - is this a function named A5BSIN or an expression A5SIN(2) ?

If a function with a name A5BSIN is defined then it should take precedence. Otherwise A*5*SIN(2) as long as A can be satisfied. If none can then it can crash.

The same should be applied for 1π where PI is defined as .setVariable("π", Expression.PI).


CSharp Samples outdated

The CSharp example seems to be outdated.
The nuget package reference and namespace and class names in the sample doesn't exist anymore.
Any hint on Signing in with new API?

The NuGet package is provided by the third party. The code in https://github.com/tdlib/td/tree/master/example/csharp is on a par with up-to-date TDLib version.

Thanks for the confirmation. I was confused between the nuget package and this repository. Now I know why the namespaces and classes in the nuget package is different.
Thanks for the prompt reply.


fix fixes for neopixels

@pelikhan,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot


I1202 Can't select .KIP as source

This Looks Great. Tested and it worked.

Let's reference the Github issue that this PR address if one exist already. If not let's create it.

Looks good. We only need GitHub ticket and document this fix in release notes.

Let's reference the Github issue that this PR address if one exist already. If not let's create it.

Was a Github ticket created for this ? If yes please reference the ticket in the description.

The PR appears to have conflicts. Please resolve.

@tst-eclamar can you please publish the doc changes to Github for easier review of the release notes changes ?

There are conflicts that were developed. Let's get them resolved.


How to expand/collapse nodes in UI?

This is a fantastic library, one of the best I've seen for mind mapping. Excellent work. This might seem like a silly question but I could not see in demo how to collapse/expand nodes. Is that done by keystroke combination, mouse, or some or other configuration? :-)

Thanks for the quick reply @ssshooter! In Chrome and Edge Chromium on Windows 10 it does not seem to work. Please see video below, perhaps I'm missing something? I tried moving the mouse over the entire node and the junctions.
https://user-images.githubusercontent.com/47497/138381339-ec673a91-2c8d-4bcc-9bb7-411f8c5baf42.mp4


stylus-loader npm package should depend on file-loader npm package

Hi,
I'm not familiar with this module, but I tried to remove file the file-loader package from a package.json of a project (the npm-check utility suggested that it may not be used), and it resulted in an error (something tried to load file-loader).
So I guess it would be reasonable to add file-loader as a dependency in stylus-loader's package.json.

Hey @attila123 thanks for opening this issue. There is nothing we need to change though since this is a matter of how you configure a webpack build.
stylus-loader has no dependency on file-loader. Through webpack you can configure it to builds files like images with file-loader, url-loader, or other more specialized loaders. But use of those loaders through webpack doesn't happen for stylus files through stylus-loader, it happens in css-loader. css-loader could be said to have a dependency on "some" loader to handles images and fonts that are referred to by url(some-image.png) calls. But its still configuration dependent and there would be no safe way for css-loader to ensure a loader is configured for such files. So that is why webpack errors when it encounters a file that it can't parse as JavaScript, because no loader has been stated or configured to be used for that type of resource.
stylus-loader has no connection to file-loader or other "file" loaders. css-loader has an indirect connection to "file" loaders but its wholly dependent on webpack configuration. Its really something to leave to documentation and not package.json.


fix(proxy) resolve DNS hostnames post-plugins

This is mainly to address the Lambda and OpenWhisk plugins not being
able to run if the specified ùpstream_url`'s hostname does not resolve,
hence resulting in an HTTP 500 error.
Ideally, this change should probably be reverted, since some future
plugins might need to know the resolved IP address or the upstream Host
header in their access phase.
No tests for now, just running the CI.

haven't looked at the code yet, but imo we should resolve as late as possible. Any plugin needing IPs should do its own resolving. So that is something we might/should enable.

I guess I should have put the "wip" label on this, this was more of a first try to test the CI build than an definitive proposal/patch. But thanks anyways :)


Add --profile-startup option

This pull request adds the --profile-startup command line option that will profile render process for the created window.
The goal here is to make it easy to see why an Atom window might be loading slowly.
It covers the time between the window.onload handler firing and the completion of the bootstrap script. So it will include deserialization, all requires, package loading, theme loading, etc.
atom --profile-startup

:clap: This sounds really cool!

:clap: :clap: Yesss, this will be super helpful!

is this still implemented?


Add FAQ section to the documentation

Go through issues here on Github and questions on my email, and compile a list of frequently asked questions.

Done: https://ondrejnepozitek.github.io/Edgar-Unity/docs/other/faq/


Code cleanup

tox.ini: add requires, drop pytest-order
add instance variables prototypes to every class
fix Sphinx roles
ignore typing code in coverage reports

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 13 (100.0%)  changed or added relevant lines in 5 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
73

Relevant Lines:
73

💛  - Coveralls

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

13 of 13 (100.0%)  changed or added relevant lines in 5 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
73

Relevant Lines:
73

💛  - Coveralls


Not displaying any data in webinterface

PDNS newest debian-package version with enabled mysql-backend
Installed newest stable pdnsmanager version by hand
Running install.php successfully
Adding new master domain including records: Successfully
They are inserted into the database, but they are not displayed in webinterface. Just empty rows, although the DNS server is fully working and the records are in fact in database and were inserted by pdnsmanager.
Strange bug ...

Naaah it's too late for me to work. If anyone every encounters that issue:
apt-get install php5-mysqlnd && php5enmod mysqlnd && service apache2 restart


Document how the TOC election process works

This is supposed to prevent a power vacuum so that a problem similar to the 2024 election does not happen again
Additionally, update current TOC members from 2023 to 2024.

PS: my vote goes only to the docs change
for the update that I will be in elected as silverwind replacement, I'm thankfull and do accept it.
but I'd like to see other maintainers lgtm as legitimation not mine ;)

8 approvals.
I don't think there will be any more.
I'm giving a deadline of 12 hours for further feedback and will then merge the PR if there's no new feedback.


TL_messageFwdHeader update

Hi!
Thnx you for you work!
Can you update TL_messageFwdHeader to recive a filds "Saved_from_peer" and "Saved_from_msg_id" ?
Please.

MTProto is defined by Telegram guys, so that the feature is out of scope of this repository.
I think you can mashup functions to get desired information.
Thanks


spotify: fix missing ffmpeg_0_10 dependency

This makes local playback work.

/cc other maintainer @ftrvxmtrx.

:+1: very good :)

@nathan7 tnx!


Installation issue with Plug

A fairly minor issue, and it could be something in my own setup that's causing this, but it seems that using Plug defaults to a branch named master rather than main. So, to install, users may need to use:
Plug 'rmehri01/onenord.nvim', { 'branch': 'main' }

Might be worth mentioning in the README.
(And thanks for putting this together! It looks fantastic.)

Ah I see, I didn't know about this. I'll add it to the readme, thanks!


[Design Inspiration] -> [Microcopy Inspirations]

removed line 938
https://www.microcopyinspirations.com it's down. Not sure how long.

Lest wait for it for a week.


Update docs staging context creation.

Already on https://github.com/ocurrent/ocurrent-deployer/pull/203


download and resend not working in combination with file time sensitive in directory listener

❗ please do not add sensitive information in issues, you can provide extra information via email using issue number as reference ❗
Describe the issue
When trying to download (or resend) an error message from the directory listener GUI the download fails because it finds a  %25 (instead of a space) this gives an error.
14-Nov-2023 10:48:42.141 SEVERE [http-nio-80-exec-9] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [IAF-API] in context with path [] threw exception
org.springframework.security.web.firewall.RequestRejectedException: The request was rejected because the URL contained a potentially malicious String "%25"
at org.springframework.security.web.firewall.StrictHttpFirewall.rejectedBlocklistedUrls(StrictHttpFirewall.java:535)
at org.springframework.security.web.firewall.StrictHttpFirewall.getFirewalledRequest(StrictHttpFirewall.java:505)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:210)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:190)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at org.apache.logging.log4j.web.Log4jServletFilter.doFilter(Log4jServletFilter.java:71)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:596)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:673)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:389)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:926)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Reporter
Eddy /Jeroen

Screenshots
If applicable, add screenshots to help explain your problem.
Configuration
<Adapter name="MFHIn_00109" active="${active.mfh}" description="Create trigger in the IBISFILETRIGGER table for legacy flow 00109">
        <Receiver pollInterval="${mfh.MFHIn_00109.pollInterval}">
            <DirectoryListener
                minStableTime ="${mfh.MFHIn_00109.minstabletime}"
                fileTimeSensitive="true"
                name="MFHIn"
                messageType="info"
                root="${mfh.MFHIn_00109.rootdir}"
                inputFolder="in"
                inProcessFolder="inProcess"
                errorFolder="error"
                createFolders="true"
                wildcard="${mfh.MFHIn_00109.wildcard}"/>
        </Receiver>

Input
Please provide an example of the input message.
Alternatively, provide a Ladybug report.

Environment
was working on 7.7.3

FF! 7.8.3: Ibis4SapCD
Running on LPCENL005288323 using Apache Tomcat/9.0.74
Java Version: OpenJDK Runtime Environment (1.8.0_292-b10)
Heap size: 357M, total JVM memory: 604M
Free disk space: 101GB, total disk space: 238GB
Up since: 2023-11-14 12:44:45 (8m)
has context menu

Additional Environment

DBMS: [e.g. Oracle, MSSQL, MariaDB, MySql, PostgreSQL]
Browser: [e.g. Chrome, Safari, Edge, Firefox]

Additional Context
Add any other context about the problem here. (f.e. ladybug report / test adapter with larva test)

Having managed to reproduce this, I see that the download-link contains %2520 where the original filename had a   (space). (There are still other spaces in the download link though, oddly enough...)
http://localhost:8080/iaf-test/iaf/api/configurations/FileHandling/adapters/MFHIn/receivers/DirectoryListener [MFHIn]/stores/Error/messages/mfh-input.xml-2023-03-08%252019_23_41.283/download
However the JSON data does not contain neither the %25 nor %20, it still has the original spaces in the filename.
So I guess the download-error is an issue with the front end doing double url-encoding on parts of the URL being constructed.

https://github.com/frankframework/frankframework/blob/bc99f5eb894ba4e2ff9014a79f6e82cd52869624/console/backend/src/main/java/nl/nn/adapterframework/management/web/CompatiblityTransactionalStorage.java#L73

So we screw up one use-case to "fix" another.
I suggest that either we put the mangling of the message-ids in the Exchange listener, or we do a safe manging and unmangling all in the back-end and not one part in the front-end and another part in the back-end.


Music not working properly on Debian 8

I never tried using music before, today I tried and only heard random noises, the song itself didn't matter.
Already tried changing Discord server location, voice channel bitrate to no avail.
Also tried rebooting the VPS my NadekoBot is being hosted on and had no changes.
Redownloaded prerequisites, redownloaded NadekoBot and still only noise.
Console log shows no errors.

Second that, it would sound like you're scraping metal against another metal surface, just that annoying squeaking/scraping noise

Can you confirm it's because of Debian 8? Maybe it's because the connection from your VPS to discord isn't stable/sufficient?

Uh this happened before, are you 100% sure you have the correct version of ffmpeg? (3.3+)
@samdivaio do you maybe know of the issue where we resolved this?

We used docker to fix music issues on debian. 🤔

Unsure on top of my head, I think both of us simply used the Bot installer to download the necessary prereqs like ffmpeg that came with it. So the version should be the one supplied by the installer.

Seems to be the ffmpeg version, tried removing and reinstalling it with apt-get and I'm getting ffmpeg 3.2...
I'll try installing docker

Just use docker on it or switch os to Ubuntu

Switched OS to ubuntu and now it works completely fine

Could have something to do with opus or sodium... Maybe it's 32bit, i really don't know. If someone else is reading this in the future, try compiling libopus and libsodium yourself, and replacing the existing ones in the output folder after building the bot.


query by upn rather than samaccount, guid or sid

I'd like to be able to search users by upn for DirectoryReplicationClient.GetAccount.

Is this what you had in mind? It will appear in a next release.

yes, that is what i was looking for.
thanks!

This change has been just released in v 2.16.

Thanks.
Sent from my Verizon, Samsung Galaxy smartphone

-------- Original message --------From: Michael Grafnetter  Date: 8/7/16  7:07 AM  (GMT-05:00) To: MichaelGrafnetter/DSInternals  Cc: kevindelafield , Author  Subject: Re: [MichaelGrafnetter/DSInternals] query by upn rather than samaccount, guid or sid (#18)

This change has been just released in v 2.16.

You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/MichaelGrafnetter/DSInternals/issues/18#issuecomment-238076376


issue matrix google spreadsheet template is private

Hi! I think the Google Doc (key: 18g7ExcIMXcSpVGMcQYAIHvmGqMzf9j15woW8UmsgmGU) for the template for the issue matrix is private.
When I run fab add_issue_matrix:[slug], I get:
Creating spreadsheet...
Error creating spreadsheet (status code 404) with message Not Found

Thanks!

Try now?

Perfect! Thanks for getting to it so quickly.


CORS error?

I haven't put much energy into tracking this down, but I figured I'd at least report getting this error.

You might need to host it with python3 -m http.server or similar, or mess with some browser settings to enable file:// file accesses; unfortunately they're limited by default.

Starting an explicit server did work.  Thanks.


Build failure for ios-armv7

./buildnativeoperations.sh -o ios-armv7

will cause
-- The C compiler identification is unknown
-- The CXX compiler identification is unknown
CMake Error at CMakeLists.txt:2 (project):
  The CMAKE_C_COMPILER:

    clang-omp

  is not a full path and was not found in the PATH.

  Tell CMake where to find the compiler by setting either the environment
  variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full path to
  the compiler, or to the compiler name if it is in the PATH.

CMake Error at CMakeLists.txt:2 (project):
  The CMAKE_CXX_COMPILER:

    clang-omp

  is not a full path and was not found in the PATH.

  Tell CMake where to find the compiler by setting either the environment
  variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path
  to the compiler, or to the compiler name if it is in the PATH.

-- Configuring incomplete, errors occurred!

This is due to two lines in cmake/ios-armv7.cmake
set(CMAKE_C_COMPILER   "clang-omp")
set(CMAKE_CXX_COMPILER "clang-omp")

There is a known relevant issue #300, where clang-omp is no longer available from brew install clang-omp.

It seems one needs to use llvm for this job. Please pardon my ignorance, but isn't clang a front-end for llvm? I was believing using clang is same as using llvm.

If I use just clang instead of clang-omp, build stops with a fatal error:
/libnd4j/include/helpers/helper_random.h:19:10: fatal error: 'mutex' file not found
#include <mutex>

We normally use gcc on mac. Is there a problem with that for ios?

I have no real knowledge on that. But a little bit of Googling tells me that there is a binary compatibility issues between clang and gcc. Based on that reason, now this is my wild guess why gcc is not for iOS. So it means objects may not be linked together well if one is compiled with clang and the other is with gcc. This is a problem on iOS, where Apple enforces static linking for the app. Everything must be linked together into a monolith one binary. No shared or dynamic libraries are allowed on iOS. Xcode and all iOS frameworks are believed to work with clang. Probably gcc will have a hard time to link with the iOS frameworks. On the other hand, one can use dynamic libraries, like libnd4jcpu.dylib on macOS, without an issue.

Trying different clangs.
Xcode 8.3.3: does not recognize -fopenmp, and stops at  mutex not found
clang --version
Apple LLVM version 8.1.0 (clang-802.0.42)
Target: x86_64-apple-darwin16.6.0

Xcode 9.0: does not recognize -fopenmp, and stops at  mutex not found
clang --version
Apple LLVM version 9.0.0 (clang-9<IP_ADDRESS>)
Target: x86_64-apple-darwin16.6.0

LLVM 4.0: does recognize -fopenmp, but stops at  mutex not found
clang --version
clang version 4.0.0 (tags/RELEASE_400/final)
Target: x86_64-apple-darwin16.6.0

Hmm - the reason I'm asking is because our mac builds use gcc and no one has complained. Clang doesn't have openmp, they broke it on mac. That's wh you're seeing these problems.

I am looking into why mutex is needed in the first place, and if there is any workaround.

Adding -stdlib=libc++, then LLVM 4.0 did compile and generated libnd4jcpu.a
For Xcode clangs, remove -fopenmp  and add -stdlib=libc++ , compiled and linked to libnd4jcpu.a
So it does build with -stdlib=libc++.