Vulkan: Investigate using VK_KHR_buffer_device_address and PhysicalStorageBuffer for vulkan memory access

Request description
Branching off issue https://github.com/openxla/iree/issues/13196 to see what it would take to implement VK_KHR_buffer_device_address and PhysicalStorageBuffer  based access to Vulkan devices. Increasing we are dealing with very large tensors (>4GB) and maxStorageBufferRange is limited to 4GB. While we explore options with https://github.com/openxla/iree/issues/13196 this feature request is to see what it would take to move us to using VK_KHR_buffer_device_address.
We are seeing increasing model sizes Stable Diffusion (768x768) , LLaMA upto 65B etc that we are unable to run on our Vulkan backend today without doing a multi-process hack. We are also starting to see 16GB+ VRAM allocations on mobile SoC devices so this a requirement across the board for vulkan devices.
Some other references:
https://gpuopen-librariesandsdks.github.io/VulkanMemoryAllocator/html/enabling_buffer_device_address.html ? There were some references to it from https://github.com/KhronosGroup/Vulkan-Docs/issues/1016
@antiagainst @benvanik @stellaraccident
What component(s) does this issue relate to?
No response
Additional context
No response

Adding @antiagainst to take a look when you can.

This would be a major change w.r.t. how we handle buffers across runtime and kernel. What the device buffer address extension does is enabling querying int64 physical GPU addresses for storage buffers so that we can populate them in uniform buffers or push constants and then let kernels directly load them as buffer pointers and do load/store thereafter. So it's pretty substantial, at least at the conceptual level.
Now, regarding what needs to be changed to support this extension, a few big parts:

Runtime-kernel boundary: we'd need a way/convention to make sure we have consistency between runtime (in IREE) and kernel CodeGen (in MLIR). E.g., using a uniform buffer or push constants for all base buffer address. Using push constants  avoids indirection but push constants already used for shape dimensions and such. If uniform buffer, which descriptor set. How buffer addresses for different descriptor (set, binding) are organized in the uniform buffer, etc.
Runtime: we'd need to change how push descriptors are handled in the Vulkan HAL driver, to query device buffer address and populate them in the above manner. This actually makes it closer to LLVMGPU side in a sense.
Kernel: we'd need to plumb through support for physical storage buffer and related capabilities in MLIR/SPIR-V CodeGen. And then follow the convention to not generate descriptors but load device pointers directly and use them.

I'd need to talk with @benvanik to get a more detailed picture w.r.t. what's need to be changed, esp. on the runtime side.

Yeah, lots tangled up here - we'll need to break it down. There's some easier solutions and some harder ones for sure :)
I suspect we'll build ArgumentBuffer-like thing in the Vulkan HAL driver and map descriptor sets into that instead of native descriptor sets.

(this was how I was imagining enabling secondary command buffer buffer substitution in iree_hal_command_buffer_execute - so doing it may get us that too!)

Took a look at what would be needed. @antiagainst articulated the major parts and then there's a few details:

need to do #7824 so that we can have alternative shaders (ones with/without device address support) - today without linking it'd create an explosion as each shader is its own executable whereas we should instead have one executable for all shaders with a given target configuration
executable target configuration for device addresses propagated to vulkan/spir-v env so we can specify targets that do/don't support them
HAL -> SPIR-V lowering needs to emit all resources as pointers loaded from a base descriptor when the configuration demands it
SPIR-V ExecutableDef flatbuffer needs a flag for whether device addresses are used (this is how the runtime knows it needs to switch modes)
FillBufferUnaligned builtin will need to support device addresses (today it's just descriptors)
direct command buffer needs two paths for push_descriptor_set: one that does what it does today with DescriptorSetArena and another that uses binding tables

The idea would be to have each command buffer have a growable set of staging buffers with uniform-buffer-upload semantics (something we'd have to benchmark, but usually device-local|host-visible|host-coherent) and as the command buffer is recorded and push_descriptor_sets is called we'd slice off some of the current staging buffer, scribble in our buffer info, and then bind the staging buffer with a dynamic offset as a normal descriptor set operation using DescriptorSetArena as today. The shaders would have a buffer declared for the descriptors and access all descriptors indirectly through it. Something like:
iree_hal_vulkan_direct_command_buffer_push_descriptor_set(...) {
  if (iree_hal_vulkan_native_pipeline_layout_has_indirect_access(pipeline_layout)) {
    ... stash on command buffer mirror of descriptor state ...
    ... mark parameters as dirty ...
  } else {
    // existing descriptor set binding path
  }
}
iree_hal_vulkan_direct_command_buffer_dispatch(...) {
  if (iree_hal_vulkan_native_pipeline_layout_has_indirect_access(pipeline_layout)) {
     if (parameters are dirty) {
       // upload a new parameters chunk by flushing the command buffer descriptor state for the pipeline layout
       // this may allocate a new staging buffer if the prior one is exhausted
       iree_hal_vulkan_direct_command_buffer_append_dispatch_parameters(pipeline_layout, &staging_descriptor_set, &staging_offset);
       // bind the root descriptor with the dynamic offset of the dispatch - should be cheap
       vkCmdBindDescriptorSets(staging_descriptor_set, staging_offset);
       ... reset parameters dirty flag ...
     }
  }
  vkCmdDispatch...
}

And the shader:
layout(buffer_reference, std430, buffer_reference_align = 16) buffer binding_f32_t {
  float data[];
};
layout(set = 3, binding = 0) buffer root_t {
    binding_f32_t binding_0;
    binding_f32_t binding_1;
    binding_f32_t binding_2;
} root;
void main() {
  root.binding_0.data[0];  // access...
}

Oh the other thing this intersects with is secondary indirect command buffers (iree_hal_command_buffer_execute_commands) - those are recorded with placeholders and then when executed the placeholders get updated to the bindings passed in via iree_hal_buffer_binding_table_t. There's a bit more bookkeeping required such that we map what a command-buffer-global binding slot is to the locations in the staging buffer that need to have that value populated. In the above we can write the device addresses in when flushing the parameters but here we'd instead scribble aside the offset into the parameter buffer where the address should be written as we don't actually have it at the time of recording. When a secondary command buffer is  scheduled with iree_hal_command_buffer_execute_commands the hosting primary command buffer would use one or more vkCmdUpdateBuffer to populate the parameters in stream order and so long as we didn't allow multiple overlapping executions of the same secondary buffer (we'll need to track) we should be safe. I call this out because the compiler side will look identical and it's just additional tracking at runtime on top of the above work to also get reusable command buffers!

One thing that may need some fiddling is how to communicate the pipeline layout mode or if we want to make it per descriptor set (I think we want to make it per descriptor set) - we can add a bit to iree_hal_descriptor_set_layout_flags_t for whether it's a native descriptor (default) or an indirect one, have the compiler emit the flag when creating such sets, and then have that be queried during dispatch. The flags are set based on the executable target which we'd know needs the flag set. when binding we'd then ignore any indirect descriptor sets as those are covered by the parameter upload path.

#14777 has disabled VMA by default and hopefully it sticks (there may be some issues). We'll let that soak a bit and get into shark before fully removing VMA.
After that #14778 makes the runtime Vulkan HAL detect support for buffer device addresses and enables the feature on the allocations we make.
The next step is to implement the indirect parameter buffer in the Vulkan HAL in preparation for the compiler using it. I've got some sketches that dovetail with indirect command buffers and will see if I can piece them apart for some incremental work.

Following up from the discussions yesterday, here's the spec I'm going to be shooting for on the compiler/runtime side outside of codegen:

new HAL_DescriptorSetLayoutFlags_Indirect/IREE_HAL_DESCRIPTOR_SET_LAYOUT_FLAG_INDIRECT flag
new temporary global compiler flag so that MaterializeInterfaces sets the flag on DescriptorSetLayoutAttr
runtime command buffer logic that uses set 3 binding N for each original set N

So what would have been:
#version 460
layout(set = 0, binding = 0, std430) buffer set_0_binding_0 { float data[]; };
layout(set = 0, binding = 1, std430) buffer set_0_binding_1 { float data[]; };
layout(set = 0, binding = 2, std430) buffer set_0_binding_2 { float data[]; };
// note no binding 0 used
layout(set = 1, binding = 1, std430) buffer set_1_binding_1 { float data[]; };
void main() {
  set_0_binding_0.data[0];
  set_0_binding_1.data[0];
  set_0_binding_2.data[0];
  set_1_binding_1.data[1] = 1.0f;
  // ...
}

->
#version 460
#extension GL_EXT_buffer_reference : require
layout(buffer_reference, std430, buffer_reference_align = 16) buffer binding_f32_t {
  float data[];
};
layout(set = 3, binding = 0) buffer set_0_t {
  binding_f32_t binding_0;
  binding_f32_t binding_1;
  binding_f32_t binding_2;
} set_0;
layout(set = 3, binding = 1) buffer set_1_t {
  binding_f32_t unused_binding_0;  // note here for alignment
  binding_f32_t binding_1;
} set_1;
void main() {
  set_0.binding_0.data[0];  // access original set(0) binding(0)
  set_0.binding_1.data[0];  // access original set(0) binding(1)
  set_0.binding_2.data[0];  // access original set(0) binding(2)
  set_1.binding_1.data[1] = 1.0f;  // access original set(1) binding(0)
  // ...
}

SPIR-V:
; SPIR-V
; Version: 1.6
; Generator: Khronos Glslang Reference Front End; 11
; Bound: 34
; Schema: 0
               OpCapability Shader
               OpCapability PhysicalStorageBufferAddresses
          %1 = OpExtInstImport "GLSL.std.450"
               OpMemoryModel PhysicalStorageBuffer64 GLSL450
               OpEntryPoint GLCompute %main "main" %set_0 %set_1
               OpExecutionModeId %main LocalSizeId %uint_1 %uint_1 %uint_1
               OpSource GLSL 460
               OpSourceExtension "GL_EXT_buffer_reference"
               OpName %main "main"
               OpName %set_0_t "set_0_t"
               OpMemberName %set_0_t 0 "binding_0"
               OpMemberName %set_0_t 1 "binding_1"
               OpMemberName %set_0_t 2 "binding_2"
               OpName %binding_f32_t "binding_f32_t"
               OpMemberName %binding_f32_t 0 "data"
               OpName %set_0 "set_0"
               OpName %set_1_t "set_1_t"
               OpMemberName %set_1_t 0 "unused_binding_0"
               OpMemberName %set_1_t 1 "binding_1"
               OpName %set_1 "set_1"
               OpMemberDecorate %set_0_t 0 Offset 0
               OpMemberDecorate %set_0_t 1 Offset 8
               OpMemberDecorate %set_0_t 2 Offset 16
               OpDecorate %set_0_t Block
               OpDecorate %_runtimearr_float ArrayStride 4
               OpMemberDecorate %binding_f32_t 0 Offset 0
               OpDecorate %binding_f32_t Block
               OpDecorate %set_0 DescriptorSet 3
               OpDecorate %set_0 Binding 0
               OpMemberDecorate %set_1_t 0 Offset 0
               OpMemberDecorate %set_1_t 1 Offset 8
               OpDecorate %set_1_t Block
               OpDecorate %set_1 DescriptorSet 3
               OpDecorate %set_1 Binding 1
       %void = OpTypeVoid
          %3 = OpTypeFunction %void
       %uint = OpTypeInt 32 0
     %uint_1 = OpConstant %uint 1
               OpTypeForwardPointer %_ptr_PhysicalStorageBuffer_binding_f32_t PhysicalStorageBuffer
    %set_0_t = OpTypeStruct %_ptr_PhysicalStorageBuffer_binding_f32_t %_ptr_PhysicalStorageBuffer_binding_f32_t %_ptr_PhysicalStorageBuffer_binding_f32_t
      %float = OpTypeFloat 32
%_runtimearr_float = OpTypeRuntimeArray %float
%binding_f32_t = OpTypeStruct %_runtimearr_float
%_ptr_PhysicalStorageBuffer_binding_f32_t = OpTypePointer PhysicalStorageBuffer %binding_f32_t
%_ptr_StorageBuffer_set_0_t = OpTypePointer StorageBuffer %set_0_t
      %set_0 = OpVariable %_ptr_StorageBuffer_set_0_t StorageBuffer
        %int = OpTypeInt 32 1
      %int_0 = OpConstant %int 0
%_ptr_StorageBuffer__ptr_PhysicalStorageBuffer_binding_f32_t = OpTypePointer StorageBuffer %_ptr_PhysicalStorageBuffer_binding_f32_t
      %int_1 = OpConstant %int 1
      %int_2 = OpConstant %int 2
    %set_1_t = OpTypeStruct %_ptr_PhysicalStorageBuffer_binding_f32_t %_ptr_PhysicalStorageBuffer_binding_f32_t
%_ptr_StorageBuffer_set_1_t = OpTypePointer StorageBuffer %set_1_t
      %set_1 = OpVariable %_ptr_StorageBuffer_set_1_t StorageBuffer
    %float_1 = OpConstant %float 1
%_ptr_PhysicalStorageBuffer_float = OpTypePointer PhysicalStorageBuffer %float
       %main = OpFunction %void None %3
          %5 = OpLabel
         %18 = OpAccessChain %_ptr_StorageBuffer__ptr_PhysicalStorageBuffer_binding_f32_t %set_0 %int_0
         %19 = OpLoad %_ptr_PhysicalStorageBuffer_binding_f32_t %18
         %21 = OpAccessChain %_ptr_StorageBuffer__ptr_PhysicalStorageBuffer_binding_f32_t %set_0 %int_1
         %22 = OpLoad %_ptr_PhysicalStorageBuffer_binding_f32_t %21
         %24 = OpAccessChain %_ptr_StorageBuffer__ptr_PhysicalStorageBuffer_binding_f32_t %set_0 %int_2
         %25 = OpLoad %_ptr_PhysicalStorageBuffer_binding_f32_t %24
         %29 = OpAccessChain %_ptr_StorageBuffer__ptr_PhysicalStorageBuffer_binding_f32_t %set_1 %int_1
         %30 = OpLoad %_ptr_PhysicalStorageBuffer_binding_f32_t %29
         %33 = OpAccessChain %_ptr_PhysicalStorageBuffer_float %30 %int_0 %int_1
               OpStore %33 %float_1 Aligned 4
               OpReturn
               OpFunctionEnd

Update that I decided flags are fine for now as this can be experimental. In #14977 I've added the --iree-vulkan-experimental-indirect-bindings=true compiler flag that changes the executable format to that required by the runtime (vulkan-spirv-fb-ptr) and sets the Indirect flag on the descriptor set layouts on the exported executable variant functions.
Next steps on the runtime side are to route iree_hal_command_buffer_push_descriptor_set calls down a special parameter buffer path when the IREE_HAL_DESCRIPTOR_SET_LAYOUT_FLAG_INDIRECT flag is set (along with some other goo), while on the compiler side the codegen lowerings will need to inspect the layout flags and when IREE::HAL::DescriptorSetLayoutFlags::Indirect lower to the above SPIR-V binding style.

(there's a lot I don't like about this approach but it's not worth me stalling any longer - we've got enough other cleanup around SPIR-V executables and extensions pending and this is such a big switch that it may help ground out discussions on next steps by being so hideous :)

Thanks @benvanik for the details! @kuhar will help to flesh out the SPIR-V part:

As we chatted, the majority of the changes can happen in the ConvertToSPIRVPass right now--there is a pre-step there analyzing subspan ops and creating SPIR-V global variables for them. We can add the logic there to perform the extra indirect indexing via the argument/parameter buffer.
The addressing model needs to be PhysicalStorageBuffer64--the spv.module op is directly created in ConvertToSPIRVPass too so should be direct to modify.
And we need to plumb through support for the extension and associated capabilties in the upstream MLIR and IREE SPIR-V CodeGen side.


Add a Decroate() method to GameModels

With this it is a bit easier to add a decorator (GoF) to an existing model. It replaces the model in the end, but the original model is passed to teh decorator.
With this tow mods could override the same model but different methods, but if the override the same method than it would depend (probably) on the load order what happens.
One question is: Should there be decorators for the common models already or must the user roll their own? Currently there are none, but could be added more-or-less easily.
Example:
In OnGameStart:
campaign.Decorate((WorkshopModel m) => new GuildWorkshopModelDecorator(m));

The decorator:
public class GuildWorkshopModelDecorator : WorkshopModel
{
    private readonly WorkshopModel model;

    public GuildWorkshopModelDecorater(WorkshopModel model)
    {
        this.model = model;
    }

    // other methods omitted  for brevity...

    public override int GetMaxWorkshopCountForPlayer()
        => int.MaxValue;
}

Beautiful work, cheers, I'll make sure to include an example of this in the docs.
I think letting people roll their own decorators is probably reasonable, given that there's quite a lot of them.
I think no matter the way any one developer implements this, collisions when it comes to models are inherently unavoidable. A system where all of the model types are pre-wrapped and can have functionality hot swapped / chained at runtime by developers would probably be the only way to solve it in a sufficiently solid way.
Perhaps something like this lazily written example where devs add their own functionality to the priority queue.
public class BannerLibXpModel : GenericXpModel
{
    private List<(int, Func<Hero, float>)> ExamplePriorityQueue;

    public override float GetXpMultiplier(Hero hero)
    {
        return ExamplePriorityQueue
            .OrderBy(x => x.Item1)
            .Select(x => x.Item2(hero))
            .Aggregate((x, y) => x + y);
    }
}

No matter what is done, devs are going to want ultimate control over the domain their mod acts over though, so I think we're probably at the limit with what we can do really.
I'm open to other solutions if you have em though.


embed_html(..., offline=True) cannot download index.js from unpkg.com for ipyvolume>0.5.0

Attempting to use the offline=True keyword argument in ipyvolume.embed.embed_html() creates an error when trying to download the index.js file from unpkg.com. This is true for versions of ipyvolume >= 0.5.0
I assume this is just a case of the index.js files needing to be uploaded to unpkg.com for ipyvolume 0.5.0 and 0.5.1 (since the index.js file is available for 0.4.5 and 0.4.6, etc.)
import ipyvolume as ipv
import numpy as np
x, y, z = np.random.rand(3, 100)
fig = ipv.scatter(x, y, z)
ipv.embed.embed_html('test_offline.html', fig, offline=True)

And this is the error message displayed in the jupyter notebook:
Downloading<EMAIL_ADDRESS>
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
<ipython-input-8-9e602383e0fe> in <module>()
----> 1 ipv.embed.embed_html('test_offline.html', fig, offline=True)

/anaconda3/envs/ipyvolume/lib/python3.6/site-packages/ipyvolume/embed.py in embed_html(filepath, widgets, makedirs, title, all_states, offline, scripts_path, drop_defaults, template, template_options, devmode, offline_cors)
    176             rel_script_path += '/'
    177
--> 178         fname_pyv = save_ipyvolumejs(scripts_path, devmode=devmode)
    179         fname_require = save_requirejs(os.path.join(scripts_path))
    180         fname_embed = save_embed_js(os.path.join(scripts_path))

/anaconda3/envs/ipyvolume/lib/python3.6/site-packages/ipyvolume/embed.py in save_ipyvolumejs(target, devmode, version, version3js)
     49         shutil.copy(devfile, pyv_filepath)
     50     else:
---> 51         download_to_file(url, pyv_filepath)
     52
     53     # TODO: currently not in use, think about this if we want to have this external for embedding,

/anaconda3/envs/ipyvolume/lib/python3.6/site-packages/ipyvolume/utils.py in download_to_file(url, filepath, resume, overwrite, chunk_size, loadbar_length)
    159     response = requests.get(url, stream=stream, headers=resume_header)
    160     # raise error if download was unsuccessful
--> 161     response.raise_for_status()
    162
    163     # get the size of the file if available

/anaconda3/envs/ipyvolume/lib/python3.6/site-packages/requests/models.py in raise_for_status(self)
    937
    938         if http_error_msg:
--> 939             raise HTTPError(http_error_msg, response=self)
    940
    941     def close(self):

HTTPError: 404 Client Error: Not Found for url<EMAIL_ADDRESS>
Maybe related: https://github.com/maartenbreddels/ipyvolume/issues/175

Yes, they are related, it seem to be an issue with unpkg: https://github.com/unpkg/unpkg.com/issues/134
I have to see if I can easily move to use jsdelivr.

I see - thanks for the quick response!
I'm sure we can make do with some kind of workaround for now, I'll let you know if I'm not able to do that. Too bad it's not quite as simple as just dropping back to 0.4.6 (the font-awesome location changed since then).
Thanks!

Forgot to say for anyone else with this problem: that workaround is using the development installation (setup instructions are in the README), and the devmode=True keyword argument in the ipv.embed.embed_html() function.

The issue on unpkg is resolved.

Just wanted to let you know I've observed more issues with unpkg - could be a recurrance of whatever this problem was last time.


Get on the Brave referral programs

To get compensated when brave users sign up, and when users browse our sites:
Links are brave.com/refer and publishers.brave.com respectively.

.


feat: Add RemoteSLURMCluster & RemoteSLURMJob #502

What does this PR change?
It adds support for RemoteSLURMCluster which spins up dask-workers through the SLURM REST API instead of through subprocess.Popen.
Why is this useful?
In environments where the python process doesn't have direct access to the SLURM CLI (f.e. a docker container) we still want to be able to use dask-jobqueue to interface with SLURM.
Open questions

I've been able to write some minimal end-to-end test. The implementation largely inherits from SLURMCluster, which is sufficiently tested. Where could we invest more in tests?
I've not been able to test this with a workin SLURM REST API cluster, since I am still waiting on my colleagues to activate it. I'm opening this PR early to receive early reviews.

Thank you for allowing me to contribute 🙇♂  @jacobtomlinson @guillaumeeb
Resolve #502

TODO: Add SLURM REST API to ci/slurm and remove mocking from test_end_to_end. We can use the slurm cluster instead.
(cc @jacobtomlinson I think this is a better idea)

Given that you've already written the mocked test there's probably little harm in leaving it in, but adding a CI test would be good.
(Also don't worry about squashing and force pushing, we will squash on merge. It actually makes it harder to review because we can't easily see what has changed between reviews)

Given that you've already written the mocked test there's probably little harm in leaving it in, but adding a CI test would be good.
(Also don't worry about squashing and force pushing, we will squash on merge. It actually makes it harder to review because we can't easily see what has changed between reviews)

I unfortunately have to remove this test because the aresponses dependency cannot be installed by conda. In any case I'm currently working on making the tests work with the ci/slurm docker environment. That should provide plenty of test coverage for us to accept this PR in good conscience.

@lesteve Would you have any tips on how I can update the ci/slurm integration test docker setup and have it be used during integration testing of slurm?
For this PR to pass we would need;

Update slurm version to 20.11 (the version the REST API was released in)
Add slurmrestd somehow to have an endpoint we can send requests to in the tests.

Update slurm
I pushed an image to dockerhub and added it to the FROM section in the ci/slurm/Dockerfile

slurmrestd
Perhaps you know, off the top of your head, where I should look for configuration examples to configure this? any direction you'd prefer would be appreciated.

I'll make time to look at this today 👍

Please stop force pushing 😂. I can't see what changes you are making.

Please stop force pushing 😂. I can't see what changes you are making.

I'll try my absolute best. 😬 it's part of my normal workflow to rebase/retouch commits to make them look nice and to the point.

Update slurm version to 20.11 (the version the REST API was released in)

As you might see, we were using images from https://github.com/giovtorres/slurm-docker-cluster, but it has not been updated on dockerhub for two years. The simplest thing to do here would probably to copy paste the Dockerfile from this project, update Slurm version, and to build and push the image through our CI to daskdev docker user. It could then be used for running Python tests.

Add slurmrestd somehow to have an endpoint we can send requests to in the tests.

Never configured a Slurm cluster, and I'm not sure any maintainer here has... So you'll have to search or try, since https://slurm.schedmd.com/rest.html does not seem to give all needed information... In the end this is probably some lines of configuration, and one more daemon to launch in the docker-compose setup.

Alternatively I could update this PR with unit tests only (with mocking) and we tackle those integration tests in another PR (due to ci.yml action needing to build the newer slurm docker image later).

This is an alternative, as updating CI here might be a bit complicated, and we've already done it. But It is a really important thing to do though.

@jacobtomlinson I got it to work with Unix sockets as authentication. Slurm does not accept unauthenticated requests. Let me know what you think of this implementation.
Something inside Slurm is making it impossible to share the unix socket via the docker-compose file (it errors as permission error), therefore we have to run those tests in the slurmrestd container directly.
JWT authentication does not seem to work yet, tomorrow I'll investigate if the latest version 20-11-8-1 can make it work. 20-11-4-1 (current version in Dockerfile) seems to be broken.
@guillaumeeb thank you for your insights.

The simplest thing to do here would probably to copy paste the Dockerfile from this project, update Slurm version, and to build and push the image through our CI to daskdev docker user. It could then be used for running Python tests.

I had the same thought and made a new slurm docker image definition for us. :)

In the end this is probably some lines of configuration, and one more daemon to launch in the docker-compose setup.

You were right 👍 however I must say that the slurm docs leave a lot to be desired. I ended up reading their code to make sense of the configuration/requests.

I've managed to also add the JWT HTTP configuration - however the integration tests fail. Given that the posted payload is correct it leads me to believe that slurmrestd doesn't actually have JWT support yet.
The NEWS in slurm mentions a couple of bugfixes in version later than 20-11-4-1 (the version currently installable in ubuntu) https://github.com/SchedMD/slurm/blob/master/NEWS. So I'll have to investigate if something is wrong with the token generation/authentication or this is a bug in Slurm.

I've managed to also add the JWT HTTP configuration - however the integration tests fail. Given that the posted payload is correct it leads me to believe that slurmrestd doesn't actually have JWT support yet.
The NEWS in slurm mentions a couple of bugfixes in version later than 20-11-4-1 (the version currently installable in ubuntu) https://github.com/SchedMD/slurm/blob/master/NEWS. So I'll have to investigate if something is wrong with the token generation/authentication or this is a bug in Slurm.

I figured out JWT authentication, now still need to figure out why we're seeing;
Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7f39d33c7048>, 1059921.899742882)]']
connector: <aiohttp.connector.UnixConnector object at 0x7f3a0578d7f0>
Task was destroyed but it is pending!
task: <Task pending coro=<AdaptiveCore.adapt() done, defined at /opt/anaconda/lib/python3.6/site-packages/distributed/deploy/adaptive_core.py:179> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f39d34bc5e8>()]> cb=[IOLoop.add_future.<locals>.<lambda>() at /opt/anaconda/lib/python3.6/site-packages/tornado/ioloop.py:688]>

🤔

@jacobtomlinson @lesteve this is now ready for a good review, all the features explained in the issue are in. Please let me know your thoughts 🙇 your thoughts help immensely.

I found a bit of a snag working with aiohttp. Due to SLURM responses sometimes (most of the time?) not being parsable the http-parser library fails on reading the response. Unfortunately I have not been able to deduce why these responses cannot be read.
In any case setting AIOHTTP_NO_EXTENSIONS as suggested by https://github.com/aio-libs/aiohttp/issues/1843 fixes this. I have however no idea how we should control for this in _submit_job. 🤔

@AlexanderVanEck I see you push some commits about a month ago. Do you still feel this PR is ready to review?

+1

This PR hasn't has any activity for a long time so I'm going to close it out.


update docs

for now, for single instance models, only pb files from sleap 1.2.9 work, not the most recent versions. Would be great to have this in the documentation (I learned the hard way by finding an issue after my training failed)

thanks emily for bringing this to our attention!

@emilyjanedennis @banchan86 this has been fixed in #21 and will be released soon together with updated documentation.


Docs / examples on project a mixamo fbx mesh on 2d image?

Hi, just wonder is there any Docs / examples on project a mixamo fbx mesh on 2d image?

Please use the discussions tab for general questions.


Fix MongoDB Bson properties with dots

This is a fixed version of the following fork - https://github.com/DarkXoro/serilog-sinks-mongodb
Proposed in PR - https://github.com/serilog/serilog-sinks-mongodb/pull/45/ (not merged)
Previous issue (closed but not resolved) - https://github.com/serilog/serilog-sinks-mongodb/issues/29
New issue - https://github.com/serilog/serilog-sinks-mongodb/issues/58

Hi @nblumhardt , I'm wondering if this this repo is still maintained?

Hi Yuriy, thanks for the nudge. :+1:
Not really, unfortunately - it's very minimally maintained.
This problem is best addressed by passing a suitable ITextFormatter in the WriteTo.MongoDB() call. It's the job of the formatter to determine how properties are represented in the event documents.
One option would be to create an ITextFormatter specifically for MongoDB, e.g. by copying the basics from https://github.com/serilog/serilog-formatting-compact/blob/dev/src/Serilog.Formatting.Compact/Formatting/Compact/CompactJsonFormatter.cs.
This could be linked from the README, or if it's widely-used, included in this package.
(Another option, entirely within the sink, would be to copy and modify the LogEvent before passing it to the formatter.)
Using RegEx for this seems like a last resort, given the usual role of ITextFormatter to do this job.

@nblumhardt FYI, recently, I did a bit of reimagining on this project. To me, it's grown too complex due to its "double conversion" ways (to Json and then to Bson). I made a version that directly stores the logging output in mongo using native Bson: https://github.com/ChangemakerStudios/serilog-sinks-mongodb/tree/dev
I prefer the more native output, and it solves some of the issues with the current "formatting" strategy, but it's a radical change -- so I'm not entirely sure what to do with the changes:

@Jaben sounds interesting... I keep forgetting that you still use this one - a refresh could be what this sink needs?
One possibility would be to introduce new overloads for WriteTo.MongoDB() that don't accept ITextFormatter, and use the new mechanism under the hood. The old overloads could then have all the parameters non-defaulted, so that existing code would still compile and link successfully, while the new overloads would be picked by the compiler unless the formatter` argument is supplied?
Just an idle thought!

@nblumhardt I don't use it, really -- Seq is too good ;) I do use MongoDb a lot, though.
So I added MongoDbBson() extensions which output using the Bson structured data. The extensions also have a simpler configuration system (fluent). The current sink extension functions are huge -- figured best to keep them API compatible, though so there are fewer surprises on upgrade.
I'll put in a PR for it.

Btw @Jaben , here's what I come up with (using your fork) - updated LogEntry.cs like so:
public BsonDocument Exception { get; set; }

Exception = logEvent.Exception == null ? null : SanitizeDocumentRecursive(logEvent.Exception.ToBsonDocument()),

private static BsonDocument SanitizeDocumentRecursive(BsonDocument document)
{
    var sanitizedElements = document.Select(e => new BsonElement(
        SanitizedElementName(e.Name),
        e.Value.IsBsonDocument ? SanitizeDocumentRecursive(e.Value.AsBsonDocument) : e.Value));

    return new BsonDocument(sanitizedElements);
}

This fixes an issue with exception serialization in test I've mentioned. What do you think about this?

Actually doing the same in my fork insted of regex matching or custom formatter does the same thing.
file MongoDBHelpers.cs, last line in method GenerateBsonDocuments
return bson["logEvents"].AsBsonArray.Select(x => SanitizeDocumentRecursive(x.AsBsonDocument)).ToList();

Btw @Jaben , here's what I come up with (using your fork) - updated LogEntry.cs like so:
public BsonDocument Exception { get; set; }

Exception = logEvent.Exception == null ? null : SanitizeDocumentRecursive(logEvent.Exception.ToBsonDocument()),

private static BsonDocument SanitizeDocumentRecursive(BsonDocument document)
{
    var sanitizedElements = document.Select(e => new BsonElement(
        SanitizedElementName(e.Name),
        e.Value.IsBsonDocument ? SanitizeDocumentRecursive(e.Value.AsBsonDocument) : e.Value));

    return new BsonDocument(sanitizedElements);
}

This fixes an issue with exception serialization in test I've mentioned. What do you think about this?

That's great! Thanks for finding and fixing this issue. I've added your fix. 👍

Sounds good, if you open a PR then this one can be closed.

Sounds good, if you open a PR then this one can be closed.

Done. It's here: https://github.com/serilog/serilog-sinks-mongodb/pull/60

Great, closing this one!


SGF FF[1]-FF[3] support

Lowercase letters should be allowed in property identifiers in order to support SGF FF[1]-FF[3].
This is related to this closed issue, where I left a comment:
https://github.com/Kashomon/glift/issues/101
For SGF files with FF[1] through FF[3], lowercase letters are allowed in the property identifier names, but are simply ignored. Hence, the property id "CoPyright" should simply be interpreted as "CP" for such files. Also, note that if an SGF file omits the FF property, FF[1] should be assumed by default.
The guide for converting older file formats discusses this issue:
http://www.red-bean.com/sgf/converting.html

Yes I agree. Thanks for the link. I agree that Glift should support FF3


function dispatch_now don't exist

Versions

PHP version: 7.0
Laravel version: Lumen 5.5
Package version: 3.1

Description
When use "Chunk reading",  unfortunately cause to exception as follows
Call to undefined function Maatwebsite\Excel\dispatch_now()

You should add these 2 methods to your helpers:
if (! function_exists('dispatch')) {
    /**
     * Dispatch a job to its appropriate handler.
     *
     * @param  mixed  $job
     * @return \Illuminate\Foundation\Bus\PendingDispatch
     */
    function dispatch($job)
    {
        if ($job instanceof Closure) {
            $job = new CallQueuedClosure(new SerializableClosure($job));
        }
        return new PendingDispatch($job);
    }
}
if (! function_exists('dispatch_now')) {
    /**
     * Dispatch a command to its appropriate handler in the current process.
     *
     * @param  mixed  $job
     * @param  mixed  $handler
     * @return mixed
     */
    function dispatch_now($job, $handler = null)
    {
        return app(Dispatcher::class)->dispatchNow($job, $handler);
    }
}

https://github.com/laravel/lumen-framework/issues/733
I'll leave the ticket open, so we can improve this for Lumen support

@patrickbrouwers thanks

Will be fixed in next release


Ideas for funny hedgehog cartoons

Every two weeks, @lottiecoxon is creating an awesome little cartoon to feature in our newsletter, and on social media etc. Here's a recent one.

We need more ideas for jokes, memes – anything we think our audience will enjoy.
Please share them here.

Throwing this in as an option for next week. A version of this...

One additional route (bit easier than writing jokes) could be topical hogs from the latest news in tech? e.g. something around Elon/Twitter

One additional route (bit easier than writing jokes) could be topical hogs from the latest news in tech? e.g. something around Elon/Twitter

Yeah, definitely a posibility, but trickier given we're doing these every two weeks.

Throwing this in as an option for next week. A version of this...

I get it, as someone who also uses a Windows machine sometimes.
That's not everyone here 🤣

@lottiecoxon So we should probably settle on something so you can get working on this.
I'm thinking, as per the Slack thread, perhaps we should do our own version of the Drake meme?
One idea I just came up with... thoughts?

i think going timely will make these not feel tired

@lottiecoxon Ok, idea for this week.
It's a riff on the people mis-pronouncing SQL as "es-queue-el".
We could do it a couple of different ways...

Two hedgehogs chatting, one reacting angrily / surprised / shocked and the caption reads "It's pronounced "es-queue-el" right?

Same basic layout but with speech bubbles, one saying "It's pronounced "es-queue-el" right?" and the other saying "We can't be friends anymore"

I've also thought of a third where's two hedgehods on a date, which might be fun.
As usual, plan is for newsletter to go out around 5pm BST on Wednesday.

I like the second idea with 'we can't be friends anymore'

Maybe a comic around how product analytics isn't evil tracking everyone but actually helps make the products you're using better & it's not about selling your data for advertising. Source of the idea: sometimes when I mention that I work on product analytics people have a negative reaction.

this is a cool idea - could do an expectation vs reality but with product analytics?

could be a cool blogpost too, if we don't already have one potentially

May be worth following this subreddit for inspo.

At Charles' suggestion, we're canning the funny cartoons mainly for time reasons. We're just going to use a new hedgehog avatar in each newsletter. Way easier for both me and Lottie.


Adding

Next up is basic arithmetic. Add this function as a library and append to interpret.rb

Adding is done


MQTTNet Server - Communication with clients from different subnet?

Describe your question
Hello,
We have a MQTTNet Server set up in our local network which contains multiple subnets. The server does not seem to recognize MQTT packets that are sent from clients from different subnets, even though the packets arrive at the host PC.
e.g. when building the broker with IPAddress lets say <IP_ADDRESS> (by using WithDefaultEndpointBoundIPAddress), all packets with source and destination on the same sub are handled successfully.
If the destination is lets say on <IP_ADDRESS> the broker does not respond even if he was build with the same IPAddress.
Are there any other broker build options we should pay attention to?
Which project is your question related to?

Server

No there are no other properties related to this. The bound IP address is used for that. Probably the documentation for .net has more information about how it works for other subnets.


Bybit Perpetual - Upgrade API to V5

Description
Bounty to update bybit_perpetual connector API to V5
Bybit had a newsletter where is officially informed that they will suspend the use of our old version openAPI service. This decision was made in order to reduce unnecessary maintenance and cost, and also to ensure the stability bybit platform.
Users can migrate to V5 API, which has the full ability.

Details regarding to offline plan:

Offline market Rest API endpoints and public websocket stream on 16 OCT 2023

USDC Option V1 Market Data Rest API Endpoints , Public Topics
USDC Perpetual V1 Market Data Endpoints , Public Topics
Spot V1 Market Data Endpoints , Public Topics V1 , Public Topics V2
Inverse Perpetual V2 Market Data Endpoints , Public Topics
Inverse Futures V2 Market Data Endpoints , Public Topics
USDT Perpetual V2 Market Data Endpoints , Public Topics

Offline private Rest API endpoints and private websocket stream on 30 OCT 2023

USDC Option V1 Account Data Rest API Endpoints , Private Topics
USDC Perpetual V1 Account Data Endpoints , Private Topics
Spot V1 Account Data Endpoints , Wallet Data Endpoints , Leveraged Token Endpoints , Cross Margin Trading Endpoints , Private Topics
Asset V1 Transfer Data Endpoints , Withdraw and Deposit Endpoints
Inverse Perpetual V2 Account Data Endpoints , Wallet Data Endpoints , Private Topics
Inverse Futures V2 Account Data Endpoints , Wallet Data Endpoints , Private Topics
USDT Perpetual V2 Account Data Endpoints , Wallet Data Endpoints , Private Topics
Thank you for your continued support and understanding.
Best regards,
The Bybit Team

Bounty
Sponsor: Funded by Hummingbot community sponsors
Bounty Amount: 470 USDC
Developer Payment: 376 USDC (80%)

hi @nikspz can i get this one please?

Assigned to @tomasgaudino, please be informed:

Since you’re assigned to this bounty exclusively, please keep us informed on your progress over the next few days
Please review and follow contribution guidelines (https://hummingbot.org/developers/contributions/)
Please review the checklist after PR submitted (https://hummingbot.org/developers/contributions/#checklist)
Let us know if you don’t have time, and we will unassigned and keep it open for other devs
Only one dev can be assigned to one bounty

Thanks for your contribution!

Thanks! I'll notify any update on this topic

https://github.com/hummingbot/hummingbot/pull/7196

should we close this one? @nikspz

@cardosofede I think there's some updates from bybit side, I created this one and asked @tomasgaudino to check if he could use the connector https://github.com/hummingbot/hummingbot/issues/7267

For me I currently failed to connect updated account (newest one)
For the previous one I was not able to place orders

 For me I currently failed to connect updated account (newest one) For the previous one I was not able to place orders

https://github.com/hummingbot/hummingbot/issues/7267
Fix merged. Update API to V5 (for unified trading accounts) available in the latest development branch (dev-2.2.0) and will be fully available in the next release 2.2.0

thanks guys!


Merged the pull-request by Brian Cox into latest code

Merged the pull request for     adding wildcard search by @bricox into latest code. I also need the same functionality in my project.
let me know if there are any changes needed.

I was waiting for @bricox to update the README.

I'm working in this feature.


[Feature Request] Add support for selecting global fvm flutter version mode.

Feature description.
I'm dealing with several projects at the same time. Sometimes, I need to test them in different flutter SDK versions which thanks to your amazing work.  However, sometimes I need to use the same flutter SDK version to run the projects.  So is that possible to add a feature that could help set all the projects to use the same flutter SDK version, like fvm global 1.17.1. To cancel the global mode then use a command like fvm disglobal or something.
Describe the solution you'd like
Could you please let me know if it's possible to add a global variable someplace, which will be checked when using fvm flutter SDK? If the user sets the global mode then we can set the global variable to true and save the SDK version user inputs. After that, every time when using fvm flutter SDK we will check the variable to decide whether we use the SDK version saved someplace.
Alternative solution
A clear and concise description of any alternative solutions or features you've considered.
Also, Can you check another possible solution? I've noticed fvm use version_num --force can set all the flutter projects which never used fvm tool to use the specified SDK version. So, maybe we can delete the .fvm folder in all the flutter projects and then use the fvm use version_num --force. But how to find all the projects using fvm? It's a problem I'm not sure.
Thanks for your work again.

@bytes-stack https://github.com/leoafarias/fvm#install-a-sdk-version on this there is a section called set global version let me know if that is what you are trying to do.

@bytes-stack https://github.com/leoafarias/fvm#install-a-sdk-version on this there is a section called set global version let me know if that is what you are trying to do.

Thanks for your quick response and solution. But I still have some questions after I tried the method.

I set the global mode and added the environment variable then I used fvm list, everything is OK. I found the version I set for global mode had the identifier (global). It's pretty good! However, I noticed that I still used the SDK version which I set in the command fvm use <version> when I tried fvm flutter run, not global version. Is that correct?
Also, as far as I'm concerned, whether we need to add a command to disable the global mode, after which we can use the previous SDK version.

Please let me know if I got any mistakes.


kind: 0.2.1 -> v0.3.0

Motivation for this change
Upgrade kind from 0.2.1 to v0.3.0.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] other Linux distributions

[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Fits CONTRIBUTING.md.

cc @gilligan and @thoughtpolice who may be interested in this

@GrahamcOfBorg build kind

LGTM.


Update MD file on new branch 2017-30-12_21-11-45

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

split/updateTestRef.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


OAuth doesn't work for Google accounts in MakeCode on some version of Android

Describe the bug
https://github.com/microsoft/pxt-microbit/issues/5590

@microbit-matt-hillsdon can reproduce this.

Seems to be Google policy to disallow in webviews. I don't understand why it works in the iOS app.
https://developers.googleblog.com/2021/06/upcoming-security-changes-to-googles-oauth-2.0-authorization-endpoint.html

I can also reproduce this for me on my Android 8.1 device even though it lets me get through the whole entering password flow etc. 403: disallowed_useragent

Reported in support ticket https://support.microbit.org/helpdesk/tickets/74336 (private)

Reported in support ticket https://support.microbit.org/helpdesk/tickets/74736 (private)

Reported from an iOS device in an in-person workshop on the 16th of July.

What is the plan here? Are you using native webview in android?
May be this link helps? https://developers.google.com/identity/android-credential-manager

Reported in support ticket https://support.microbit.org/helpdesk/tickets/83590

This is because the use of unsupported versions of browser/webview and google doesn't allow to login. We can't do much here.

@eanders-ms can you pitch in here?

On the device where it's failing for MakeCode, does Google auth work for other 3rd party (non-Google) apps?

Seems to be Google policy to disallow in webviews.
https://developers.googleblog.com/2021/06/upcoming-security-changes-to-googles-oauth-2.0-authorization-endpoint.html
This might help - I haven't read it yet!
https://developer.android.com/identity/sign-in/credential-manager-webview

We may need to leverage the native Android app to manage the authentication process, which would pass the final redirect URL back to the WebView. From there we can get the auth token and cookie. Custom Tabs might be a suitable API for handling the oauth flow from the native side.

Reported in support ticket https://support.microbit.org/helpdesk/tickets/83933 (private)


Remove standard values for environment vars in docker file

Breaking change

Proposed change

Fixes the problem of double or randon values if users add vars with different case
Type of change

[ ] Dependency upgrade
[x] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (which adds functionality to an existing integration)
[ ] Breaking change (fix/feature causing existing functionality to break)
[ ] Code quality improvements to existing code or addition of tests

Additional information

This PR fixes or closes issue: fixes #
This PR is related to issue:
Link to documentation pull request:

Checklist

[ ] The code change is tested and works locally.
[ ] Local tests pass. Your PR cannot be merged unless tests pass
[ ] There is no commented out code in this PR.
[ ] I have followed the [development checklist][dev-checklist]
[ ] The code compiles without warnings (code quality chek)
[ ] Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

[ ] Documentation added/updated for www.home-assistant.io

Codecov Report

Merging #643 (1a1c837) into dev (0bd811e) will increase coverage by 0.06%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##              dev     #643      +/-   ##
==========================================
+ Coverage   80.08%   80.15%   +0.06%
==========================================
  Files         111      111
  Lines        2898     2898
  Branches      311      311
==========================================
+ Hits         2321     2323       +2
+ Misses        450      447       -3
- Partials      127      128       +1

Flag
Coverage Δ

unittests
80.15% <ø> (+0.06%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...mon.HassClient/Internal/HomeAssistantConnection.cs
85.71% <0.00%> (-0.96%)
:arrow_down:

...mon.HassClient/Internal/Net/WebSocketClientImpl.cs
69.44% <0.00%> (+8.33%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0bd811e...1a1c837. Read the comment docs.


[WIP] Lidar aggregation and visualization

Add a utility class to aggregate lidar data. Modify drake-visualizer to use said class to visualize lidar data.
This doesn't handle bidirectional scans yet, nor does it copy all of the available data when building a vtkPolyData from the aggregated data. (On second thought, maybe this should be optional?)

Note: requires RobotLocomotion/drake#1699 and building director with USE_DRAKE=ON.

Looks great!  A couple optimizations that could be added (probably not critical yet):

ensure extractPolyData is only only called if there is new data
move the work performed by extractPolyData off the main thread

I will hold off merging this while the lcm type discussion is ongoing.  I think it will be easy to update this branch once everything is settled, very nice.

ensure extractPolyData is only only called if there is new data

Hmm, will think about how to do this. Should be do-able, but:

data(...) will have to either also return a 'last modified' time or take a 'since' time and return nothing if last <= since. (Latter is probably more efficient.)
extractPolyData(...) will definitely need to take a 'since', as I don't want to enforce a one-to-one mapping of consumers to aggregators.

move the work performed by extractPolyData off the main thread

This requires either spinning up an additional new thread and moving the update loop into the same, or else somehow making the call able to execute asynchronously.


Resize Image - Callback

This feature request is for extension: _ (storage-resize-images)
It would be great to see a callback included in the extension to be able to get information about the new image that has been created.
For example,

you have the extension configured on a bucket
you upload the original image to the bucket and await a callback
when the new resized image has been created the callback includes information about that new image

This would make the extension more useable and make it a lot easier to know when the image has been created, and easier to go fetch it.
Thanks.

Hey @ltbmedia
Just so I understand correctly, do you mean a data record/list of resized images? So, if I upload an image it gets resized but also a records gets added to a database somewhere?
Thanks!

Hi there,
That would be one solution I guess.
But a cleaner approach in what I was asking would be -->
`storageRef.put(ORIGINAL_IMAGE).then((snapshot) => {
  // as this image was uploaded to a bucket with the resize-function configured
  // wait for callback to let us know the resized image(s) have completed
 // include data about the new resized image(s) so we can store & fetch

});`
Hope this helps,
Thanks

Hm I don't know where I need to write, but I agree with you.
Currently I am using React Native app with firebase inside, We added resize images extension to the project and found that when we got resized image we can't get url for it because it don't have access token(download URL) in Firebase Storage, while the parent image (which is not resized) have it.

Hi Aliaksei,
You can get the resized image by using ->
let gsReference = this.storage.refFromURL('gs://your-bucket-name/' + filename + '_200x200' + fileExtension)
You can only call this and get the resized image from the bucket once the resize-function has completed, otherwise you get an error because the new resized image does not exist yet - only workaround is using a delay before calling that which isn't a great solution.
This is why we need a callback in the function to let us know that the new image exists, and has been completed. We also need the new image information so that we don't have to string together a reference url!
Hope this helps

@ltbmedia , very strange.

I added image directly to storage for testing resizing... and original image has download url, but resized not...
But on videos here , resized images have donwload URL... https://www.youtube.com/watch?v=zrpydDVFa3I&t=263s , https://www.youtube.com/watch?v=Sx7ijOnob54 , https://www.youtube.com/watch?v=9o8oenmUBYg&t=17s .
I will try to use your workaround, but for me it looks like a bug....

@AliakseiPaseishvili the download url is only added after a call to getDownloadUrl() is made, as mentioned here

Hi All - this is an interesting topic and is relatable across other Extensions. This is being reviewed internally for now, we'll let you know as soon as possible on how to correctly handle such a use-case. Thanks!

A callback would be very useful.  I have done this as below but I found sometimes it takes longer to resize than 5 seconds so I increased it to 10 seconds.  This is still no guarantee it won't break.
Is there no way to get a notification/callback of some kind?
 setTimeout(() => {
    this.downloadURL = this.storage.ref(thumbPath).getDownloadURL();
}, 5000);

@ltbmedia What if the extension can optionally publish to a PubSub topic after it finishes resizing the image? Would that solve your use case?

Hi @laurenzlong, I guess if that is a way of knowing when the resize has completed then that would solve the initial issue.
Would the published message also contain information about the resized image so that an additional call does not have to be made?
Thanks

@ltbmedia What kind of information would you want?

as a workaround, I used firebase function that works for this case :  functions.storage.object().onFinalize(callback) - here I wrote logic, where we check what file is uploaded  and if it is an image - I check is it resized or not - if yes  - I am calling getSignedUrl method for our path and then add it to the firestore, where it should be.

@laurenzlong downloadURL would be ideal or some reference to the image

@ltbmedia If your desire is to simply know how to download the newly resized image, and not do any additional processing, then perhaps https://github.com/firebase/extensions/issues/11 is more suitable. As well, it would not be a good idea for us to auto generate download URLs on the resized image, since you should be in control of the access model of your file, there's a good explanation of the different types of download URLs in this SO comment, and we would not want  to be making the decision for you of what kind of a download URL to generate.

@laurenzlong I don't think https://github.com/firebase/extensions/issues/11#issue-499273584 would solve this issue as there is still no "notification" of the change.  Also I delete the originals.
If @AliakseiPaseishvili's solution works then it would solve the problem for me I think.

Hi @laurenzlong, the biggest issue is not knowing when the resized image has completed - that’s why a callback or function as you suggested is needed.
If a downloadURL or other reference cannot be supplied I can use my own method to fetch the file, but at least I would know that it is ready.

Thanks for the additional context @ltbmedia! We'll take this into consideration as we plan out our roadmap.

Hi @ltbmedia thanks so much for this feature request!
We've reviewed this as a team and think that the best solution long term is better lifecycle management for an extension: a way to understand the status of an extension (when it's executing and when it has finished) and a way to trigger an action based on a status change from an extension. We're thinking through this problem long term and to @Ehesp's point, this is a feature that would benefit many extensions.
Unfortunately, in the short-term, there is no easy fix for this. I'm closing this for now but we will be tracking this internally in our long term roadmaps.
Thanks again for the feedback!
Kara

Hi @karayu, this is great news and look forward to hearing any progress on this 👍

I have found the solution to it like i have to comment out the line which deleting the access token in the cloud function and now i have access to token of image after resizing but there is problem that if i have to change the configurations of extension the code is updated in cloud function so i have to change this line every time i change the configuration. i have attached the snippet of code which i have to change.
this solution works for me well but its not a solid solution.
its simple process

after installing the extension go to functions
then select view details usage its opens a screen from where you have to select source
form source find file named FUNCTIONS/LIB/INDEX.JS , Go to line 152 then comment this out
code on line 152 : delete metadata.metadata.firebaseStorageDownloadTokens;

@Mian-Zaid You might be interested in tracking this PR which deletes that token: https://github.com/firebase/extensions/pull/279

html solution
<picture>
  <source media="(max-width: 799px)" srcset="elva_800x800.jpg">
  <source media="(min-width: 800px)" srcset="elva.jpg">
  <img src="elva.jpg" alt="Elva">
</picture>

anyone found the solution for this? Any callbacks to download the resized image?

I still can't find a viable solution to this issue. As many others have requested, it would be ideal to have a callback with the resized image's metadata that is invoked when the image processing is complete. Users would have complete control of what to do with the new image when it is done processing or they can simply ignore the callback.

@TravisJN
I have an Angular solution which involves polling for the file to see when it exists.  Not ideal but it works ok.
ngOnInit() {
    this.bucket = this.afStorage.storage.app.storage(environment.storageBucketChatFiles);
    // this.defaultBucket = this.afStorage.storage.app.storage();
    this.useDateInFileName = this.data.useDateInFileName;
    this.pathDirectory = this.data.pathDirectory;
    this.entityId = this.data.entityId;
  }

  toggleHover(event: boolean) {
    this.isHovering = event;
  }

  startUpload(event: FileList) {
    // The File object
    const file = event.item(0);
    if (file.size > (environment.imageUploadSizeLimit * 1024 * 1024)) {
      this.notificationService.alertError(`Please ensure that the size < ${environment.imageUploadSizeLimit}Mb`, 'File Size to Large');
      return;
    }
    if (this.supportedContentTypes.includes(file.type)) {
      // Image will be resized
      this.sizeNameSuffix = '_800x800';
    }
    // The storage path
    const path = this.generateStoragePath(file);
    // The main task
    let storageRef: firebase.storage.Reference;
    try {
      this.spinner.show('spinner-upload');
      storageRef = this.bucket.ref().child(path);
      if (this.sizeNameSuffix.length > 0) {
        this.fileUpload.storagePath = this.addResizeExtensionToName(path);
      } else {
        this.fileUpload.storagePath = path;
      }
      this.fbTask = storageRef.put(file);
      this.fbTask.catch(err => {
        console.log(err);
        this.notificationService
        .alertError(`Please check the type of the file and that the size < ${environment.imageUploadSizeLimit}Mb`, 'Error Uploading File');
        this.fileUpload.storagePath = null;
        return;
      });

      // Progress monitoring
      // this.percentage = this.task.percentageChanges();
      // this.snapshot   = this.task.snapshotChanges();

      // The file's download URL
      this.fbTask.then(async result => {
        if (this.sizeNameSuffix.length > 0) {
          // Is Image
          this.spinner.hide('spinner-upload');
          this.spinner.show('spinner-resize');
          const storageRefResized: firebase.storage.Reference = this.bucket.ref().child(this.fileUpload.storagePath);
          this.firebaseStorageService.keepTrying(10, storageRefResized).then(downloadUrl => {
            this.fileUpload.url = downloadUrl;
            this.spinner.hide('spinner-resize');
            this.dialogRef.close( { fileUpload: this.fileUpload } );
          }, err => {
            console.log(err);
          });
        } else {
          // Other type of file
          this.fileUpload.url = await storageRef.getDownloadURL();
          this.spinner.hide('spinner-upload');
          this.dialogRef.close( { fileUpload: this.fileUpload } );
        }
      });
        // Remove extension and add size
    } catch (err) {
      console.log(err);
      this.notificationService.alertError('Error', 'Error Uploading File');
      this.fileUpload.storagePath = null;
      return;
    } finally {
      this.spinner.hide('spinner-upload');
    }
  }

  // Determines if the upload task is active
  isActive(snapshot) {
    return snapshot.state === 'running' && snapshot.bytesTransferred < snapshot.totalBytes;
  }

  keepTrying(triesRemaining, storageRef: firebase.storage.Reference): Promise<any> {
    if (triesRemaining < 0) {
      return Promise.reject('out of tries');
    }
    return storageRef.getDownloadURL().then((downloadUrl) => {
      return downloadUrl;
    }, error => {
      switch (error.code) {
        case 'storage/object-not-found':
          return this.delay(2000).then(() => {
            return this.keepTrying(triesRemaining - 1, storageRef);
          });
        default:
          console.log(error);
          return Promise.reject(error);
      }
    });
  }

Hello, any progress on the callback or lifecycle management status? I am running into the same issue as others on this thread - no ability to get the downloadURL + token for a resized image. Thank you!

Really interested to know if a solid solution to this has been developed. Having to set timeout delay before checking for the download URL. Works in development, but really nervous putting this into production.

@jaysonfrazer I am still using my polling method in production but check out this release and see if it has what we want.
https://github.com/firebase/extensions/releases/tag/storage-resize-images-v0.1.28
Let me know!

I want to only update the firestore document when the resize is finished, since setting a timeout is a very bad solution.


fix: missing build step

:tada: This PR is included in version 2.3.1 :tada:
The release is available on:

npm package (@next dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


[4.0.0-rc-1] Image upload for default content

Images can be uploaded for new content but not for the default content that is created during the installation.

Please check if 974860c fixes this issue

Fixed, please check the new release v4.0.0 rc3.
Feel free to open the ticket if the issue is still there, if a new issue please open a new ticket.


Add average SNVs per person to stats page

To be calculated by production team and added to Stats page after
Related: https://github.com/broadinstitute/gnomad_production/issues/1438

Per Person stats are out:
Exomes: https://docs.google.com/spreadsheets/d/12AI_MMB-USEaGq0iAocwhLRzsn7leUL0sLjGBj5AjNI/edit?gid=855071952#gid=855071952
Genomes: https://docs.google.com/spreadsheets/d/1laHF0g8zXubvLKyXBRP1JYaHXfMMQu99clyH9Jt6XuA/edit?gid=1493364530#gid=1493364530

Mega Summary is that, for Autosomal+Par Exomes, for variants passing variant qc:
gnomAD: 94,247.60 SNVs/person
non-UKB: 45360 SNVs/person
UKB: 173830	SNVs/person
for Autosomal+SNP Exomes, for all variants passing variant qc :
gnomAD: 4,749,800.00 SNVs/person
all stats are for all imputed genetic ancestries


:memo: Feat: Update jsdoc

Description
Concise description of proposed changes
Testing
Explain the quality checks that have been done on the code changes
Additional Information

[ ] I read the contributing docs (if this is your first contribution)

Your ENS/address:

#1235  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @roninjin10 and the rest of your teammates on  Graphite


Add support for query languages like XPath and JSONPath

Description
Typst is missing query languages for the various data formats it supports. The most prominent ones being XPath and JSONPath:

https://www.w3.org/TR/1999/REC-xpath-19991116/
https://goessner.net/articles/JsonPath/

The example on https://typst.app/docs/reference/data-loading/xml/ could look like this instead:
#let data = xml("example.xml")
#let articles = xpath(data, "//article")
#for article in articles {
  let title = xpath(article, "./title").first()
  let author = xpath(article, "./author").first()
  // ..
}

Use Case
Query languages would definitely make it easier to interact with data formats like XML and JSON. Currently it's very tedious to work with them.

This seems like something libraries should implement, not typst itself.
As a proof of concept, I implemented some features of JSONPath here: https://gist.github.com/frozolotl/a527ecd9716fe9eb84e0f66b4dcfbc86

I agree that this is out of scope for the core compiler.


CreateAppend not meet GSP-134 Write Behavior Consistency

AppendObject doesn't support overwrite. CreateAppend to an existing appendable object with size greater than 0 will get "code":"invalid_argument","message":"position must equal to the current object size, position=0, object size=xxx", which not meet GSP-134: Write Behavior Consistency.

I think we need to remove the file before creating it.

Fixed in #84


Bug: endTime 23:59 is automatically changed to 24:00

Hi there! :)
Thank you for creating and maintaining this nice package!
Unfortunately we have found an issue, that is bugging us:
When calling showTimeRangePicker with start = 0:00 and end = 23:59, the endTime is changed to 24:00. This seems to happen in time-range-dialog.dart in initState() of _TimeRangePickerState.
If I'm not mistaken, it is not required to manipulate the end time, we can simply use it as it is:
Line 300 should be _endTime = endTime; instead of _endTime = _roundMinutes(endTime.hour * 60 + endTime.minute * 1.0);
It might be related to this issue?
Thank you!

Hi,
did you check the interval parameter?
It is a duration (default 5 minutes) that all times (start, end & selection) are rounded to. So if you set 23:59 it will round to 24:00.
Can you try to set the interval to 1 Minute or smaller and report back in case this problem persists?

Hi!
Thank you so much for your quick reply! Yes, that works like a charm! :)
I'm sorry, I overlooked that parameter.

No problem, good luck with your project.


[Brightcove] support bostonglobe.com's use

BrightcoveNewIE.extract_urls(): Handle player data when it is all in
attributes of the  tag, and also the id is
data-brightcove-video-id not data-video-id.
Add test to generic extractor, note HEAD gives 404, no checksum computed(?).

Can someone explain to me why md5 checksums do not appear to be computed by the test harness in this case? Is this an issue with playlists, and can we document that?  I don't see an open issue, but it seems problematic if playlist tests can't be checksummed.

As a side note, this pattern is quite similar to #12005:
        <video data-video-id="5311302538001"
               data-account="800000640001"
               data-player="HyenP6LuN"
               data-embed="default"
               class="video-js apply-ratio"
                               autoplay                preload
               controls></video>

a similar approache has been used from the beginning when i added the extractor but it was changed in 536f819eda975224666374a9ce83cc3472f5aa5a, i guess because this attributes can be used by other websites for other contexts.
also data-brightcove-video-id is not used by the standard embeds https://docs.brightcove.com/en/perform/brightcove-player/guides/in-page-embed-player-implementation.html.

Requested changes done. Also an addition to _generic_id() to avoid ending up with 'story' as the ID for all Boston Globe urls. I was surprised I couldn't  find other examples in tests that have this sort of problem.
It does look like @remitamine's original approach was more general than what is there now. I'm not sure what's going on there.
I agree data-brightcove-video-id isn't in the docs, but it still seems to me that it's appropriate to look for the more-specific identifier before the generic one. But I'm happy to flip the order of the test if someone has a better argument.

Also, can anyone please help with:

Can someone explain to me why md5 checksums do not appear to be computed by the test harness in this case? Is this an issue with playlists, and can we document that?  I don't see an open issue, but it seems problematic if playlist tests can't be checksummed.

Thanks.

md5 checksum will be checked if it's available in the test case even in playlists:https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/orf.py#L30.

md5 checksum will be checked if it's available in the test case even in playlists:

When I added a dummy md5 checksum (either in the playlist entry or in the overall test), no error was reported. Did I do it wrong?

Specifically, with an md5 field inside the playlist info_dict:
======================================================================
FAIL: test_BostonGlobe (__main__.TestDownload)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_download.py", line 210, in test_template
    expect_info_dict(self, info_dict, tc.get('info_dict', {}))
  File "/Users/jhawk/src/youtube-dl/test/helper.py", line 177, in expect_info_dict
    expect_dict(self, got_dict, expected_dict)
  File "/Users/jhawk/src/youtube-dl/test/helper.py", line 173, in expect_dict
    expect_value(self, got, expected, info_field)
  File "/Users/jhawk/src/youtube-dl/test/helper.py", line 167, in expect_value
    'Invalid value for field %s, expected %r, got %r' % (field, expected, got))
AssertionError: Invalid value for field md5, expected u'dummy1', got None

And an md5 field in the test itself is ignored:
    _TEST = {
            'url': 'http://www.bostonglobe.com/metro/2017/02/11/tree-finally-succumbs-disease-leaving-hole-neighborhood/h1b4lviqzMTIn9sVy8F3gP/story.html',
            'md5': 'dummy3',
...

i'm not sure why it's not working for you, but i just tested it with an md5 value in list item, it raise an error when the md5 is wrong.

I tried to run the orf.py test, but
Skipping ORFTVthek: Blocked outside of Austria / Germany

and uncommenting the skip directive I get a 404.
Do you have another example?

python test_download.py TestDownload.test_Seeker_1
[Seeker] changes-expected-at-zoos-following-recent-gorilla-lion-shootings: Downloading webpage
[download] Downloading playlist: After Gorilla Killing, Changes Ahead for Zoos
[Seeker] playlist After Gorilla Killing, Changes Ahead for Zoos: Collected 1 video ids (downloading 1 of them)
[download] Downloading video 1 of 1
[revision3:embed] 67558: Downloading JSON metadata
[revision3:embed] 67558: Downloading m3u8 information
[info] Writing video description metadata as JSON to: 67558.info.json
[debug] Invoking downloader on 'http://videos.revision3.com/revision3/web/dnews/2260/dnews--2260--the-pros-cons-of-zoos--hd720p30.h264.mp4'
[download] Destination: 67558.mp4
[download] 100% of 10.00KiB in 00:01
[download] Finished downloading playlist: After Gorilla Killing, Changes Ahead for Zoos
.
----------------------------------------------------------------------
Ran 1 test in 12.183s

OK

changing the md5:
python test_download.py TestDownload.test_Seeker_1
[Seeker] changes-expected-at-zoos-following-recent-gorilla-lion-shootings: Downloading webpage
[download] Downloading playlist: After Gorilla Killing, Changes Ahead for Zoos
[Seeker] playlist After Gorilla Killing, Changes Ahead for Zoos: Collected 1 video ids (downloading 1 of them)
[download] Downloading video 1 of 1
[revision3:embed] 67558: Downloading JSON metadata
[revision3:embed] 67558: Downloading m3u8 information
[info] Writing video description metadata as JSON to: 67558.info.json
[debug] Invoking downloader on 'http://videos.revision3.com/revision3/web/dnews/2260/dnews--2260--the-pros-cons-of-zoos--hd720p30.h264.mp4'
[download] Destination: 67558.mp4
[download] 100% of 10.00KiB in 00:00
[download] Finished downloading playlist: After Gorilla Killing, Changes Ahead for Zoos
F
======================================================================
FAIL: test_Seeker_1 (__main__.TestDownload)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_download.py", line 202, in test_template
    self.assertEqual(md5_for_file, tc['md5'])
AssertionError: '83bcd157cab89ad7318dd7b8c9cf1306' != '83bcd57cab89ad7318dd7b8c9cf1306'
- 83bcd157cab89ad7318dd7b8c9cf1306
?      -
+ 83bcd57cab89ad7318dd7b8c9cf1306

----------------------------------------------------------------------
Ran 1 test in 7.770s

FAILED (failures=1)

Thanks, @remitamine, I feel dumb!
I had tried it in all 3 wrong places: the top-level test, the top-level info_dict, and the playlist info_dict. But the right place is the playlist outside the info_dict, per 349ea41bb8902fb76aca37d4c0dd4cfbe1498717.
Thanks for leading me to the right place.

I believe this is all set now.
I do have a reservation, which I expressed before:

seems to … [be] more fragile (what happens when there are other kinds of videos on bostonglobe.com that the Generic extractor would have caught, but this one won't).

Indeed, I found some examples that use YouTube like
https://www.bostonglobe.com/lifestyle/names/2017/02/17/does-ben-affleck-play-matt-damon-favorite-version-batman/ruqkc9VxKBYmh5txn1XhSI/story.html
and
https://www.bostonglobe.com/sports/2017/02/18/racecar-that-runs-chocolate-may-not-sweet-idea/rIj2EAeCXZz3VM1w0obAaO/story.html
that the GenericIE supports but that support now breaks.  I don't know how to handle this. Extending bostonglobe.py to know explicitly about YouTube seems wrong. What if there are other types? Indeed
https://www.bostonglobe.com/sports/bruins/2017/02/18/ottawa-acquisition-mike-condon-pays-off/76G520N2pqH5bnbHYjRBfL/story.html
seems to be a Twitter embedded video, although this one the Generic extractor doesn't catch (a bug I would assume?)
Should I do what lcp.py does:
https://github.com/rg3/youtube-dl/blob/c58b7ffef43f60fa6a183c849cfdca42e36eae0c/youtube_dl/extractor/lcp.py#L71-L76
It seems a cheap trick, and doesn't seem to be done very often (but maybe it should be?)
I could talk about how I don't understand the code philosophy, but you knew that already :)

It's been a while, I haven't heard anything. Is there anything I can do to speed this up?
Given no feedback, I went ahead and implemented the fallback to the Generic extractor that I asked about last month.

Thanks, @yan12125. All set on your review

I'm confused about where the HEAD requests are generated from, but you're right that it didn't cause a problem for the first test. But the second test I just added did have that problem (the YouTube embed). Weird.
_og_search_title() seems less fragile so I went with that. Not sure how to choose when they're both valid. I don't see any guidance in the code on priority.
Youtube example added.

Please fix coding styles.
$ flake8
./youtube_dl/extractor/bostonglobe.py:8:1: F401 '..utils.orderedSet' imported but unused
./youtube_dl/extractor/bostonglobe.py:13:1: E302 expected 2 blank lines, found 1
./youtube_dl/extractor/bostonglobe.py:19:17: F601 dictionary key 'title' repeated with different values
./youtube_dl/extractor/bostonglobe.py:22:17: F601 dictionary key 'title' repeated with different values
./youtube_dl/extractor/bostonglobe.py:28:9: E122 continuation line missing indentation or outdented
./youtube_dl/extractor/bostonglobe.py:44:9: E123 closing bracket does not match indentation of opening bracket's line
./youtube_dl/extractor/bostonglobe.py:51:1: W293 blank line contains whitespace
./youtube_dl/extractor/bostonglobe.py:57:21: E221 multiple spaces before operator
./youtube_dl/extractor/bostonglobe.py:58:23: E221 multiple spaces before operator
./youtube_dl/extractor/bostonglobe.py:59:22: E221 multiple spaces before operator
./youtube_dl/extractor/bostonglobe.py:60:18: E221 multiple spaces before operator
./youtube_dl/extractor/generic.py:2497:20: F821 undefined name '_playlist_from_matches'

I don't see any guidance in the code on priority.

If an approach works and is easy to understand, it's good. No priorities are enforced.

All set, hopefully.

If an approach works and is easy to understand, it's good. No priorities are enforced.

Well, it just seems like it's a bad plan to go from a flexible mechanism with a good fallback to a fragile mechanism that's more likely to break.
Maybe not this extractor or this website, but across the project, at some point title extraction is going to fail in some places. Why should the only the sites depending on the Generic extractor get the benefit of this?
Ah, well, philosophy :)

Why should the only the sites depending on the Generic extractor get the benefit of this?

For metadata, we want accuracy instead of flexible. In Bostonglobe there happens to be not much difference.

Last final issue! Could you squash commits?

Was I supposed to enter a comment after squashing? Squashed.

Merged. Thanks for the patience and the work!


Remove Unncessary Text

Describe the feature
Hello Project Admin
I found an unnecessary text under Navbar as you can see in the below page
I would like to remove that text (village)
Can you assign me this issue under GSSOC'24 @Kalivarapubindusree
Add ScreenShots

Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'24 contributor
[X] I want to work on this issue

please remove two feedback forms and give them proper alignment @sailaja-adapa

Yeah okk @Kalivarapubindusree

I will do it by today and update you @Kalivarapubindusree


Create a poe command for compiling Cairo1 contracts

Feature Request
Currently many compiled contracts are in /precompiled directory.

Good idea, but this would have to somehow detect cairo1 compiler directory. Probably can be done, but not sure how easily.


fix(ui-templates)!: switch to useHead import

this will be a breaking change but will hide the warning useMeta has been renamed to useHead
Might be worth delaying merging this until just before we bump to @nuxt/ui-templates (without edge).

Merged too fast!! Canceled CI. Fully agree about doing with rc but anyway old users are bound to the latest edge version.


Android : Camera setPreview or setParameters Error

Android : Camera setPreview or setParameters Error
What's happening?

I tried to move from one screen to the other  where both have QRCodeScanners and when I come back to the first screen , I got the camera1 setPreview error followed by setParameters Error
How can it be reproduced?

Build details?

I have the same problem


add veno token

hello.. what the logo images failed.?

Incorrect information submitted, please go through our guide and check again.


Install specific version

Hi,
I ran scoop install rsync which install version 3.1.1. On my servers running under Debian stable, I have rsync 3.0.9 (latest available there) which has a compatibility issue with 3.1.1.
How can I install 3.0.9 on my local computer ? Is there something like scoop install<EMAIL_ADDRESS>

Hi @vpratfr
There is not currently any way to install a specific version of a manifest in scoop. The current closest solution is to have a manifest specifically for that version. E.g. manifests for git and git19
You could create a rsync30 manifest in this case to solve your problem

@vpratfr Did you end up resolving your issue?

I did that manually. I still think there should be a way to specify a version number for a package, this would be helpful to solve cases like that.


would it be possible to use the below two projects to make a GUI?

https://github.com/brummer10/Ratatouille.lv2
for ratatouille, the Neural-Amp-Modeler and the RTNeural loaders into self contained C++ classes it wont be to hard to reuse them in other projects. Same is true for the convolver class and the dcblocker.
https://github.com/Chowdhury-DSP/BYOD
for BYOD, there is a signal chain type GUI that already accepts IRs and will just need to accept NAM files
originally posted:
https://github.com/mikeoliphant/neural-amp-modeler-lv2/issues/67

Fell free to use those classes and do so. But note, RTNeural is, as well as BYOD, original written by jatinchowdhury18, so, implementing a model loader into BYOD as plugin would be straight forward for him.

Feel free to use those classes and do so. But note, RTNeural is, as well as BYOD, original written by jatinchowdhury18, so, implementing a model loader into BYOD as plugin would be straight forward for him.

okay ive contacted them about it, thanks!


Release v8.0.0

Simple tracking issue for the v8.0.0 release. Now that v1.11 has been released, we should release client-go v8.0.0 too.
TODO:

[x] Smoke test with https://gist.github.com/sttts/d05106dae85a79f1ca513f1c71260d56
[x] Create the tag/release.
[x] Update CHANGELOG and README: https://github.com/kubernetes/client-go/pull/429
[x] Link the changelog on the releases page: https://github.com/kubernetes/client-go/releases/tag/v8.0.0.

/cc @sttts @caesarxuchao

https://gist.github.com/sttts/d05106dae85a79f1ca513f1c71260d56 finished successfully

This is now done! :tada:


Multi item

/Users/runner/work/tppsss/tppsss/src/main.cpp:110:17: warning: 'auto' type specifier is a C++11 extension [-Wc++11-extensions]
            for(auto equity: assets){
                ^
/Users/runner/work/tppsss/tppsss/src/main.cpp:110:28: warning: range-based for loop is a C++11 extension [-Wc++11-extensions]
            for(auto equity: assets){
                           ^
/Users/runner/work/tppsss/tppsss/src/main.cpp:337:37: warning: 'auto' type specifier is a C++11 extension [-Wc++11-extensions]
                                for(auto equity: assets){
                                    ^
/Users/runner/work/tppsss/tppsss/src/main.cpp:337:48: warning: range-based for loop is a C++11 extension [-Wc++11-extensions]
                                for(auto equity: assets){
                                               ^
/Users/runner/work/tppsss/tppsss/src/main.cpp:396:59: error: implicit instantiation of undefined template 'std::basic_stringstream<char>'
                                        std::stringstream toInt(input);
                                                          ^
/Applications/Xcode_13.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.1.sdk/usr/include/c++/v1/iosfwd:139:[32](https://github.com/trackpadpro/tppsss/runs/6549871910?check_suite_focus=true#step:3:33): note: template is declared here
    class _LIBCPP_TEMPLATE_VIS basic_stringstream;
                               ^
4 warnings and 1 error generated.

Clang64 does not appreciate the auto iteration of the vector.
Clang64 errors at basic_stringstream


Create 0347-top-k-frequent-elements.dart

File(s) Modified: 0347-top-k-frequent-elements.dart
Language(s) Used: dart
Submission URL: _https://leetcode.com/problems/top-k-frequent-elements/submissions/1228155773/_

Important
Please make sure the file name is lowercase and a duplicate file does not already exist before merging.

Hello @a93a, could you please take a look at this?


css: Use Source Sans 3 VF.

This is to overcome the limitations of previous static font, which didn't allow us to use various font widths.
There are no visual changes that I can see.
discussion: https://chat.zulip.org/#narrow/stream/101-design/topic/font.20weight.20500

There are so extra matches here since there no space after font-weight: in some svg files. So, this is the correct grep:
git grep -nP 'font-weight:(?! ?([234679]00|normal|bold|initial|inherit)( !important)?;)'
Binary file docs/images/architecture_simple.svg matches
web/styles/app_components.css:785:                font-weight: lighter;
web/styles/app_components.css:818:            font-weight: 500;
web/styles/left_sidebar.css:24:            font-weight: 800;
web/styles/left_sidebar.css:31:    font-weight: 800;
web/styles/lightbox.css:161:            font-weight: 100;
web/styles/popovers.css:245:            font-weight: 550;
web/styles/portico/billing.css:58:            font-weight: 500;
web/styles/portico/landing_page.css:312:                    font-weight: 100;
web/styles/portico/landing_page.css:760:            font-weight: 500;
web/styles/portico/landing_page.css:987:                font-weight: 500;
web/styles/portico/landing_page.css:1028:            font-weight: 800;
web/styles/portico/landing_page.css:2580:    font-weight: 500;
web/styles/portico/landing_page.css:3297:                font-weight: 100;
web/styles/portico/landing_page.css:3872:    font-weight: 800;
web/styles/portico/landing_page.css:3925:        font-weight: 800;
web/styles/portico/landing_page.css:3966:        font-weight: bolder;
web/styles/portico/landing_page.css:4105:                    font-weight: 550;
web/styles/portico/landing_page.css:4329:                    font-weight: 550;
web/styles/portico/portico.css:333:        font-weight: lighter;
web/styles/portico/portico_signin.css:36:    font-weight: 500;
web/styles/portico/portico_signin.css:52:    font-weight: 500;
web/styles/portico/portico_signin.css:439:        font-weight: 500;
web/styles/portico/portico_signin.css:1168:            font-weight: 500;
web/styles/settings.css:298:        font-weight: 500;
web/styles/settings.css:1481:            font-weight: lighter;
web/styles/subscriptions.css:79:    font-weight: 500;
web/styles/subscriptions.css:835:                font-weight: 800;
web/styles/zulip.css:1778:                font-weight: 800;
web/third/bootstrap/css/bootstrap.css:1490:  font-weight: 100;

I will try to see what changes these have on rendered pages and push commits accordingly.

web/third/bootstrap/css/bootstrap.css:1490:  font-weight: 100;

Only the above font weight is remaining which I didn't modify since it is in bootstrap and for carousel code.
.carousel-control {
  position: absolute;
  top: 40%;
  left: 15px;
  width: 40px;
  height: 40px;
  margin-top: -20px;
  font-size: 60px;
  font-weight: 100;
  line-height: 30px;
  color: #ffffff;
  text-align: center;
  background: #222222;
  border: 3px solid #ffffff;
  -webkit-border-radius: 23px;
  -moz-border-radius: 23px;
  border-radius: 23px;
  opacity: 0.5;
}

@andersk this is ready for another review.

Only the above font weight is remaining which I didn't modify since it is in bootstrap and for carousel code,

For the carousel, I believe this PR removes the only carousel we had, so we should be able to delete that part of Bootstrap.

Looks good, merged, thanks @amanagr!


Lazy rendering.

I'm beginner in VueJs. I've been trying to optimize some 2 level nested tabs with subtabs that have at least 6 highcharts each one.  Currently vue-tabs is lazy rendering, which is good for some scenarios, but thinking how to experiment if still doing lazy rendering but with render only once, so the active tab is not destroyed after transition so next time is not going to be rendered again. Any ideas ?

@vadimpopa This can be achieved with https://vuejs.org/v2/api/#keep-alive

Yes I saw that, but did a few experiments, haven't noted any difference. So according to you, in the following template where I should add it ?
<template>

    <vue-tabs>
      <v-tab v-for="(tab, i) in tabs" :key="i" :title="tab.name">
        <vue-tabs>
          <v-tab v-for="(subTab, j) in tab.subTabs" :key="j" :title="subTab.name">

              <chart class="Grid-cell" :query="tab.searchQuery" :type="type"></chart>

          </v-tab>
        </vue-tabs>
      </v-tab>
    </vue-tabs>

</template>

Let's create a jsfiddle and start from there. Please Create one with your example and maybe put something else than charts and point what you want to cache.

I'll work on it as soon as I'll get more time to play. I noticed that keep-alive is not working well with the v-for, especially in such tabs context. Caching worked but it rendered only the first tab.

Let's start from here https://jsfiddle.net/b44cc4dq/87/

Ok, see the screenshot, this is what I've got.

Maybe I'm missing something but tabs are already kept in the DOM. Internally they use v-show to display content which sets display:none on the tab.
You can see that all tabs are present in the DOM but only one is visible. The rest have aria-hidden="true"
This also applies to sub-tabs so you might not need keep-alive at all. So

so the active tab is not destroyed after transition so next time is not going to be rendered again
Is actually true by default since tabs are rendered only once and they are present in the DOM

Yep, I you missed pushing a new release with latest changes, and I missed as well this but only now noticed. The npm version is from 9 May, but that fiddle is working with the latests which has the 10 days ago changes. The npm version is working like with v-if.

Oh ok. Thanks for pointing that out. I will try to do small releases with like 1-2 issues from now on so I don't encounter such situations later on. For now you can point to a github repo in package.json so you get the latest version or wait for the next release in 1-2 days.

Nah, since yours is not going to have lazy rendering it's not suitable anymore for me.  But besides lazy rendering I need the "render only once"/deferred rendering feature: so when the tab first is activated the content is rendered, next time when the tab is activated again the content should be in DOM already. This called "deferred rendering", ExtJS has this and works very well. I may need to implement the tabs myself with this feature.

Sure, putem vb<EMAIL_ADDRESS>I will think over the issue and see if I can find some solution for that

Ai incercat sa te uiti peste https://vuejs.org/v2/api/#v-once ?

I have v-once, but is not helping much....anyways looks like vuikit (https://github.com/vuikit/vuikit) does deferred rendering, I haven't looked much into, just did a quick tests.

    @Prop({ default: true }) keepAlive!: boolean;
    @Prop({ default: true }) lazy!: boolean;
    loaded = false;

    render() {
        let loadContentFlag: boolean = false;
        let needToFigureIsActiveAndKeepAlive: boolean = false;

        if (this.lazy) {
            if (this.loaded) {
                needToFigureIsActiveAndKeepAlive = true;
            }
        } else {
            needToFigureIsActiveAndKeepAlive = true;
        }

        if (needToFigureIsActiveAndKeepAlive) {
            if (this.active) {
                loadContentFlag = true;
            } else {
                if (this.keepAlive) {
                    loadContentFlag = true;
                }
            }
        }

        return (
            <section
                class="tab-container"
                id={`p-${this.tabId}`}
                aria-labelledby={`t-${this.tabId}`}
                role="tabpanel"
                v-show={this.active}
            >
                   {loadContentFlag && this.$slots.default}

        );
    }


Add more tests

The test suite is sparse.

Still need to expand out the test suite

We don't have tests at the moment so this is moot


Services component: services that user has given consent to

Show services that user already uses and has given consent to.
This component will be used in Mydata profile services tab.


Update List Exp API with new query params

This PR fixes #722

@khansaad - Please include a design doc for listExperiments with all the supported input parameters (results, recommendations, latest, experiment_name). Include functional tests to test the below with single and multiple experiments.

/listExperiments without any parameters

validate only experiment details are displayed and they match with the created experiments
validate none of the results or recommendations are displayed

/listExperiments?results=true

validate latest results are displayed without recommendations and results are as expected
Post multiple results and validate latest result is shown for the experiments

/listExperiments?results=true&latest=false

validate all the results are displayed without recommendations

/listExperiments?recommendations=true

validate only if the latest recommendation is displayed without any results

/listExperiments?recommendations=true&latest=false

validate if all recommendations are displayed for all the experiments and without results.

/listExperiments?results=true&recommendations=true&latest=false

validate if all the results and recommendations are displayed for all the experiments

The above tests need to be included with experiment_name parameter & without experiment_name (are we supporting this?)
Negative cases:

Check the response codes and error messages with invalid values for the input parameters:

If experiment_name is supported check the behavior with a non-existing experiment_name
Invalid values for results, recommendations

Delete a few experiments and ensure it is not listed as part of listExperiments, what is the behavior if we query with results or recommendations to true and latest = false?
Create experiments, without update results

validate no results are displayed when results=true and results=true&latest=false
validate no recommendations are displayed with recommendations=true and recommendations=true&latest=false
validate listExperiments without any parameters
updateResults for a few exps and validate if results / recommendations are shown only for those

@khansaad - Please include a design doc for listExperiments with all the supported input parameters (results, recommendations, latest, experiment_name). Include functional tests to test the below with single and multiple experiments.

/listExperiments without any parameters

validate only experiment details are displayed and they match with the created experiments
validate none of the results or recommendations are displayed

/listExperiments?results=true

validate latest results are displayed without recommendations and results are as expected
Post multiple results and validate latest result is shown for the experiments

/listExperiments?results=true&latest=false

validate all the results are displayed without recommendations

/listExperiments?recommendations=true

validate only if the latest recommendation is displayed without any results

/listExperiments?recommendations=true&latest=false

validate if all recommendations are displayed for all the experiments and without results.

/listExperiments?results=true&recommendations=true&latest=false

validate if all the results and recommendations are displayed for all the experiments

The above tests need to be included with experiment_name parameter & without experiment_name (are we supporting this?)
Negative cases:

Check the response codes and error messages with invalid values for the input parameters:

If experiment_name is supported check the behavior with a non-existing experiment_name
Invalid values for results, recommendations

Delete a few experiments and ensure it is not listed as part of listExperiments, what is the behavior if we query with results or recommendations to true and latest = false?

Create experiments, without update results

validate no results are displayed when results=true and results=true&latest=false
validate no recommendations are displayed with recommendations=true and recommendations=true&latest=false
validate listExperiments without any parameters
updateResults for a few exps and validate if results / recommendations are shown only for those

Updated the MointoringAPI doc to include the new listExperiments changes.
Working on the test changes in a separate PR.

@khansaad Still seeing one issue with the latest changes
/listExperiments?experiment_name=quarkus-resteasy-kruize-min-http-response-time-db_0&recommendations=true
This should provide only the latest recommendation for the given experiment and not all. I am seeing all of the recommendations. (default should be latest=false)