llvm_mode floating point split failure

It's better to open an issue from the float related bug that I found while trying to compile ffmpeg for the issue #85 .
I attach the command line, the preprocessed C file and the generated bitcode.
Note that if I use this command line to generate the bitcode is fine (change -emit-obj to -S -emit-llvm).
tmp.tar.gz
LLVM version 7.0.0, Ubuntu 18.04.3.

When compiling the produced bitcode with llc-7, I have this error:
llc-7: error: llc-7: af_adeclick-edd99e.ll:5257:16: error: invalid cast opcode for cast from 'i1' to '<2 x i32>'
  %2431 = zext i1 %2365 to <2 x i32>, !dbg !1038
               ^

Removing this snippet seems to not solve the crash (but I'm sure that now the generated code is wrong):
      case CmpInst::FCMP_ONE:
      case CmpInst::FCMP_UNE:
        /* unequal signs are unequal values */
        /* goto true branch */
        PN->addIncoming(ConstantInt::get(Int1Ty, 1), bb);
        /* unequal exponents are unequal values, too */
        PN->addIncoming(ConstantInt::get(Int1Ty, 1), signequal_bb);
        /* fractions comparison */
        PN->addIncoming(icmp_fraction_result, middle_bb);
        break;

Removing the float related code solves the segfaults but there also always other errors like the following:
[+] Instrumented 1093 locations (non-hardened mode, ratio 100%).
fatal error: error in backend: Cannot select: 0x55b01c3f3840: ch = br_cc 0x55b01c404ee8, setogt:ch, 0x55b01c3f3d88, 0x55b01c404a08, BasicBlock:ch< 0x55b01c4024b0>
  0x55b01c3f3d88: v4f32,ch = load<(load 16 from %ir.2889, align 4, !tbaa !838, !alias.scope !847)> 0x55b01c3f3d20, 0x55b01c3f3978, undef:i64, libavfilter/af_asoftclip.c:144:21
    0x55b01c3f3978: i64 = add 0x55b01c3f3770, 0x55b01c3d46f0, libavfilter/af_asoftclip.c:144:21
      0x55b01c3f3770: i64,ch = CopyFromReg 0x55b01c10cbd8, Register:i64 %413, libavfilter/af_asoftclip.c:144:21
        0x55b01c4040b0: i64 = Register %413
      0x55b01c3d46f0: i64 = shl 0x55b01c3f4060, Constant:i8<2>, libavfilter/af_asoftclip.c:144:21
        0x55b01c3f4060: i64,ch = CopyFromReg 0x55b01c10cbd8, Register:i64 %446, libavfilter/af_asoftclip.c:144:21
          0x55b01c3d4a30: i64 = Register %446
        0x55b01c3f37d8: i8 = Constant<2>
    0x55b01c404db0: i64 = undef
  0x55b01c404a08: v4f32 = bitcast 0x55b01c3f3c50
    0x55b01c3f3c50: v4i32 = BUILD_VECTOR Constant:i32<0>, Constant:i32<0>, Constant:i32<0>, Constant:i32<0>
      0x55b01c3d4b00: i32 = Constant<0>
      0x55b01c3d4b00: i32 = Constant<0>
      0x55b01c3d4b00: i32 = Constant<0>
      0x55b01c3d4b00: i32 = Constant<0>
In function: filter_flt
clang: error: clang frontend command failed with exit code 70 (use -v to see invocation)
clang version 7.0.0-3~ubuntu<IP_ADDRESS> (tags/RELEASE_700/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

The source file from the tmp.tar.gz does not look preprocessed to me:
/usr/include/string.h:155:59: error: unknown type name 'locale_t'
extern int strcoll_l (const char *__s1, const char *__s2, locale_t __l)
                                                          ^
/usr/include/string.h:160:5: error: unknown type name 'locale_t'
                         locale_t __l) __THROW __nonnull ((2, 4));
                         ^
/usr/include/string.h:427:40: error: unknown type name 'locale_t'
extern char *strerror_l (int __errnum, locale_t __l) __THROW;
                                       ^
In file included from libavfilter/af_adeclick.c:21:
In file included from ./libavutil/audio_fifo.h:30:
./libavutil/avutil.h:344:1: error: unknown type name 'AVRational'
AVRational av_get_time_base_q(void);
^
In file included from libavfilter/af_adeclick.c:22:
./libavutil/opt.h:270:9: error: unknown type name 'AVRational'
        AVRational q;
        ^
...

Could you produce one, thanks.

IMO @hexcoder- you should use another env var to enable float splits like AFL_LLVM_LAF_SPLIT_FLOATS

The source file from the tmp.tar.gz does not look preprocessed to me:
Use the sh script after replacing the /home/andrea/Desktop/AFLplusplus strings with your path to afl++

Could it be, that the compiler does not support native 64 bit ints?

Idk, seems a laf-intel bug, it is present also if I completely remove the call to splitFPCompares

Have you tried to compile ffmpeg with your LLVM version?

Please don't force me to download the whole package, preprocessed source please.

LOL it is also present when I force "NOT Running split-compare-pass" with an invalid bitw. The bug is present also when I only execute simplifyCompares(M)

Regarding AFL_LLVM_LAF_SPLIT_FLOATS:
that is difficult to separate, since it uses AFL_LLVM_LAF_SPLIT_COMPARES splitting later on.
But I get your point.

The problem seems this code in simplyfyCompares:
          if (selectcmpInst->getPredicate() == CmpInst::FCMP_OGE ||
              selectcmpInst->getPredicate() == CmpInst::FCMP_UGE ||
              selectcmpInst->getPredicate() == CmpInst::FCMP_OLE ||
              selectcmpInst->getPredicate() == CmpInst::FCMP_ULE) {

            auto op0 = selectcmpInst->getOperand(0);
            auto op1 = selectcmpInst->getOperand(1);

            Type *TyOp0 = op0->getType();
            Type *TyOp1 = op1->getType();

            /* this is probably not needed but we do it anyway */
            if (TyOp0 != TyOp1) { continue; }

            fcomps.push_back(selectcmpInst);

          }```
After removing it and the call to splitFPCompares the issue does not appear anymore.

ok, good progress. If you could provide something smaller and preprocessed, I might see what is going on. Then it is the sign extraction part of simplifyCompares(). Probably would happen with FP splitting too.
Please lets isolate it.

Yes, I'm trying to select two testcases cause seems two separate bugs. One when the >= and <= float comp are splitted and the other on the produced code in splitFPCompares.

semplify_bug.tar.gz
You have a testcase for the first bug, this is for the second.

But I don't have PREPROCESSED source. Seems to be really difficult...

Oh yes sorry I thought that LLVM debug C files were already preprocessed.
Download it from https://drive.google.com/file/d/1Yl7TiWsrDcVmHMxxqC9-166JAdS6ouBo/view?usp=sharing (too large for github)

@hexcoder- as quick and dirty workaround to have laf-intel working I added the AFL_LLVM_LAF_SPLIT_FLOATS env var that enables the splitting of float comparisons when AFL_LLVM_LAF_SPLIT_COMPARES is also set.

fixed with https://github.com/vanhauser-thc/AFLplusplus/commit/3ad5316dd1d7a4512b5acc17381a634a82901406. Please recheck.

AFL_LLVM_LAF_SPLIT_FLOATS is a good option to have anyway.
is it fixed?

Like I said.
The preprocessed code from @andreafioraldi  does compile now with FP splitting enabled.
The treatment of FP splitting is unchanged, but we now filter out comparisons with operands of type vector/array of double, float, etc.
These types are not easily splittable (for me).

I can't test it now, but I will do it tomorrow probably. Btw I pushed float compcov for qemu mode if you want to try it.

Negative, it fails again. @hexcoder- take this preprocessed C https://drive.google.com/file/d/1-zA8Vo_DyRrKxk-EKQjk4hGdSTZFGKwK/view?usp=sharing

Thanks for testing!

ok, the second fix is in https://github.com/vanhauser-thc/AFLplusplus/commit/38d74f0ad56a74cfed1eec04a8a46b4f8e4ec824. I could compile the whole ffmpeg with AFL_LLVM_LAF_SPLIT_COMPARES=1
and AFL_LLVM_LAF_SPLIT_FLOATS=1.


Part of :  https://github.com/owncloud/QA/issues/703

@SwikritiT "something happened" and all the master stuff seems to be in this PR to edge
You need to checkout the edge branch, then make a branch from there and make the commit id change.

@SwikritiT "something happened" and all the master stuff seems to be in this PR to edge
You need to checkout the edge branch, then make a branch from there and make the commit id change.

yes, I forgot to checkout the new branch from the edge and did it from master instead. Fixed in the latest push.


Enabling TAGS_SAVE_AS causes site build to fail

I'm trying to enable the Pelican tag-based navigation for www.apache.org
(https://github.com/apache/www-site/issues/196) and as a first step I tried enabling the TAGS_SAVE_AS feature, by hardcoding it for now, setting TAGS_SAVE_AS = 'tags.html' in bin/pelican.auto.ezt
Just doing this causes a site build of the sibling template-site repository to fail with
WARNING  Caught exception: log.py:91 'page' is undefined".

It looks like something in this module is getting in the way, but I know little about Pelican and even less about Python so I'm not sure where to look. Pointers are welcome.
Here's my failure scenario:
$ cd infrastructure-pelican
$ git diff bin/pelican.auto.ezt
 ...
-TAGS_SAVE_AS = ''
+TAGS_SAVE_AS = 'tags.html'
...

$ docker build -t pelican-asf2 .

# try to build https://github.com/apache/template-site
$ cd ../template-site
$ rm -rf site-generated && docker run -it -p12345:8000 -v $PWD:/site pelican-asf2
...
WARNING  Caught exception:
    page' is undefined".

And the site-generated folder is not created, nothing is generated.

Another option might be to fork the templates for the www-site project and work there to enable tags. I don't know how much interest there is in tags from other ASF sites using the infrastructure-pelican templates, but I think having tags for www-site would be very helpful.
However, as mentioned, I'm not a Pelican or Python expert by far, so I'm not sure of the consequences of such a fork.


[docs] Improve search experience

there is extra header- prepend on hash which does not correspond to the real heading in TOC (only for h2, h3). Fixed in https://github.com/mui-org/material-ui/pull/28801/commits/2799d554ad63b34c8475a90846ca676dcf5656c2

Update styling to use title first. @danilo-leal is the change looks good to you? https://github.com/mui-org/material-ui/pull/28801/commits/69930c4cd010140aa3a7de04612c35d3793ec3d3

Before
After

~- a keyword like "but" should display Button first, but currently it shows word with spaces first (I suggest to fix it by remove spaces in h1).~ Drop this afford from this PR and will try to find a way without removing spaces in heading (using typoTolerance: false does not work)

Before
After

[x] I have followed (at least) the PR section of the contributing guide.

Regarding the configuration, here is one example of a param that can be configured at the indexation and query time: https://www.algolia.com/doc/api-reference/api-parameters/typoTolerance/?client=javascript#override-default-typo-tolerance-mode-for-the-current-search.
For the change in the docs' h1 https://mui.com/components/xx pages, I think that we should get the perspective of @mbrookes.

here is one example of a param that can be configured at the indexation and query time

Okay, I will try to see if it works or not.

FloatingActionButton (FAB)

The component name is Fab, so I"m not sure it makes sense to use FloatingActionButton in the header and nav.
(Incidentally, I noted that we're inconsistent in both the headings and nav on capitalisation)

Comment added.


A question of Data preparation in the Vehicle Re-Identification

Hi,
When I work on the Data preparation step in the Vehicle Re-Identification, I cannot not find the the text file "data/makes.txt"  which is referred to the final step to merge CompCars and VMMRdb.
Dose anyone can help? I will appreciate for it.

Hi,
The correct relative path of the mentioned directory data is pytorch_toolkit/object_reidentification/vehicle_reidentification/data.

OK, I have found the makes.txt.
Thanks a lot.


Set data-gutter on exported code element

The exported code element is missing line numbers, This PR adds the data-gutter attribute to the exportDOM() method of CodeNode to fix this issue.

This data attribute is only used internally to render line numbers, what's the reason for it to be exported?

This data attribute is only used internally to render line numbers, what's the reason for it to be exported?

The line numbers are displayed via content: attr(data-gutter); style declaration, if the attribute is missing no content is displayed.

In this case it should be fixed in plugin to make it generate gutter data for newly created code node

In this case it should be fixed in plugin to make it generate gutter data for newly created code node

everything works as expected, but the exported html element doesn't have the data-gutter attribute.
It seems obvious to add it to the exportDOM() method.

In this case it should be fixed in plugin to make it generate gutter data for newly created code node

everything works as expected, but the exported html element doesn't have the data-gutter attribute. It seems obvious to add it to the exportDOM() method.

As Maks said, this property is not part of the public API - on paste, these attributes can be regenerated - they don't need to be transported as part of the paste payload. They'd mean nothing to any other editor because they're an implementation detail.

@ibastawisi Hi. I'm on same issue and found this PR. How did you make it use data-gutter on html when exporting without this PR?


Ubuntu 18.04 LTS

Does vpnfailsafe work for Ubuntu 18.04 LTS without any issues?
I asking this because they replaced ifupdown, resolvconf, and others with networkd, resolved, and their homegrown netplan.
Can anyone using vpnfailsafe confirm that it works on Ubuntu 18.04 LTS without issues?

Did you get anywhere with this?

Hi @askgitquestions. I don't use Ubuntu, or any of these services, so I can't answer that. If you encounter any problems with this setup, please report them.


Incorrect alignment requirement for buffer-to-image copy

Hi, I'm getting an assertion which appears to be incorrect while copying from a staging buffer to an image.
The application creates a buffer with a 4-byte aligned allocation. This is the alignment reported as required for STORAGE | TRANSFER_SRC. The buffer is filled with BC3 compressed data, and a vkCmdCopyBufferToImage command is issued (in a compute queue).
The driver then traps here because (view.gpuAddr % requiredAlignment) != 0 inside IsValidTypedBufferView.
Here's the full stack trace:
[amdvlk64.so] Gfx10CreateTypedBufferViewSrds gfx9Device.cpp:3531
[amdvlk64.so] CreateTypedBufferViewSrds palDevice.h:3884
[amdvlk64.so] CopyBetweenMemoryAndImageCs rsrcProcMgr.cpp:1236
[amdvlk64.so] Pal::RsrcProcMgr::CopyBetweenMemoryAndImage(Pal::GfxCmdBuffer *, const Pal::ComputePipeline *, const Pal::GpuMemory &, const Pal::Image &, Pal::ImageLayout, bool, bool, unsigned int, const Pal::MemoryImageCopyRegion *, bool) const rsrcProcMgr.cpp:1305
[amdvlk64.so] CmdCopyMemoryToImage rsrcProcMgr.cpp:968
[amdvlk64.so] CmdCopyMemoryToImage gfx9RsrcProcMgr.cpp:3832
[amdvlk64.so] CmdCopyMemoryToImage gfxCmdBuffer.cpp:358
[amdvlk64.so] CopyBufferToImage<VkBufferImageCopy> vk_cmdbuffer.cpp:3524

requiredAlignment here is bytesPerPixel, which is actually more like "bytes per texel block" (for BC3 this value is 16).
20.3. Copying Data Between Buffers and Images specifies how offsets are calculated, but unlike e.g. 22.4. Vertex Input Address Calculation, it does not seem to impose any alignment requirements on texel addresses. The only obviously applicable alignment requirement I could find in the spec. is that of a TRANSFER_SRC buffer. Perhaps there is also the natural alignment of image component data types, but I couldn't find any explicit mention of it (except for vertex input attributes), and I'm not sure what that would be for block compressed formats.
Judging by the stack trace, buffer-to-image copies use the same machinery as texel buffers. But i couldn't find any connection between the two in the spec. Also, even for texel buffers, this alignment requirement is excessive because minTexelBufferOffsetAlignment reports 4.

AMDVLK is built from the latest sources (except LLVM) with assertions and ASAN enabled.
GPU: RX 6900 XT
Linux: 6.0.8-1-MANJARO
KDE on X11

@glebov-andrey
Hi, Could you please provide your application test?
Thanks

Of course, here is a simplified example:
// This assumes some boilerplate setup has already been done.
// `device` has at a compute queue (has COMPUTE_BIT, but not GRAPHICS_BIT, in my case family 1).
// `command_buffer` was allocated from a VkCommandPool, which was created for that queue.
// Error handling has been omitted.
void test(const VkDevice device,
          const VkPhysicalDeviceMemoryProperties memory_properties,
          const VkCommandBuffer command_buffer) {
    const auto find_memory_type = [&](const std::uint32_t memory_type_mask,
                                      const VkMemoryPropertyFlags required_properties) {
        for (std::uint32_t idx = 0; idx != memory_properties.memoryTypeCount; ++idx) {
            if (memory_type_mask & (std::uint32_t{1} << idx)) {
                if ((memory_properties.memoryTypes[idx].propertyFlags & required_properties) == required_properties) {
                    return idx;
                }
            }
        }
        assert(!"memory type not found");
    };

    constexpr std::uint32_t image_extent = 16;
    // For BC3_UNORM_BLOCK:
    constexpr std::uint32_t block_extent = 4;
    constexpr vk::DeviceSize block_size = 16;
    constexpr auto block_count = (image_extent / block_extent) * (image_extent / block_extent);
    constexpr auto buffer_size = block_count * block_size;

    VkBuffer buffer = nullptr;
    const VkBufferCreateInfo buffer_create_info = {
            .sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO,
            .pNext = nullptr,
            .flags = {},
            .size = buffer_size,
            .usage = VK_BUFFER_USAGE_STORAGE_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_SRC_BIT,
            .sharingMode = VK_SHARING_MODE_EXCLUSIVE,
            .queueFamilyIndexCount = 0,
            .pQueueFamilyIndices = nullptr};
    vkCreateBuffer(device, &buffer_create_info, nullptr, &buffer);
    VkMemoryRequirements buffer_memory_requirements = {};
    vkGetBufferMemoryRequirements(device, buffer, &buffer_memory_requirements);
    assert(buffer_memory_requirements.alignment == 4);
    const auto buffer_memory_type = find_memory_type(buffer_memory_requirements.memoryTypeBits,
                                                     VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT);
    VkDeviceMemory buffer_memory = nullptr;
    const VkMemoryAllocateInfo buffer_allocate_info = {
            .sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO,
            .pNext = nullptr,
            .allocationSize = buffer_memory_requirements.size + buffer_memory_requirements.alignment,
            .memoryTypeIndex = buffer_memory_type};
    vkAllocateMemory(device, &buffer_allocate_info, nullptr, &buffer_memory);
    vkBindBufferMemory(device, buffer, buffer_memory, buffer_memory_requirements.alignment); // minimum valid alignment

    VkImage image = nullptr;
    const VkImageCreateInfo image_create_info = {.sType = VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO,
                                                 .pNext = nullptr,
                                                 .flags = {},
                                                 .imageType = VK_IMAGE_TYPE_2D,
                                                 .format = VK_FORMAT_BC3_UNORM_BLOCK,
                                                 .extent = {.width = image_extent, .height = image_extent, .depth = 1},
                                                 .mipLevels = 1,
                                                 .arrayLayers = 1,
                                                 .samples = VK_SAMPLE_COUNT_1_BIT,
                                                 .tiling = VK_IMAGE_TILING_OPTIMAL,
                                                 .usage = VK_IMAGE_USAGE_TRANSFER_DST_BIT | VK_IMAGE_USAGE_SAMPLED_BIT,
                                                 .sharingMode = VK_SHARING_MODE_EXCLUSIVE,
                                                 .queueFamilyIndexCount = 0,
                                                 .pQueueFamilyIndices = nullptr,
                                                 .initialLayout = VK_IMAGE_LAYOUT_UNDEFINED};
    vkCreateImage(device, &image_create_info, nullptr, &image);
    VkMemoryRequirements image_memory_requirements = {};
    vkGetImageMemoryRequirements(device, image, &image_memory_requirements);
    const auto image_memory_type = find_memory_type(image_memory_requirements.memoryTypeBits,
                                                    VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT);
    VkDeviceMemory image_memory = nullptr;
    const VkMemoryAllocateInfo image_allocate_info = {.sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO,
                                                      .pNext = nullptr,
                                                      .allocationSize = image_memory_requirements.size,
                                                      .memoryTypeIndex = image_memory_type};
    vkAllocateMemory(device, &image_allocate_info, nullptr, &image_memory);
    vkBindImageMemory(device, image, image_memory, 0);

    const VkCommandBufferBeginInfo begin_info = {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO,
                                                 .pNext = nullptr,
                                                 .flags = {},
                                                 .pInheritanceInfo = nullptr};
    vkBeginCommandBuffer(command_buffer, &begin_info);
    // The application actually dispatches a compute pipeline to fill the buffer, but it doesn't matter for this case.
    const VkImageMemoryBarrier image_memory_barrier = {.sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER,
                                                       .pNext = nullptr,
                                                       .srcAccessMask = {},
                                                       .dstAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT,
                                                       .oldLayout = VK_IMAGE_LAYOUT_UNDEFINED,
                                                       .newLayout = VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL,
                                                       .srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED,
                                                       .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED,
                                                       .image = image,
                                                       .subresourceRange = {.aspectMask = VK_IMAGE_ASPECT_COLOR_BIT,
                                                                            .baseMipLevel = 0,
                                                                            .levelCount = 1,
                                                                            .baseArrayLayer = 0,
                                                                            .layerCount = 1}};
    vkCmdPipelineBarrier(command_buffer,
                         {},
                         VK_PIPELINE_STAGE_TRANSFER_BIT,
                         {},
                         0,
                         nullptr,
                         0,
                         nullptr,
                         1,
                         &image_memory_barrier);
    const VkBufferImageCopy copy_region = {.bufferOffset = 0,
                                           .bufferRowLength = 0,
                                           .bufferImageHeight = 0,
                                           .imageSubresource = {.aspectMask = VK_IMAGE_ASPECT_COLOR_BIT,
                                                                .mipLevel = 0,
                                                                .baseArrayLayer = 0,
                                                                .layerCount = 1},
                                           .imageOffset = {},
                                           .imageExtent = image_create_info.extent};
    vkCmdCopyBufferToImage(command_buffer, buffer, image, VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL, 1, &copy_region); // FAILS HERE

    vkEndCommandBuffer(command_buffer);
    vkDestroyImage(device, image, nullptr);
    vkFreeMemory(device, image_memory, nullptr);
    vkDestroyBuffer(device, buffer, nullptr);
    vkFreeMemory(device, buffer_memory, nullptr);
}

@glebov-andrey Hello, I can't reproduce this issue. Could you please provide a complete test code?
Thanks

@MrRobbin I will make a full example and post it as a Gist or something, but I'll only be able to after the 8th of January.
Until then my best guess is that it might be device-dependent and/or queue family dependent - it only seems to happen in the compute queue.

@MrRobbin The following VkInstance and VkDevice initialization code is enough to cause the assertion (when combined with the previously provided code):
void test() {
    VkInstance instance = nullptr;
    const VkApplicationInfo application_info = {.sType = VK_STRUCTURE_TYPE_APPLICATION_INFO,
                                                .pNext = nullptr,
                                                .pApplicationName = "test-app",
                                                .applicationVersion = VK_MAKE_API_VERSION(0, 0, 1, 0),
                                                .pEngineName = "test-engine",
                                                .engineVersion = VK_MAKE_API_VERSION(0, 0, 1, 0),
                                                .apiVersion = VK_API_VERSION_1_3};
    const VkInstanceCreateInfo instance_create_info = {.sType = VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO,
                                                       .pNext = nullptr,
                                                       .flags = {},
                                                       .pApplicationInfo = &application_info,
                                                       .enabledLayerCount = 0,
                                                       .ppEnabledLayerNames = nullptr,
                                                       .enabledExtensionCount = 0,
                                                       .ppEnabledExtensionNames = nullptr};
    vkCreateInstance(&instance_create_info, nullptr, &instance);

    std::uint32_t physical_device_count = 0;
    vkEnumeratePhysicalDevices(instance, &physical_device_count, nullptr);
    assert(physical_device_count == 1);
    VkPhysicalDevice physical_device = nullptr;
    vkEnumeratePhysicalDevices(instance, &physical_device_count, &physical_device);

    std::uint32_t queue_family_count = 0;
    vkGetPhysicalDeviceQueueFamilyProperties(physical_device, &queue_family_count, nullptr);
    assert(queue_family_count == 3);
    std::array<VkQueueFamilyProperties, 3> queue_family_properties = {};
    vkGetPhysicalDeviceQueueFamilyProperties(physical_device, &queue_family_count, queue_family_properties.data());

    constexpr std::uint32_t queue_family_index = 0;
    constexpr VkQueueFlags expected_queue_flags = VK_QUEUE_GRAPHICS_BIT | VK_QUEUE_COMPUTE_BIT | VK_QUEUE_TRANSFER_BIT |
                                                  VK_QUEUE_SPARSE_BINDING_BIT;
    assert(static_cast<VkQueueFlags>(queue_family_properties[queue_family_index].queueFlags) == expected_queue_flags);

    VkPhysicalDeviceMemoryProperties memory_properties = {};
    vkGetPhysicalDeviceMemoryProperties(physical_device, &memory_properties);

    VkDevice device = nullptr;
    constexpr auto queue_priority = 1.0f;
    const VkDeviceQueueCreateInfo device_queue_create_info = {.sType = VK_STRUCTURE_TYPE_DEVICE_QUEUE_CREATE_INFO,
                                                              .pNext = nullptr,
                                                              .flags = {},
                                                              .queueFamilyIndex = queue_family_index,
                                                              .queueCount = 1,
                                                              .pQueuePriorities = &queue_priority};
    const VkPhysicalDeviceFeatures enabled_features_1_0 = {};
    const VkDeviceCreateInfo device_create_info = {.sType = VK_STRUCTURE_TYPE_DEVICE_CREATE_INFO,
                                                   .pNext = nullptr,
                                                   .flags = {},
                                                   .queueCreateInfoCount = 1,
                                                   .pQueueCreateInfos = &device_queue_create_info,
                                                   .enabledLayerCount = 0,
                                                   .ppEnabledLayerNames = nullptr,
                                                   .enabledExtensionCount = 0,
                                                   .ppEnabledExtensionNames = nullptr,
                                                   .pEnabledFeatures = &enabled_features_1_0};
    vkCreateDevice(physical_device, &device_create_info, nullptr, &device);

    VkCommandPool command_pool = nullptr;
    const VkCommandPoolCreateInfo command_pool_create_info = {.sType = VK_STRUCTURE_TYPE_COMMAND_POOL_CREATE_INFO,
                                                              .pNext = nullptr,
                                                              .flags = VK_COMMAND_POOL_CREATE_TRANSIENT_BIT,
                                                              .queueFamilyIndex = queue_family_index};
    vkCreateCommandPool(device, &command_pool_create_info, nullptr, &command_pool);
    VkCommandBuffer command_buffer = nullptr;
    const VkCommandBufferAllocateInfo command_buffer_allocate_info = {
            .sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_ALLOCATE_INFO,
            .pNext = nullptr,
            .commandPool = command_pool,
            .level = VK_COMMAND_BUFFER_LEVEL_PRIMARY,
            .commandBufferCount = 1};
    vkAllocateCommandBuffers(device, &command_buffer_allocate_info, &command_buffer);

    // ... the code from my previous comment ...

    vkDestroyCommandPool(device, command_pool, nullptr);
    vkDestroyDevice(device, nullptr);
    vkDestroyInstance(instance, nullptr);
}

I've confirmed that the issue is queue family dependent - the assertion only fails in a GRAPHICS (family 0) or COMPUTE (family 1) queue.
The GPU I'm using is an AMD Radeon RX 6900 XT (0x73bf).
I've also rebuilt AMDVLK from scratch using the current dev branch, and the issue still exists.
Here's the full stack trace from an unoptimized debug build:
[amdvlk64.so] Pal::Gfx9::Device::Gfx10CreateTypedBufferViewSrds gfx9Device.cpp:3747
[amdvlk64.so] Pal::IDevice::CreateTypedBufferViewSrds palDevice.h:3926
[amdvlk64.so] Pal::RsrcProcMgr::CopyBetweenMemoryAndImageCs rsrcProcMgr.cpp:1235
[amdvlk64.so] Pal::RsrcProcMgr::CopyBetweenMemoryAndImage rsrcProcMgr.cpp:1304
[amdvlk64.so] Pal::RsrcProcMgr::CmdCopyMemoryToImage rsrcProcMgr.cpp:967
[amdvlk64.so] Pal::Gfx9::RsrcProcMgr::CmdCopyMemoryToImage gfx9RsrcProcMgr.cpp:3918
[amdvlk64.so] Pal::GfxCmdBuffer::CmdCopyMemoryToImage gfxCmdBuffer.cpp:358
[amdvlk64.so] vk::CmdBuffer::PalCmdCopyMemoryToImage vk_cmdbuffer.cpp:1317
[amdvlk64.so] vk::CmdBuffer::CopyBufferToImage<VkBufferImageCopy> vk_cmdbuffer.cpp:3548
[amdvlk64.so] vk::entry::vkCmdCopyBufferToImage entry.cpp:370
[libvulkan.so.1] vkCmdCopyBufferToImage 0x00007ffff7ceb41e

@glebov-andrey Thanks, I have reproduced this issue.

@glebov-andrey Hello, this assert is not necessary, Could you please try the release driver?

@MrRobbin Everything appears to work with a release build of the driver with assertions disabled.
My concern was that either this was a driver/hardware constraint which was not disclosed by the properties/limits API, or I misunderstood the specification regarding copyBufferToImage alignment requirements.
Just to confirm, this means that the only alignment requirement for the buffer in this case is that returned by vkGetBufferMemoryRequirements?

@glebov-andrey Sorry for the late response. Let me make a brief summary.
At first, I don't find the  'alignment requirement' in Vulkan spec. I can only find  from the 'vkCmdCopyBufferToImage(...)' is that 'Copy regions for the image must be aligned to a multiple of the texel block extent in each dimension, except at the edges of the image, where region extents must match the edge of the image' but this is not releated to this issue in my view.
Back to the assert failed itself, here is the BufferViewInfo structure, the assert failed at '(view.gpuAddr % requiredAlignment) != 0' .
Struct BufferViewInfo {
Gpusize gpuAddr ;    //Gpu memory virtual address where the buffer view starts, in bytes. Must be aligned to bytes-per-element for typed access.
…
}

In amdvlk, we just do a simple solution about the alignment. We make the memory's virtual address has a large alignment when it was allocated and this seems can guarantee all formats assert success. But your app's  vkBindBufferMemory(…, buffer_memroy_requirements.alignment) will destory the alignment. The minium memory alignment and the
view.gpuAddr's required alignment have different meanings. Sometimes, we can use the vkBindBufferMemory(...) to satisfy the assert. For example, the 'BC3_UNORM_BLOCK' block size is equals to 16, so the bind offset should be multiple of 16. And we can combine the vkGetBufferMemoryRequirements(...) and the block size to determine a specified block's bind offset. In this way, it can avoid assert failures.
To sum up, Vulkan spec don't explicitly specify the alignment, so it doesn't matter.

Please check the fix with  2023.Q1.4 release

Thanks, it's working fine now.


SDPA integration for nvFuser

This PR:

Adds SDPA operator for nvFuser -- currently it is behind the nv_enable_sdpa=True compile flag, and only supports flash attention.
Moves register_operator to Executor class to allow use for nvFuserExecutor.
Moves some common utility functions from sdpaex.py to utils.py

CC: @jjsjann123 @wujingyue

Cool! I'll try it with https://github.com/NVIDIA/Fuser/blob/13ed2bc60836c107d3edfddc5c1bfc4deb79ffb8/benchmarks/python/test_transformer.py#L5-L39 next week.

It seems to work for me! At least, I'm seeing one fusion for forward and one for backward, and they both contain sdpfa nodes.
$ grep fusion_id /tmp/stdout
def nvfuser_fusion_id1(fd : FusionDefinition) -> None :
    nvfuser_fusion_id1(fd)
def nvfuser_fusion_id2(fd : FusionDefinition) -> None :
    nvfuser_fusion_id2(fd)
def nvfuser_fusion_id2(fd : FusionDefinition) -> None :
    nvfuser_fusion_id2(fd)
def nvfuser_fusion_id2(fd : FusionDefinition) -> None :
    nvfuser_fusion_id2(fd)
def nvfuser_fusion_id2(fd : FusionDefinition) -> None :
    nvfuser_fusion_id2(fd)
def nvfuser_fusion_id2(fd : FusionDefinition) -> None :
    nvfuser_fusion_id2(fd)
def nvfuser_fusion_id2(fd : FusionDefinition) -> None :
    nvfuser_fusion_id2(fd)
def nvfuser_fusion_id2(fd : FusionDefinition) -> None :
    nvfuser_fusion_id2(fd)
def nvfuser_fusion_id2(fd : FusionDefinition) -> None :
    nvfuser_fusion_id2(fd)
def nvfuser_fusion_id2(fd : FusionDefinition) -> None :
    nvfuser_fusion_id2(fd)
def nvfuser_fusion_id2(fd : FusionDefinition) -> None :
    nvfuser_fusion_id2(fd)
def nvfuser_fusion_id2(fd : FusionDefinition) -> None :
    nvfuser_fusion_id2(fd)
def nvfuser_fusion_id2(fd : FusionDefinition) -> None :
    nvfuser_fusion_id2(fd)
def nvfuser_fusion_id2(fd : FusionDefinition) -> None :
    nvfuser_fusion_id2(fd)
def nvfuser_fusion_id2(fd : FusionDefinition) -> None :
    nvfuser_fusion_id2(fd)
def nvfuser_fusion_id2(fd : FusionDefinition) -> None :
    nvfuser_fusion_id2(fd)
def nvfuser_fusion_id2(fd : FusionDefinition) -> None :
    nvfuser_fusion_id2(fd)
def nvfuser_fusion_id2(fd : FusionDefinition) -> None :
    nvfuser_fusion_id2(fd)
def nvfuser_fusion_id2(fd : FusionDefinition) -> None :
    nvfuser_fusion_id2(fd)
def nvfuser_fusion_id2(fd : FusionDefinition) -> None :
    nvfuser_fusion_id2(fd)
def nvfuser_fusion_id2(fd : FusionDefinition) -> None :
    nvfuser_fusion_id2(fd)

$ grep sdpfa /tmp/stdout
    T87, T88, T89, T90 = fd.ops.sdpfa_fwd(T77, T70, T84, S85, S86, None)
    T336, T337, T338 = fd.ops.sdpfa_bwd(T333, T75, T68, T82, T7, T22, S334, S335, T23, T24, None)
    T336, T337, T338 = fd.ops.sdpfa_bwd(T333, T75, T68, T82, T7, T22, S334, S335, T23, T24, None)
    T336, T337, T338 = fd.ops.sdpfa_bwd(T333, T75, T68, T82, T7, T22, S334, S335, T23, T24, None)
    T336, T337, T338 = fd.ops.sdpfa_bwd(T333, T75, T68, T82, T7, T22, S334, S335, T23, T24, None)
    T336, T337, T338 = fd.ops.sdpfa_bwd(T333, T75, T68, T82, T7, T22, S334, S335, T23, T24, None)
    T336, T337, T338 = fd.ops.sdpfa_bwd(T333, T75, T68, T82, T7, T22, S334, S335, T23, T24, None)
    T336, T337, T338 = fd.ops.sdpfa_bwd(T333, T75, T68, T82, T7, T22, S334, S335, T23, T24, None)
    T336, T337, T338 = fd.ops.sdpfa_bwd(T333, T75, T68, T82, T7, T22, S334, S335, T23, T24, None)
    T336, T337, T338 = fd.ops.sdpfa_bwd(T333, T75, T68, T82, T7, T22, S334, S335, T23, T24, None)
    T336, T337, T338 = fd.ops.sdpfa_bwd(T333, T75, T68, T82, T7, T22, S334, S335, T23, T24, None)
    T336, T337, T338 = fd.ops.sdpfa_bwd(T333, T75, T68, T82, T7, T22, S334, S335, T23, T24, None)
    T336, T337, T338 = fd.ops.sdpfa_bwd(T333, T75, T68, T82, T7, T22, S334, S335, T23, T24, None)
    T336, T337, T338 = fd.ops.sdpfa_bwd(T333, T75, T68, T82, T7, T22, S334, S335, T23, T24, None)
    T336, T337, T338 = fd.ops.sdpfa_bwd(T333, T75, T68, T82, T7, T22, S334, S335, T23, T24, None)
    T336, T337, T338 = fd.ops.sdpfa_bwd(T333, T75, T68, T82, T7, T22, S334, S335, T23, T24, None)
    T336, T337, T338 = fd.ops.sdpfa_bwd(T333, T75, T68, T82, T7, T22, S334, S335, T23, T24, None)
    T336, T337, T338 = fd.ops.sdpfa_bwd(T333, T75, T68, T82, T7, T22, S334, S335, T23, T24, None)
    T336, T337, T338 = fd.ops.sdpfa_bwd(T333, T75, T68, T82, T7, T22, S334, S335, T23, T24, None)
    T336, T337, T338 = fd.ops.sdpfa_bwd(T333, T75, T68, T82, T7, T22, S334, S335, T23, T24, None)
    T336, T337, T338 = fd.ops.sdpfa_bwd(T333, T75, T68, T82, T7, T22, S334, S335, T23, T24, None)
    # (t43, t44, t45, t46) = nv_sdpfa_fwd(t34, t26, t42, 0.1, True, scale=None)
    # (t563, t564, t565) = nv_sdpfa_bwd(t562, t34, t26, t42, t43, t44, 0.1, True, t45, t46, scale=None)

This looks nice to me, I'm wondering if you happen to have run some benchmark to compare the performance with and without nvfuser SDPA?

@wujingyue has a benchmark in this PR: https://github.com/NVIDIA/Fuser/pull/2787.
I will run some benchmarks later this week.

This looks nice to me, I'm wondering if you happen to have run some benchmark to compare the performance with and without nvfuser SDPA?

@wujingyue has a benchmark in this PR: NVIDIA/Fuser#2787. I will run some benchmarks later this week.

I'm not sure if we are talking about the same thing. NVIDIA/Fuser#2787 compares SDPA vs the decomposed version of that (transpose, matmul, softmax, matmul etc). @crcrpar, is that what you were interested in? Or do you want to compare with and without Thunder's SDPA executor?

Excuse my unclear question, I'm interested in the performance benefit of using nvfuser's SDPA in thunder's benchmarks, e.g. https://github.com/Lightning-AI/lightning-thunder/blob/e88966f2f35433e167f40d26e1330684b8ee3fc1/thunder/benchmarks/benchmark_litgpt.py.
E.g., how would it affect the performance of llama-2-7b-hf / llama-3-8B with FSDP zero2?
An example command would be
$ torchrun --nproc-per-node=8 --local-ranks-filter 0 --role rank --tee 3 thunder/benchmarks/benchmark_litgpt.py --compile=thunder_inductor_cat --distributed_mode=fsdp --model_name=Llama-2-7b-hf

Excuse my unclear question, I'm interested in the performance benefit of using nvfuser's SDPA in thunder's benchmarks, e.g. https://github.com/Lightning-AI/lightning-thunder/blob/e88966f2f35433e167f40d26e1330684b8ee3fc1/thunder/benchmarks/benchmark_litgpt.py.
E.g., how would it affect the performance of llama-2-7b-hf / llama-3-8B with FSDP zero2? An example command would be
$ torchrun --nproc-per-node=8 --local-ranks-filter 0 --role rank --tee 3 thunder/benchmarks/benchmark_litgpt.py --compile=thunder_inductor_cat --distributed_mode=fsdp --model_name=Llama-2-7b-hf

I am yet to run these end-to-end benchmarks.


UseDotNet@2 task should clarify that global.json file contents are largely ignored

In the documentation page for the task it says
"For more information on global.json, see Select the .NET version to use."
This appears to indicate that the task would use the global.json file as specified. However it appears that the task only uses the "version" property of the global.json file and ignores other properties (such as "rollForward"), as I think this comment in the pipeline task itself is indicating:
https://github.com/microsoft/azure-pipelines-tasks/issues/19292#issuecomment-1831304847
This is extremely misleading (and left me confused for a long time). So if this is the case, can these limitations in the global.json support be clarified in the documentation?

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: a9c6264c-6eed-8e94-0d93-47983e69060c
Version Independent ID: a9c6264c-6eed-8e94-0d93-47983e69060c
Content: UseDotNet@2 - Use dotnet v2 task
Content Source: task-reference/use-dotnet-v2.md
Service: azure-devops-pipelines
Sub-service: azure-devops-pipelines-tasks
GitHub Login: @steved0x
Microsoft Alias: sdanie

Thank you for your feedback. We are migrating this issue to our new documentation feedback system, and closing this issue.
As we phase out GitHub issues for this repository, we encourage you to propose changes using public PRs or submit feedback through our updated control at the top of the page. For more information on our feedback systems, see https://aka.ms/contentuserfeedback.


chore: Add retry logic for the ci workflows

Add retry logic for maven install to fix the random issues pulling maven dependencies from maven central.

@meltsufin Could you take another look when you get a chance?


Using generics as record keys doesn't narrow types correctly

Bug Report
🔎 Search Terms
assigning a narrowed type to a variable behaves as original type
reassigning a narrowed type loses narrowed type
record with generic keys narrowed type lost
🕗 Version & Regression Information

This is the behaviour in every version I tried, and I reviewed the FAQ for entries about narrowing types

⏯ Playground Link
Playground link with relevant code
💻 Code
type TestProps<T extends string> = {
    genericKey: T;
    value: Record<T, Date | undefined>
}

const test = <T extends string>(props: TestProps<T>) => {
    const {genericKey, value} = props;
    const fromDate = value[genericKey];

    if (!fromDate) return 'no value';

    const usedDate = fromDate.getDate();

    const triedToAssignDate: Date = fromDate;

    return value;
}

🙁 Actual behaviour
The variable triedToAssignDate is not using the narrowed type from fromDate and is resorting to the Date | undefined of the original Record value.
🙂 Expected behaviour
I expect to be able to assign fromDate without any issues and use it as a Date object.

This is fixed in the nightly build


[Backport to main] Change the ziputil dependency to fix a potential security concern

Description
This is a backport PR for #824
Check List

[x] New functionality includes testing.

[x] All tests pass

[x] New functionality has been documented.

[x] New functionality has javadoc added

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #831 (6554b2e) into main (35dc179) will decrease coverage by 0.28%.
The diff coverage is 80.00%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff              @@
##               main     #831      +/-   ##
============================================
- Coverage     85.08%   84.80%   -0.28%
- Complexity     1612     1632      +20
============================================
  Files           135      135
  Lines          5973     6082     +109
  Branches        570      596      +26
============================================
+ Hits           5082     5158      +76
- Misses          649      666      +17
- Partials        242      258      +16

Flag
Coverage Δ

ml-commons
84.80% <80.00%> (-0.28%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

.../java/org/opensearch/ml/engine/utils/ZipUtils.java
79.16% <79.16%> (ø)

...a/org/opensearch/ml/engine/algorithms/DLModel.java
85.43% <100.00%> (-0.10%)
:arrow_down:

... and 31 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Fail to restore file share snapshot with error 'Unexpected Cancel'

Storage Explorer Version: 1.33.0-dev (96)
Build Number: 20231119.2
Branch: main
Platform/OS: Windows 10/Linux Ubuntu 20.04/MacOS Sonoma 14.0(Apple M1 Pro)
Architecture: x64/x64/arm64
How Found: From running test cases
Regression From: Previous release (1.32.1)
Steps to Reproduce

Expand one storage account -> File Shares.
Open one file share -> Upload a file.
Right click the file share -> Click 'Create Snapshot'.
Switch to snapshots view -> Open the file share snapshot.
Select the file -> Click 'Restore Snapshot'.
Check whether succeeds to restore snapshot.

Expected Experience
Succeed to restore snapshot.
Actual Experience
Fail to restore snapshot.

A fix has been merged.

Verified this issue on the build 20231121.2. Fixed. So, we are going to close it.


Add support for "Compile File" command

Fixes #931.
This PR changes how the calculate_output_groups.py script works, having it use knowledge of the destination platform to select the correct BAZEL_TARGET_IDs for all configuredTargets, and only using the scheme target ids as a backup.
It also has an unused implementation to pick the library target of a merged target. It's unused because Xcode doesn't update the buildRequest reliably for it, and we can get stuck building the wrong output group.

@BalestraPatrick @jpsim @erikkerber Can y'all test this in your projects? This is on the riskier side and I want to have good coverage.

@brentleyjones Seems to work fine for me. Tested in two different projects containing extensions and tests and couldn't spot any regression.

I think it's the first time in my 11 years of using Xcode I used that command 😅

Yep, same.

Late here but our project does work fine with this as well. It does seem to link like you noted in the issue.
Now I need to see if this should fit into my (or others) workflows 😂

I think using it to confirm a module you are working on compiles is a nice workflow. This way you don't have to switch your scheme away from the App scheme.


Create a presentation of the project to fulfill the assignment guidelines

The presentation of the project should include the fundamental underlying concepts and ideas.
The code structure should also be briefly addressed.
A important component of the presentation involves introducing the software and showcasing the achieved results, complemented by a live demonstration.
The live showcase aims to provide an authentic understanding of the software's functionality in action.
We've got serveral diagrams and graphics in our issues already, they will be a great addition!
Format: PowerPoint

i think it would be good if we form a team of 2 people primarily for this issue 🤔

@heyitsalina we can work on it togather 😎

YUHU


Generate DOIs on release

See https://docs.github.com/en/repositories/archiving-a-github-repository/referencing-and-citing-content

Done


Move prometheus server to main

Even if we have no encode_prom, define the prometheus endpoint so that operational metrics can be reported.
Define prefix for operational metrics.
Remove addr:port from encode_prom definition.
Allow multiple instantiations of encode_prom with different parameters.
All encode_prom instances use the same global prometheus endpoint.

Thanks @KalmanMeth for this! :)

@KalmanMeth the PR looks good to me, just a few minor remarks above.
Since it's a breaking change we must put it on hold because changes are necessary on the operator side as well
/hold

@KalmanMeth I mentioned this shouldn't be merged before the operator PR is ready to address the breaking changes; currently operator deployment is broken, so we need to do either a quick fix on the operator side, or rollback this one

@jotak Sorry for the misunderstanding. I saw that you had removed the the no-merge label and added the label lgtm.

the do-not-merge label is still there :)
no worries .. should be easy to fix in the operator


NCCL error in distributed run: unhandled cuda error

I have the following error while trying to run distributed setup:
The commands are these:
python3.6 train.py \
			--n_gpus=2 \
			--rank=0 \
			--output_directory=outdir_en_22 \
			--log_directory=logdir \
			--hparams=batch_size=$batch_size,distributed_run=True,dist_backend="nccl",dist_url="tcp://localhost:54321" &
python3.6 train.py \
			--n_gpus=2 \
			--rank=1 \
			--output_directory=outdir_en_22 \
			--log_directory=logdir \
			--hparams=batch_size=$batch_size,distributed_run=True,dist_backend="nccl",dist_url="tcp://localhost:54321" &

Other hparams are default (as in master).
The error is the same if I use file:// or tcp:// distributed urls.
The log is:
FP16 Run: False
Dynamic Loss Scaling: True
Distributed Run: True
cuDNN Enabled: True
cuDNN Benchmark: False
Initializing Distributed
Done initializing distributed
FP16 Run: False
Dynamic Loss Scaling: True
Distributed Run: True
cuDNN Enabled: True
cuDNN Benchmark: False
Initializing Distributed
Done initializing distributed
Traceback (most recent call last):
  File "train.py", line 294, in <module>
    args.warm_start, args.n_gpus, args.rank, args.group_name, hparams)
  File "train.py", line 179, in train
Traceback (most recent call last):
    model = load_model(hparams)
  File "train.py", line 294, in <module>
  File "train.py", line 87, in load_model
    args.warm_start, args.n_gpus, args.rank, args.group_name, hparams)
    model = apply_gradient_allreduce(model)
  File "train.py", line 179, in train
  File "/home/ttsynth/tacotron2-nvidia-new/tacotron2/distributed.py", line 135, in apply_gradient_allreduce
    model = load_model(hparams)
    dist.broadcast(p, 0)
  File "train.py", line 87, in load_model
  File "/home/ttsynth/tacotron2-nvidia-new/tacotron2/p3.6_venv/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 723, in broadcast
    model = apply_gradient_allreduce(model)
    work = _default_pg.broadcast([tensor], opts)
  File "/home/ttsynth/tacotron2-nvidia-new/tacotron2/distributed.py", line 135, in apply_gradient_allreduce
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:252, unhandled cuda error
    dist.broadcast(p, 0)
  File "/home/ttsynth/tacotron2-nvidia-new/tacotron2/p3.6_venv/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 723, in broadcast
    work = _default_pg.broadcast([tensor], opts)
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:252, unhandled cuda error

The platform is Linux, Ubuntu 16, python3.6; I installed pytorch 1.0.0 from nightly build just today; I am using 1080 cards. tacotron2 repo was updated just now, on master.
What can be wrong? Is it pytorch bug?

For those who encounter the same error: you really need to use python -m multiproc train.py ... instead of just python train.py .... Still, the error message is very misleading :(


Investigate workarounds for lack or per-resource schemas

SURE-8995
Is your feature request related to a problem? Please describe.

This issue specifically relates to the issue of users that are Project Members of a project (not owners) and also have the Manage Project Members custom role.
Users with this combination cannot exercise the manage clusters feature

They cannot edit the project and manage users given they don't have permissions to go down the edit project path
They cannot ad-hoc manage roles given they don't have permissions to to to Cluster / Cluster and Project Members

Describe the solution you'd like
There are two lo-bar possible workarounds

Users can get to the edit project area but can only manage members
Users with access to Project Role Template Bindings an get to the cluster and project members page

currently this is only open to those with access to Cluster Role Template Bindings

We need to understand what's feasible

Edit Project without editing project

UX wise does this look ok?
Can we offer a different UI that just manages members?

Cluster and Project Members

UX wise what happens, these users can see and attempt to manage project members for projects they don't have access to?
how many types of user has access to PRTBs, if it's lots of others who don't have project owner or manage project members then we've exposed a lot of stuff that isn't going to work

In all cases....

How can we efficiently work out require permissions (we should not fetch ALL PRTB to see if user has manage project members role for specific cluster)

Once way forward is clear, take back to JIRA and raise another issue

@richard-cox do we have a 2.11.0 ticket for this too? and how doable is this for December?

Updated to align correctly with milestone process (enhancements in a minor version, label if backport candidate)


[STORM-4016] KafkaSpout: moving to poll(Duration) due to deprecation of poll(long)

What is the purpose of the change
Details here: https://issues.apache.org/jira/browse/STORM-4016
How was the change tested
mvn clean install was ran locally

@reiabreu feel free to merge it yourself ;-)

@reiabreu You just need to link your ASF ID with GitHub via https://gitbox.apache.org/boxer/ (than it will automatically provide your GitHub account with the required permissions)

@reiabreu Great. Don't forget to assign the jira to you + set the fix version and close it :)


mini-batch with no duplicate speaker_id

Hi Wespeaker team,
Thank you for sharing your amazing work. I am trying to use your work in my research. However, I am not familiar with the data sampler using your toolkit. Could I ask how to modify your data sampler so that in a mini-batch, each training sample has a different speaker_id (no duplicate speaker_id in a mini-batch)?
Thank you for your support!

I think one simple method is to re-define your collate_fn in the dataloader, and filter the training samples of the same spk_id with a batch. Although the batch_size could be a little smaller, the training process should be normal.
You can refer our similar usage of customized collate_fn in ssl/dataset.py. Hope this solve your problem.

Thank you for your suggestion! I will try that.

Then if I want to use triplet loss and a batch contains N speakers with M different utts for each speaker, can I modify your data sampler?


Launch Party - Calculate APY by compounding

Right now the APY is calculated without compounding taken into account.
Right now it's calculated like this:
const apy = (reward.toNumber() / 1_000_000_000 / 5) * 365 * 100;

We gotta calculate it as if we compounded it every 5 days instead.

I'll share the math here, tag Christo and if everything checks out we'll close it.

Alright, so the math is very simple:
let multiplier = reward / 1_000_000_000
let apy = multiplier ** (365 / 5)

The reward is based on the contract reward that's stored as parts per billion. So a 1_100_000_000 reward multiplies the deposit by 1.1 over the course of 5 days. So to simulate compounding every five days for a year we just elevate it to (365/5). Is this math sound?
I mean, sure, it doesn't make much sense because you cannot actually compound the reward because it's on a different currency.
@hpiyankov could I get your input here?

Your multiplier ** (365 / 5) logic is correct as long as for a 10% multiplier you are using 1.1
Yes you need to subtract 1 at the end to get the APY

For the rest, I cannot comment, because it has been such a long time, that this is totally out of context for me.


Error: StartService(), Cannot start the driver: A device attached to the system is not functioning.

Windows 10 x64 (Insider)
Surface Book 2, 16GB RAM
VSM enabled: Yes
Cred. Guard enabled: Yes
HVCI enabled: No
C:\Windows\Temp>winpmem.exe -o c:\Windows\Temp\test.dmp -ddd
2020-05-23 16:19:46 I This is The WinPmem memory imager. version 3.3rc2
2020-05-23 16:19:46 D Creating intermediate directories C:\
2020-05-23 16:19:46 D Creating intermediate directories C:\Users\
2020-05-23 16:19:46 D Creating intermediate directories C:\Users\USER\
2020-05-23 16:19:46 D Creating intermediate directories C:\Users\USER\AppData\
2020-05-23 16:19:46 D Creating intermediate directories C:\Users\USER\AppData\Local\
2020-05-23 16:19:46 D Creating intermediate directories C:\Users\USER\AppData\Local\Temp\
2020-05-23 16:19:46 I Extracted 45368 bytes into C:\Users\USER\AppData\Local\Temp\pme861B.tmp
2020-05-23 16:19:46 I Driver Unloaded.
2020-05-23 16:19:46 E Error: StartService(), Cannot start the driver: A device attached to the system is not functioning.

2020-05-23 16:19:46 I Trying to load non-attestation signed driver.
2020-05-23 16:19:46 D Creating intermediate directories C:\
2020-05-23 16:19:46 D Creating intermediate directories C:\Users\
2020-05-23 16:19:46 D Creating intermediate directories C:\Users\USER\
2020-05-23 16:19:46 D Creating intermediate directories C:\Users\USER\AppData\
2020-05-23 16:19:46 D Creating intermediate directories C:\Users\USER\AppData\Local\
2020-05-23 16:19:46 D Creating intermediate directories C:\Users\USER\AppData\Local\Temp\
2020-05-23 16:19:46 I Extracted 36176 bytes into C:\Users\USER\AppData\Local\Temp\pme867A.tmp
2020-05-23 16:19:47 I Driver Unloaded.
2020-05-23 16:19:47 E Error: StartService(), Cannot start the driver: A device attached to the system is not functioning.

IO_ERROR: at win_pmem.cc: 695
2020-05-23 16:19:47 E Imaging failed with error: IO_ERROR
2020-05-23 16:19:47 I Removed C:\Users\USER\AppData\Local\Temp\pme861B.tmp
2020-05-23 16:19:47 I Removed C:\Users\USER\AppData\Local\Temp\pme867A.tm
```p

I did hear there were some issues loading the driver on Windows insider edition - Maybe Microsoft tightened down driver loading this the new version?
Is there any clue in the event log why the driver won't load?

@scudette
Had the same problem on Version 2004 (Os Build 10.0.19041.329)

Ah interesting - it looks like it is unable to import a function it needs. Possibly one of the acquisition methods is no longer possible since the function it needs is not exported.

@scudette
actually no, that log is being printed before the GetProcAddress stuff.
It fails to find the Kernel Base used in the GetProcAddress.
You should probably use NtQuerySystemInformation with the SystemModuleInformation to get the kernel base

Hmm yeah good point. The reason is calls getprocaddress in the first place is to try to be somewhat sneaky to try and prevent kernel mode rootkits from identifying that we are an imaging application by looking at import tables.
Maybe this is too complicated and we should stop doing that. I can prepare a fix but I am not able to sign the new driver. We could maybe ask our friends at binalyze to sign it again?

Hello everyone,
Is there any progress on this issue?
I've just got Windows 10 2004 update and winpmem has stopped working giving the same error (or maybe it's a different one?):
c:\Program Files\Vound\w4-portable-x64-1.1.1-SNAPSHOT-rev47454\bin>winpmem.exe -ddd -o r:\tmp\image.tmp
2020-07-20 17:33:55 I This is The WinPmem memory imager. version 3.3rc2
2020-07-20 17:33:55 D Creating intermediate directories C:\
2020-07-20 17:33:55 D Creating intermediate directories C:\Users\
2020-07-20 17:33:55 D Creating intermediate directories C:\Users\igorr\
2020-07-20 17:33:55 D Creating intermediate directories C:\Users\igorr\AppData\
2020-07-20 17:33:55 D Creating intermediate directories C:\Users\igorr\AppData\Local\
2020-07-20 17:33:55 D Creating intermediate directories C:\Users\igorr\AppData\Local\Temp\
2020-07-20 17:33:55 I Extracted 45368 bytes into C:\Users\igorr\AppData\Local\Temp\pme82B9.tmp
2020-07-20 17:33:55 I Driver Unloaded.
2020-07-20 17:33:55 E Error: StartService(), Cannot start the driver: A device attached to the system is not functioning.

2020-07-20 17:33:55 I Trying to load non-attestation signed driver.
2020-07-20 17:33:55 D Creating intermediate directories C:\
2020-07-20 17:33:55 D Creating intermediate directories C:\Users\
2020-07-20 17:33:55 D Creating intermediate directories C:\Users\igorr\
2020-07-20 17:33:55 D Creating intermediate directories C:\Users\igorr\AppData\
2020-07-20 17:33:55 D Creating intermediate directories C:\Users\igorr\AppData\Local\
2020-07-20 17:33:55 D Creating intermediate directories C:\Users\igorr\AppData\Local\Temp\
2020-07-20 17:33:55 I Extracted 36176 bytes into C:\Users\igorr\AppData\Local\Temp\pme82D9.tmp
2020-07-20 17:33:55 I Driver Unloaded.
2020-07-20 17:33:55 E Error: StartService(), Cannot start the driver: Windows cannot verify the digital signature for this file. A recent hardware or software change might have installed a file that is signed incorrectly or damaged, or that might be malicious software from an unknown source.

IO_ERROR: at win_pmem.cc: 695
2020-07-20 17:33:55 E Imaging failed with error: IO_ERROR
2020-07-20 17:33:56 I Removed C:\Users\igorr\AppData\Local\Temp\pme82B9.tmp
2020-07-20 17:33:56 I Removed C:\Users\igorr\AppData\Local\Temp\pme82D9.tmp

@ilay122 Thanks - https://docs.microsoft.com/en-us/windows/win32/api/winnt/nf-winnt-rtlpctofileheader indicates that this function is a userspace function so wont work in a kernel module.
I dont think it is worth obfuscating anyway - I will remove that code and go back to basics.

@scudette sure, do what you think is right.
But ntoskrnl.exe does export RtlPcToFileHeader and I have used it before.

@ilay122 Ah ok good to know! If it is available then it would be obviously the easiest solution :-) Thanks for the tip :+1:

Any news on this fix?

Sorry not just yet. I will have to port the project to the latest VS to
make it work with the latest DDK so it might be a bit more complex than I
first thought. Hoping to make some progress soon.
Mike Cohen
Digital Paleontologist,
Velocidex Enterprises
M  +61 470 238 491  <+61+470+238+491>
E<EMAIL_ADDRESS><EMAIL_ADDRESS>On Thu, Aug 27, 2020 at 10:53 PM EccoTheFlintstone<EMAIL_ADDRESS>wrote:

Any news on this fix?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Velocidex/c-aff4/issues/144#issuecomment-681929767,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA5NRIV3BSFUT4KWQ42S733SCZJOFANCNFSM4NIPXVAA
.

Hi all,
I just confirmed that Winpmem 3.3rc3 has this issue on ANY Windows 10 version 2004.
Basically, I created a brand new VM using the Win 10 v.2004 ISO and the right out of the box Winpmem was failing to start the driver.
After that, I had my existing VMs with Windows 10 v.1909 where Winpmem always worked, and to confirm the issue I decided to execute the upgrade to version 2004, and immediately after the first restart Winpmem was failing to load the driver.
In the end, I've confirmed it fails in Windows 10 Enterprise and Windows 10 Professional for Workstations editions.
I have attached a screenshot just to confirm the Windows build number and also Winpmem output

I'll be very interested in seeing a fix for this issue, and I'll be happy to test any new release in the same VMs to confirm if the issue persists.
Regards,
Alex
Automated QA Engineer

There has been a lot of work done on Winpmem by Viviane here https://github.com/Velocidex/WinPmem and it is almost ready for the next release. The new code not only works on all architectures but it is also significantly faster.

Just an update: 4.0rc1 release mentioned above fixes the problem with Win10 2004 update, but it doesn't support VSM mode yet. So you have a PC with both Win10 2004 and VSM, there is no solution for that yet.

@igorrogov
why would not it work? the driver loaded fine on my PC, which has HYPER-V and VSM enabled.
Reading memory was also working.
or maybe I don't understand what you mean by VSM mode ?
I have Windows Version 2004, 19041.508 (intel x64)

Just to clarify. VSM (sometimes called memory guard) uses a hypervisor and in that case reading memory actually fails. The imager has to go back and mark certain pages as unreadable.
This is now fixed in the rc2 binary in the release page so have a good a testing it. You will see when a page is unreadable the progress symbol will be an x icterus l instead of a .

error win_pmem.cc: 695 again

-
System

-
Provider

[
Name]
Service Control
Manager

[
Guid]
{555908d1-a6d7-4695-8e1e-26931d2012f4}

[
EventSourceName]
Service Control
Manager

-
EventID
7000

[
Qualifiers]
49152

Version
0

Level
2

Task
0

Opcode
0

Keywords
0x8080000000000000

-
TimeCreated

[
SystemTime]
2022-06-11T07:17:54.6947851Z

EventRecordID
93674

Correlation

-
Execution

[
ProcessID]
952

[
ThreadID]
6944

Channel
System

Computer
HP-4730-W-10

Security

-
EventData

param1
pmem

param2
%%31

70006D0065006D000000

Binäre Daten:
In Wörtern
0000: 006D0070 006D0065 0000
In Bytes
0000: 70 00 6D 00 65 00 6D 00
p.m.e.m.0008: 00 00                     ..

The winpmem project moved here https://github.com/Velocidex/WinPmem

I believe this is a driver naming conflict. When Windows' pmem driver (located at C:\Windows\System32\Drivers\pmem.sys) is running, the winpmem_mini_x64_rc2.exe service cannot start because the driver included with winpmem_mini_x64_rc2.exe is also named pmem. When the Windows pmem driver is not active, winpmem_mini_x64_rc2.exe runs normally.

我相信这是驱动程序命名冲突。当 Windows 的 pmem 驱动程序（位于 C：\Windows\System32\Drivers\pmem.sys）运行时，winpmem_mini_x64_rc2.exe服务无法启动，因为 winpmem_mini_x64_rc2.exe 附带的驱动程序也命名为 pmem。当 Windows pmem 驱动程序未处于活动状态时，winpmem_mini_x64_rc2.exe正常运行。

you can run ```sc qurey pmem```` in power shell,when winpmem_mini_x64_rc2.exe not work


Semver

I saw that @ScratchCat458's PR was merged -- very happy to see that.
However, it contains API breaking changes (see new() method) but the published version was bumped from 0.1.0 to 0.1.1, which is not semver compliant.
Please consider being semver compatible in the future -- it makes life easier for us downstream users of your crate.
I figure this was just an oversight, but I wanted to mention it just in case.

Good point and apologies! I haven't read up on SemVer in a while and probably should have checked prior to the push.
I agree that we should more strictly follow it in future, especially considering the way cargo matches version numbers.


Wrong query types generated by typescript-react-apollo-template

Hey,
First off, thank u so much for this project and for the react-apollo-template. Works very nicely.
Issue
graphql-code-generator Version: 0.10.7 (latest as of 8/12/2018)
Node Version: 10.8.0
Platform: Ubuntu 18.04
Using command:
npx gql-gen --template graphql-codegen-typescript-react-apollo-template --config gqlgen.json --out __generated/queries.ts src/queries/**/*.graphql --schema __generated/schema.json
gqlgen.json:
{
  "generatorConfig": {
    "noNamespaces": true,
    "resolvers": false
  }
}

For query
query Games {
  games {
    id
    token
    name
  }
}

Result (showing only relevant lines):
export type GamesVariables = {};

export type GamesQuery = {
  __typename?: 'Query';
  games: GamesGames[];
};

export type GamesGames = {
  __typename?: 'Game';
  id: Uuid;
  token: string;
  name: string;
};

import * as ReactApollo from 'react-apollo';

export class GamesComponent extends ReactApollo.Query<Query, GamesVariables> {}

Expected behavior:
class GamesComponent should extend GamesQuery, not Query
I have to use the no-namespace version, since I'm using babel with typescript preset that doesn't support namespaces.
Is this an issue, or am I doing something wrong?
Thank u for your time!

It's been fixed by
https://github.com/dotansimha/graphql-code-generator/commit/e5feb631d4383046b9cc2b00c4fad893ec1567a7#diff-60c0e6de76c51ffb8188d0076519b572R11

Done! 0.11 is available now. @mcongy


[Question]: Question about docker-compose way to run compass want to monitor local spark on yarn

Contact Details
No response
What would you like to ask or discuss?
if you use docker-compose start compass how to connect to my local hadoop and spark it, my local hdfs is a pseudo-cluster mode, yarn is the default configuration, spark running on the yarn cluster, the version number is: spark3.3.1, hadoop3.2.3
The configuration file is as follows
hdfs-site.xml
<configuration>
  <property>
    <name>dfs.replication</name>
    <value>2</value>
  </property>
  <property>
  <name>dfs.namenode.name.dir</name>
  <value>/Users/kaitsan/data/hadoop/hdfs/namenode</value>
  </property>
  <property>
  <name>dfs.datanode.data.dir</name>
  <value>/Users/kaitsan/data/hadoop/hdfs/datanode</value>
  </property>
</configuration>

core-site.xml
<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>
    <property>
        <name>hadoop.proxyuser.kaitsan.hosts</name>
        <value>*</value>
    </property>
    <property>
        <name>hadoop.proxyuser.kaitsan.groups</name>
        <value>*</value>
    </property>
</configuration>

mapred-site.xml
<configuration>
    <property>
         <name>mapreduce.framework.name</name>
         <value>yarn</value>
     </property>
   <property>
         <name>mapreduce.map.java.opts</name>
         <value>-Xmx1024m</value>
     </property>
   <property>
         <name>mapreduce.reduce.java.opts</name>
         <value>-Xmx1024m</value>
     </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>localhost:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>localhost:19888</value>
    </property>
</configuration>

yarn-site.xml
<configuration>

    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
                  <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>

        <property>
                <name>yarn.scheduler.maximum-allocation-mb</name>
                <value>19000</value>
        </property>
<property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>16</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>19000</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-vcores</name>
<value>16</value>
</property>
        <!--Spark Yarn-->
    <!-- 是否开启聚合日志 -->
    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
    <!-- 配置日志服务器的地址,work节点使用 -->
    <property>
        <name>yarn.log.server.url</name>
        <value>http://loaclhost:19888/jobhistory/logs/</value>
    </property>
    <!-- 配置日志过期时间,单位秒 -->
    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>2592000</value>
    </property>
</configuration>

spark-defaults/conf
# Example:
spark.master                    yarn
spark.eventLog.enabled          true
spark.eventLog.compress         true
# 保存在hdfs上
spark.eventLog.dir  hdfs://localhost:9000/tmp/spark-yarn-logs
spark.history.fs.logDirectory hdfs://localhost:9000/tmp/spark-yarn-logs
spark.yarn.historyServer.address=localhost:18080
spark.history.ui.port=18080

I follow https://github.com/cubefs/compass/blob/main/docker/playground/README.md
Started compass
docker compose --profile compass up -d --build
But the Web UI : http://<IP_ADDRESS>:7075/compass is empty.
Is there any other config file I need to change?

@liewkaitsan You can change local host to hostname or ip, usually we map ip to hostname in /etc/hosts. By the way , do you use scheduler?

@liewkaitsan You can change local host to hostname or ip, usually we map ip to hostname in /etc/hosts. By the way , do you use scheduler?
After I change the hostname or ip, do I need to change any configuration files?
Must have access to scheduling tools, I just want to implement spark task monitoring.

@liewkaitsan You can change local host to hostname or ip, usually we map ip to hostname in /etc/hosts. By the way , do you use scheduler?
can

@liewkaitsan You can change local host to hostname or ip, usually we map ip to hostname in /etc/hosts. By the way , do you use scheduler?

There was a problem with the layout, so I've changed it.

@liewkaitsan You can change local host to hostname or ip, usually we map ip to hostname in /etc/hosts. By the way , do you use scheduler?

There was a problem with the layout, so I've changed it.

@liewkaitsan You can change local host to hostname or ip, usually we map ip to hostname in /etc/hosts. By the way , do you use scheduler?
Can I add your wechat or qq?

@liewkaitsan You can change local host to hostname or ip, usually we map ip to hostname in /etc/hosts. By the way , do you use scheduler?
Can I add your wechat or qq?

Yes, Please add wechat ID zebozhuang

@liewkaitsan You can use the one click module to diagnose spark tasks.

You can use the one click module to diagnose spark tasks

like this?
it's error

@liewkaitsan yes, check whether the application-hadoop.yml configuration is correct, and check whether the task-metadata module is running.

@liewkaitsan yes, check whether the application-hadoop.yml configuration is correct, and check whether the task-metadata module is running.

I'm not sure how these should be configured
yarn - clusterName I don't know where to configure this, I don't have yarn HA turned on.
hdfs uses a local pseudo-cluster, localhost:9000.
nameservices. I didn't set that up either
hadoop:
  namenodes:
    - nameservices: log-hdfs
      namenodesAddr: [ "localhost"]
      namenodes: [ "localhost" ]
      user: kaitsan
      password:
      port: 9000
      # scheduler platform hdfs log path keyword identification, used by task-application
      matchPathKeys: [ "flume" ]
      # kerberos
      enableKerberos: false
      # /etc/krb5.conf
      krb5Conf: ""
      #<EMAIL_ADDRESS>      principalPattern:  ""
      # admin
      loginUser: ""
      # /var/kerberos/krb5kdc/admin.keytab
      keytabPath: ""

  yarn:
    - clusterName: "bigdata"
      resourceManager: [ "localhost:8088" ]
      jobHistoryServer: "localhost:19888"

  spark:
    sparkHistoryServer: [ "localhost:18018" ]

@liewkaitsan There is a problem with using localhost. It is best to change it to the hostname or IP. Yarn can also be used without turning on HA. ClusterName is optional.


Fix navigation bars padding and colors

#198  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @ksharma-xyz and the rest of your teammates on  Graphite


Cannot use SuperSafe with Scala 2.12.9

There doesn't appear to be a Scala 2.12.9-targeted artifact available for SuperSafe 1.1.7, so it's not currently possible to use it with this new Scala release (released on Aug 5th).
I'm seeing the following trying to perform a build:
[warn] ==== Artima Maven Repository: tried
[warn]   http://repo.artima.com/releases/com/artima/supersafe/supersafe_2.12.9/1.1.7/supersafe_2.12.9-1.1.7.pom
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	::          UNRESOLVED DEPENDENCIES         ::
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	:: com.artima.supersafe#supersafe_2.12.9;1.1.7: not found
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] 	Note: Unresolved dependencies path:
[warn] 		com.artima.supersafe:supersafe_2.12.9:1.1.7 (Defaults.scala#L3331)

Thanks for all your great work!

@MichaelJAllen I'll release for 2.12.9 today, sorry for the inconvenience caused.

Hey, that was hardly any inconvenience at all! Thanks very much! I'll keep an eye out for it...

@MichaelJAllen fyi @bvenners has deployed SuperSafe 1.1.8 that includes support for Scala 2.13.0 and 2.12.9, can you try to see if it works for you?
Thank you.

@cheeseng @bvenners Works great! Thanks for all your efforts!


Files get re-downloaded always

Describe the bug
I have the performers folder with all the files and when i wanna download the newest files and choose to "not re-download files" when being asked by the script it will still redownload all files, even when the are already in the folder.
To Reproduce
Steps to reproduce the behavior:
I have moved the files outside of the folder before but when i run the script i will put them back in place exactly like they were and this still happens.
Expected behavior
It should only download new files that are not already in the folder.
Screenshots/Logs
https://privatebin.io/?00d65c30613da279#GzachShyL9f8S66PFiZYqwQTqGQrNUTYjBzHcVqvoqFh
Config
{
"main_profile": "main_profile",
"metadata": "{configpath}/{profile}/.data/{model_id}",
"discord": "",
"file_options": {
"save_location": "\\NAS\Medien\Video\XXX_tmp\#OnlyFans_ManyVids_etc\#OFScraper",
"dir_format": "{model_username}/{responsetype}/{mediatype}/",
"file_format": "{filename}.{ext}",
"textlength": 0,
"space_replacer": " ",
"date": "MM-DD-YYYY",
"text_type_default": "letter",
"truncation_default": true
},
"download_options": {
"filter": [
"Videos"
],
"auto_resume": true,
"system_free_min": 0,
"max_post_count": 0
},
"binary_options": {
"ffmpeg": ""
},
"cdm_options": {
"private-key": null,
"client-id": null,
"key-mode-default": "cdrm",
"keydb_api": ""
},
"performance_options": {
"download_sems": 6,
"thread_count": 2,
"download_limit": 0
},
"content_filter_options": {
"block_ads": false,
"file_size_max": 0,
"file_size_min": 0,
"length_max": null,
"length_min": null
},
"advanced_options": {
"code-execution": false,
"dynamic-mode-default": "datawhores",
"backend": "aio",
"downloadbars": false,
"cache-mode": "sqlite",
"appendlog": true,
"custom_values": null,
"sanitize_text": false,
"temp_dir": null,
"remove_hash_match": null,
"infinite_loop_action_mode": false,
"enable_auto_after": true,
"default_user_list": "main",
"default_black_list": ""
},
"scripts_options": {
"post_download_script": null,
"post_script": null
},
"responsetype": {
"timeline": "Posts",
"message": "Messages",
"archived": "Archived",
"paid": "Messages",
"stories": "Stories",
"highlights": "Stories",
"profile": "Profile",
"pinned": "Posts",
"streams": "Streams"
},
"overwrites": {
"audios": {},
"videos": {},
"images": {},
"text": {}
}
}
System Info

Win10
Firefox
Version 3.12.9
python
pip

I'm not experiencing this issue here and haven't seen anyone else mention this recently ether. What command are you using when you run ofscraper?

just ofscraper but it seems to work like expected when i add the --after 2000. does the script check against the filer that are there or some DB it created?

Yes, there is a model DB file that is created which logs everything getting scraped from that model so it should only scrape new content.

This commit I authored should fix it:
https://github.com/datawhores/OF-Scraper/commit/d185f62d39bbbbe2fe5d64355c26cb691303da98
Either wait for a new stable release, install the developement branch, or edit the file manually on your system.

ah i see thank you. so does it only check against the DB or also against the files? Like i said i moved the files outside of the folder at some point and probably when i was using a older version of the script too. now after idk maybe 1 year i come back and subscribe to a model again and scrape. if i make sure the files are there but there is no DB entries(because i used a old version before probably) what will happen? I was hoping it would be enough if the files are there without any DB updates etc.

ah i see thank you. so does it only check against the DB or also against the files? Like i said i moved the files outside of the folder at some point and probably when i was using a older version of the script too. now after idk maybe 1 year i come back and subscribe to a model again and scrape. if i make sure the files are there but there is no DB entries(because i used a old version before probably) what will happen? I was hoping it would be enough if the files are there without any DB updates etc.

If you move files, they won't get redownloaded because it checks against the databases in the meta directory (depending where you saved them. It is set with the config parameter metadata, for exemple, metadata": "ofscraper2/meta/{model_username}_{model_id}/").
If you moved the metadata folder or the databases, things will get redownloaded because ofscrapper won't find the db and recreate them.
You can also force redownloading with --after 2000 --force-all  too.

that's what i mean what if the DB is not updated because it was a old version of ofscraper i downloaded the files with.? now i have a new version installed without DB entries but the files are there. so will it always download again even if the files are already there if the DB is empty?

that's what i mean what if the DB is not updated because it was a old version of ofscraper i downloaded the files with.? now i have a new version installed without DB entries but the files are there. so will it always download again even if the files are already there if the DB is empty?

yeah, I think so. But there is a metadata command to reconstruct your databases:
ofscraper metadata --help                                                                                 38s 439ms  fish
Usage: OF-Scraper metadata [OPTIONS]

  Uses API to modify db files without the need for downloading

metadata_options:
  -md, --metadata METADATA MODE

                                  METADATA MODES

                                  check:

                                  Updates metadata fields for the specified area
                                  via an API

                                  This option does not check for new filenames

                                  update: This mode performs two actions:

                                  1. Updates download status and filenames based
                                  on the presence of actual files

                                  2. Updates metadata fields for the specified
                                  area via an API

                                  complete:

                                  Marks the download as complete and updates
                                  metadata fields via an API

                                  It also uses a new filename if one is
                                  available
  -ms, --mark-stray-locked, --mark-stray
                                  Sets unmatched media items as locked
                                  This is done per api type excluding labels,
                                  and is limited to --after and --before range
  -an, --anon                     Use anonymise creditional
                                  forces --individual flag
                                  force --list flag to be removed
                                  most media is ignored

Try running ofscraper metadata -md 1 assuming the files are still correctly located, or  ofscraper metadata -md 2 followed by options on which models you want  the metadata to be be filed if the files have been moved.

thank you this is very helpful! :)


exposing the SwiftUI Image in WhatsNew.Feature.Image

Is your feature request related to a problem?
In addition to showing a sheet, I'm trying to create a simple List view that shows all the updates by enumerating the WhatsNewItems. One issue I ran into is that the WhatsNew.Feature.Image is essentially a type erased image, so you can't apply Image specific SwiftUI modifiers to it.
What solution would you like?
It would be great if somehow the SwiftUI Image was exposed so that one can apply Image specific modifiers.
What alternatives have you considered?
N/A
Any additional context?

The above is the screen I'm trying to create.

Since the underlying image is retained via a view builder closure there is currently no direct way in retrieving the SwiftUI.Image instance.
Please feel free to open up a PR.


'There are no more files'

Whenever I run my WPF app I get this exception in the call to HockeyClient.Current.Configure.SendCrashesAsync(): System.IO.IOException: 'There are no more files.
This is the value of the maybeFullPath variable at the top of the stack:
HockeyCrashes\CE-BF-A7-1E-54-C7-5A-B7-C2-FC-0B-EA-EF-03-6A-59\crashinfo_*.log
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.IsolatedStorage.IsolatedStorageFile.GetFileDirectoryNames(String path, String userSearchPattern, Boolean file)
   at System.IO.IsolatedStorage.IsolatedStorageFile.GetFileNames(String searchPattern)
   at Microsoft.HockeyApp.HockeyPlatformHelperWPF.<GetFileNamesAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.HockeyApp.HockeyClient.<GetCrashFileNamesAsync>d__103.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.HockeyApp.HockeyClient.<AnyCrashesAvailableAsync>d__105.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.HockeyApp.HockeyClientWPFExtensions.<SendCrashesAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

In order to work around this I've written my call to send crashes async like this:
try
{
    await HockeyClient.Current.SendCrashesAsync();
}
catch (Exception ex)
{
    // ...
}

I'm using HockeySDK.WPF v4.1.6

Hi @jgcoded,
thx for getting in touch and the inconvenience this causes. We're looking at this and I'll update the ticket accordingly.
Best,
Benjamin

Hi @jgcoded
Your path seems to be right. Seems like IsolatedStorage is throwing this error which is strange. Analyzing information in this issue and issue #136 there is something unique in your project setup that is causing this two errors. So could you please provide more info on how is your project setup, what other dependencies you are using and everything that might help to localize the root of these problems.
Thanks,
Murat

Thanks for getting back to me Murat. My project is setup as a wpf app that gets converted to a uwp app with the Desktop Bridge tool. The issue only occurred when I was debugging the Desktop Bridge version of the app in VS 2017. Release mode didn't throw these exceptions.
I don't use HockeySDK anymore, but at that time my only other dependency was StructureMap iirc.


RetryStrategy can not handle thrown DBConnection errors

Due to the way DBConnection.ConnectionError can be sometimes be thrown instead of returned (https://github.com/lexhide/xandra/issues/119), RetryStrategy does not handle them. Please see run_with_retrying, which does not try/rescue the call to fun here: https://github.com/lexhide/xandra/blob/2ba4701930ec0c515070f15fd217b870deb57560/lib/xandra/retry_strategy.ex#L126-L127
I ran into this situation:
On two different processes nearly the same error was returned on one and thrown on the other from the same call spot at nearly the same time, which was very unexpected and confusing.
The retry strategy could handle one and I logged it as a warning, but the thrown error just killed the process.

@mezz I think it would make sense to rescue DBConnection.ConnectionErrors since they seem to be raised. Would you like to submit a PR? I don't have cycles currently to work on Xandra :)

Thanks @whatyouhide, I have just created a PR for this here: https://github.com/lexhide/xandra/pull/184

On two different processes nearly the same error was returned on one and thrown on the other from the same call spot at nearly the same time, which was very unexpected and confusing.

I think I'm more inclined towards this being a DBConnection issue rather than a Xandra issue. 🤔

@qcam I am not entirely sure this is an inconsistency in DBConnection, it seems to be in how Xandra handles calling it in different situations.
I found this previous issue, where it seems like Xandra previously chose to handle this case inconsistently: https://github.com/lexhide/xandra/issues/119

I don't think that's Xandra handling the return differently. Xandra does handle when the call returns with {:error, error} (as what the DBConnection docs says), but rather DBConnection would raise when it fails to check out a connection. The raising behavior (what it raises on, what it raises with) is not really documented on DBConnection, which means technically that could change any time. I personally don't see this as a bug on Xandra :).
I guess the question becomes if we would want to retry on checkout failure. IMO if we do, we might want to guard against this behavior explicitly, rather than rescuing every DBConnection.ConnectionError exception.

Thanks @qcam, I think you have better context on this issue than I do.
I don't think it makes sense to raise some errors and return others. They should be handled one documented way and differentiated by the type of error instead.
If this DBConnection behavior can be changed without breaking things significantly, I agree it would be best to fix it there.
If in general DBConnection leaves errors and retries up to the caller, I think that by consistently returning the error with {:error, error} it  would allow consumers like Xandra to handle the error however they want, either by retrying or failing.

@qcam raises good points. @mezz it would be great if you could report this issue to DBConnection? In the meantime, I'll close this issue and #184.


Glift for Drupal

Hello,
As you know I am developing Glift for Drupal at https://www.drupal.org/sandbox/varenius/2345991. I started off by solving a few basic things myself to get a feeling for Drupal module writing and Glift. Now when I have a basic version working, I decided to look at the wordpress code for inspiration. I find it very well structured and clear to read - well done! I notice you solved a few things in a better way than I did, and other things that I thought about you have also adressed. I fact, I decided to incorporate most of it into the Drupal version. Since the Wordpress development seems closely tied with the Glift development, I think that it is good if the Drupal development can be synchronised as closely as possible to this wordpress plugin development. Hence, in the coming days I will try to modify the Drupal module code to be an interface to the classes and tools already written within the wordpress plugin. That way, I hope that once glift is updated, it will be reflected in the wordpress plugin, and some updates could then be very easily (perhaps straight copy-paste or file-replacement) transferred from the Wordpress plugin to the Drupal module.
Again, great work - I will keep you posted on the news from the Drupal side.
Best,
Eskil

Thanks Eskil,
Sorry for not replying to this comment earlier. I was very busy helping customers etc in the months before Christmas last year and never managed to get back to you.
I agree that it would be better if the code could be reused as much as possible and I'm happy if you can use some of my work to save time. You may have noticed that the WordPress plugin makes fairly heavy use of native WordPress functions and APIs. Because of that, I wonder whether it would be better to make the code more modular and move all the platform dependent stuff to one place. For example, maybe we could write some wrappers for the functions that are native to WordPress/Drupal/Joomla etc and keep the rest generic? Please let me know whether you think that would be feasible in this case.
I've tried my best to make the code as clear and readable as possible, but I think it could still be improved a lot in terms of architecture and modularity. This is still a very early version.
Also, you'll notice in the latest version in the develop branch that there's a lot of new code to add a settings page in WordPress. This is an example of something that would have to be totally rewritten for Drupal and I was worried about that when I originally saw your comment last year. Still, I think these issues could be solved with a smarter design :)

Hi again,
Good points and ideas. Unfortunately my time is very limited at the moment so I do not have time to dive in to this or come with suggestions right now. If I find the time later this spring I might contact you again to discuss the best way to share code between the modules.
Best,
Eskil

No worries. Thanks Eskil and sorry that the timing didn't work out well between us. Feel free to get in touch any time and hopefully I'll see you over at GGG in any case :)
David


The social icon is not update on your side bar section

Description
In the sidebar section there are social media icon but the twitter logo is not updated
Please assign this to me and also add label
Screenshots
Here's the image of it:

Any additional information?
No response
What browser are you seeing the problem on?
No response
Checklist

[X] I have checked the existing issues.
[X] I have read the Code Of Conduct
[X] The changes don't break the code
[X] Please attach the screenshots and video of the development work once it is done

i want to work on this project. plaese. assign it to me

@Muskan753951 this issue is already solved
#220


Add high-performance sprite option

This would include Canvas and WebGL support, and would be able to handle multiple different sprites (e.g. without creating Scenery nodes for each, since those have a lot of overhead).

There were some complications with setting up the variable-length buffers needed, and a lot of code design decisions have complicated trade-offs. Finding the right level of abstraction that makes things simple, but also performant is trickier.

Implemented, closing.

Reopening for main input-based features being added, along with a demo and image hit testing changes. Requested for Natural Selection.

Hit testing integration with Sprites and associated types should be set up, including per-pixel hit testing in SpriteImage. Demo for scenery-phet available as scenery-phet/scenery-phet_en.html?brand=phet&ea&screens=3&component=Sprites

@pixelzoom can you verify that the above demo is working on your end, and look at integration into natural selection?

Deployed a demo that could be useful for performance testing this general feature: https://phet-dev.colorado.edu/html/scenery-phet/1.0.0-dev.20/phet/scenery-phet_en_phet.html?screens=3&component=Sprites

On 7/15/20, @jonathanolson and I met on Zoom, and made a couple of minor tweaks.  Otherwise this is working nicely in Natural Selection, see phetsims/natural-selection#128.   Unassigning myself.

@jonathanolson if this is ready for review, set label to "status:ready-for-review" and assign to @ariel-phet.

Since @pixelzoom used this feature recently in Natural selection he is probably the most appropriate reviewer.

All looks good here.  Nicely documented, and using this API in Natural Selection was relatively straightforward.
Two items for @jonathanolson. Feel free to close this issue after addressing them.

[ ] I'm curious about why SpritesWebGLDrawable and SpritesCanvasDrawable are implemented using inherit instead of class.

[ ] There is an outstanding TODO in SpritesWebGLDrawable that needs to be addressed.

I'm curious about why SpritesWebGLDrawable and SpritesCanvasDrawable are implemented using inherit instead of class.

Scenery Drawables currently use inherit, and due to mixins/traits it will be beneficial to "upgrade" all of those to class at the same time (so we don't have to have separate inherit/class versions of traits).

The above should be handled, closing. Thanks!


Add ARM support

Motivation
As Falco supports ARM, I guess it would be nice for sidekick to support ARM as well.
Feature
Support ARM CPUs
Alternatives
N/A
Additional context

Nevermind, I've seen it supports ARM... what doesn't support ARM is the redis dependency. I guess I'll open the issue in the helm chart repo instead. Sorry for the noise.


[SMALLFIX] Update summary javadoc of GCSUnderFileSystem

ixes Alluxio/new-contributor-tasks#81
New contributor task: Alluxio/new-contributor-tasks#81

Can one of the admins verify this patch?

Can one of the admins verify this patch?

@bf8086
Can you take a look please?

@bf8086
Can you take a look please?

alluxio-bot, merge this please

alluxio-bot, merge this please


HOME icon

Hello, I am using ghost 0.7.9 and  the clickable home icon doesn't show at the top left of the page.
How should configure to make the icon appear ?

Heya, as far as I remember this home link is based on your logo uploaded in the settings.

Oh I see, thanks : )
2016年5月23日星期一，Hayden Bleasel<EMAIL_ADDRESS>写道：

Heya, as far as I remember this home link is based on your logo uploaded
in the settings.
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/haydenbleasel/ghost-themes/issues/31#issuecomment-220935187


Refine GitHub actions

Apply latest version of gradle-enterprise-convention plugin.
Publish build scan to public instance in community PR.
Don't add custom values which depend on TeamCity environment
variables.

@bot-gradle test and merge

Your PR is queued. See the queue page for details.

Pre-tested commit build failed.


Promise.withResolvers is not a function

Getting stuck in the ingestion step with the following error:
@jimmyland22 ➜ /workspaces/azure-openai-rag-workshop (main) $ ./scripts/ingest-data.sh
Loading azd .env file from current environment
Installing dependencies and building CLI
up to date, audited 699 packages in 2s
155 packages are looking for funding
run npm fund for details
found 0 vulnerabilities

<EMAIL_ADDRESS>build
tsc

Running "ingest-files" CLI tool
Ingesting 3 file(s)...
Ingesting file "./data/privacy-policy.pdf"...
Error ingesting files: Error ingesting file "./data/privacy-policy.pdf": Unknown server error: Promise.withResolvers is not a function

Run into the same situation as @jimmyland22 but pinning the pdf-js version didn't help(tried various pdf-js versions, all with same output):

<EMAIL_ADDRESS>build
tsc

Running "ingest-files" CLI tool
Ingesting 13 file(s)...
Ingesting file "./data/xxxxxxxxxxxxxxxxxxxx.pdf"...
Error ingesting files: Error ingesting file "./data/Bxxxxxxxxxxxxxxxxxxxxx.pdf": Unknown server error: Promise.withResolvers is not a function
npm list pdfjs-dist
<EMAIL_ADDRESS>/workspaces/azure-openai-rag-workshop
└─┬<EMAIL_ADDRESS>-> ./src/ingestion
└──<EMAIL_ADDRESS>

The ingestion service has been rewritten entirely, so this issue should not occur anymore.
Thanks for reporting