Add RowsStreamingWindowBuild to avoid OOM in Window operator

Unlike StreamingWindowBuild, RowLevelStreamingWindowBuild  in this PR is capable of processing window functions as rows arrive within a single partition, without the need to wait for the entire partition to be ready. This approach can significantly reduce memory usage, especially when a single partition contains a large amount of data. It is particularly suited for optimizing rank and row_number functions, as well as aggregate window functions with a default frame.
The detailed discussions is here. The design doc is here.

@mbasmanova @aditi-pandit Can you help to review? Thanks for your help.

@JkSelf : Had a high level question about this PR.
My understanding is that :

You are trying to simulate an end of partition by artificially adding a partitionStartRow offset for each input block.
Each partition has an offset indicating which row position is the first one.
So for every input block you might naturally encounter end of partition if there is such a row. But you always reach end of partition at the end of the block.
The next input block is a sort of new partition but carries the offset in the partition.

That would mean that resetPartition function is called at end of each input block. That is changing the semantics of WindowFunction. That is misleading. We shouldn't do that.
Instead we should enhance WindowPartition structure to model a partially filled partition. The code in Window operator that iterates in callApplyLoop would have logic to handle paritial WindowPartitions.

@aditi-pandit Make sense to me. Updated based on your suggestions and make the WindowPartition be a partially partition. Can you help to review again? Thanks.

@JkSelf Would you update the PR description to provide some context on the changes here. Specifically, please, describe the overall design you implemented.

@rui-mo @PHILO-HE Folks, would you help review this PR?

HI @JkSelf : Sorry for the delay in my review. I got swamped with many things the last 2 days
Agree with others about having a design doc. Putting a doc together will give an opportunity to think about this more clearly from first principles.
Had a suggestion for you about this PR as well:
i) Split this PR into multiple parts. One specifically to add the streaming metadata to the function. Like @mbasmanova pointed out even aggregate functions are streaming for default window frame. So it would be better to add an API for WindowFunction to get if the function implementation in streaming. The parameters of this method include the windowFrame.
ii) The second enhances WindowPartition with the partial data. Ensure this part works with both Streaming and SortWindowBuild.
iii) I feel that RankLikeWindowBuild is not separately needed. We could enhance StreamingWindowBUild to accomodate the changes directly. Maybe give that a thought as well.
Thanks.

@mbasmanova @aditi-pandit @rui-mo Thanks for your review and suggestions.
I have written a design document about streaming processing for Rank and row_number here. Correct me if something wrong. Thanks.

Had a suggestion for you about this PR as well:
i) Split this PR into multiple parts. One specifically to add the streaming metadata to the function. Like @mbasmanova pointed out even aggregate functions are streaming for default window frame. So it would be better to add an API for WindowFunction to get if the function implementation in streaming. The parameters of this method include the windowFrame.

Sure. I will split one PR to solve the streaming metadata into WindowFunction.

ii) The second enhances WindowPartition with the partial data. Ensure this part works with both Streaming and SortWindowBuild.

The patch already support WindowPartition with partial input. And can pass the window test with StreamingWindowBuild and SortWindowBuild. The failed unit tests is not related with this PR.

iii) I feel that RankLikeWindowBuild is not separately needed. We could enhance StreamingWindowBUild to accomodate the changes directly. Maybe give that a thought as well.

It is feasible to put the current RankLikeWindowBuild into StreamingWindowBuild, but it will increase the complexity of the StreamingWindowBuild code. Personally, I feel that keeping them separate would be clearer.

@mbasmanova @rui-mo @aditi-pandit Resolved all your comments. Can you help to review again? Thanks for your help.

@mbasmanova @aditi-pandit @rui-mo  Already support the agg window function with default frame in this PR. Please help to review. Thanks.

@mbasmanova @aditi-pandit @rui-mo @PHILO-HE Can you help to review? The failed unit tests is not related with this PR. Thanks.

@aditi-pandit I have resolved all your comments. Can you help to review again? Thanks.

@PHILO-HE I have resolved all your comments. Can you help to review again? Thanks.

@aditi-pandit Thanks for your review. I have resolved all your comments. Can you help to review again? Thanks.

@mbasmanova @aditi-pandit @PHILO-HE @rui-mo Can you help to review this PR? Thanks for your help.

@mbasmanova @aditi-pandit @PHILO-HE @rui-mo Can you help to review this PR? Thanks for your help.

@mbasmanova @aditi-pandit Do you have more comments? The PR is to fix the issue of TPCDS Q67.

@mbasmanova, @FelixYBW : Sorry folks on the delay on this PR. I fell quite unwell last week and am recovering now. Am catching up with all reviews now.

@JkSelf : I looked at this code again but want to brainstorm more design with you to finalize this. I've started some notes at https://github.com/facebookincubator/velox/discussions/8975#discussioncomment-9131351, but am not completed.
Are you open to doing a video call to discuss more design and what I'm thinking ? Would really help.

@JkSelf : I looked at this code again but want to brainstorm more design with you to finalize this. I've started some notes at #8975 (comment), but am not completed.
Are you open to doing a video call to discuss more design and what I'm thinking ? Would really help.

@aditi-pandit I apologize for the delayed response.
I find your proposal to be excellent as it addresses the fundamental issue of window operator out-of-memory (OOM) problems. Currently, even with the RowsStreamingWindow  introduced in this PR, some of Gluten's customers still face unresolved cases, about 20%, with typical SQL scenarios involving aggregate window functions with "Rows between Unbounded Preceding and Unbounded Following" clauses. However, I believe that if we support window spilling, the OOM issues associated with these queries can also be resolved effectively.
Nonetheless, we are aware that enabling spilling can impact performance. In contrast, the RowsStreamingWindow optimization in this PR not only solves the OOM issue but also avoids performance loss by not requiring data to be spilled to disk, which I consider to be a superior solution. For cases where RowsStreamingWindow optimization cannot be applied, we can resort to the window spilling strategy. Therefore, my current thought is that we should proceed with the current PR as per our previous discussions. For those use cases where RowsStreamingWindow is not applicable, we can then address them using window spilling.
@mbasmanova @FelixYBW  @zhouyuan @PHILO-HE @rui-mo What do you think?

Thanks @JkSelf. I'm glad you are liking that proposal. I'll get to spend more time refining that doc on Friday. Will update you on completion. Lets connect next week to discuss next steps.

@JkSelf : My main design concern in this PR is mostly around WindowPartition and the new virtual functions that were added to it. I feel they are not needed.
I tried to draft something that would use the WIndowPartition class itself, but only extend it a bit to also allow for partial partitions. https://github.com/facebookincubator/velox/pull/9661/commits/d757b124f55ef53370ffdf240b8cf808d6cc619d
This is not a complete implementation but gives the main idea. It would vastly simplify the code. Let me know what you think.

@JkSelf : My main design concern in this PR is mostly around WindowPartition and the new virtual functions that were added to it. I feel they are not needed.
I tried to draft something in this PR code that would use the WIndowPartition class itself, but only extend it a bit to also allow for partial partitions. d757b12
This is not a complete implementation but gives the main idea. It would vastly simplify the code. Please feel free to pick up those changes in your PR. Let me know what you think.

@aditi-pandit I apologize for the late response as I was on vacation recently. Thank you very much for your improvement.  Such changes indeed save many unnecessary methods in WindowPartition. I have made the modifications according to your commit. Could you please help review it again? Thanks.

@aditi-pandit Can you help to review again? Thanks.

Might be better to consolidate its logic in RowsStreamingWindowBuild instead d757b12 tries to do that.

@aditi-pandit  Delete RowsStreamingWindowPartition  class.

Might be better to consolidate its logic in RowsStreamingWindowBuild instead d757b12 tries to do that.

@aditi-pandit Delete RowsStreamingWindowPartition class.

@JkSelf : This is great. I'll do a review today or tomorrow latest.

@aditi-pandit Thanks for your review. I have resolved all your comments. Can you help to review again? Thanks.

@aditi-pandit Can you help to review again? Thanks for you help.

@aditi-pandit Resolved all your comments. Can you help to review again? Thanks.

@aditi-pandit I have resolved all your comments. Can you help to review again? Thanks.

@JkSelf : Another high level comment : Might be a good idea to add plans with RowsStreamingWindowBuild in WindowFuzzer testAlternativePlans https://github.com/facebookincubator/velox/blob/main/velox/exec/fuzzer/WindowFuzzer.cpp#L293

@JkSelf : It would be great to add WindowFuzzer test. But other than that this PR is looking good to me.

@aditi-pandit When we run the TPC-DS test, we occasionally encounter the following error. This issue is unstable and cannot be consistently reproduced. According to the error log stack trace, the problem seems to occur when determining whether the last row of the previous batch and the first row of the current batch belong to the same peer. Do you have any suggestions? Alternatively, is there another method we can use to determine if the data belongs to the same peer?
ISSUE 1:
C  [libvelox.so+0xd903934]  xsimd::batch<signed char, xsimd::fma3<xsimd::avx2> > xsimd::kernel::load_unaligned<xsimd::fma3<xsimd::avx2>, signed char, void>(signed char const*, xsimd::kernel::convert<signed char>, xsimd::avx const&)+0x3e
C  [libvelox.so+0xd9020a8]  xsimd::batch<signed char, xsimd::fma3<xsimd::avx2> > xsimd::batch<signed char, xsimd::fma3<xsimd::avx2> >::load_unaligned<signed char>(signed char const*)+0x46
C  [libvelox.so+0xd8ff97d]  facebook::velox::simd::detail::CopyWord<xsimd::batch<signed char, xsimd::fma3<xsimd::avx2> >, xsimd::fma3<xsimd::avx2> >::apply(void*, void const*)+0x37
C  [libvelox.so+0xd8fc794]  bool facebook::velox::simd::detail::copyNextWord<xsimd::batch<signed char, xsimd::fma3<xsimd::avx2> >, xsimd::fma3<xsimd::avx2> >(void*&, void const*&, int&)+0x3f
C  [libvelox.so+0xd8f990a]  void facebook::velox::simd::memcpy<xsimd::fma3<xsimd::avx2> >(void*, void const*, int, xsimd::fma3<xsimd::avx2> const&)+0x4a
C  [libvelox.so+0x17eeb28e]  facebook::velox::ByteOutputStream::appendStringView(std::basic_string_view<char, std::char_traits<char> >)+0xca
C  [libvelox.so+0x17eeb1bc]  facebook::velox::ByteOutputStream::appendStringView(facebook::velox::StringView)+0x46
C  [libvelox.so+0x17ef046a]  facebook::velox::HashStringAllocator::copyMultipartNoInline(facebook::velox::StringView const&, char*, int)+0x106
C  [libvelox.so+0x15d48e5c]  facebook::velox::HashStringAllocator::copyMultipart(facebook::velox::StringView const&, char*, int)+0x64
C  [libvelox.so+0x15d42a77]  facebook::velox::exec::RowContainer::storeVariableSizeAt(char const*, char*, unsigned int)+0x11b
C  [libvelox.so+0x15d43508]  facebook::velox::exec::RowContainer::storeSerializedRow(facebook::velox::FlatVector<facebook::velox::StringView> const&, int, char*)+0x24c
C  [libvelox.so+0x15e81095]  facebook::velox::exec::WindowPartition::computePeerBuffers(int, int, int, int, int*, int*)+0x32f
C  [libvelox.so+0x15f8a42f]  facebook::velox::exec::Window::computePeerAndFrameBuffers(int, int)+0x2f9
C  [libvelox.so+0x15f8a80e]  facebook::velox::exec::Window::callApplyForPartitionRows(int, int, int, std::shared_ptr<facebook::velox::RowVector> const&)+0x56
C  [libvelox.so+0x15f8aa81]  facebook::velox::exec::Window::callApplyLoop(int, std::shared_ptr<facebook::velox::RowVector> const&)+0x111
C  [libvelox.so+0x15f8ad1b]  facebook::velox::exec::Window::getOutput()+0x1c3
C  [libvelox.so+0x15cf742f]  facebook::velox::exec::Driver::runInternal(std::shared_ptr<facebook::velox::exec::Driver>&, std::shared_ptr<facebook::velox::exec::BlockingState>&, std::shared_ptr<facebook::velox::RowVector>&)+0xf8b
C  [libvelox.so+0x15cf5455]  facebook::velox::exec::Driver::next(std::shared_ptr<facebook::velox::exec::BlockingState>&)+0xbf
C  [libvelox.so+0x15dbcc30]  facebook::velox::exec::Task::next(folly::SemiFuture<folly::Unit>*)+0x63c

ISSUE 2:
C  [libvelox.so+0x17ee5c60]  facebook::velox::memory::Allocation::~Allocation()+0x32
C  [libvelox.so+0xdc0a015]  void std::_Destroy<facebook::velox::memory::Allocation>(facebook::velox::memory::Allocation*)+0x1c
C  [libvelox.so+0xdc083be]  void std::_Destroy_aux<false>::__destroy<facebook::velox::memory::Allocation*>(facebook::velox::memory::Allocation*, facebook::velox::memory::Allocation*)+0x32
C  [libvelox.so+0xdc05f17]  void std::_Destroy<facebook::velox::memory::Allocation*>(facebook::velox::memory::Allocation*, facebook::velox::memory::Allocation*)+0x27
C  [libvelox.so+0xdc03c77]  void std::_Destroy<facebook::velox::memory::Allocation*, facebook::velox::memory::Allocation>(facebook::velox::memory::Allocation*, facebook::velox::memory::Allocation*, std::allocator<facebook::velox::memory::Allocation>&)+0x2b
C  [libvelox.so+0x17e9c670]  std::vector<facebook::velox::memory::Allocation, std::allocator<facebook::velox::memory::Allocation> >::_M_erase_at_end(facebook::velox::memory::Allocation*)+0x66
C  [libvelox.so+0x17e99818]  std::vector<facebook::velox::memory::Allocation, std::allocator<facebook::velox::memory::Allocation> >::clear()+0x26
C  [libvelox.so+0x17ee7fcc]  facebook::velox::memory::AllocationPool::clear()+0x20
C  [libvelox.so+0x17eed45f]  facebook::velox::HashStringAllocator::clear()+0x1dd
C  [libvelox.so+0x15d44e50]  facebook::velox::exec::RowContainer::clear()+0x1c6
C  [libvelox.so+0x15d3f842]  facebook::velox::exec::RowContainer::~RowContainer()+0x1c
C  [libvelox.so+0xdb399b0]  std::default_delete<facebook::velox::exec::RowContainer>::operator()(facebook::velox::exec::RowContainer*) const+0x26
C  [libvelox.so+0xdb3320e]  std::unique_ptr<facebook::velox::exec::RowContainer, std::default_delete<facebook::velox::exec::RowContainer> >::~unique_ptr()+0x56
C  [libvelox.so+0x15e81223]  facebook::velox::exec::WindowPartition::computePeerBuffers(int, int, int, int, int*, int*)+0x4bd
C  [libvelox.so+0x15f8a42f]  facebook::velox::exec::Window::computePeerAndFrameBuffers(int, int)+0x2f9
C  [libvelox.so+0x15f8a80e]  facebook::velox::exec::Window::callApplyForPartitionRows(int, int, int, std::shared_ptr<facebook::velox::RowVector> const&)+0x56
C  [libvelox.so+0x15f8ab0d]  facebook::velox::exec::Window::callApplyLoop(int, std::shared_ptr<facebook::velox::RowVector> const&)+0x19d
C  [libvelox.so+0x15f8ad1b]  facebook::velox::exec::Window::getOutput()+0x1c3
C  [libvelox.so+0x15cf742f]  facebook::velox::exec::Driver::runInternal(std::shared_ptr<facebook::velox::exec::Driver>&, std::shared_ptr<facebook::velox::exec::BlockingState>&, std::shared_ptr<facebook::velox::RowVector>&)+0xf8b
C  [libvelox.so+0x15cf5455]  facebook::velox::exec::Driver::next(std::shared_ptr<facebook::velox::exec::BlockingState>&)+0xbf
C  [libvelox.so+0x15dbcc30]  facebook::velox::exec::Task::next(folly::SemiFuture<folly::Unit>*)+0x63c

@aditi-pandit When we run the TPC-DS test, we occasionally encounter the following error. This issue is unstable and cannot be consistently reproduced. According to the error log stack trace, the problem seems to occur when determining whether the last row of the previous batch and the first row of the current batch belong to the same peer. Do you have any suggestions? Alternatively, is there another method we can use to determine if the data belongs to the same peer?
Related code are computePeerBuffers and previousGroupLastRow_
ISSUE 1:
C  [libvelox.so+0xd903934]  xsimd::batch<signed char, xsimd::fma3<xsimd::avx2> > xsimd::kernel::load_unaligned<xsimd::fma3<xsimd::avx2>, signed char, void>(signed char const*, xsimd::kernel::convert<signed char>, xsimd::avx const&)+0x3e
C  [libvelox.so+0xd9020a8]  xsimd::batch<signed char, xsimd::fma3<xsimd::avx2> > xsimd::batch<signed char, xsimd::fma3<xsimd::avx2> >::load_unaligned<signed char>(signed char const*)+0x46
C  [libvelox.so+0xd8ff97d]  facebook::velox::simd::detail::CopyWord<xsimd::batch<signed char, xsimd::fma3<xsimd::avx2> >, xsimd::fma3<xsimd::avx2> >::apply(void*, void const*)+0x37
C  [libvelox.so+0xd8fc794]  bool facebook::velox::simd::detail::copyNextWord<xsimd::batch<signed char, xsimd::fma3<xsimd::avx2> >, xsimd::fma3<xsimd::avx2> >(void*&, void const*&, int&)+0x3f
C  [libvelox.so+0xd8f990a]  void facebook::velox::simd::memcpy<xsimd::fma3<xsimd::avx2> >(void*, void const*, int, xsimd::fma3<xsimd::avx2> const&)+0x4a
C  [libvelox.so+0x17eeb28e]  facebook::velox::ByteOutputStream::appendStringView(std::basic_string_view<char, std::char_traits<char> >)+0xca
C  [libvelox.so+0x17eeb1bc]  facebook::velox::ByteOutputStream::appendStringView(facebook::velox::StringView)+0x46
C  [libvelox.so+0x17ef046a]  facebook::velox::HashStringAllocator::copyMultipartNoInline(facebook::velox::StringView const&, char*, int)+0x106
C  [libvelox.so+0x15d48e5c]  facebook::velox::HashStringAllocator::copyMultipart(facebook::velox::StringView const&, char*, int)+0x64
C  [libvelox.so+0x15d42a77]  facebook::velox::exec::RowContainer::storeVariableSizeAt(char const*, char*, unsigned int)+0x11b
C  [libvelox.so+0x15d43508]  facebook::velox::exec::RowContainer::storeSerializedRow(facebook::velox::FlatVector<facebook::velox::StringView> const&, int, char*)+0x24c
C  [libvelox.so+0x15e81095]  facebook::velox::exec::WindowPartition::computePeerBuffers(int, int, int, int, int*, int*)+0x32f
C  [libvelox.so+0x15f8a42f]  facebook::velox::exec::Window::computePeerAndFrameBuffers(int, int)+0x2f9
C  [libvelox.so+0x15f8a80e]  facebook::velox::exec::Window::callApplyForPartitionRows(int, int, int, std::shared_ptr<facebook::velox::RowVector> const&)+0x56
C  [libvelox.so+0x15f8aa81]  facebook::velox::exec::Window::callApplyLoop(int, std::shared_ptr<facebook::velox::RowVector> const&)+0x111
C  [libvelox.so+0x15f8ad1b]  facebook::velox::exec::Window::getOutput()+0x1c3
C  [libvelox.so+0x15cf742f]  facebook::velox::exec::Driver::runInternal(std::shared_ptr<facebook::velox::exec::Driver>&, std::shared_ptr<facebook::velox::exec::BlockingState>&, std::shared_ptr<facebook::velox::RowVector>&)+0xf8b
C  [libvelox.so+0x15cf5455]  facebook::velox::exec::Driver::next(std::shared_ptr<facebook::velox::exec::BlockingState>&)+0xbf
C  [libvelox.so+0x15dbcc30]  facebook::velox::exec::Task::next(folly::SemiFuture<folly::Unit>*)+0x63c

ISSUE 2:
C  [libvelox.so+0x17ee5c60]  facebook::velox::memory::Allocation::~Allocation()+0x32
C  [libvelox.so+0xdc0a015]  void std::_Destroy<facebook::velox::memory::Allocation>(facebook::velox::memory::Allocation*)+0x1c
C  [libvelox.so+0xdc083be]  void std::_Destroy_aux<false>::__destroy<facebook::velox::memory::Allocation*>(facebook::velox::memory::Allocation*, facebook::velox::memory::Allocation*)+0x32
C  [libvelox.so+0xdc05f17]  void std::_Destroy<facebook::velox::memory::Allocation*>(facebook::velox::memory::Allocation*, facebook::velox::memory::Allocation*)+0x27
C  [libvelox.so+0xdc03c77]  void std::_Destroy<facebook::velox::memory::Allocation*, facebook::velox::memory::Allocation>(facebook::velox::memory::Allocation*, facebook::velox::memory::Allocation*, std::allocator<facebook::velox::memory::Allocation>&)+0x2b
C  [libvelox.so+0x17e9c670]  std::vector<facebook::velox::memory::Allocation, std::allocator<facebook::velox::memory::Allocation> >::_M_erase_at_end(facebook::velox::memory::Allocation*)+0x66
C  [libvelox.so+0x17e99818]  std::vector<facebook::velox::memory::Allocation, std::allocator<facebook::velox::memory::Allocation> >::clear()+0x26
C  [libvelox.so+0x17ee7fcc]  facebook::velox::memory::AllocationPool::clear()+0x20
C  [libvelox.so+0x17eed45f]  facebook::velox::HashStringAllocator::clear()+0x1dd
C  [libvelox.so+0x15d44e50]  facebook::velox::exec::RowContainer::clear()+0x1c6
C  [libvelox.so+0x15d3f842]  facebook::velox::exec::RowContainer::~RowContainer()+0x1c
C  [libvelox.so+0xdb399b0]  std::default_delete<facebook::velox::exec::RowContainer>::operator()(facebook::velox::exec::RowContainer*) const+0x26
C  [libvelox.so+0xdb3320e]  std::unique_ptr<facebook::velox::exec::RowContainer, std::default_delete<facebook::velox::exec::RowContainer> >::~unique_ptr()+0x56
C  [libvelox.so+0x15e81223]  facebook::velox::exec::WindowPartition::computePeerBuffers(int, int, int, int, int*, int*)+0x4bd
C  [libvelox.so+0x15f8a42f]  facebook::velox::exec::Window::computePeerAndFrameBuffers(int, int)+0x2f9
C  [libvelox.so+0x15f8a80e]  facebook::velox::exec::Window::callApplyForPartitionRows(int, int, int, std::shared_ptr<facebook::velox::RowVector> const&)+0x56
C  [libvelox.so+0x15f8ab0d]  facebook::velox::exec::Window::callApplyLoop(int, std::shared_ptr<facebook::velox::RowVector> const&)+0x19d
C  [libvelox.so+0x15f8ad1b]  facebook::velox::exec::Window::getOutput()+0x1c3
C  [libvelox.so+0x15cf742f]  facebook::velox::exec::Driver::runInternal(std::shared_ptr<facebook::velox::exec::Driver>&, std::shared_ptr<facebook::velox::exec::BlockingState>&, std::shared_ptr<facebook::velox::RowVector>&)+0xf8b
C  [libvelox.so+0x15cf5455]  facebook::velox::exec::Driver::next(std::shared_ptr<facebook::velox::exec::BlockingState>&)+0xbf
C  [libvelox.so+0x15dbcc30]  facebook::velox::exec::Task::next(folly::SemiFuture<folly::Unit>*)+0x63c

@JkSelf : As I look at the code more, there are few points of vulnerability:
i) It might be better to make the rowContainer and previousGroupRow member variables of WindowPartition so that memory is allocated only once for them.
ii) Pass the operator memory pool from the WindowBuild to WindowPartition and use it directly in constructing the above so that there isn't a dependency on data_->pool(). I feel the second crash might be because one has gone prior to the other.

@aditi-pandit When we run the TPC-DS test, we occasionally encounter the following error. This issue is unstable and cannot be consistently reproduced. According to the error log stack trace, the problem seems to occur when determining whether the last row of the previous batch and the first row of the current batch belong to the same peer. Do you have any suggestions? Alternatively, is there another method we can use to determine if the data belongs to the same peer?
Related code are computePeerBuffers and previousGroupLastRow_
ISSUE 1:
C  [libvelox.so+0xd903934]  xsimd::batch<signed char, xsimd::fma3<xsimd::avx2> > xsimd::kernel::load_unaligned<xsimd::fma3<xsimd::avx2>, signed char, void>(signed char const*, xsimd::kernel::convert<signed char>, xsimd::avx const&)+0x3e
C  [libvelox.so+0xd9020a8]  xsimd::batch<signed char, xsimd::fma3<xsimd::avx2> > xsimd::batch<signed char, xsimd::fma3<xsimd::avx2> >::load_unaligned<signed char>(signed char const*)+0x46
C  [libvelox.so+0xd8ff97d]  facebook::velox::simd::detail::CopyWord<xsimd::batch<signed char, xsimd::fma3<xsimd::avx2> >, xsimd::fma3<xsimd::avx2> >::apply(void*, void const*)+0x37
C  [libvelox.so+0xd8fc794]  bool facebook::velox::simd::detail::copyNextWord<xsimd::batch<signed char, xsimd::fma3<xsimd::avx2> >, xsimd::fma3<xsimd::avx2> >(void*&, void const*&, int&)+0x3f
C  [libvelox.so+0xd8f990a]  void facebook::velox::simd::memcpy<xsimd::fma3<xsimd::avx2> >(void*, void const*, int, xsimd::fma3<xsimd::avx2> const&)+0x4a
C  [libvelox.so+0x17eeb28e]  facebook::velox::ByteOutputStream::appendStringView(std::basic_string_view<char, std::char_traits<char> >)+0xca
C  [libvelox.so+0x17eeb1bc]  facebook::velox::ByteOutputStream::appendStringView(facebook::velox::StringView)+0x46
C  [libvelox.so+0x17ef046a]  facebook::velox::HashStringAllocator::copyMultipartNoInline(facebook::velox::StringView const&, char*, int)+0x106
C  [libvelox.so+0x15d48e5c]  facebook::velox::HashStringAllocator::copyMultipart(facebook::velox::StringView const&, char*, int)+0x64
C  [libvelox.so+0x15d42a77]  facebook::velox::exec::RowContainer::storeVariableSizeAt(char const*, char*, unsigned int)+0x11b
C  [libvelox.so+0x15d43508]  facebook::velox::exec::RowContainer::storeSerializedRow(facebook::velox::FlatVector<facebook::velox::StringView> const&, int, char*)+0x24c
C  [libvelox.so+0x15e81095]  facebook::velox::exec::WindowPartition::computePeerBuffers(int, int, int, int, int*, int*)+0x32f
C  [libvelox.so+0x15f8a42f]  facebook::velox::exec::Window::computePeerAndFrameBuffers(int, int)+0x2f9
C  [libvelox.so+0x15f8a80e]  facebook::velox::exec::Window::callApplyForPartitionRows(int, int, int, std::shared_ptr<facebook::velox::RowVector> const&)+0x56
C  [libvelox.so+0x15f8aa81]  facebook::velox::exec::Window::callApplyLoop(int, std::shared_ptr<facebook::velox::RowVector> const&)+0x111
C  [libvelox.so+0x15f8ad1b]  facebook::velox::exec::Window::getOutput()+0x1c3
C  [libvelox.so+0x15cf742f]  facebook::velox::exec::Driver::runInternal(std::shared_ptr<facebook::velox::exec::Driver>&, std::shared_ptr<facebook::velox::exec::BlockingState>&, std::shared_ptr<facebook::velox::RowVector>&)+0xf8b
C  [libvelox.so+0x15cf5455]  facebook::velox::exec::Driver::next(std::shared_ptr<facebook::velox::exec::BlockingState>&)+0xbf
C  [libvelox.so+0x15dbcc30]  facebook::velox::exec::Task::next(folly::SemiFuture<folly::Unit>*)+0x63c

ISSUE 2:
C  [libvelox.so+0x17ee5c60]  facebook::velox::memory::Allocation::~Allocation()+0x32
C  [libvelox.so+0xdc0a015]  void std::_Destroy<facebook::velox::memory::Allocation>(facebook::velox::memory::Allocation*)+0x1c
C  [libvelox.so+0xdc083be]  void std::_Destroy_aux<false>::__destroy<facebook::velox::memory::Allocation*>(facebook::velox::memory::Allocation*, facebook::velox::memory::Allocation*)+0x32
C  [libvelox.so+0xdc05f17]  void std::_Destroy<facebook::velox::memory::Allocation*>(facebook::velox::memory::Allocation*, facebook::velox::memory::Allocation*)+0x27
C  [libvelox.so+0xdc03c77]  void std::_Destroy<facebook::velox::memory::Allocation*, facebook::velox::memory::Allocation>(facebook::velox::memory::Allocation*, facebook::velox::memory::Allocation*, std::allocator<facebook::velox::memory::Allocation>&)+0x2b
C  [libvelox.so+0x17e9c670]  std::vector<facebook::velox::memory::Allocation, std::allocator<facebook::velox::memory::Allocation> >::_M_erase_at_end(facebook::velox::memory::Allocation*)+0x66
C  [libvelox.so+0x17e99818]  std::vector<facebook::velox::memory::Allocation, std::allocator<facebook::velox::memory::Allocation> >::clear()+0x26
C  [libvelox.so+0x17ee7fcc]  facebook::velox::memory::AllocationPool::clear()+0x20
C  [libvelox.so+0x17eed45f]  facebook::velox::HashStringAllocator::clear()+0x1dd
C  [libvelox.so+0x15d44e50]  facebook::velox::exec::RowContainer::clear()+0x1c6
C  [libvelox.so+0x15d3f842]  facebook::velox::exec::RowContainer::~RowContainer()+0x1c
C  [libvelox.so+0xdb399b0]  std::default_delete<facebook::velox::exec::RowContainer>::operator()(facebook::velox::exec::RowContainer*) const+0x26
C  [libvelox.so+0xdb3320e]  std::unique_ptr<facebook::velox::exec::RowContainer, std::default_delete<facebook::velox::exec::RowContainer> >::~unique_ptr()+0x56
C  [libvelox.so+0x15e81223]  facebook::velox::exec::WindowPartition::computePeerBuffers(int, int, int, int, int*, int*)+0x4bd
C  [libvelox.so+0x15f8a42f]  facebook::velox::exec::Window::computePeerAndFrameBuffers(int, int)+0x2f9
C  [libvelox.so+0x15f8a80e]  facebook::velox::exec::Window::callApplyForPartitionRows(int, int, int, std::shared_ptr<facebook::velox::RowVector> const&)+0x56
C  [libvelox.so+0x15f8ab0d]  facebook::velox::exec::Window::callApplyLoop(int, std::shared_ptr<facebook::velox::RowVector> const&)+0x19d
C  [libvelox.so+0x15f8ad1b]  facebook::velox::exec::Window::getOutput()+0x1c3
C  [libvelox.so+0x15cf742f]  facebook::velox::exec::Driver::runInternal(std::shared_ptr<facebook::velox::exec::Driver>&, std::shared_ptr<facebook::velox::exec::BlockingState>&, std::shared_ptr<facebook::velox::RowVector>&)+0xf8b
C  [libvelox.so+0x15cf5455]  facebook::velox::exec::Driver::next(std::shared_ptr<facebook::velox::exec::BlockingState>&)+0xbf
C  [libvelox.so+0x15dbcc30]  facebook::velox::exec::Task::next(folly::SemiFuture<folly::Unit>*)+0x63c

@JkSelf : As I look at the code more, there are few points of vulnerability: i) It might be better to make the rowContainer and previousGroupRow member variables of WindowPartition so that memory is allocated only once for them. ii) Pass the operator memory pool from the WindowBuild to WindowPartition and use it directly in constructing the above so that there isn't a dependency on data_->pool(). I feel the second crash might be because one has gone prior to the other.
Let's evaluate after these code changes.

@aditi-pandit  I can resolve Issue 2 by moving the parameters into member variables. However, Issue 1 persists, and it appears to be related to the SIMD commands. To address this, I've implemented an alternative method for the same functionality, which ensures that the previous row is not deleted in the last batch within the clearOutputRows method. Instead, it is now erased after the peer group comparison. This new approach has allowed the gluten CI tests to pass successfully. Additionally, I've added window fuzzer test. Could you hep to review again? Thanks.

@JkSelf : This code is looking good now. Lets get a review from Masha.

@aditi-pandit I have resolved all your comments. Can you help to review again? Thanks.

@aditi-pandit Could you review again? The feature is essential to Gluten customers

@aditi-pandit Could you review again? The feature is essential to Gluten customers

@FelixYBW : Apologize if there was some misunderstanding. I'm complete with my review and wanted @mbasmanova to do next round.
The changes look good.

@JkSelf : Your code is looking good. I had only one pending comment from https://github.com/facebookincubator/velox/pull/9025#discussion_r1652013149

@mbasmanova @aditi-pandit I have resolved all your comments. Can you help to review again? Thanks.

@aditi-pandit @mbasmanova Can you help to review again? Thanks.

@JkSelf : Changes look good.

@mbasmanova Do you have any further comment? Thanks.

@mbasmanova Can you help to review again? Thanks.

@mbasmanova Any more comments? I have used the PR on a customer's cluster for some time, it works well so far.

@aditi-pandit Thanks for your review. I have resolved all your comments. Can you help to review again? Thanks.

@aditi-pandit Thanks for your review. I have resolved all your comments. Can you help to review again? Thanks.

Thanks @JkSelf. Changes look good.

@JkSelf : Can you merge your commits and put out a single one now. Would be great for subsequent reviews.

@xiaoxmeng @aditi-pandit I resolved all your comments. Can you help to review again? Thanks.

Hi @JkSelf, thank you for adding RowsStreamingWindowBuild. Could you also run the window fuzzer that compares Velox results against Presto for 2+ hours? To run that, you need to first start a Presto server following steps 1--10 here (but change the version 0.284 to 0.288): https://github.com/facebookincubator/velox/issues/8111. Then, you can start velox_window_fuzzer_test with command-line arguments --enable_window_reference_verification --presto_url=http://<IP_ADDRESS>:8080 --logtostderr=1 --minloglevel=0 --duration_sec=7200 .
Please let me know if you have any questions with running the fuzzer test. Thanks!

Hi @JkSelf, thank you for adding RowsStreamingWindowBuild. Could you also run the window fuzzer that compares Velox results against Presto for 2+ hours? To run that, you need to first start a Presto server following steps 1--10 here (but change the version 0.284 to 0.288): #8111. Then, you can start velox_window_fuzzer_test with command-line arguments --enable_window_reference_verification --presto_url=http://<IP_ADDRESS>:8080 --logtostderr=1 --minloglevel=0 --duration_sec=7200 .
Please let me know if you have any questions with running the fuzzer test. Thanks!

@kagamiori Thanks for your review. I will try to execute the window fuzzer test locally following the steps you provided.

@kagamiori @xiaoxmeng
I encountered the following error during the execution process, but this error doesn't seem to be related to my PR. I tried running it without my PR, and the same error occurred.
I0822 23:45:06.386909 2118850 WindowFuzzer.cpp:277] ==============================> Done with iteration 228
I0822 23:45:06.387009 2118850 WindowFuzzer.cpp:217] ==============================> Started iteration 229 (seed:<PHONE_NUMBER>)
I0822 23:45:06.413928 2118850 AggregationFuzzerBase.cpp:427] Executing query plan:
-- Window[1][partition by [p0, p1] order by [s0 ASC NULLS FIRST, s1 DESC NULLS FIRST, row_number ASC NULLS LAST] w0 := approx_percentile(ROW["c0"],ROW["c1"],ROW["c2"]) ROWS between k0 FOLLOWING and -8268856359831993385 FOLLOWING] -> c0:DOUBLE, c1:BIGINT, c2:DOUBLE, s0:ROW<f0:BOOLEAN,f1:INTERVAL DAY TO SECOND,f2:BOOLEAN,f3:VARCHAR,f4:INTEGER>, s1:SMALLINT, p0:ARRAY<DOUBLE>, p1:ARRAY<DOUBLE>, k0:INTEGER, row_number:BIGINT, w0:DOUBLE
  -- Values[0][1000 rows in 10 vectors] -> c0:DOUBLE, c1:BIGINT, c2:DOUBLE, s0:ROW<f0:BOOLEAN,f1:INTERVAL DAY TO SECOND,f2:BOOLEAN,f3:VARCHAR,f4:INTEGER>, s1:SMALLINT, p0:ARRAY<DOUBLE>, p1:ARRAY<DOUBLE>, k0:INTEGER, row_number:BIGINT
I0822 23:45:06.414655 2121801 Task.cpp:1920] Terminating task test_cursor 1170 with state Failed after running for 0ms
I0822 23:45:06.414741 2121801 Task.cpp:1155] All drivers (1) finished for task test_cursor 1170 after running for 0ms
E0822 23:45:06.414860 2121801 MemoryPool.cpp:448] [MEM] Memory leak (Used memory): Memory Pool[op.1.0.0.Window LEAF root[query.TaskCursorQuery_1169.1619] parent[node.1] MALLOC track-usage thread-safe]<unlimited max capacity unlimited capacity used 128B available 1023.88KB reservation [used 128B, reserved 1.00MB, min 0B] counters [allocs 9, frees 8, reserves 0, releases 1, collisions 0])>
E0822 23:45:06.414930 2121801 Exceptions.h:67] Line: /mnt/DP_disk3/jk/projects/fb-velox/velox/velox/common/memory/MemoryArbitrator.cpp:100, Function:removePool, Expression: pool->reservedBytes() == 0 (1048576 vs. 0), Source: RUNTIME, ErrorCode: INVALID_STATE
terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
  what():  Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: (1048576 vs. 0)
Retriable: False
Expression: pool->reservedBytes() == 0
Function: removePool
File: /mnt/DP_disk3/jk/projects/fb-velox/velox/velox/common/memory/MemoryArbitrator.cpp
Line: 100
Stack trace:
# 0  facebook::velox::VeloxException::VeloxException(char const*, unsigned long, char const*, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, bool, facebook::velox::VeloxException::Type, std::basic_string_view<char, std::char_traits<char> >)
# 1  void facebook::velox::detail::veloxCheckFail<facebook::velox::VeloxRuntimeError, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(facebook::velox::detail::VeloxCheckFailArgs const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
# 2  facebook::velox::memory::(anonymous namespace)::NoopArbitrator::removePool(facebook::velox::memory::MemoryPool*)
# 3  facebook::velox::memory::MemoryManager::dropPool(facebook::velox::memory::MemoryPool*)
# 4  facebook::velox::memory::MemoryPoolImpl::~MemoryPoolImpl()
# 5  std::_Sp_counted_ptr<facebook::velox::core::QueryCtx*, (__gnu_cxx::_Lock_policy)2>::_M_dispose()
# 6  std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() [clone .part.0]
# 7  facebook::velox::exec::Task::~Task()
# 8  std::_Sp_counted_ptr<facebook::velox::exec::Task*, (__gnu_cxx::_Lock_policy)2>::_M_dispose()
# 9  std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() [clone .part.0]
# 10 std::_Sp_counted_ptr<facebook::velox::exec::Driver*, (__gnu_cxx::_Lock_policy)2>::_M_dispose()
# 11 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() [clone .part.0]
# 12 unsigned long folly::detail::function::DispatchSmall::exec<facebook::velox::exec::Driver::enqueue(std::shared_ptr<facebook::velox::exec::Driver>)::{lambda()#1}>(folly::detail::function::Op, folly::detail::function::Data*, folly::detail::function::Data)
# 13 folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&)
# 14 folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)
# 15 void folly::detail::function::call_<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)>, true, false, void>(, folly::detail::function::Data&)
# 16 0x00000000000d6df3
# 17 start_thread
# 18 clone

*** Aborted at<PHONE_NUMBER> (Unix time, try 'date -d @1724341506') ***
*** Signal 6 (SIGABRT) (0x2054c2) received by PID 2118850 (pthread TID 0x7f7e077fe700) (linux TID 2121801) (maybe from PID 2118850, UID 0) (code: -6), stack trace: ***
    @<PHONE_NUMBER>449637 folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
                       /mnt/DP_disk3/jk/projects/gluten/ep/build-velox/build/velox_ep/folly/folly/experimental/symbolizer/SignalHandler.cpp:453
    @<PHONE_NUMBER>01441f (unknown)
    @<PHONE_NUMBER>04300b gsignal
    @<PHONE_NUMBER>022858 abort
    @<PHONE_NUMBER>09e8d0 (unknown)
    @<PHONE_NUMBER>0aa37b (unknown)
    @<PHONE_NUMBER>0a9358 (unknown)
    @<PHONE_NUMBER>0a9d10 __gxx_personality_v0
    @<PHONE_NUMBER>003f88 __libunwind_Unwind_Resume
    @<PHONE_NUMBER>3e7d28 facebook::velox::memory::MemoryManager::dropPool(facebook::velox::memory::MemoryPool*) [clone .cold]
    @<PHONE_NUMBER>395017 facebook::velox::memory::MemoryPoolImpl::~MemoryPoolImpl()
    @<PHONE_NUMBER>1d6a98 std::_Sp_counted_ptr<facebook::velox::core::QueryCtx*, (__gnu_cxx::_Lock_policy)2>::_M_dispose()
    @<PHONE_NUMBER>4332d9 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() [clone .part.0]
    @<PHONE_NUMBER>c75269 facebook::velox::exec::Task::~Task()
    @<PHONE_NUMBER>c92975 std::_Sp_counted_ptr<facebook::velox::exec::Task*, (__gnu_cxx::_Lock_policy)2>::_M_dispose()
    @<PHONE_NUMBER>4332d9 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() [clone .part.0]
    @<PHONE_NUMBER>d0fdd3 std::_Sp_counted_ptr<facebook::velox::exec::Driver*, (__gnu_cxx::_Lock_policy)2>::_M_dispose()
    @<PHONE_NUMBER>4332d9 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() [clone .part.0]
    @<PHONE_NUMBER>ce647f unsigned long folly::detail::function::DispatchSmall::exec<facebook::velox::exec::Driver::enqueue(std::shared_ptr<facebook::velox::exec::Driver>)::{lambda()#1}>(folly::detail::function::Op, folly::detail::function::Data*, folly::detail::function::Data)
    @<PHONE_NUMBER>43f420 folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&)
                       /mnt/DP_disk3/jk/projects/gluten/ep/build-velox/build/velox_ep/folly/folly/Function.h:639
                       -> /mnt/DP_disk3/jk/projects/gluten/ep/build-velox/build/velox_ep/folly/folly/executors/ThreadPoolExecutor.cpp
    @<PHONE_NUMBER>427d25 folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)
                       /mnt/DP_disk3/jk/projects/gluten/ep/build-velox/build/velox_ep/folly/folly/executors/CPUThreadPoolExecutor.cpp:350
    @<PHONE_NUMBER>445d2d void folly::detail::function::call_<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)>, true, false, void>(, folly::detail::function::Data&)
                       /usr/include/c++/9/bits/invoke.h:73
                       -> /mnt/DP_disk3/jk/projects/gluten/ep/build-velox/build/velox_ep/folly/folly/executors/ThreadPoolExecutor.cpp
    @<PHONE_NUMBER>0d6df3 (unknown)
    @<PHONE_NUMBER>008608 start_thread
    @<PHONE_NUMBER>11f352 clone
Aborted (core dumped)

enable_window_reference_verification

This seems to be memory leak somewhere, and I didn't see that in Meta internal run against duckDB. Not sure if it relates to Presto verification. I will try with enableWindowVerification set with duckDB.

enable_window_reference_verification

This seems to be memory leak somewhere, and I didn't see that in Meta internal run against duckDB. Not sure if it relates to Presto verification. I will try with enableWindowVerification set with duckDB.

@xiaoxmeng @kagamiori
I have discovered that the memory leak occurs during the creation of the frame.
Detected total of 1 leaked allocations:
======== Leaked memory from 1 total allocations of 96B total size ========
# 0  facebook::velox::memory::MemoryPoolImpl::recordAllocDbg(void const*, unsigned
# 1  facebook::velox::memory::MemoryPoolImpl::allocate(long)
# 2  boost::intrusive_ptr<facebook::velox::Buffer> facebook::velox::AlignedBuffer::e<int>(unsigned long, facebook::velox::memory::MemoryPool*, std::optional<int> cons
# 3  std::shared_ptr<facebook::velox::BaseVector> facebook::velox::createEmpty<(facvelox::TypeKind)3>(int, facebook::velox::memory::MemoryPool*, std::shared_ptr<faceblox::Type const> const&)
# 4  facebook::velox::BaseVector::createInternal(std::shared_ptr<facebook::velox::Tst> const&, int, facebook::velox::memory::MemoryPool*)::{lambda()#1}::operator()() {lambda()#1}::operator()() const
# 5  facebook::velox::BaseVector::createInternal(std::shared_ptr<facebook::velox::Tst> const&, int, facebook::velox::memory::MemoryPool*)::{lambda()#1}::operator()()
# 6  facebook::velox::BaseVector::createInternal(std::shared_ptr<facebook::velox::Tst> const&, int, facebook::velox::memory::MemoryPool*)
# 7  std::shared_ptr<facebook::velox::BaseVector> facebook::velox::BaseVector::creabook::velox::BaseVector>(std::shared_ptr<facebook::velox::Type const> const&, int, k::velox::memory::MemoryPool*)
# 8  facebook::velox::exec::Window::createWindowFrame(std::shared_ptr<facebook::vele::WindowNode const> const&, facebook::velox::core::WindowNode::Frame const&, std::ptr<facebook::velox::RowType const> const&)::{lambda(std::shared_ptr<facebook::velo::ITypedExpr const> const&)#1}::operator()(std::shared_ptr<facebook::velox::core::Ipr const> const&) const
# 9  facebook::velox::exec::Window::createWindowFrame(std::shared_ptr<facebook::vele::WindowNode const> const&, facebook::velox::core::WindowNode::Frame const&, std::ptr<facebook::velox::RowType const> const&)
# 10 facebook::velox::exec::Window::createWindowFunctions()
# 11 facebook::velox::exec::Window::initialize()
# 12 facebook::velox::exec::Driver::initializeOperators()
# 13 facebook::velox::exec::Driver::runInternal(std::shared_ptr<facebook::velox::exver>&, std::shared_ptr<facebook::velox::exec::BlockingState>&, std::shared_ptr<faceelox::RowVector>&)
# 14 facebook::velox::exec::Driver::run(std::shared_ptr<facebook::velox::exec::Driv
# 15 facebook::velox::exec::Driver::enqueue(std::shared_ptr<facebook::velox::exec::)::{lambda()#1}::operator()() const
# 16 void folly::detail::function::call_<facebook::velox::exec::Driver::enqueue(stdd_ptr<facebook::velox::exec::Driver>)::{lambda()#1}, true, false, void>(, folly::deunction::Data&)
# 17 folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor:: const&, folly::ThreadPoolExecutor::Task&&)
# 18 folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecuread>)
# 19 void folly::detail::function::call_<std::_Bind<void (folly::ThreadPoolExecutorly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::ptr<folly::ThreadPoolExecutor::Thread>)>, true, false, void>(, folly::detail::functta&)
# 20 folly::detail::function::FunctionTraits<void ()>::operator()()
# 21 folly::NamedThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}tor()()
# 22 void std::__invoke_impl<void, folly::NamedThreadFactory::newThread(folly::Funcid ()>&&)::{lambda()#1}>(std::__invoke_other, folly::NamedThreadFactory::newThread(Function<void ()>&&)::{lambda()#1}&&)
# 23 std::__invoke_result<folly::NamedThreadFactory::newThread(folly::Function<void::{lambda()#1}>::type std::__invoke<folly::NamedThreadFactory::newThread(folly::Funoid ()>&&)::{lambda()#1}>(std::__invoke_result&&, (folly::NamedThreadFactory::newThlly::Function<void ()>&&)::{lambda()#1}&&)...)
# 24 void std::thread::_Invoker<std::tuple<folly::NamedThreadFactory::newThread(folction<void ()>&&)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>)
# 25 std::thread::_Invoker<std::tuple<folly::NamedThreadFactory::newThread(folly::F<void ()>&&)::{lambda()#1}> >::operator()()
# 26 std::thread::_State_impl<std::thread::_Invoker<std::tuple<folly::NamedThreadFanewThread(folly::Function<void ()>&&)::{lambda()#1}> > >::_M_run()
# 27 0x00000000000d6df3
# 28 start_thread
# 29 clone

The frame start has already been established and a buffer has been allocated here. However, when creating the frame end, the offset is negative, which triggers an error here. At this point, the buffer allocated for the frame start has not been released in time, resulting in the reported memory leak. The issue can fixed by converting the negative to positive in WindowFuzzer.cpp.


Some image name is not working

docker.io/golang
registry:5000/google/cadvisor

@mv012004 docker.io is your local registry or the image name?

For official images, I think we can't pull layer.  Need to double check. However, I will put better log/error for that.
Moreover,  official images are already tested by docker itself.
Finally, docker.io in the name is handle as official image (docker.io is removed from the name when it's tagged)
registry:5000/google/cadvisor name is handle correctly

nope.. i have my registry running on a different server, like abc.com/google/cadvisor

Yes for google/cadvisor it's another problem then the image naming.
Is it the official one that you pushed on your own registry?
Le mar. 5 avr. 2016 22:40, mv012004<EMAIL_ADDRESS>a écrit :

nope.. i have my registry running on a different server, like
abc.com/google/cadvisor
—
You are receiving this because you modified the open/close state.
Reply to this email directly or view it on GitHub
https://github.com/wemanity-belgium/hyperclair/issues/48#issuecomment-205978374


Standardize builders

We have builders with all manners of differences and inconsistencies between them, even inside the same builder, we should probably agree on a standard, and change our builders to mostly follow that.
(such as the command parser)
let mut parser_config = CommandParserConfig::new();
parser_config.command("help").add();
parser_config.command("ping").add();
parser_config.add_prefix("kona");

i vote for whatever isn't &mut self

I've made a list of all the builders I could find and their styles. Altogether, we've got 4 (or 5 in a way) unique styles across all crates.
twilight-builders
The embed builder uses an.. interesting, consuming and commit style. It looks like:
use twilight_builders::embed::EmbedBuilder;

let embed = EmbedBuilder::new()
    .color(7)
    .author()
    .name("author name")
    .url("a url")
    .commit()
    .add_field("name", "value")
    .inline(true)
    .commit()
    .build();

Interestingly add_field is the only method that has the action in the method name, which is kind of inconsistent. commit is chainable and returns the overarching embed builder.
command-parser
There's two builders here, the parser config builder and the command builder. Altogether, this looks like the OP:
let mut parser_config = CommandParserConfig::new();
parser_config.command("help").add();
parser_config.command("ping").add();
parser_config.add_prefix("kona");

The parser config also has the action in the method name, and is reference style (&mut self). The command builder is consuming style, and works like the embed builder's commit with add, except unlike the command builder it's not chainable.
gateway
The gateway has two builders. The cluster config builder is consuming. The shard config builder is also consuming, except notably the intents method which is referencing. Here's the code:
/// The HTTP client to be used by the shard for getting gateway information.
pub fn http_client(mut self, http_client: HttpClient) -> Self {
    self.0.http_client = http_client;

    self
}

/// Sets the gateway intents.
pub fn intents(&mut self, intents: Option<GatewayIntents>) -> &mut Self {
    self.0.intents = intents;

    self
}

http
http's client builder is referencing. Request builders are consuming, used like:
let messages = client.channel_messages(channel_id).limit(5).await?;

Closed by #361 and #364.


Virtual Threads Integration

Hello engineers,
I would like to understand if you are going to integrate the virtual threads into Gatling based on JDK21?

Hello,
This would be best suited on our community forum as a general discussion: https://community.gatling.io/categories, not on the bug tracker.
There's no such plan for now, in particular as there's no clear benefit for us as of now for the main protocols we support.


Unresolved warnings on build

Using Apple M1 Max.
npm run initialize runs / completes fine (once I initialized the ipfs daemon -- that wasn't mentioned in Readme but dug through the error messages)
I get the following error when I run npm run build-general:

should note that I am able to use snek just fine by just aliasing snek to node ~/snek/index.js $argv

Can you confirm that npm run build-general doesn't produce a usable snek binary in the same directory as as the repo? I get similar errors on mac but still get snek built.
these warnings should be looked at, just want to understand the issue.

I think building a single binary like this is pretty non-standard, wondering if we can't just go with npm i -g type of thing. @stobiewan thoughts

@dmfxyz you are correct, the binary actually does build; apparently I missed / made assumptions based on the errors. Updated issue title accordingly.

Originally added "build-linux" script just because it's helpful for testing and using myself. Something like npm install -g would be ideal, assume vyper+python devs are OK with using npm to get snek.


fixed doc comment ByteBuffer#readString(length: Int)

Fixed Doc per discussion with @Lukasa.
Motivation:
The accuracy of documentation so developers such as myself using swift-nio will be able to use the method ByteBuffer#readString(length: Int) with confidence, was the motivation for this patch.
Modifications:
Changed the doc comment, to indicate that the aforementioned method is documented correctly.
Result:
Applying this patch does not change the code base logically, it only changes the documentation.

@swift-nio-bot test this please


Fix to CPU model parsing

It wasn't possible to filter cpu models on prometheus because of trailing spaces. At least a \n was present, but the bug fix removes any whitespace character.
Also, I cleaned the code a bit.

LGTM


[L10N][Sample file]: The string "Extension Manager" should be localized in the sample html file.

Steps:

Launch Brackets and open the sample html file
Hover on the Extension Manager sidebar icon.

Result:
The string " Extension Manager" is not localized in the sample file.
Expected:
The string should be localized and keep consistent with sidebar icon tooltip.
ENV: Win7 and Mac 10.9 JP & FR os
Build: 1.2.0-15675
Screenshots:
ENU reference:

FR:

JP:

fixed with PR https://github.com/adobe/brackets/pull/10572

Extension Manager is a product name and normally they stay in English. Is this not the case?

@lmariz It looks like the dialog name is translated in most Brackets translations.  It's not really a product name, it's the name of a feature / command / screen, and normally those are translated.  Same for other Brackets commands like Quick Edit.

OK, thanks for confirming @peterflynn

@peterflynn thanks for confirming.

For FR please change 'Extension Manager' to: Gestionnaire d’extensions. A curly quote should be used to keep consistency with the rest of the text. Other than that, the FR string is correct.

fix is in PR https://github.com/adobe/brackets/pull/10624

@julieyuan Fix in PR #10625 was merged -- please verify on the release branch

Thanks @pantkowiak @peterflynn , verified the FR string is correct.
@pantkowiak ,please also help to change the same FR string with other two string ID to: Gestionnaire d’extensions. Their string ID is: CMD_EXTENSION_MANAGER,EXTENSION_MANAGER_TITLE. And please notes that the translation for EXTENSION_MANAGER_TITLE is not same as the translation for the string 'Extension Manager' right now, could you please help to check whether it need to be kept consistency? Thanks.

@julieyuan issue is fixed with PR #10767, please verify.

Thanks @pantkowiak .Confirmed it is fixed with latest build 1.3.0-15767 (master 2d880bcbe), closing as fixed.


🔧  config alignment

Codecov Report

Merging #65 (f31a3b4) into main (cc9d59d) will not change coverage.
The diff coverage is 100.00%.

:exclamation: Current head f31a3b4 differs from pull request most recent head 13541e1. Consider uploading reports for the commit 13541e1 to get more accurate results

@@           Coverage Diff           @@
##             main      #65   +/-   ##
=======================================
  Coverage   74.69%   74.69%
=======================================
  Files           2        2
  Lines          83       83
=======================================
  Hits           62       62
  Misses         21       21

Impacted Files
Coverage Δ

iamlistening/__init__.py
100.00% <100.00%> (ø)


fix spacing part 2

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Current dependencies on/for this PR:

main

PR #468   👈

This stack of pull requests is managed by Graphite.


node crashes if any node subscribes to OccupancyGrid

It crashes in map_builder_bridge.cc#170.

rviz was subscribing to occupancy grid topic.
trajectory_id 0 was for loaded map from serialized data.

F0705 19:26:54.898905 26894 map_builder_bridge.cc:170] Check failed: trajectory_options_.count(0) == 1 (0 vs. 1)
[/cartographer_node FATAL<PHONE_NUMBER>.899195430,<PHONE_NUMBER>.110075237]: F0705 19:26:54.000000 26894 map_builder_bridge.cc:170] Check failed: trajectory_options_.count(0) == 1 (0 vs. 1)
*** Check failure stack trace: ***575.107289   Duration: 1.908086 / 591.608141
    @     0x7f9bec9f95cd  google::LogMessage::Fail()
    @     0x7f9bec9fb433  google::LogMessage::SendToLog()
    @     0x7f9bec9f915b  google::LogMessage::Flush()
    @     0x7f9bec9fbe1e  google::LogMessageFatal::~LogMessageFatal()
    @           0x51fe83  cartographer_ros::MapBuilderBridge::BuildOccupancyGrid()
    @           0x4fa7ee  cartographer_ros::Node::SpinOccupancyGridThreadForever()
    @     0x7f9be9064c80  (unknown)
    @     0x7f9bec5b76ba  start_thread
    @     0x7f9be8ad33dd  clone
    @              (nil)  (unknown)

This is sorta expected behavior. I am unclear what should be done here....
The problem is that we do not have scans from the loaded data, so we cannot generate an occupancy grid (since we do not have trajectory_nodes). And even if we had the scans, we would still not know which trajectory builder options (in this case: resolution and laser_scan_insert options) we should use for generating the ROS map.
I am not sure how this should be properly handled... probably by configuring the parameters for the published map through ROS somehow.

One option would be having a separate ROS node that generates OccupancyGrid from submaps as you  suggested in #343. In this case, we can probably extend this node to provide other representations like gridmap for the future use case too.
Would it generate different map from the current occupancy grid map generation approach?

Would it generate different map from the current occupancy grid map generation approach?

Probably. Currently we generate the occupancy grid by replaying all scans that we have in memory and inserting them using the current opinion of where they belong. A separate node would either

listen to the lasers (or other sources of points) themselve and use Cartographer's trajectory to periodically generate an occupancy grid. This would give the most similar results.
listen to the submaps similar to how the rViz plugin does it and create the occupancy grid by blending the submaps together (like the rViz plugin does). This would result in a map that is more similar to the visualization, but maybe not as good of quality.

I encountered similar behavior with attempting to use WriteAssets when a map had been loaded - it dies at https://github.com/googlecartographer/cartographer_ros/blob/master/cartographer_ros/cartographer_ros/map_builder_bridge.cc#L86
For that particular use case, I was able to work around it by amending the WriteAssets service to accept a trajectory ID - This allows an occupancy grid to be written, however it will only write an occupancy grid for the current run's trajectory.  Anything from the loaded map that does not exist in the current trajectory will not be written out - and after reading through the discussion here I understand why.  I need to talk to management internally as to what company policy is with regards to me pushing my changes out to a public location, I'll publish those to a branch somewhere if I can obtain permission.
Is my understanding of the commit history for cartographer correct in that you now have scans from loaded data, leaving only the issue of unknown options remaining?  What about assuming that the existing scans were using the same options as for the currently running trajectory (typically in this case trajectory ID 1)?

This is now fixed by publishing a map using the Texture information from Cartographer directly. This does allow to generate a ROS map by launching Cartographer with a serialized state and running the occupancy_grid_node. If an offline ROS map (PGM and YAML) is desired, the map_saver can be used.


Question: Redirect to Shopify page with success/error message

I am wondering if it's possible to redirect to a Shopify page from a controller? And if it possible to add a message (success/error/info) to the session?
Say for example that we set up an Admin link in the App Extension section:

This will implement an action in the order page which points to any route we specify:

If this route is pointed to a controller which would update the order and redirect back to the order page with either a popup success message like the example below showing the success message

Line items have been fulfilled

or by showing a sticky success message in the top of the screen like this:

If this is possible and anyboy have some information about this, then I will happily add it to the documentation.

@ohmybrew thank you for the answer. I could not find any documentation about this anywhere, but your answer clarifies this.
Maybe the handling of Admin link should be added to the documentation (How to information) and there this could be mentioned as a side note?
Otherwise, it could be beneficial to include this in a section about templates and how to use the Shopify theme?

Possibly, I think it's better left to Shopify's own documentation for this since it would just be a repeat.


Enforce types on builtins

Accept strings during conversions.
Make most builtins vectorize over arrays.
refs #182, #200, #237

Not perfect, but hopefully better.

The exported functions should have clean versions so that the cursed vectorized types can be kept internal.


Resolve #234: Change "this.Goto(typeof(T));" to "this.Goto();" wit…

…h " where T: MachineState" (or MonitorState)

@TedHartMS @akashlal I don't think that we should remove the old version, this will break compatibility with existing code (both ours and our users). We could just add the new version alongside the current method. By the way, I do not fully understand the reasoning behind this request. Is this just a syntactic preference? Or there is some stronger reason behind? Is this because they do not have access to typeof()? But in either case, I would keep both. Any thoughts?
@Ted: assuming we all agree on the above, could you please make this change (which should hopefully simplify the commit too) and rebase so I can check it again?

Agreed--will make the change.
typeof() is always available. Goto<T>() will error at compile time if T is not a MachineState (or MonitorState).

Restored as deprecated in #ee013c7. This will require disabling warning 618 if warnings are treated as errors.
PSharpRuntime has several methods that take (Type type, ...):

the CreateMachine variants
RegisterMonitor
InvokeMonitor This is the only one that also has a generic overload: InvokeMonitor<T>(Event e).
also INetworkProvider.RemoteCreateMachine

For consistency should we add the generic form for these? Deprecate the non-generic form?

What you just said would be great Ted, please feel free to open an issue and work on it on a separate branch. But maybe put it as a low priority item in your TODO list, since no one asked for it directly (yet)?


rust analyzer not showing completion for custom messages but shows it for "built-in" messages like std_msgs/Header

Hi,
At our company, we're considering using rosrust for message generation and ser-deserialisation of messages in order to interop with a large codebase of c++ code.
Message generation works great, for example I could  generate an EnergySpeed message form a .msg file. But, when hovering over it or ctrl+clicking into it I don't get anything. But when I do it for Header (which is a ros builtin type) I get this

and I can click on it which takes me to a lib.rs file located in the rosrust_msg crate which contains this code:
include!(concat!(env!("OUT_DIR"), "/", "messages.rs"));
Is this somehow a limitation of rust-analyzer or a way with which rosrust is currently designed ? @adnanademovic
Thank you for the amazing work!

It might be a Rust Analyzer limitation. It might not be having a good time with finding or traversing to generated files within the target directory.
If it's any consolation, generating the documentation via cargo doc --open will give you documentation for all messages generated within rosrust_msg.

Thanks for your response! :)


Add institutodh.net domain

Proposes to add the institutodh.net domain to the swot list
URL of the School website https://blogsaverroes.juntadeandalucia.es/institutodh/

@PepeDev26 Please provide us with:

the school official website URL
the school street address, including city and country
an URL of a page on the official website where the school offers an IT-related long-term (>1 year) course
a proof, which shows that the school recognizes the domain you are submitting as an official email domain for the students.

The official website is https://blogsaverroes.juntadeandalucia.es/institutodh/ and also the domain that I'm trying to register (institutodh.net) redirects to it
The adress is C. Dalí, 9, 21710 Bollullos Par del Condado, Huelva, Spain
Here you have the educative offer of the center in Spanish which includes 3 IT Vocational School courses (Formación Profesional) https://blogsaverroes.juntadeandalucia.es/institutodh/oferta-educativa/
Which kind of proof you need, I dont understand

@PepeDev26  Thank you for the clarifications. Pull request merged. Please start requesting the licenses in about an hour, to let the changes to propagate through our system.


Change "Review Date" from Epoch

On the Assessment Summary page, the Review Date is showing at Epoch.  Need to change to a standard DTG.

For example: http://pathfinder-ui-vft-dashboard.int.open.paas.redhat.com/pathfinder-ui/assessments-v2.jsp?customerId=ba3e2f67-7f15-4c4f-ba29-e14fc8b71ff7
Shows Review Date as<PHONE_NUMBER>536 which should be:  Saturday, 30 June 2018 04:19:55

fyi - date format config options are here: https://github.com/phstc/jquery-dateFormat


apt does not maintain list of installed applications

apt is installing the applications without any errors. It is able to perform all of its actions excepting apt remove <pkg-name>
It is not maintaining any list on the system of the applications installed. This bug has to be resolved before the next release.

To test the functionality of apt remove, first install the test application from the repository. Execute the following steps:
apt update
apt install example-helloworld
apt remove example-helloworld

This bug has been fixed and the changes have been uploaded to dpkg branch.


Add option to open payment dialog without needing a button

It would look something like:
PayButton.openDialog( {   to: 'address' } );

Extract the Dialog from here https://github.com/PayButton/paybutton/blob/master/react/src/components/PayButton/PayButton.tsx#L69 into its own component so that it can be rendered without a Button beside it

Add the openDialog function here: https://github.com/PayButton/paybutton/blob/master/paybutton/src/index.tsx#L115 , which will create a new container div and append it to document.body, and then render the dialog component from step 1 inside it, with an onClose handler that deletes the container


{App Service} Fix #25616: az webapp up: Add help message to clarify that the cloud shell is not supported

Related command

az webapp up --launch-browser
Description

resolves https://github.com/Azure/azure-cli/issues/25616
Testing Guide

az webapp up --launch-browser
History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change
[Component Name 2] az command b: Add some customer-facing feature

This checklist is used to make sure that common guidelines for a pull request are followed.

[x] The PR title and description has followed the guideline in Submitting Pull Requests.

[x] I adhere to the Command Guidelines.

[x] I adhere to the Error Handling Guidelines.

App Service


Fix typos in documentation

The documentation for the bitwise or operator described the behavior of the bitwise and operator.

The prose in wren/doc/site/embedding/calling-c-from-wren.markdown at line 115 is a bit mangled, but I'm not sure what it's supposed to say so I can't submit a fix for that.

Oops, thanks!


Mobile menu won't open

Tapping "menu" icon in the header doesn't open a menu.

@pmmck I'm finding this as working.

@fuhton you are correct, it is working on native mobile devices.  I tested both iOS and Android.  It seems to only be an issue on desktop with small viewport sizes.   Both Chrome and FF on OSX.

@pmmck If you resize and then refresh the page is it working?

@fuhton yep!


Drop BytesIO from CodeStream, ~7.5% import block speedup

What was wrong?
While inlining CodeStream.__next __(), it started to make sense to drop BytesIO altogether. We were basically doing all the same work anyway, to avoid any slowdowns caused by dispatching to CodeStream.
How was it fixed?

Use slots to speed up attribute/method access
Drop io.BytesIO and just keep a pc and bytes buffer explicitly
Added SlowCodeStream as a reference implementation for testing, until this new one "burns in" for a while
(Bonus) try to skip the ljust in push_xx opcodes because it's typically not needed.

All added to a nice speed bump:
__iter__  4.53% -> 1.58% = 2.95%
read      2.31% -> 0.56% = 1.75%
pc()      1.12% -> 0%    = 1.12%
peek      1.07% -> 0.09% = 0.98%

push_xx   5.40% -> 4.57% = 0.83%
--------------------------------
Total lift                 7.6%

TODO:

[x] see how bad it would be to drop the jump-dest min() (it was proactive, since pc is not being explicitly fenced anymore)
[x] add notes about why performance decisions were made
[x] squash
[x] More explicit commit message about speedup

Cute Animal Picture

@pipermerriam okay, actually ready for review now


Space and Time complexity

Considering space and time complexity is a significant aspect of programming that ensures code efficiency, scalability, and maintainability. Having a brief introduction to these concepts, followed by example, can greatly benefit JavaScript developers.

I would love to take this issue.

There are some basic details I want to know before I start working on this.

[ ] Can I change the chapter numbers as I want to put the complexity chapter after the loops, functions and objects chapter?
[ ] Till what details should I go into the complexity chapter?
[ ] Can you tell whether the given structure is satisfactory or not?

en/complexity
├── README.md
├── space-complexity.md
└── time-complexity.md

Certainly! Considering space and time complexity is crucial in programming for efficient, scalable, and maintainable code.
Would like to contribute to this issue.

@sumn2u There are some basic details I want to know before I start working on this.

[ ]  Can I change the chapter numbers as I want to put the complexity chapter after the loops, functions and objects chapter?

Yes

[ ]  Till what details should I go into the complexity chapter?

Surface level that gives readers context would be enough.

[ ]  Can you tell whether the given structure is satisfactory or not?

en/complexity
├── README.md
├── space-complexity.md
└── time-complexity.md

Looks good.

Surface level that gives readers context would be enough.

Will do that. Thanks for the info.

@sumn2u can I pick this?

Hey @sumn2u, is this issue still open. Can i participate to resolve it?

Hey @sumn2u, is this issue still open. Can i participate to resolve it?

Yes, we already have the structure in place. We just need to add the content.

@sumn2u  Thanks for assigning me for this. Looking forward to work on this.

Hey @sumn2u,  I've added content in the time and space complexity sections. Please have a look and tell if i need to do some adjustments or it will do the job.


updated c++ basics

#4 ,issue resolved , please merge (so much effort given)..

@UjjwalSharma01 please merge , took so much effort

yes we can see the usage of AI in this


Provider Default in UI

Hi, I'm using this plugin on Ubuntu Linux w/ Obsidian V1.6.7
Each time I open the plugin in the UI it forgets the previous settings, so I have to reset them each time.
I want it to default to grocq, but it opens with openai each time.
Should it work this way or can we have the previous settings remembered for next use?

Im going to close this out, but you can use the version 1.0.5 (this version im using for post release changes) this version will give you all the features ive added and issues ive fixed recently.


Import version from setup.py into bot.self.version

When the bot starts up, I'd like to write the version to the log file.  There are a few strategies for this on stackoverflow.

Great! It will prevent from using the old versions with bugs :)


You were added to awesome-humane-tech

This is just a FYI issue to notify that you were added to the curated awesome-humane-tech in the 'Analytics' category, and - if you like that - are now entitled to wear our badge:

By adding this to the README:
[![Awesome Humane Tech](https://raw.githubusercontent.com/humanetech-community/awesome-humane-tech/main/humane-tech-badge.svg?sanitize=true)](https://github.com/humanetech-community/awesome-humane-tech)

https://github.com/humanetech-community/awesome-humane-tech

@aschrijver Thanks Arnold! I've added the badge to readme - https://github.com/sheshbabu/freshlytics/commit/3ad75dac53f373ac07df70701fcaef3a394129e9 :)


Incrementing flannel to the latest in the store, and updating readme.

This will point to the most current flannel charm.

+1 LGTM


What's the Max accounts and biggest radius possible?

I added 55 accounts and jumped it to R15. Scanner never worked. I'm pretty sure I got an IP ban straight up. Wondering what's the most accounts and biggest radius someone has achieved and still had a functional scanner.

Got it working at 15 accounts R7 and this time I'm going to attempt to build it up slowly by adding 3-5 accounts daily and increase radius every 10 extra accounts i make.. Hopefully I can get it to a travel radius of 15 minutes in all directions and 3-4 minute full scan 😄

how i add more ptc's accounts?

i working on super scanner, now working with r30, 50ptc, working good 2 weeks.

If I could make a suggestion, I did have several of my accounts get banned.  In replacing those accounts, I logged into them thru the app.  After catching the stater mon, I also caught another mon.  Since then, those accounts were not banned.  I also created some other accounts but did not take this extra step of catching an additional pokemon.  Give it a try if you are getting bans.

Hi I have working 50 PTC in range 13 and seems to work well at first, the issue comes when the second scan or attempt begins to put more accounts or rank as the trail gradually lost, holes appear in the scan and loses speed when it should be the opposite. I have 4 GB of RAM and Windows 10 for if you ask, and I would like to know what is prohibited. :D Saludos desde Argentina- Mar del Plata


omniauth_success.html.erb JSON bug

I realized that, in the current way omniauth_success.html.erb is written (copied below), the JSON returned gives booleans and numbers as strings, e.g. "false", rather than false or "5" rather than 5. This gave me some problems cause I use coffeescript, which only supports the === or !== equalities which means that "5" !== 5. Any way to fix this easily?
<% @resource.as_json.each do |attr, val| %>
  "<%= attr %>": "<%= val %>",
<% end %>

"auth_token": "<%= @token %>",
"message":    "deliverCredentials",
"client_id":  "<%= @client_id %>",
"expiry":     "<%= @expiry %>",
"config":     "<%= @config %>"

You could override this view with something like:
<% @resource.as_json.each do |attr, val| %>

    # Check for number/boolean
    <% if val.is_a?(Numeric) || !!val == val %>
        "<%= attr %>": <%= val %>,
    <% else %>
        "<%= attr %>": "<%= val %>",
    <% end %>
<% end %>

"auth_token": "<%= @token %>",
"message":    "deliverCredentials",
"client_id":  "<%= @client_id %>",
"expiry":     "<%= @expiry %>",
"config":     "<%= @config %>"

Then you'd just need to make sure that your override view is rendered in OmniauthCallbacksController#omniauth_success. I had previously written an override for this controller, so I just replaced this line with:
render :layout => "layouts/omniauth_response", :template => "path_to_my_view_overrides/omniauth_success"

Hope this helps.

ahhh, good idea :) thanks!

Here is a more general solution, which worked for me with array values as well:
    <% @resource.as_json.each do |attr, val| %>

        "<%= attr %>": <%= val.to_json.html_safe %>,
    <% end %>

    "auth_token": "<%= @token %>",
    "message":    "deliverCredentials",
    "client_id":  "<%= @client_id %>",
    "expiry":     "<%= @expiry %>",
    "config":     "<%= @config %>"


Add overlay of tide range to be selected over tide height plot in Coastal_Erosion.ipynb

Suggestion that it would be good to see the range of the tide selected for analysis over the plot of all tide heights that the user creates halfway down the notebook. This would allow the user to better customise the portion of the tide range they are selecting.
https://github.com/GeoscienceAustralia/dea-notebooks/blob/master/DEA_sandbox/Case_Studies/Coastal_Erosion.ipynb

Fixed in #334


Add shell completion script generation to the CLI

clap provides a package for this called available at https://crates.io/crates/clap_complete.
The package would be added as a runtime dependency. The CLI could then provide a new command called completion, accepting the shell name and printing the completion script to stdout.
If this feature is desired, I can open a PR.

Seems reasonable to me to add! If you're up for sending a PR I'm happy to give it a review


Redesign navigation

Remove the app bar from the top. Keep the settings button and uploading button to the left.
Below, add the navigation toolbar. It should have, right to left, [filter, micro, poi, entrances, map layer].

Done in 0.4.


work around Chrome WebXR bug

Fixes #2060
Fixes #2071
Here is the bug: https://bugs.chromium.org/p/chromium/issues/detail?id=1184085
It will be fixed soon, but as it'll take time to roll out to stable, we're putting in this workaround in the meanwhile.

FYI @klausw, this seems to fix it on my Pixel 3. Thanks!


Add test for :focus-within CSS pseudo-class

CSS Selectors Level 4 defines a :focus-within pseudo-class.
A test should be added for it.
Sebastian

Fair enough. I've filed https://github.com/LeaVerou/css3test/issues/142 instead.
Sebastian


zh-cn zh-tw

it will take chinese as japanese,so it will translate many times.

Two things:

Which version was this a problem in? From 3.3.0+, an anti-spam safeguard was implemented that SHOULD prevent the exact thing that is described here. Basically, it is the function "IsTranslatable":

      private bool IsTranslatable( string str )
      {
         return TextHelper.ContainsJapaneseSymbols( str ) && str.Length <= Settings.MaxCharactersPerTranslation && !_translatedTexts.Contains( str );
      }

As you can can see, there is a check that disallows the plugin from translating something that is has already translated something else into. (the exact solution described by Tidyzq, except it is in a different method).

I see your point about ContainsJapanese. Tbh, I hope there is a better way to implement this, but the check is rather important right now because often games outputs a crapton of trash to text UI (depending on the game!), that there is no meaning in translating.

Here's a solution I see. If the FromLanguage is ja(-JP), then the check will be made. If we really want this to be generic, we should consider adding a check for languages that uses different alphabets. Perhaps this is overkill though. :)

i dont know . almost every autotranslator plugins use google has this bug,like cm3d2's autotranslator in hongfire.
but in zodgame forum, someone  said add [cn] behinde translated text can fix it.
also they add baidu to translate text.
https://github.com/xddxdd/CM3D2.AutoTranslate
this is link

Could you try the following version and report back with whether or not you have the same problem, as you previously reported?
XUnity.AutoTranslator-IPA-2.3.2.zip
XUnity.AutoTranslator-BepIn-2.3.2.zip
The fix Tidyzq suggested is absolutely present in this version (although it should have been present in 2.3.0 as well). In addition I have changed the "ContainsJapaneseCharacters" check to a check that is based on the "FromLanguage" param.

i had try ,but still......

Then I will need you to be very specific about the problem you are facing and provide me with steps to reproduce, because when I test it, it works. It works in the sense that nothing is being re-translated twice.
There is a problem in KK, with the font being used. (apparently it is unable to display some of the required kanji).

Actually, i am using 2.3.0 with IPA version, but the problem occurs as usual.
I'm sorry because it is hard to give you a retrieval path to test it as the conversation text is random in the game( the game is Koikatu from Illusion ), and i can't tell which ui component the game is using.
All i can observe from it is that: no error message printed in the IPA console, and the http request did send to google translation server.

The screenshot above is part of my _AutoGeneratedTranslations.zh-tw, as you can see the text wrapped in red is exactly the same, and the second one takes it as untranslated and re-translated it again.
The text being re-translated is “今天將會是美好的一天”, you can set it as the translated text of some words, restart the game, and try if the problem will come up. But again, i don't know which ui component the game is using, so maybe you'll need to try them all.
If you can add some log points and provide me a compiled DLL, then i can provide more
info for you ( i'm sorry i don't have the compiling environment right now ).

Here's another attempt. I have added an additional check to "IsTranslatable" after the text "debounce" function.
What I suspect was happening is:

Text is set to something that is not translated.
Immediately afterwards it is set to something that is translated, and as such it gets "around" the initial check in the "TranslateOrQueueWebJobImmediate" method.

Could you try the following and tell me if the problem still occurs?
XUnity.AutoTranslator-BepIn-2.3.2.zip
XUnity.AutoTranslator-IPA-2.3.2.zip

Stop. I uploaded wrong file! Let me just try again

Try this instead:
XUnity.AutoTranslator-IPA-2.3.2.zip
XUnity.AutoTranslator-BepIn-2.3.2.zip

Great! The re-translation problem never shows again!
But, it is seems it's not exactly the behavior you expected. Like a conversation dialog component, the text slowly shows character by character, the text is already been translated before the whole text is shown, as you expected to see the untranslated one.
In addition, in such case as text shows character by character, the ToggleTranslation method is not working, and the IPA console gives the following error message:
System.NullReferenceException:
  at (wrapper managed-to-native) UnityEngine.Component:get_transform ()
  at XUnity.AutoTranslator.Plugin.Core.TranslationInfo.ResizeUI (System.Object graphic) [0x00000] in <filename unknown>:0
  at XUnity.AutoTranslator.Plugin.Core.AutoTranslationPlugin.SetText (System.Object ui, System.String text, Boolean isTranslated, XUnity.AutoTranslator.Plugin.Core.TranslationInfo info) [0x00000] in <filename unknown>:0
  at XUnity.AutoTranslator.Plugin.Core.AutoTranslationPlugin.ToggleTranslation () [0x00000] in <filename unknown>:0
  at XUnity.AutoTranslator.Plugin.Core.AutoTranslationPlugin.Update () [0x00000] in <filename unknown>:0

And it reminds me that all of the re-translating problem occurs in the conversation dialog!

"But, it is seems it's not exactly the behavior you expected. Like a conversation dialog component, the text slowly shows character by character, the text is already been translated before the whole text is shown, as you expected to see the untranslated one."
This is actually OK. There is some strange behaviour going on when the text is scrolling in, and if I am able to catch a translated string immediately from my cached data, I will translate it immediately without debounce. In this case the "scrolling in" of text is actually based on the translation, and not the original untranslated text.
In addition, in such case as text shows character by character, the ToggleTranslation method is not working, and the IPA console gives the following error message:
Can this be consistently reproduced?

In the Koikatu game, yes, it occurs consistently. But i can't tell if it will occurs in other games.

Here's another version that I'd like you to test:
XUnity.AutoTranslator-IPA-2.3.2.zip
XUnity.AutoTranslator-BepIn-2.3.2.zip
I suspect the problem was that there was a lot of referenced components outside of the current scene that were somehow not attached to the scene/game or whatever anymore.

i had try the newest version,also would re-translate
_AutoGeneratedTranslations.zh-tw.txt
193 and 249  .210 and 250

_AutoGeneratedTranslations.zh-tw.txt
3011 line(the last)
1.i always remove DynamicTranslationLoader while i use autotranlate
2.i had try,this is text......still

use unicode to distinguish chinese or japanese

Finally.it work almost perfect,text dont re-translate .but “☐” still  exist.

still.ignore it.can you add baidu server to translate .in zh-cn.baidu is well than google.

@halulin The only reason why characters were rendered as "☐" is the game doesn't support the character set, you should never blame the translator.
And maybe you should use baidu to translate yourself's words.

Have I missed something here?
Anyway, as Tidyzq says, the font used in the game appears to not support all Kanji, which makes them appear as squares. It's probably possibly to change that, but I would need to research it, as the way to do it would probably be different for each text framework.
As for providing Baidu... Well there's some challenges here.
First of all, I cannot read the documentation at all. Now, this is probably not that much a problem, as I can probably rely on the CM3D2 implementation (if that implementation still works!).
Second of all, I do not know how to obtain an appId/appSecret, and I would absolutely need this in order to implement it.
If you are willing to provide me with this information, I can provide you with an email to send it to (do NOT post it here). Once I have it working, I will delete the appId/appSecret.

ok, I have appid

Alright. Please send the appId and appSecret to mrtranslate (at) protonmail.com  (spam protected notation).

I received it.

FYI, baidu is not like google which doesn't need an appid ( i see you are sending requests to the API used by google translate extension for chrome ), you must write it in your code. So i think there is no difference between putting the appid in your code or in this issue, therefore i will leave it that way XD

Thanks, but it is not needed. I have already received a key and have already have confirmed it is working. Currently working on integrating it. I removed the appId/appSecret from your above comment.

Are you sure? Because baidu's API is not for free, you have to charge it once people actually using your plugin.

I will simply implement support for Baidu translate API.
If the user of the plugin wants to use it with baidu he must provide his own appId/appSecret.

Let's just say that Baidu is being for more explicit by actually requiring an appId/appSecret.

It works perfectly.

Although it also would re-translate,it Not messing up the text,so,in a sense,this bug was fix.


Implements/heartbeat delegate

I basically took the logic Brennan built out for the android component and implemented it here.

This isn't the same as what we did for Android. On Android the delegate is provided via a factory during the "Video Playback Started" event. In here, the delegate is created during initialization instead.

See here https://github.com/segment-integrations/analytics-android-integration-adobe-analytics/blob/master/src/main/java/com/segment/analytics/android/integrations/adobeanalytics/AdobeIntegration.java#L574-L575

@f2prateek updated with requested changes — let me know if I'm on the right track here!

@f2prateek  added tests

@f2prateek I think the issue was I needed to set these values here.
I thought maybe I needed to initialize each instance variable in my SEGPlaybackDelegate implementation with 0, but that didn't seem to do the trick. Only by setting these values in the tests prior to asserting against the outcome did this seem to pass.
My understanding is the the instance variables are automatically initialized with a value of 0 with the instance ([SEGPlaybackDelegate alloc] init] in this case)

I thought maybe I needed to initialize each instance variable in my SEGPlaybackDelegate implementation with 0, but that didn't seem to do the trick. Only by setting these values in the tests prior to asserting against the outcome did this seem to pass.

That seems like a bug, no? The tests are only passing under these certain conditions setup in tests - but these conditions are not being setup in a real app so that means in a real app this behaviour would break.

@f2prateek Updates from yesterday!


Solving the responsiveness issue of the content page

The content page responsiveness is not good and it can be improved. I can solve this issue please assign me this task
@subhadipbhowmik Please add gssoc-ext, hacktoberfest-accepted , also level this issue add this labels

@subhadipbhowmik I have fixed the responsive issue of the content page. Please merge it


Need a way to regularly test multimachine configurations

Not sure what this looks like but somewhere we need to have regular automated test runs with multiple machines using the different communication protocols in different configurations.

This will require some additional thought

I think this will need to be addressed in the example repo, in the tests we run in that repository, since I think it will be easier to test through there.

moving to 1.3 since it likely will require the example repo being operational and some other considerations

This is on hold until the examples Repo is ready then we can discuss again.

This issue will get moved to the HELICS-Examples repo

@kdheepak,  @nightlark  I was working with some folks at UTK today.  They showed me some stuff on mininet.  It looks like a tool that allows you to create a virtual network on a pc or laptop.  We talked about it in the context of using it as part of HELICS, but I don't think that will work very well,  it isn't a simulator per se.  But one thing it might be very useful for us is as a platform for testing the multimachine configurations for HELICS.    I can imagine setting up a couple dockers or  VM's with Mininet,  then maybe HELICS-Runner or something else can kick off a series of test simulations using the multimachine configurations inside the mininet processes,  and maybe we can run this on travis or one of the other services, without actually needing multiple machines.    @kdheepak, @nightlark  can you take a look and see what you think?  I do think we are at the point where we need to be more extensively testing the multimachine stuff and this could be a way to do it.

It looks like the mininet cli (and api?) can be used to run commands in the context of a host on the virtual network.

It looks like Docker Compose is included in Travis build environments. Assuming you're OK with not having Windows as part of your multi-machine testing, couldn't you just use multiple Docker containers orchestrated by Docker Compose in Travis? Each container gets its own IP address and is ran in its own network namespace (unless you configure it for host networking), effectively making each container look like a separate machine from a HELICS application perspective.
Just my two cents, and I know I'm likely missing something obvious given my infant familiarity with HELICS.


Streamer : ShortThrowToFDepth

Hi, I'm trying to send HoloLens ShortThrowToFDepth data through the Streamer project and receive it with the Receiver on the desktop.
Here is what I've been trying to do so far, but I think this is wrong.
Would there be anyone who can help with this issue?
Thank you.

I have just pushed some changes that split the Receiver and Streamer tools into two versions, one handling the photo/video camera, and one handling the new research mode sensors (specifically, the 4 visible light cameras). Enabling the latter required fixing a bug in the receiver that was not unpacking the frames correctly. I haven't tried streaming depth and reflectivity yet, but there's a good chance it should work with some amount of tweaking.
The specific issue you are hitting seems to be something else, and it looks like you're on the right track -- XAML only allows Bgra8 images, so you need to convert the Gray16 stream to Bgra8. There is some code in the streamer that should do it (look for TransformBitmap).

It works!. Thank you for your help!


Disable for specific domains

I'm a big fan of this extension; however, it conflicts with Google Doc's "Strikethrough" text formatting option.
Is it possible to disable this extension for specific domains?

Hmm, I'm not sure. Personally I re-bind mine to Cmd+Shift+P so I haven't run into issues. It's been years now so I can't remember why the default is Cmd/Control+Shift+X but it was related to some other keybinding or something I didn't want to trample for everyone.

alt+p is also unused, and my personal go-


[feat] make landing page responsive for tablet and mobile view

Hello @hparashar27
I would like to work on this issue. kindly assign me!
Thanks!

@Mansi168 sure you can start working on it👍

should we use tailwind css for ui