ARROW-?????: [C++] as-of-join backpressure for large sources

Added support for making generator sources suitable for as-of-join. Also refactored a bit and added docstrings.

@icexelloss, @westonpace: I've added a experimentation tool for backpressure with large sources. Here's an example invocation:
$ (export ARROW_BACKPRESSURE_DEMO_NUM_BATCHES=10000; export ARROW_BACKPRESSURE_DEMO_BATCH_SIZE=1000; ./release/arrow-compute-asof-join-node-test --gtest_filter=AsofJoinTest.BackpressureDemoWithBatchesGen)
Running main() from ./googletest/src/gtest_main.cc
Note: Google Test filter = AsofJoinTest.BackpressureDemoWithBatchesGen
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from AsofJoinTest
[ RUN      ] AsofJoinTest.BackpressureDemoWithBatchesGen
[       OK ] AsofJoinTest.BackpressureDemoWithBatchesGen (115623 ms)
[----------] 1 test from AsofJoinTest (115623 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (115623 ms total)
[  PASSED  ] 1 test.

Trying various parameterizations, I've observed runs with:

low batch size leading to reasonably capped memory (VIRT and RES) usage;
high batch size leading to exhausted memory, perhaps before reaching the maximum.

I intend to investigate further.

This CI job failure suggests that the detach code path in ~AsofJoinNode, which occurs when the process thread invokes the destructor, leads to a race condition, i.e., the destructor completes too early. Given this, I don't yet have a solution for the case of the process thread invoking the destructor.

This CI job failure suggests that the detach code path in ~AsofJoinNode, which occurs when the process thread invokes the destructor, leads to a race condition, i.e., the destructor completes too early. Given this, I don't yet have a solution for the case of the process thread invoking the destructor

Is this an issue because of using the serial executor? I don't recall having this issue before

Is this an issue because of using the serial executor? I don't recall having this issue before

No, the issue is due to a fix, the detach call, I added in the destructor, which I did because of a deadlock problem (process thread trying to join on itself) that the slow-source experiment revealed. So, my fix is not good, and while the adding of the detach call can be reverted, I'm not sure how to properly fix the original issue.

Ok. As far as I recall this is a new issue that got introduced on this branch. Perhaps it is because of the serial execution but I am not sure. Do you know why would the process thread try to join itself? As far as I remember the destructor should be called on the thread that owns the Plan then join the process thread.

Perhaps @westonpace has some insight what might be going on

Ah, we probably need to move line 952 up to line 950 in asof_join_node.cc so that the node gets mark finished on a plan thread and not on the process thread.  I'll reproduce / fix that this evening (I'm in meetings until about 2PM PST so I won't be able to look at it until then).  I'll do a quick review and merge then too.

Aha - that makes sense
On Thu, Oct 27, 2022 at 10:58 AM Weston Pace @.***>
wrote:

Ah, we probably need to move line 952 up to line 950 in asof_join_node.cc
so that the node gets mark finished on a plan thread and not on the process
thread. I'll reproduce / fix that this evening (I'm in meetings until about
2PM PST so I won't be able to look at it until then). I'll do a quick
review and merge then too.
—
Reply to this email directly, view it on GitHub
https://github.com/westonpace/arrow/pull/21#issuecomment-1293653614, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGBXLDJIZGWTRJ67EYBUO3WFKKBXANCNFSM6AAAAAARPN4O44
.
You are receiving this because you were mentioned.Message ID:
@.***>

This sounds in the right direction, however:

The code in line 950 should ensure that MarkFinished is
The destructor code should be reverted to just joining, without a detach cud path.

Just moving MarkFinished into the plan task caused a small problem because then the process thread might run again before the plan task had a chance to launch and it might try to mark the node finished a second time.  I changed the process thread to return a bool so that it returns false if it is finished (and then this exits the thread).  I don't think RAII is needed anymore since, as soon as the if (state_.at(0)->Finished()) check returns true we know that Process() will return false and the thread will end so there is no risk of deadlock if the MarkFinished call fails.

Also, I was able to reproduce the bug by repeatedly running the new backpressure test under stress and confirmed that this change prevented this bug.

@westonpace, your recent commit appears to cause an as-of-join test to hang on my machine. The issue doesn't seem to be deterministic; I see a hang in a "random" AsofJoinBasicTest.TestBasic* test each time. Did you check these tests?

Below is a gdb session of a hanging test. I waited maybe a minute before manually interrupting.
$  gdb --args  ./debug/arrow-compute-asof-join-node-test --gtest_filter='AsofJoinNodeTest/AsofJoinBasicTest.TestBasic*'
GNU gdb (Ubuntu 12.0.90-0ubuntu1) 12.0.90
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./debug/arrow-compute-asof-join-node-test...
(gdb) run
Starting program: /mnt/user1/tscontract/github/westonpace/arrow/cpp/build/debug/debug/arrow-compute-asof-join-node-test --gtest_filter=AsofJoinNodeTest/AsofJoinBasicTest.TestBasic\*
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff2dff640 (LWP 248925)]
Running main() from ./googletest/src/gtest_main.cc
Note: Google Test filter = AsofJoinNodeTest/AsofJoinBasicTest.TestBasic*
[==========] Running 36 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 36 tests from AsofJoinNodeTest/AsofJoinBasicTest
[ RUN      ] AsofJoinNodeTest/AsofJoinBasicTest.TestBasic1/0
[New Thread 0x7ffff25fe640 (LWP 248926)]
[New Thread 0x7ffff17ff640 (LWP 248927)]
[Thread 0x7ffff25fe640 (LWP 248926) exited]
[New Thread 0x7ffff25fe640 (LWP 248928)]
[Thread 0x7ffff25fe640 (LWP 248928) exited]
[New Thread 0x7ffff25fe640 (LWP 248929)]
[Thread 0x7ffff25fe640 (LWP 248929) exited]
[New Thread 0x7ffff25fe640 (LWP 248930)]
[Thread 0x7ffff25fe640 (LWP 248930) exited]
[New Thread 0x7ffff25fe640 (LWP 248931)]
[Thread 0x7ffff25fe640 (LWP 248931) exited]
[New Thread 0x7ffff25fe640 (LWP 248932)]
[Thread 0x7ffff25fe640 (LWP 248932) exited]
^C
Thread 1 "arrow-compute-a" received signal SIGINT, Interrupt.
__futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555557bd330) at ./nptl/futex-internal.c:57
57	./nptl/futex-internal.c: No such file or directory.
(gdb) info threads
  Id   Target Id                                            Frame
* 1    Thread 0x7ffff7e281c0 (LWP 248922) "arrow-compute-a" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555557bd330)
    at ./nptl/futex-internal.c:57
  2    Thread 0x7ffff2dff640 (LWP 248925) "jemalloc_bg_thd" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffff3416630)
    at ./nptl/futex-internal.c:57
  4    Thread 0x7ffff17ff640 (LWP 248927) "jemalloc_bg_thd" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffff3416704)
    at ./nptl/futex-internal.c:57
(gdb) thread apply all bt

Thread 4 (Thread 0x7ffff17ff640 (LWP 248927) "jemalloc_bg_thd"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffff3416704) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffff3416704) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffff3416704, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff3869ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffff3416748, cond=0x7ffff34166d8) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x7ffff34166d8, mutex=mutex@entry=0x7ffff3416748) at ./nptl/pthread_cond_wait.c:627
#5  0x00007ffff69e50b3 in background_thread_sleep (tsdn=<optimized out>, interval=<optimized out>, info=<optimized out>) at src/background_thread.c:137
#6  background_work_sleep_once (ind=<optimized out>, info=<optimized out>, tsdn=<optimized out>) at src/background_thread.c:229
#7  background_work (ind=<optimized out>, tsd=<optimized out>) at src/background_thread.c:419
#8  background_thread_entry (ind_arg=<optimized out>) at src/background_thread.c:444
#9  0x00007ffff386ab43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007ffff38fca00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7ffff2dff640 (LWP 248925) "jemalloc_bg_thd"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7ffff3416630) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7ffff3416630) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffff3416630, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff3869ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffff3416678, cond=0x7ffff3416608) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x7ffff3416608, mutex=mutex@entry=0x7ffff3416678) at ./nptl/pthread_cond_wait.c:627
#5  0x00007ffff69e58b5 in background_thread_sleep (tsdn=<optimized out>, interval=<optimized out>, info=<optimized out>) at src/background_thread.c:137
#6  background_work_sleep_once (ind=0, info=<optimized out>, tsdn=<optimized out>) at src/background_thread.c:229
#7  background_thread0_work (tsd=<optimized out>) at src/background_thread.c:374
#8  background_work (ind=<optimized out>, tsd=<optimized out>) at src/background_thread.c:412
#9  background_thread_entry (ind_arg=<optimized out>) at src/background_thread.c:444
#10 0x00007ffff386ab43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007ffff38fca00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7ffff7e281c0 (LWP 248922) "arrow-compute-a"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5555557bd330) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5555557bd330) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5555557bd330, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffff3869ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5555557bd2e0, cond=0x5555557bd308) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x5555557bd308, mutex=0x5555557bd2e0) at ./nptl/pthread_cond_wait.c:627
#5  0x00007ffff56b763d in std::condition_variable::wait<arrow::internal::SerialExecutor::RunLoop()::<lambda()> >(std::unique_lock<std::mutex> &, struct {...}) (this=0x5555557bd308, __lock=..., __p=...) at /usr/include/c++/11/condition_variable:103
#6  0x00007ffff56b56e0 in arrow::internal::SerialExecutor::RunLoop (this=0x7fffffffca40) at /mnt/user1/tscontract/github/westonpace/arrow/cpp/src/arrow/util/thread_pool.cc:173
#7  0x00007ffff7a6ed38 in arrow::internal::SerialExecutor::Run<arrow::internal::Empty, arrow::Status>(arrow::internal::FnOnce<arrow::Future<arrow::internal::Empty> (arrow::internal::Executor*)>) (this=0x7fffffffca40, initial_task=...) at /mnt/user1/tscontract/github/westonpace/arrow/cpp/src/arrow/util/thread_pool.h:400
#8  0x00007ffff7a68587 in arrow::internal::SerialExecutor::RunInSerialExecutor<arrow::internal::Empty, arrow::Future<arrow::internal::Empty>, arrow::Status>(arrow::internal::FnOnce<arrow::Future<arrow::internal::Empty> (arrow::internal::Executor*)>) (initial_task=...) at /mnt/user1/tscontract/github/westonpace/arrow/cpp/src/arrow/util/thread_pool.h:297
#9  0x00007ffff7a610e2 in arrow::internal::RunSynchronously<arrow::Future<arrow::internal::Empty>, arrow::internal::Empty>(arrow::internal::FnOnce<arrow::Future<arrow::internal::Empty> (arrow::internal::Executor*)>, bool) (get_future=..., use_threads=false) at /mnt/user1/tscontract/github/westonpace/arrow/cpp/src/arrow/util/thread_pool.h:510
#10 0x00007ffff7a5373c in arrow::compute::StartAndFinish (plan=0x5555558ae070, use_threads=false) at /mnt/user1/tscontract/github/westonpace/arrow/cpp/src/arrow/compute/exec/test_util.cc:178
#11 0x00007ffff7a539f6 in arrow::compute::StartAndCollect(arrow::compute::ExecPlan*, std::function<arrow::Future<std::optional<arrow::compute::ExecBatch> > ()>, bool) (plan=0x5555558ae070, gen=..., use_threads=false) at /mnt/user1/tscontract/github/westonpace/arrow/cpp/src/arrow/compute/exec/test_util.cc:189
#12 0x0000555555662dee in arrow::compute::CheckRunOutput (l_batches=..., r0_batches=..., r1_batches=..., exp_batches=..., join_options=...) at /mnt/user1/tscontract/github/westonpace/arrow/cpp/src/arrow/compute/exec/asof_join_node_test.cc:238
#13 0x0000555555663f86 in arrow::compute::CheckRunOutput (l_batches=..., r0_batches=..., r1_batches=..., exp_batches=..., time=..., key=..., tolerance=1000) at /mnt/user1/tscontract/github/westonpace/arrow/cpp/src/arrow/compute/exec/asof_join_node_test.cc:262
#14 0x000055555567e6ba in arrow::compute::BasicTest::RunSingleByKey()::{lambda(arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema)#1}::operator()(arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema) const (__closure=0x7fffffffd1b8, l_batches=..., r0_batches=..., r1_batches=..., exp_nokey_batches=--Type <RET> for more, q to quit, c to continue without paging--c
..., exp_emptykey_batches=..., exp_batches=...) at /mnt/user1/tscontract/github/westonpace/arrow/cpp/src/arrow/compute/exec/asof_join_node_test.cc:459
#15 0x00005555556a68cc in arrow::compute::BasicTest::RunTypes<arrow::compute::BasicTest::RunSingleByKey()::{lambda(arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema)#1}>(arrow::compute::BasicTestTypes, arrow::compute::BasicTest::RunSingleByKey()::{lambda(arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema)#1}) (this=0x7fffffffdc10, basic_test_types=..., batches_runner=...) at /mnt/user1/tscontract/github/westonpace/arrow/cpp/src/arrow/compute/exec/asof_join_node_test.cc:628
#16 0x000055555568e21c in arrow::compute::BasicTest::RunBatches<arrow::compute::BasicTest::RunSingleByKey()::{lambda(arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema)#1}>(arrow::compute::BasicTest::RunSingleByKey()::{lambda(arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema, arrow::compute::BatchesWithSchema)#1}) (this=0x7fffffffdc10, batches_runner=...) at /mnt/user1/tscontract/github/westonpace/arrow/cpp/src/arrow/compute/exec/asof_join_node_test.cc:591
#17 0x000055555567e749 in arrow::compute::BasicTest::RunSingleByKey (this=0x7fffffffdc10) at /mnt/user1/tscontract/github/westonpace/arrow/cpp/src/arrow/compute/exec/asof_join_node_test.cc:457
#18 0x000055555567e768 in arrow::compute::BasicTest::DoSingleByKey (basic_tests=...) at /mnt/user1/tscontract/github/westonpace/arrow/cpp/src/arrow/compute/exec/asof_join_node_test.cc:463
#19 0x00005555556dbb25 in std::__invoke_impl<void, void (*&)(arrow::compute::BasicTest&), arrow::compute::BasicTest&> (__f=@0x7fffffffdcd0: 0x55555567e74c <arrow::compute::BasicTest::DoSingleByKey(arrow::compute::BasicTest&)>) at /usr/include/c++/11/bits/invoke.h:61
#20 0x00005555556d703b in std::__invoke_r<void, void (*&)(arrow::compute::BasicTest&), arrow::compute::BasicTest&> (__fn=@0x7fffffffdcd0: 0x55555567e74c <arrow::compute::BasicTest::DoSingleByKey(arrow::compute::BasicTest&)>) at /usr/include/c++/11/bits/invoke.h:111
#21 0x00005555556d126f in std::_Function_handler<void (arrow::compute::BasicTest&), void (*)(arrow::compute::BasicTest&)>::_M_invoke(std::_Any_data const&, arrow::compute::BasicTest&) (__functor=..., __args#0=...) at /usr/include/c++/11/bits/std_function.h:290
#22 0x0000555555694677 in std::function<void (arrow::compute::BasicTest&)>::operator()(arrow::compute::BasicTest&) const (this=0x7fffffffdcd0, __args#0=...) at /usr/include/c++/11/bits/std_function.h:590
#23 0x0000555555668689 in arrow::compute::AsofJoinBasicTest_TestBasic1_Test::TestBody (this=0x5555557ae560) at /mnt/user1/tscontract/github/westonpace/arrow/cpp/src/arrow/compute/exec/asof_join_node_test.cc:662
#24 0x00007ffff7b1422f in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) () from /mnt/user1/tscontract/github/westonpace/arrow/cpp/build/debug/debug/libarrow_testing.so.1000
#25 0x00007ffff7b08dd6 in testing::Test::Run() () from /mnt/user1/tscontract/github/westonpace/arrow/cpp/build/debug/debug/libarrow_testing.so.1000
#26 0x00007ffff7b08f55 in testing::TestInfo::Run() () from /mnt/user1/tscontract/github/westonpace/arrow/cpp/build/debug/debug/libarrow_testing.so.1000
#27 0x00007ffff7b09509 in testing::TestSuite::Run() () from /mnt/user1/tscontract/github/westonpace/arrow/cpp/build/debug/debug/libarrow_testing.so.1000
#28 0x00007ffff7b09c0f in testing::internal::UnitTestImpl::RunAllTests() () from /mnt/user1/tscontract/github/westonpace/arrow/cpp/build/debug/debug/libarrow_testing.so.1000
#29 0x00007ffff7b147f7 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) () from /mnt/user1/tscontract/github/westonpace/arrow/cpp/build/debug/debug/libarrow_testing.so.1000
#30 0x00007ffff7b0901c in testing::UnitTest::Run() () from /mnt/user1/tscontract/github/westonpace/arrow/cpp/build/debug/debug/libarrow_testing.so.1000
#31 0x000055555565fb04 in main ()
(gdb) Quit
(gdb)

Another issue seems to be that TSAN is reporting a race condition because the backpressure concurrent queue's DoHandle is accessing UnsyncSize which doesn't use any lock.  I'll work on the deadlock first and then look at the race condition later.

I'll commit shortly with a possible fix. @westonpace, please review.

Your fix is ok but that condition will never evaluate to false so if your fix works then we should just get rid of that if check entirely.

*never evaluate to true

Btw, here is the race condition:
WARNING: ThreadSanitizer: data race (pid=20018)
  Read of size 8 at 0x7b4c00001f90 by main thread:
    #0 std::operator-(std::_Deque_iterator<std::shared_ptr<arrow::RecordBatch>, std::shared_ptr<arrow::RecordBatch>&, std::shared_ptr<arrow::RecordBatch>*> const&, std::_Deque_iterator<std::shared_ptr<arrow::RecordBatch>, std::shared_ptr<arrow::RecordBatch>&, std::shared_ptr<arrow::RecordBatch>*> const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_deque.h:358:25 (libarrow.so.1000+0x1481995)
    #1 std::deque<std::shared_ptr<arrow::RecordBatch>, std::allocator<std::shared_ptr<arrow::RecordBatch> > >::size() const /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_deque.h:1233:40 (libarrow.so.1000+0x1481840)
    #2 std::queue<std::shared_ptr<arrow::RecordBatch>, std::deque<std::shared_ptr<arrow::RecordBatch>, std::allocator<std::shared_ptr<arrow::RecordBatch> > > >::size() const /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_queue.h:209:18 (libarrow.so.1000+0x14817f5)
    #3 arrow::compute::ConcurrentQueue<std::shared_ptr<arrow::RecordBatch> >::UnsyncSize() const /home/pace/dev/arrow/cpp/src/arrow/compute/exec/asof_join_node.cc:128:45 (libarrow.so.1000+0x14817b5)
    #4 arrow::compute::BackpressureConcurrentQueue<std::shared_ptr<arrow::RecordBatch> >::DoHandle::DoHandle(arrow::compute::BackpressureConcurrentQueue<std::shared_ptr<arrow::RecordBatch> >&) /home/pace/dev/arrow/cpp/src/arrow/compute/exec/asof_join_node.cc:297:45 (libarrow.so.1000+0x1481524)
    #5 arrow::compute::BackpressureConcurrentQueue<std::shared_ptr<arrow::RecordBatch> >::Push(std::shared_ptr<arrow::RecordBatch> const&) /home/pace/dev/arrow/cpp/src/arrow/compute/exec/asof_join_node.cc:318:14 (libarrow.so.1000+0x14a7346)
    #6 arrow::compute::InputState::Push(std::shared_ptr<arrow::RecordBatch> const&) /home/pace/dev/arrow/cpp/src/arrow/compute/exec/asof_join_node.cc:550:14 (libarrow.so.1000+0x14a68d8)
    #7 arrow::compute::AsofJoinNode::InputReceived(arrow::compute::ExecNode*, arrow::compute::ExecBatch) /home/pace/dev/arrow/cpp/src/arrow/compute/exec/asof_join_node.cc:1278:31 (libarrow.so.1000+0x14695c5)

  Previous write of size 8 at 0x7b4c00001f90 by thread T3 (mutexes: write M860041654558527520, write M859197195268661200):
    #0 std::deque<std::shared_ptr<arrow::RecordBatch>, std::allocator<std::shared_ptr<arrow::RecordBatch> > >::pop_front() /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_deque.h:1539:6 (libarrow.so.1000+0x1481f46)
    #1 std::queue<std::shared_ptr<arrow::RecordBatch>, std::deque<std::shared_ptr<arrow::RecordBatch>, std::allocator<std::shared_ptr<arrow::RecordBatch> > > >::pop() /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_queue.h:301:4 (libarrow.so.1000+0x1481aa5)
    #2 arrow::compute::ConcurrentQueue<std::shared_ptr<arrow::RecordBatch> >::TryPop() /home/pace/dev/arrow/cpp/src/arrow/compute/exec/asof_join_node.cc:112:14 (libarrow.so.1000+0x1481609)
    #3 arrow::compute::BackpressureConcurrentQueue<std::shared_ptr<arrow::RecordBatch> >::TryPop() /home/pace/dev/arrow/cpp/src/arrow/compute/exec/asof_join_node.cc:329:32 (libarrow.so.1000+0x14813bc)
    #4 arrow::compute::InputState::Advance() /home/pace/dev/arrow/cpp/src/arrow/compute/exec/asof_join_node.cc:496:38 (libarrow.so.1000+0x146f830)
    #5 arrow::compute::InputState::AdvanceAndMemoize(unsigned long) /home/pace/dev/arrow/cpp/src/arrow/compute/exec/asof_join_node.cc:534:7 (libarrow.so.1000+0x1475f5c)
    #6 arrow::compute::AsofJoinNode::UpdateRhs() /home/pace/dev/arrow/cpp/src/arrow/compute/exec/asof_join_node.cc:845:7 (libarrow.so.1000+0x146e9a8)
    #7 arrow::compute::AsofJoinNode::ProcessInner() /home/pace/dev/arrow/cpp/src/arrow/compute/exec/asof_join_node.cc:883:7 (libarrow.so.1000+0x146cd8e)
    #8 arrow::compute::AsofJoinNode::Process() /home/pace/dev/arrow/cpp/src/arrow/compute/exec/asof_join_node.cc:937:53 (libarrow.so.1000+0x146be04)

@westonpace, if the as-of-join tests work with your fix, then let's go with it. I'd just suggest adding this patch to it.

To be sure, is the race condition related to the hang and fixed by your hang-fixing code?

Another issue seems to be that TSAN is reporting a race condition because the backpressure concurrent queue's DoHandle is accessing UnsyncSize which doesn't use any lock. I'll work on the deadlock first and then look at the race condition later.

Never mind, now I see this.

I'll work on the deadlock first and then look at the race condition later.

Ah, the DoHandle logic should be ran inside the mutex lock of ConcurrentQueue. I'll work on this.

@rtpsw It looks like this PR is a mixture of
(1) Various refactors
(2) Fix for a deadlock / race condition.
Can you better title the PR and summarize the change so I can understand how urgent/important this issue is?

This PR started with refactoring and adding testing of as-of-join backpressure for large sources. During discussions, a couple of side-issues were handled:

A destructor issue: This issue led to a couple of attempted fixes. Originally, the destructor caused a deadlock when it was called from the process thread. Then, a detach code path attempt was made, but it led to a (first) race condition. Finally, Weston said he has a fix that remains to be integrated. @westonpace, are you happy with this fix? and intend to add it? or should I?
A TSAN (second) race condition issue: My recent commit should fix this issue. @westonpace, could you run you ASAN test again on this commit or let me know how to run it?

Given the above, I think this PR is important to finish soon. If we had ample time, we could have had a separate PR for each side-issue.

A destructor issue: This issue led to a couple of attempted fixes. Originally, the destructor caused a deadlock when it was called from the process thread. Then, a detach code path attempt was made, but it led to a (first) race condition. Finally, https://github.com/westonpace/arrow/pull/21#issuecomment-1295180514 he has a fix that remains to be integrated. @westonpace, are you happy with this fix? and intend to add it? or should I?

@rtpsw I put my fix in and added your defer suggestion.  If custom code throws exceptions it will probably still break things.  I don't think we are going to solve that today.  But at least we can be a little bit safer :)

A TSAN (second) race condition issue: My recent commit should fix this issue. @westonpace, could you run you ASAN test again on this commit or let me know how to run it?

I ran my test repeatedly and did not see the issue.  I was actually running TSAN, just with stress (an ubuntu program which stresses the CPU).  To do this I run stress -c 32 in a separate window (I have 16 cores so 32 stressors jams the system up pretty good which is the goal).  Sometimes I also find it useful to limit the # of cores a test runs on.  I do this with taskset -c 0,1 TEST_NAME.  That would limit the test to cores 0 & 1.  This kind of thing can help mimic the CI environment.
If you want to run ASAN you can (there is a CI job which runs it as well) by setting ARROW_USE_ASAN in the cmake options.  For good measure I went ahead and ran the asof join tests through ASAN a few times and didn't see anything.

I went ahead and merged this since we seemed to be in a stable state.  Let me know if there is anything else I need to look at.


[ldd-rings] Request for version 1G00_1B00

An R&A data provider requires version 1G00_1B00 of the rings dictionary to be made available at https://pds.nasa.gov/datastandards/dictionaries. I can find 1G00_1A00 at https://pds.nasa.gov/datastandards/dictionaries/index-<IP_ADDRESS>.shtml#rings, but I cannot find 1G00_1B00.
The version is already built at https://github.com/pds-data-dictionaries/ldd-rings/tree/main/build/release/<IP_ADDRESS>, but when I try to run the validate tool on the R&A data provider's bundle, there is an error:
 Error occurred while loading schematron: Schematron 'https://pds.nasa.gov/pds4/rings/v1/PDS4_RINGS_1G00_1B00.sch' error: Cannot read schematron from URL https://pds.nasa.gov/pds4/rings/v1/PDS4_RINGS_1G00_1B00.sch
LDD Dictionary Version
PDS4_RINGS_1G00_1B00

Would Richard Chen be able to fix this? @rchenatjpl

@mace-space this is actually @c-suh . even though the release was tagged, it looks like it was tagged as part of the 1H00_1B00 release, but we only grab the latest LDD released when we post online. Any past versions need to be manually released online.
You can do so here: https://github.com/NASA-PDS/operations/issues/new?template=-ldd-request--pds-ldd-release.md

Thanks @jordanpadams. I've requested it via your link

Addressed in https://github.com/NASA-PDS/operations/issues/352 (now closed).

thanks @c-suh


.every() .find() .filter() methods fail with "This expression is not callable." error on union of array types

Bug Report

🔎 Search Terms

This expression is not callable
none of those signatures are compatible with each other

🕗 Version & Regression Information

This is the behavior in every version I tried, and I reviewed the FAQ for entries about union types. I also reviewed previous related issues.

⏯ Playground Link

.every() does not work
.filter() does not work
.find() does not work
.reduce() also doesn't work, but there is already an issue for that one. (Perhaps we should merge it into this?)

💻 Code

interface Fizz {
    id: number;
    fizz: string;
}

interface Buzz {
    id: number;
    buzz: string;
}

([] as Fizz[] | Buzz[]).filter(item => item.id < 5);

🙁 Actual behavior
Compilation fails with the following error:
This expression is not callable.
  Each member of the union type '{ <S extends Fizz>(predicate: (value: Fizz, index: number, array: Fizz[]) => value is S, thisArg?: any): S[]; (predicate: (value: Fizz, index: number, array: Fizz[]) => unknown, thisArg?: any): Fizz[]; } | { ...; }' has signatures, but none of those signatures are compatible with each other.

🙂 Expected behavior
Compilation succeeds, as it does for map and several other array functions.
🔖 Useful links

Original issue
PR that closed original issue

Duplicate of #36390
https://github.com/microsoft/TypeScript/issues?q=is%3Aissue+is%3Aopen+array+"This+expression+is+not+callable"

Hi @DanielRosenwasser, please review my comment on that issue (also linked in the writeup above). This issue is related to #36390, but it is not a duplicate. This issue tracks the handful of array functions which still don't work with union types post-#31023.

@weswigham what's the current state of what we might be able to do here?

Well, we'd need to come up with sensible rules for merging overload lists. Conceptually, we could (attempt to) make a list of 4 overloads by combining each of the input signatures, except there's no inherent ordering to those 4 signatures, unlike a normal overload set, so it's unclear how we'd choose which signature to resolve against, ultimately.

@weswigham Is there anything in particular that makes methods like filter difficult to work with compared to map? I haven't looked at the type definition in TS. But from a JS standpoint, the two signatures are the same; so it's a bit surprising that the methods involving predicates are more complex.

Maybe something could be done specifically for arrays instead of worrying about merging arbitrary signatures?  Like, widen A[] | B[] to readonly (A | B)[] for these failing method calls?  🤷♂

Can we add concat to that list above, too?
I have a typescript-playground showing this here.

.reduce() also doesn't work, but there is already an issue for that one. (Perhaps we should merge it into this?)

I think it should be merged; if not then ita description "Array.reduce doesn't work" needs to be improved. See 44063.

Probably known, but also affects:
const arr = [] as Array<number> | Uint8Array;
arr.reduce(...)

Hey @DanielRosenwasser, any estimates when could that be fixed?

For anyone looking for an easy workaround:
interface Fizz {
    id: number;
    fizz: string;
}

interface Buzz {
    id: number;
    buzz: string;
}

const arr: Fizz[] | Buzz[] = [];

arr.filter(item => item.id < 5); // This is an error

const arrFixed: Array<typeof arr[number]> = arr; // This is the fix

arrFixed.filter(item => item.id < 5); // This works!

It's not exactly equivalent but should work for most cases. Hope this helps!

@gaberogan this just makes arrFixed type to any

@gaberogan this just makes arrFixed type to any

What are you talking about? Why would that result in any?
declare const a: string[] | number[]

const b = a.filter(Boolean); // Error. Should work and result in `string[] | number[]`
const c = (a as (typeof a[number])[]).filter(Boolean) as typeof a // Works, but wrong type. `c` is (string | number)[]
const d = (a as (typeof a[number])[]).filter(Boolean) as typeof a // Works but has two typecasts. Cumbersome.

Playground link

Hello @weswigham, please has this issue been resolved?

@Emmahchinonso you can see a few lines above your comment is a linked pull request that will close the issue. This issue is still open, meaning it is still open, but you can subscribe to notifications if you want.

will this also gonna work after this is merged?
const arr: string[] | number[] = [];
const val: string | number = 5;
const index = arr.indexOf(val);

@gegxen Doesn't seem to work.
https://www.staging-typescript.org/play?ts=5.1.0-pr-53489-26#code/MYewdgzgLgBAhgJwQLhtBBLMBzA2gXRgB8YwBXAWwCMBTBAmAXhgIG4AoUSWANzgBtU6LNmKlKtBExgBWDl2gwsAExoAPaYgQA6FeoDyAMwAUffgEoOQA

@gegxen why don’t you clone the commit in the linked PR and try it?

Copying snippet from @gaberogan
It's better to union the type before typing it as Array.
interface Fizz {
    id: number;
    fizz: string;
}

interface Buzz {
    id: number;
    buzz: string;
}

TFizzBuzz = Fizz | Buzz
const arr: TFizzBuzz[] = [];

arr.filter(item => item.id < 5); // This works for me

@ericbf No it seems fine to me, arrFixed is not any

@ericbf No it seems fine to me, arrFixed is not any

Exactly my point—I stated that arrFixed in the example was not any, contrary to @bmiakoun's comment above. Anyways, this issue is closed now—if you find something that needs to be discussed or fixed, probably open a new issue for that.

For anyone looking for an easy workaround:
interface Fizz {
    id: number;
    fizz: string;
}

interface Buzz {
    id: number;
    buzz: string;
}

const arr: Fizz[] | Buzz[] = [];

arr.filter(item => item.id < 5); // This is an error

const arrFixed: Array<typeof arr[number]> = arr; // This is the fix

arrFixed.filter(item => item.id < 5); // This works!

It's not exactly equivalent but should work for most cases. Hope this helps!

Or just do a simple change like this:
_ Use this: const arr: (Fizz | Buzz)[] = [];
_ Instead of this: const arr: Fizz[] | Buzz[] = [];

Hello @Jv7680, this issue has been closed for a while now.
But, for the record, the issue with that suggestion is that the type would not actually be correct. But that's been discussed already further up in the thread.


WIP: Start hosts file with reg-ex-ed domains from

Hi! Following the suggestions in #9 and https://github.com/StevenBlack/hosts/issues/720, I wanted to test whether this works for subscribing to such a file in adblockers. Yes, it does:

So, this PR suggests to add a hosts file to /_data, which I manually merged from domains I extracted from the journals & publishers lists. It probably contains a few false-positives still, so please don't merge until we discussed how to auto-generate such a file if it is desired.
For now, it can be tested though by adding this URL https://raw.githubusercontent.com/stop-predatory-journals/stop-predatory-journals.github.io/0e64ce25fa147df7d0a79660ec91526c6436eb88/_data/hosts to adblockers like uBlock. In case some do not support this hosts file format, we can find a more widely supported format and update this PR.

Hi, great idea!
I'm interested in flagging URLs/URIs from predatory journals on Cobaltmetrics.com. I was about to generate a list of hosts too, good thing I checked the PRs.
I reviewed your list, and I have one suggestion regarding false positives. Some hosts host multiple journals, and I think there are cases where we don't want to block all URLs from a given host  (does one bad apple spoil the whole bunch?).
For example, journals.csv includes http://journals.sfu.ca/africanem/index.php/ajtcam/index (cf. line 20), but I don't think we want to include journals.sfu.ca in the list of hosts (cf. line 1581).
What about focusing first on empty paths, / (HTTP defines an empty path to be equivalent to / anyway) and a few obvious root-like paths, e.g. anything that matches ^/(default|index|home)\.(aspx|html?|php)$ in a case-insensitive way?
Maybe also add the constraint that URLs should have no query component to be included in the list? To avoid false positives when an acceptable host hosts multiple journals, and the name of the journal is given in the query string, e.g. https://goodhost/index.php?journal=badjournal.

Good point! Using domains for blocking is rather coarse, and probably too broad. Since there has been no reaction from @stoppredatoryjournals, I guess this can be closed as out of scope.
Maybe a better approach would be to PR a conversion pipeline from _data/*.csv to an adblocker-compatible file format.
Could such a pipeline then handle the don't-block-after-all features you mention?

I don't know much about the internals of adblockers, e.g. if the biggest adblockers use the same syntax for their filters, but I eyeballed a few lists from https://filterlists.com/, and including paths (rather than just domains and hosts) seems possible.
In any case, yes, if the list you want to build is derived from the "main" lists, I think the code would be more valuable than the result.

Oh, and I extracted the set of paths from all URLs in _data/*.csv, lowercased everything and filtered out paths that contain acronyms or what looked like site- or journal-specific information. Here's the Gist: https://gist.github.com/lucboruta/0ea6ab3adac42f8eba6237ee9847c308
The list isn't very long, but there is more variation than I expected. We can't know for sure whether we can block the whole domain without some kind of manual validation.


chef-client ignores --chef-zero-host

chef-client is ignoring the command line option --chef-zero-host. See output below. I pass <IP_ADDRESS> to --chef-zero-host, and it ignores it and uses <IP_ADDRESS> instead (which is also an incorrect IP. I don't have that IP on my local system). It does honor the --chef-zero-port option on port 9000 however.
doug@ip-192-168-5-113 slice-java (dev) (master)*$ CHEF_DRIVER=docker chef-client -z recipes/docker.rb --chef-zero-host <IP_ADDRESS> --chef-zero-port 9000
[2017-01-13T07:15:43-08:00] INFO: Started chef-zero at chefzero://localhost:9000 with repository at /Users/doug/git/chef/chef-sauce
  One version per cookbook
  cookbooks at /Users/doug/git/chef/chef-cookbooks

[2017-01-13T07:15:43-08:00] INFO: Forking chef instance to converge...
Starting Chef Client, version 12.15.19
[2017-01-13T07:15:43-08:00] INFO: *** Chef 12.15.19 ***
[2017-01-13T07:15:43-08:00] INFO: Platform: x86_64-darwin13
[2017-01-13T07:15:43-08:00] INFO: Chef-client pid: 91986
[2017-01-13T07:15:47-08:00] INFO: Run List is []
[2017-01-13T07:15:47-08:00] INFO: Run List expands to []
[2017-01-13T07:15:47-08:00] INFO: Starting Chef Run for doug
[2017-01-13T07:15:47-08:00] INFO: Running start handlers
[2017-01-13T07:15:47-08:00] INFO: Start handlers complete.
[2017-01-13T07:15:47-08:00] INFO: HTTP Request Returned 404 Not Found: Object not found:
resolving cookbooks for run list: []
[2017-01-13T07:15:48-08:00] INFO: Loading cookbooks []
Synchronizing Cookbooks:
Installing Cookbook Gems:
Compiling Cookbooks...
[2017-01-13T07:15:48-08:00] WARN: Node doug has an empty run list.
Converging 1 resources
Recipe: @recipe_files::/Users/doug/git/chef/chef-cookbooks/slice-java/recipes/docker.rb
  * machine_image[ssh_server] action create[2017-01-13T07:15:48-08:00] INFO: Processing machine_image[ssh_server] action create (@recipe_files::/Users/doug/git/chef/chef-cookbooks/slice-java/recipes/docker.rb line 3)
[2017-01-13T07:15:48-08:00] INFO: HTTP Request Returned 404 Not Found: Object not found: chefzero://localhost:9000/data/machine_image/ssh_server
[2017-01-13T07:15:48-08:00] INFO: HTTP Request Returned 404 Not Found: Object not found: chefzero://localhost:9000/data/images/ssh_server
[2017-01-13T07:15:48-08:00] INFO: Processing chef_node[ssh_server] action create (basic_chef_client::block line 57)

    - update node ssh_server at chefzero://localhost:9000
    -   update normal.chef_provisioning.reference.allocated_at from "2017-01-13 15:07:02 UTC" to "2017-01-13 15:15:48 UTC"[2017-01-13T07:15:48-08:00] INFO: Processing chef_node[ssh_server] action create (basic_chef_client::block line 57)
[2017-01-13T07:15:48-08:00] INFO: Processing chef_client[ssh_server] action create (basic_chef_client::block line 146)
[2017-01-13T07:15:48-08:00] INFO: Processing chef_node[ssh_server] action create (basic_chef_client::block line 157)
[2017-01-13T07:15:48-08:00] ERROR: Proxy thread unable to start: Can't assign requested address - bind(2) for "<IP_ADDRESS>" port 9000

doug@ip-192-168-5-113 slice-java (dev) (master)*$ chef -v
Chef Development Kit Version: 0.19.6
chef-client version: 12.15.19
delivery version: master (802e801d920ea6b6d48db735aa7c6e7a6194bea4)
berks version: 5.1.0
kitchen version: 1.13.2

It looks like as part of our security cleanup of chef-zero we left behind the options for the functionality which we removed. The current setup of chef-zero is to start server and bind to it socketless. This way there isn't a chef-zero server running on your host, bound to a port which an outside user could access. We still need to circle back and remove these deprecated options.

The workaround for starting a server on a particular IP and port and then connecting to that is to start the chef-zero binary with the --host and --port option and then connect a chef-client to that by specifying the chef_server_url option.

Closing this out since we have a workaround and haven't heard enough interest in this one to dig further in fixing it.


Roll Flutter Engine from 6670f92130d9 to 8cf55ccef913 (13 revisions)

https://github.com/flutter/engine/compare/6670f92130d9...8cf55ccef913
2022-06-13<EMAIL_ADDRESS>Remove use of deprecated, unused dart io properties. (flutter/engine#34010)
2022-06-13<EMAIL_ADDRESS>Start uploading artifacts to GCS. (flutter/engine#33970)
2022-06-13<EMAIL_ADDRESS>Disable compressed pointers on iOS. (flutter/engine#34001)
2022-06-13<EMAIL_ADDRESS>All layers with raster cache should use integer CTM (flutter/engine#33981)
2022-06-13<EMAIL_ADDRESS>Roll Skia from d3d09d660c9f to 55f9ee1e3562 (1 revision) (flutter/engine#33998)
2022-06-13<EMAIL_ADDRESS>[web] Migrate Flutter Web DOM usage to JS static interop - 28. (flutter/engine#33356)
2022-06-13<EMAIL_ADDRESS>[fuchsia] Remove to be deprecated pixel_scale field (flutter/engine#33932)
2022-06-13<EMAIL_ADDRESS>Roll Skia from eeb50b00494a to d3d09d660c9f (1 revision) (flutter/engine#33997)
2022-06-13<EMAIL_ADDRESS>Roll Dart SDK from ec56a51b9cdd to 410431ef7575 (1 revision) (flutter/engine#33996)
2022-06-13<EMAIL_ADDRESS>Roll Fuchsia Mac SDK from 2alXmAhdn... to GjdmK8tOM... (flutter/engine#33995)
2022-06-13<EMAIL_ADDRESS>Roll Skia from a913ef51bb2b to eeb50b00494a (1 revision) (flutter/engine#33994)
2022-06-13<EMAIL_ADDRESS>[android] Fixes 'drawRenderNode called on a context with no surface' crash (flutter/engine#33655)
2022-06-13<EMAIL_ADDRESS>Roll Dart SDK from ab8eafda59f7 to ec56a51b9cdd (1 revision) (flutter/engine#33991)
Also rolling transitive DEPS:
fuchsia/sdk/core/mac-amd64 from 2alXmAhdnH0r to GjdmK8tOMq93
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose
To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

Commit queue failed; closing this roll.


Fixes for localiztion

Fixed #127
Fixed #128
This should allow for proper localization of these items. @l0g0sys can you double check to make sure this will work?

Yep it works. If this Tag could be used with ComboBoxItem as well, it would also solve next issue with combobox'es like Class and Filter.

I solved that by using the name, like every other place in code. Content
is for display only and name should be the value associated with the combo
On Jun 19, 2015 8:36 PM, "l0g0sys"<EMAIL_ADDRESS>wrote:

Yep it works. If this Tag could be used with ComboBoxItem as well, it
would also solve next issue with combobox'es like Class and Filter.
—
Reply to this email directly or view it on GitHub
https://github.com/EmmittJ/PoESkillTree/pull/140#issuecomment-113702349.


Add DUPLICATE status count to the list list page and as a filter option on the list detail page

NOTE The duplicate count was added to the serializer in #2044. If that PR has not yet been merged, start your work from the associated branch.
Duplicate filtering was added in #2032
The PR for this feature should be connected to #2021 as it is continued work

 on the same feature

@jwalgran, @obrienad, the list status page is still not displaying this duplicate column - is this expected at this point?

OK, thanks, they weren't showing up on Friday (when the above screen shot is from) but I see the updated columns now.


error checking for backend miner availability (404)

nothing happens when i try to right click on my gpu

error checking for backend miner availability (404)

Are you able to get into the program I am, but once I do I get this error and it seems it also can't find my GPU for mining not sure if the two are related, but if you have the same issue it could be

I solved the error in this way: also my system is 64 bit I downloaded the BFG Miner 32 bit version and seems to work. Now multiminer recognizes my GPU (old gtx560ti !) More details here https://bit.ly/3ef7T1P


Fix variable name in example

the variable is called isValid, not validJson 🤓

Great catch. Thanks!


MAINT: Fixes for Python 3.13

Working toward #86

@jan-janssen you can see the error can be replicated here:
___________________________ test_state_with_pathlib ____________________________
h5io/tests/test_io.py:415: in test_state_with_pathlib
    path_obj_reload = read_hdf5(
h5io/_h5io.py:502: in read_hdf5
    return _read(fid)
h5io/_h5io.py:496: in _read
    return _triage_read(fid[title], slash=slash)
h5io/_h5io.py:563: in _triage_read
    return _setstate(
h5io/_h5io.py:888: in _setstate
    setattr(obj, k, v)
E   AttributeError: property '_raw_path' of 'PosixPath' object has no setter

this is with the setstate / getstate stuff, can you take a look?


Enable pagination of games

A common request is that, instead of using the feed for games, we should have a paged view.
I'm aware that the intent of the feed is to be more mobile-friendly, so we might want to add a dedicated games list for the event that can be paginated. If you have any other ideas please add them here.

I want to repeat my point there as I did in LD feed:
In my opinion(and I don't think it is only mine) we badly need proper paged game list as it was on the old ludumdare.com. There are many points to it:

The feed is uncomfortable to use. Every single human I've talked about it found it uncomfortable and counter-intuitive.
It makes extremely hard to look up for games at the middle of the list. Spending half a hour clicking "Load More" and scroll through entries that you already saw and want just skip over? I don't find it funny.
With the lack of search functionality the current implementation makes it so that many entries will never be seen at all. There are plenty of creators who does not care about socialization and just make game and upload it. And some of them create pretty interesting and original games that is seen only by several people at the beginning and after that linger forever at the middle of the list, not showing with any sort settings due to low karma and the fact that most people doesn't go futher than dozen pages. I for one liked very much to search such hidden gems in the middle of the list. Now I can't do it, sob.
Please correct me if I wrong but I don't quite understand the point of making list mobile-friendly. Especially by the means of making it PC-unfriendly. The vast majority of the games on LD are downlodable PC builds. The main target platform of jammers is still PC. While nobody restricts making mobile or web games, this is not a mobile jam and mobile platforms are minority. So the main use of the game list at the current moment is: Browsing entries on your PC, downloading builds, and play them still on your PC. I really doubd that someone browse entries on mobile device just to find and dl them later on PC(since it is really bothersome and sometimes impossible task to find some specific game in the game feed). So tell me please, why are we hindering the actual jam participants and make it harder for them to browse and rate games? Is it only to make the third parties life easier?
I actually tried to load and scroll though game feed on mobile device(iPad). Surprisingly it was even worse than on PC browser. Again, old list on ludumdare.com is way more convinient to use even on the mobile device. So I'm at a little loss now - why the hell do we need that game feed at all???

There was a pagination effort at some point (Possibly by @Noojuno?). Not sure what happened to that though.

Yeah, I started with pagination, but then we decided to actually properly plan out the feed endpoint before I did it how I wanted to :P It's currently number 3 on my //TODO: list, so pretty soon I'll be taking a look at it.

I feel like hacking it in to the current feed (probably add a new query term for "get count") is a good idea for LD40, it won't be hard to refactor when we rework the feeds. I can do the backend part.

@MeiHouwang I'm curious - What is your ping to the site? (ldjam.com or api.ldjam.com), I expect it's pretty significant, I'd like to simulate the additional ping to see what we can do to improve performance in the short term.


Remove stopwords from the generated word clouds

Missing functionality
Word clouds contain the most common words, and for free text fields, these words are often: 'and', 'to', 'the', 'from' etc. Which provide no meaningful insight into the data.
Proposed feature
Include an option to remove stopwords when generating word clouds. Perhaps by incorporating the nltk package to identify the list of stopwords.
Example:
from nltk.corpus import stopwords

stop = stopwords.words('english')

Alternatives considered
Removing stopwords from a pandas dataframe prior to generating the report wouldn't suffice in this case, as it would affect the 'samples' in the report
Additional context
No response

Hi @maxdavidson91 ,
thank you for the suggestion. Let me know If you would be interested in contributing to Ydata-profiling, by developing this feature request!

I'd be happy to contribute


[Browse] update name search box to use typeahead

see https://github.com/clearlydefined/service/issues/429 for some context
We want to put in exact matches to the search box for names and typeahead should be able to help us with this.
this is rough idea of what I'm thinking

Call https://api.clearlydefined.io/definitions/?pattern=jqu like we do from /definitions
parse each coordinate response into EntityCoordinates
make a distinct list of coordinates.name
let the user choose one of the names
update the query with ?name=

This isn't ideal, but may be a good stopgap

I see. I've tested the behavior on the UI and i didn't notice before that the browse list returns some data only while the name is matching the name of a definition.
Your idea seems to solve the problem.
I will implement this.


Updatable help for PowerShellGet 3.0.0-x

Expected behavior
Updatable help for PowerShellGet should now include the new functionality for:

Find-PSResource
Get-PSResourceRepository
Get-PSResource
Install-PSResource
Register-PSResourceRepository
Save-PSResource
Set-PSResourceRepository
Publish-PSResource
Uninstall-PSResource
Unregister-PSResourceRepository
Update-PSResource

Actual behavior
Currently includes updatable help for PowerShellGet 2.x.x functions.
Environment data
This applies to all versions of PowerShell.

/cc @sdwheeler

@sdwheeler Can we close the issue?

@sdwheeler Can we close the issue?

@iSazonov @SydneyhSmith @alerickson
My PR to create the MD stub files was merged but the content is incomplete. That should probably be a new issue, but should that be here or in the PowerShellGet repository?

@iSazonov @SydneyhSmith @alerickson
My PR to create the MD stub files was merged but the content is incomplete. That should probably be a new issue, but should that be here or in the PowerShellGet repository?

If we should fix nothing in the PowerShell repo it is better to track the work in Doc repo.

If we should fix nothing in the PowerShell repo it is better to track the work in Doc repo.


Enforce a specific pin strength

One could use a simple equation like $E = log_2(R^L)$ to enforce strong pins (also known as passwords. It's not a password-less protocol after all)
https://www.pleacher.com/mp/mlessons/algebra/entropy.html

This is missing some context: For starters,

what character encodings do you assume or support (assume character means grapheme cluster here)

if normalization and removal of empty space symbols etc must be handled by user or not, see https://github.com/matu3ba/chepa#utf8-whitespace-characters

pattern influence (leet speech, adding extra symbols in between words) etc

Also, please use the simpler formula E = L × log2(R). See https://www.omnicalculator.com/other/password-entropy.
Unless you use LUTs or SIMD, performance will suffer for utf8 and I believe this should not be part of "core" and rather an "extension" due to the size or portability shortcomings.

Thanks for the comment 👍 I was just brainstorming when I wrote this down. Will look into it. You're right that this should be an extension (of the library).

The CTAP2.1 spec states for setting a new pin and changing a existing pin that: "An authenticator MAY impose arbitrary, additional constraints on PINs. If newPin fails to satisfy such additional constraints, the authenticator returns CTAP2_ERR_PIN_POLICY_VIOLATION."

newPin is encoded in UTF-8
newPin must fit into 63 bytes
the impl currently expects only ASCII chars which is actually a BUG. The min pin length is 4 utf-8 characters and not 4  bytes

I'll add further details soon...

I've added an optional callback that is called when setting/ changing a PIN. This should allow users to add additional constraints.
I see three options:

We add an extensions folder (library extension; NOT CTAP extension!) somewhere to lib and implement a function that calculates E = L × log2(R)
We do the same but add it to the Linux reference impl (work in progress) in platform-auth
Just offer the opportunity to set a callback

I'll close this for now


oem-ibm: Add new file io completion code

The IBM hypervisor sends 0x8B meaning 'full file discarded' when it doesn't have room to store a file.  In this case specifically it's for when they are receiving event logs from the BMC.
Change-Id: I21cf1cd4c7034237d47d3855397627712bb2fe17

This had some unrelated clang-format issues that failed CI.  I pushed up a new commit to fix them.


Possible performance improvement

Store OutputHandler instances insteadof classes, removes need to create instances on the fly.
For more info, see TODO item in HttpHandler

implemented, does not seem to have a big impact.


Updating contributing/install docs

CATEGORY
Choose one

[ ] Bug Fix
[x] Enhancement (new features, refinement)
[ ] Refactor
[ ] Add tests
[ ] Build / Development Environment
[ ] Documentation

SUMMARY

I've updated the docs for the 0.31 release for developer setup and installation. These directions should work for people on late model Ubuntu/OS X systems.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

Manual verification
ADDITIONAL INFORMATION

[ ] Has associated issue: #7373
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

REVIEWERS
@mistercrunch
Related Issues
Fixes #7386
Fixes #7399

Codecov Report

Merging #7386 into release--0.31 will not change coverage.
The diff coverage is n/a.

@@              Coverage Diff               @@
##           release--0.31    #7386   +/-   ##
==============================================
  Coverage          55.85%   55.85%
==============================================
  Files                531      531
  Lines              23721    23721
  Branches            2792     2792
==============================================
  Hits               13249    13249
  Misses             10061    10061
  Partials             411      411

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 700cbf4...dc928fa. Read the comment docs.

Fixed feedback to PR in https://github.com/apache/incubator-superset/pull/7386/commits/f3cfdcb34e256d644d0d9c17a8dcc6cdfb7a8ad8#diff-6a3371457528722a734f3c51d9238c13L300

Let's target master

Closed in favor of master pull https://github.com/apache/incubator-superset/pull/7426/files


Feature request: scan for changes in group folders

Hi. Is that possible to config the script to also scan for changes in group folders (created with Group folders plugin)?
They are located in [path-to-ncdata]/__groupfolders/[1, 2, ...]. Tried "external_storage = yes" option, with no luck.
But if I then change something in user's folder, all changes in group folders are detected for some reason.
If this is not possible at the moment, it would be great if you add this feature.
By the way, thanks a lot for this script, it works great, now Nextcloud is really convenient to use.

OK this is the second time I receive this feature request (first is #11) so I probably should consider making group folders work :) Will look into it when I have time. Thanks for the feedback!


webgl  Rendering efficiency lower than canvas

hello:
When i set rendering method is webGl,the Rendering efficiency is low,if set rendering method is canvas the Rendering efficiency higher than webGl,How to improve efficiency of webgl?
Thank!

And there may be some mistakes in your code ,  I think .
So , show me your code , please.
P.S.: Are you come from China ,  Mr. 冯方勇 ?

Usually if this is true, you are either:

Using extremely large textures, or
Using software rendering for WebGL which is super slow.

You can check for the later by looking at chrome://gpu/

@finscn
HI
code:

js test

<script src="https://cdnjs.cloudflare.com/ajax/libs/pixi.js/4.0.0/pixi.min.js"></script>

Fourth parameters of autoDetectRenderer is false Rendering with webgl,is true rendering with canvas,Webgl render time longer,if the number of graphics or sprite is increase,the Render time longer.
Yes,i am from china ,Thank!

Are you using software rendering for WebGL?

@englercj
Wow should i do?How to rendering for webGl

You can check if you are using software rendering by looking at chrome://gpu/


Migrator touchup and testing

The intent here is to ensure that any breaking changes to the v1 API will require a migrator to be added in order for CI to pass.

[x] add more examples to either the v1 README or a separate file to cover the v1 API (pending #440 merge)
[x] add migrators for remaining v1 ➡ v2 changes (pending #444)
[x] reimplement script in go with proper gofix-like AST awareness (save for later)

This is great, I was just thinking about how to best ease the migration path and this is exactly it.

:raised_hands: :confetti_ball:

:tada: :bomb:
On Tue, Jun 7, 2016, 00:19 Jesse Szwedko<EMAIL_ADDRESS>wrote:

Merged #439 https://github.com/urfave/cli/pull/439.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/urfave/cli/pull/439#event-683851207, or mute the
thread
https://github.com/notifications/unsubscribe/AACwVxS3eNrHfFQwuwH8-MZkjNYJo6_3ks5qJPFfgaJpZM4Ip9YP
.


Add v1 APIs

Issue: https://github.com/stolostron/backlog/issues/19241
Adds a v1 version of discoveryconfig and discoveredcluster and serves both
v1alpha1 and v1, but uses v1 as the storage version.
Because it is only a version change and no schema change a conversion webhook
is not necessary. By default Kubernetes will add a conversion strategy of None.
Signed-off-by: Jakob Gray<EMAIL_ADDRESS>

/lgtm


Update clap dependency from v3 to v4.1

The current clap version could make it difficult to improve the CLI's DevX with the newest CLI practices.
The update by itself already makes some improvements, but I'm looking forward to using this new version to add new capabilities.
Here's the full changelog: https://github.com/clap-rs/clap/blob/master/CHANGELOG.md#400---2022-09-28
Leaving this as draft for now while I test this further, but it seems to be working fine.
Also decided to depend on #1052 to test these crate updates together rather than after merge.
Signed-off-by: Matheus Cardoso<EMAIL_ADDRESS>

Tests are failing in CI but passing locally, so I'm opening this up for review while I figure out if it's related to any changes here.

Closing this since it's fallen behind. Will open a new one soon.

Thanks @cardoso and apologies -- this has definitely been a busy period for the Spin project and particularly the CLI.  Happy to help in any way I can.


description of probe type

example: userParam name="ProbeHead" value="cid"/
The VAgilant parameter "probe" is free text, thus not more than a nickname. More (and more precise) information is provided by the parameters probetype (e.g. "liquids") and probestyle (e.g. "cold" for a cold probe or "one" for a OneNMR probe).

What is the status in the @DSchober ?


🚀 Feature: Add more properties to Span Attributes for OpenAI instrumentation

Which component is this feature for?
OpenAI Instrumentation
🔖 Feature description
The OpenAI instrumentation is great, but is missing some span attributes compared with the httpx instrumentor that would be useful to capture.  I'd like to know what the client attributes were for the OpenAI client like the host and the deployment name. Those are in SpanAttributes.HTTP_METHOD and SpanAttributes.HTTP_URL.
🎤 Why is this feature needed ?
I'm using different OpenAI backends (Azure OpenAI) and want to capture the deployment name, the host name and other information that is in the client.
✌ How do you aim to achieve this?
Add some additional span attributes to the trace that capture the properties of the open ai client object.
🔄 Additional Information
No response
👀 Have you spent some time to check if this feature request has been raised before?

[X] I checked and didn't find similar issue

Are you willing to submit PR?
Yes I am willing to submit a PR!

Thanks @tonybaloney for flagging this. It totally makes sense and we should add these attributes.

Actually, looking at the source code, you've mostly got the implementation but it would only work with the old OpenAI SDK  (pre v1)
_set_api_attributes These attributes aren't used in the new SDK. I'm happy to raise a PR to fix this. Have all the pieces running already to test it

Sounds good @tonybaloney
Yeah I had some vague memory we implemented it once but might be for the old OpenAI APIs indeed.

See #522


Updated to support 1.3.0.RC3 ref tests

Documentation

[X] I thought about documentation and added the doc-change-required label to this PR if updates are required.

Changelog

[X] I thought about adding a changelog entry, and added one if I deemed necessary.

Execution payload things are fixed in my PR https://github.com/ConsenSys/teku/pull/6840
I'm not sure what is a best way to commit all

Execution payload things are fixed in my PR #6840 I'm not sure what is a best way to commit all. Both are broken being alone

I'll make ours compliant then leave it at rc2, I tried to cherry-pick yours and it didnt look clean, so will let you deal with whatever is going on there...


add kernel version to telemetry traces.

heartbeat already comes with kernel version attribute; https://github.com/microsoft/retina/blob/5ff1567e8a988209494b0cddd34f3939ac51e53c/pkg/telemetry/telemetry.go#L170
We would like to add it to traces as well in order to directly pinpoint what kernel version is producing a specific log.

Hi @jimassa, I would like to contribute to this

Hello @jimassa I have made the necessary changes and committed a PR. Kindly review it. Thank you.

isn't it possible to correlate pod name with kernel version in heartbeat?

Thank you for the suggestion! I will add the pod name to the heartbeat properties to enable correlation with the kernel version. I'll update the PR shortly. Please let me know if there are any other changes/features to be implemented.

podname should already be a part of telemetry properties, so it should be all good
https://github.com/microsoft/retina/blob/8963dd561d4d5b52c1f711dcbc13fe9f616bd360/pkg/telemetry/telemetry.go#L155

Yes, In the TrackEvent method, t.properties is merged with the provided event properties, so podname will automatically be included in the heartbeat telemetry.


AssertionError: AbstractConv shape mismatch: shape of filters does not match given kshp.

AssertionError: AbstractConv shape mismatch: shape of filters does not match given kshp.
Apply node that caused the error: Assert{msg='AbstractConv shape mismatch: shape of filters does not match given kshp.'}(InplaceDimShuffle{3,2,0,1}.0, Elemwise{eq,no_inplace}.0, Elemwise{eq,no_inplace}.0, Elemwise{eq,no_inplace}.0, Elemwise{eq,no_inplace}.0)
Toposort index: 2069
Inputs types: [TensorType(float32, 4D), TensorType(bool, scalar), TensorType(bool, scalar), TensorType(bool, scalar), TensorType(bool, scalar)]
Inputs shapes: [(3L, 64L, 7L, 7L), (), (), (), ()]
Inputs strides: [(196L, 588L, 4L, 28L), (), (), (), ()]
Inputs values: ['not shown', array(False, dtype=bool), array(False, dtype=bool), array(True, dtype=bool), array(True, dtype=bool)]
Outputs clients: [[Subtensor{::, ::, ::int64, ::int64}(Assert{msg='AbstractConv shape mismatch: shape of filters does not match given kshp.'}.0, Constant{-1}, Constant{-1})]]

Make sure the inputs to the model is of dimension (batch_size, 224, 224, 3) for tensorflow and (batch_size, 3, 224, 224) for theano backend

make sure that reduction=0.5

Make sure the inputs to the model is of dimension (batch_size, 224, 224, 3) for tensorflow and (batch_size, 3, 224, 224) for theano backend

hi，i try to run the test code,but i meet a error. the error as follow:
TypeError: 'module' object is not callable
can you help me to resolve it？ thanks.


isPasscodeSet returning false occasionally

We have had an issue in production where isPasscodeSet is returning false and prompting the user to create a passcode even though they have created a passcode before, it seems to popup semi regularly for some users, and never for others. We are unsure what is causing this behavior, but it seems to be due to inactivity at times. Like waiting a few days, then opening the app and bam.

was my mistake, sorry!


Add Mr-LLLLL/lualine-ext.nvim

Repo URL:
https://github.com/Mr-LLLLL/lualine-ext.nvim
Checklist:

[x] The plugin is specifically built for Neovim, or if it's a colorscheme, it supports treesitter syntax.
[x] The lines end with a .. This is to conform to awesome-list linting and requirements.
[x] The title of the pull request is Add/Update/Remove `username/repo`  (notice the backticks around `username/repo`) when adding a new plugin.
[x] The description doesn't mention that it's a Neovim plugin, it's obvious from the rest of the document. No mentions of the word plugin unless it's related to something else.
[x] The description doesn't contain emojis.
[x] Neovim is spelled as Neovim (not nvim, NeoVim or neovim), Vim is spelled as Vim (capitalized), Lua is spelled as Lua (capitalized), Tree-sitter is spelled as Tree-sitter.
[x] Acronyms should be fully capitalized, for example LSP, TS, YAML, etc.

Thanks for the PR!


Settings flash when checking/unchecking an option

Pretty sure this belongs here and not in settings-view.  Atom 0.184.0 safe mode on Ubuntu 14.10.

Looks like it might be due to the menu bar changing state? On OS X, I don't have that setting and other settings don't repro the issue you're showing.

It happens with any checkbox (and maybe even strings but I can't reproduce that right now).

I'm not seeing it with 0.184.0 in Win7 with a default (dark) UI theme.

Hmm, I can't reproduce this in Ubuntu 14.04.

Still happening for me on 0.186.0 :(.  I wonder if it's something to do with my graphics card then?

What is your UI theme? Maybe the theme is bugged?

One Light.
...and I can't reproduce it anymore.  Hopefully it won't come back.


switch from npm to yarn

This will allow bringing node 8 back into the automated build.

May I take this and #29?

Have at it!


Makes the probes valid for Gitlab 10+

The Problem
The latest version of the Gitlab Runner has the main process renamed from gitlab-ci-multi to gitlab-runner.
Our probe was always killing our pod after a while because of this.

Hi,
Thank you for update


[bug]: file name show on windows fix (#73)

Fixes #73
What was changed?
showing file name instead of file path for windows.
Here, describe what part of the application you changed (e.g. login page, database, etc.). If possible, mention what specific components were added, removed, or modified.
client-app/src/components/DraftOrcaDashboard.js modified.
Here, describe the issue that you are fixing with this code, and why your code fixes it.
Input box was showing file path in windows system instead of file name, so I changed the input value, which will now show the file name.

@tanishq-singh-2407 - Great effort! However, I am getting the following error when running the code. Can you please look into it?

Closing this PR as this is fixed as part of #94


Does anyone have the old default prompt or improved one I forgot to save it before I changed

Just checking to see if anyone has the old prompt template I attempted to experiment with a few things and forgot to save it want to ensure i get the best experience out of this tool.

The default prompt is in openai.ts.  It's currently:

You're a frontend web developer that specializes in tailwindcss.
Given a description or an image, generate HTML with tailwindcss. You should support
both dark and light mode. It should render nicely on desktop, tablet, and mobile.
Keep your responses concise and just return HTML that would appear in the
no need for  or . Use placehold.co for placeholder images. If the user asks for
interactivity, use modern ES6 javascript and native browser apis to handle events.
Do not generate SVG's, instead use an image tag with an alt attribute of the same
descriptive name, i.e.:
<img aria-hidden="true" alt="check" src="/icons/check.svg" />


Automate Labeling of Issues Based on Comments from Non-Core Members

Is your feature request related to a problem? Please describe.
Currently, there is no way to track issues with unresolved comments. Contributors, interns, or anyone other than core team members may have shared a comment that requires action from the core team, but it might be missed, causing the issue to stall.
Describe the solution you'd like
To identify such cases, a label can be automatically added to the issue whenever someone other than a core team member comments.
Feature Requirements:

Automatically add a "question" label to an issue whenever a non-core team member comments.
Once a core team member replies or comments, the "question" label should be automatically removed.

@aravindm4 lets just add the tag, lets not make the removal automatic.
name: Notify Core Team on Non-Core Questions

on:
  issue_comment:
    types: [created]

jobs:
  notify_core_team:
    runs-on: ubuntu-latest
    steps:
      - name: Check if commenter is non-core member
        uses: actions/github-script@v6
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          script: |
            const coreMembers = ['coreMember1', 'coreMember2']; // Define core members
            const commenter = context.payload.comment.user.login;
            if (!coreMembers.includes(commenter)) {
              return commenter; // Proceed if not core member
            }

      - name: Check if comment is a question
        id: check_question
        run: |
          echo "${{ github.event.comment.body }}" | grep -Eq "(\?|what|how|why|when|who)"

      - name: Notify core team
        if: ${{ steps.check_question.outcome == 'success' }}
        run: |
          curl -X POST -H 'Content-type: application/json' --data '{"text":"A non-core member posted a question: https://github.com/${{ github.repository }}/issues/${{ github.event.issue.number }}"}' ${{ secrets.SLACK_WEBHOOK }}

      - name: Add question label (optional)
        if: ${{ steps.check_question.outcome == 'success' }}
        uses: actions/github-script@v6
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          script: |
            await github.issues.addLabels({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.payload.issue.number,
              labels: ['question'],
            });

Something like this with options for questions and core configurable via actions env


Fix href attribute for sequential link elements

I use a browser extension (Tridactyl) that makes it easy to navigate backwards and forwards using the URLs specified by <link> elements. This commit add the .html suffix for href attributes of external resource links when the relationship indicates sequential access, i.e., “prev”, “next”.

Good catch, also what's the Tridactyl keybinding for this, please advise.

Good catch, also what's the Tridactyl keybinding for this, please advise.

It's [[ and ]] to go backwards and forwards, respectively.
I vaguely recall keyboard short-cuts for the same feature being available for browsers in the late 90s / early 2000s. I’m thinking of Netscape but it could have been Konqeror or Opera.
I was interested to see how you used make with Pandoc to generate the HTML so I was happy to be able to help make it better.


Allow for a configurable package ensure parameter

I like this change (I really should have done it this way to start), but it seems slightly redundant to have both a package_ensure parameter and a n install_package parameter.
I think that I'm going to mark install_package as deprecated for the next release in favor of this (I won't remove it until later).


BadRequest since yesterday

There are some change about authtoken process?

May be, not remind it's.


Googletest - Xcode integration improvements

I added case to support tests that are skipped in google test so Xcode does not show them as fails.
I also removed use of deprecated unarchiveObjectWithFile method, however this needs to be tested.
scripts/check.sh did not show any problems, but I do not know if edited code is being tested anywhere and I was not able to run any google tests which would use the edited code flow (in Xcode GUI, I specified unit tests to run, disabled selected tests, and so on, but each call of the edited method failed because of empty 'filePath' in both code versions).

@TheKuko Thanks for the PR. It looks like there is a problem with the macos build. PTAL
Also, the travis Xcode 10 build doesn't find XCSkip, but we're going to turn those tests off soon since we know longer support Xcode 10.


Implement eth_syncing

Implement the eth_syncing API. If it exists, add the eth_subscrbe version of this too.
Discuss in slack how to determine whether a node is syncing or not. I can think of a few options with different tradeoffs. Some research needed.

@chetan-zilliqa If its okay, I'll change this ticket to specify implement eth_syncing which should give us all this information. I believe there is also a websocket version which a command line client could use to get live updates on sync status.

@JamesHinshelwood  - naively:

we are syncing if highest_known_view > current_view
startingBlock - is the block number at the highest confirmed view when we last started syncing.
currentBlock is the block number at the highest confirmed view
highestBlock - is currentBlock + (highest_known_view - currentBlock.view)

Does that seem sane? If you'd prefer something else, let me know...

@rrw-zilliqa Yes, that sounds sane to me. I think its unavoidable that a malicious node could cause us to get confused about highest_known_view - We can't know the committee in which a supermajority would be required in order to trust a block, without knowing that block's parent, which obvious requires syncing :)


Write guides to setup the system and how to set the env up

[x] For developers
[x] Move developer guide to doc folder
[x] For admins with screenshots, ...

The README of V2 contains guides for developers and a "getting started" section for production mode.


feat: switch from rows to columns max-width: 768px

closes #19

@marilari88 thanks for the approval, I don't have the right permissions and I don't see any merge button, so you have always to merge these for me!

Thanks! We should fix the impossibility to scroll the value section in case of values > 2. But we can afford in a new PR later

Thanks! We should fix the impossibility to scroll the value section in case of values > 2. But we can afford in a new PR later

Yes I've noticed it too


index2.htl will not work.

Hi!
I have downloaded the index2.html file from your github.  But when  I run it  I get a lot of errors in the console.  Any suggestions?
I got the followinfg error (in console):    Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'setAttribute')
at PyScript.evaluate (base.ts:128:79)
Error in py-script:
JsException(PythonError: Traceback (most recent call last): File "/lib/python3.10/asyncio/futures.py", line 201, in result raise self._exception File "/lib/python3.10/asyncio/tasks.py", line 232, in __step result = coro.send(None) File "/lib/python3.10/site-packages/_pyodide/_base.py", line 500, in eval_code_async await CodeRunner( File "/lib/python3.10/site-packages/_pyodide/_base.py", line 237, in init self.ast = next(self._gen) File "/lib/python3.10/site-packages/_pyodide/_base.py", line 141, in _parse_and_compile_gen mod = compile(source, filename, mode, flags | ast.PyCF_ONLY_AST) File "", line 14 if file_input.value is not None: ^^ IndentationError: expected an indented block after function definition on line 13 )

I got it weork. Was some problems with intendation.


Add Goldpinger as a feature

Is your feature request related to a problem? Please describe.
Goldpinger  was investigated as an feature to add to Apps here
We concluded in this arch topic that we will add it as a feature to Apps.
Describe the solution you'd like
Please add Goldpinger as a feature in Apps.
Additional context
Goldpinger is  installed on the long-running cluster.
Please check how many series is generated by this application in a normal sized cluster so we know if we need to resize the cluster resources.
Definition of done:
Goldpinger is added as a feature to Apps

##Decision:

We enable this in ck8s and create a roadmap item to implement this.

Action Items

[ ] create a roadmap item for the implementations @lucianvlad


Multilanguage

Description
Hey !
It could be nice to be able to choose your language in the configuration.
I'm sure many people would like to participate in the translation.
Priority
Medium (Would be very useful)

That's actually a good idea ! I have not yet used internalization in React, but I'll take a look at it 😉

I agree too! I can help with Spanish.

Me for the french language, if needed!

I can help with Italian. Any news about?

I could do german, any update would be very much appriciated :)

Thank you so much everyone for your enthusiasm. The most important right now is someone that can extract all the strings from the code and extract them into a JSON with an understandable name that would correspond to the English string (That would be the base English translation, it just takes a lot of time and isn't my priority right now)

Well, I could take a look at it maybe.
By the other hand, a good platform to manage translations for a project, is Crowdin.

Well, I could take a look at it maybe.
By the other hand, a good platform to manage translations for a project, is Crowdin.

Yeah I've seen it before. Only problem is that we don't have a base configuration at the moment

Actually afer using i18n in a personal project I can say that this is very annoying to do. I might include it later with only english translation and use some external tool so that everyone can contribute

I have been working on this with #357 (it was a pain). There are still quite a few bugs that need to be fixed before release.
Please don't create any commits with translation files yet. We'll add a tool later for making translations online.
I won't contribute for a while, because I have personal priorities. Thus, @ajnart will take over here.

Hi again,
we've successfully implemented translations 🚀. You can read our blog post to see how you can contribute your language - no programming skills required.
https://homarr.vercel.app/blog/translations

I'll close this issue now, since translations are up and running.


HTML to Docx Converter

Hi
I wonder if you could help in converting HTML to DOCX in android ?
thanks

I don't understand this issue...  this module is a plugin for the Apache Isis framework, which doesn't run on Android, so I don't see how it relates?


fix device registration mistakes

When registering your device, you should use the device's MAC address, not the WiFi destination.

Hello,
Thank you for your comment, I thought the hardware MAC was some internal feature and I preferred to stick to wifi MAC since it should be more recognizable.
Do you think the wifi mac is not 'stable' enough to let it work as an hardware identifier in the device registry?

Hello @krahabb,
I'm on hass core-2021.3.4, but at least in this environment, devices with the same wifi mac will be recognized as one device.

I've encountered the issue too: devices get coalesced into a single instance because the wifi MAC relates to the router MAC and not the plug wifi interface. I didn't really noticed while testing the first version..I was probably messing up too much with reconfigurations.

@krahabb Thank you for merging! 👍


Add percentage(default) or raw reading option to currentLoad function

I've added idle field for the currentLoad() and options object that allows the user to specify the return value (raw or percentage[default]).
eg.
si.currentLoad({
  raw: true
})
  .then(function(data) {
    console.log(data); // returns raw reading
  });

I really like these changes, in some situations it's really helpful to get the raw idle & load values. @sebhildebrandt what do you think?

Hi, I definitely like these changes! The only reason I did not merged the pull request is, that it changes the outside interface and then would be a breaking change. I was a little bit busy, so I did not worked on a solution for that. But I am now working on a solution that will also provide raw data without changing the interface. Will publish it on Monday.

Just published a version that includes also raw data for currentLoad.


[Request] Akkoma Bubble Support

Is your feature request related to a problem? Please describe.
I miss the bubble timeline from akkoma. It is a timeline with a set of instances choosen by the instance admin.
Describe the solution you'd like
please add support for the akkoma bubble timeline, it is well documented in akkoma docs.

Might take this up sometime soon :)


validating objects with nested array of objects

let data = {
  name: 'foo',
  items: [
    { name: 'bar', amount<EMAIL_ADDRESS>baz: true},
    { name: 'bok', amount: 60, baz: 'not a boolean'},
    { name: 'bog', amount: 60, baz: false},
  ]
}

Does validate.js handle writing a validation schema for this?
Does validate.js handle declaring errors as: { 'items.0.amount': [ 'must be a number'] } ?

This is poorly supported at this moment, sorry.
You can track the progress in #1


wayland-utils: 1.1.0 -> 1.2.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/project/wayland-utils/versions.
meta.description for wayland-utils is: Wayland utilities (wayland-info)
meta.homepage for wayland-utils is: https://gitlab.freedesktop.org/wayland/wayland-utils
Updates performed

Version update

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 1.2.0 in filename of file in /nix/store/jxbjz6bra5dr72ydq5nq3ysrv4cj6p7c-wayland-utils-1.2.0

Rebuild report (if merged into master) (click to expand)

3 total rebuild path(s)

3 package rebuild(s)

First fifty rebuilds by attrpath
libsForQt5.kinfocenter
plasma5Packages.kinfocenter
wayland-utils

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/jxbjz6bra5dr72ydq5nq3ysrv4cj6p7c-wayland-utils-1.2.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A wayland-utils https://github.com/r-ryantm/nixpkgs/archive/9729b9c202af3002c774b83d8cf8c842586e497c.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/jxbjz6bra5dr72ydq5nq3ysrv4cj6p7c-wayland-utils-1.2.0
ls -la /nix/store/jxbjz6bra5dr72ydq5nq3ysrv4cj6p7c-wayland-utils-1.2.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

2 packages built:

libsForQt5.kinfocenter
wayland-utils

Maintainer pings
cc @primeos for testing.

nixpkgs-reviewd here. Worked fine. I will submit a PR for wayland-utils.metadata.mainProgram

Announcement/changelog: https://lists.freedesktop.org/archives/wayland-devel/2023-June/042736.html

mainProgram change https://github.com/NixOS/nixpkgs/pull/237037


Refactor HTTP pipelines

UgcHttpClient.GetUgcFileAsync() now returns byte[].

Codecov Report

Merging #18 into master will decrease coverage by 1.25%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master      #18      +/-   ##
==========================================
- Coverage   87.52%   86.26%   -1.26%
==========================================
  Files          40       40
  Lines         569      517      -52
  Branches       56       53       -3
==========================================
- Hits          498      446      -52
  Misses         53       53
  Partials       18       18

Impacted Files
Coverage Δ

...z.NecroDancer.Leaderboards/ContentLengthHandler.cs
100% <100%> (ø)

...oDancer.Leaderboards/ProgressReporterHttpClient.cs
100% <100%> (ø)

...fz.NecroDancer.Leaderboards/Steam/UgcHttpClient.cs
100% <100%> (ø)
:arrow_up:

...cer.Leaderboards/Steam/WebApi/SteamWebApiClient.cs
100% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4c44cd9...4d80fab. Read the comment docs.


no remote found for dependency "kpwebb/select2"

Make fails with the following message:
fatal : no remote found for dependency "kpwebb/select2".
Looks like the former repo https://github.com/kpwebb/select2 is not available any more.

Same PB !


Add screen

Closes #104.

Codacy reports issues with the skin files, but that's not really a problem. The skin was copied as an asset so it is not the responsibility of this project to format it properly.


feat: no-restricted-vi-method rule

This PR fixes/implements the following bugs/features

[ ] Bug
[x] Feature
[ ] Breaking changes

Closing issues

Fixes #

There is a bug in parseVitestFnCall(node, context) function.

There is a bug in parseVitestFnCall(node, context) function. I am working on addressing it, but if you have time, that's the right place to look trouble shooting.

I believe I found the issue and opened a different PR with a fix. -- https://github.com/veritem/eslint-plugin-vitest/pull/55


nz-cascader does not support case insensitive search..

What problem does this feature solve?
nz-cascader having property nzShowSearch when it set to true i can search in data but its case sensitive search also containing NzShowSearchOptions but can't understand how to use this property.
can't find a solution, please help
What does the proposed API look like?
[nzShowSearch]="{case-sensitive:true}"

How do you use this? What's 'someInput'?

Why not? The documentation is ambiguous and makes no sense.
I can't understand this at all:

And I thought that this sort of configuration should be based on a predicate reference, yet your "someInput" in your comment above comes from somewhere I don't understand.
Does this mean that the Cascader component is unable to resolve option label values against the internal search input value?

I modified that demo above. Hopefully it would make more sense to you.
The reason why we wouldn't support the API is that  [nzShowSearch]="{ 'case-sensitive': true }" is not a generic solution to all kinds of customized searching. What if users wants to use fuzzy searching here? Shall we add a configure like { fuzzy: true }? IMHO, we could not cover all scenario by just adding configuration.
Hope this helps. :p

Why not? The documentation is ambiguous and makes no sense.
I can't understand this at all:

And I thought that this sort of configuration should be based on a predicate reference, yet your "someInput" in your comment above comes from somewhere I don't understand.
Does this mean that the Cascader component is unable to resolve option label values against the internal search input value?

I modified that demo above. Hopefully it would make more sense to you.
The reason why we wouldn't support the API is that [nzShowSearch]="{ 'case-sensitive': true }" is not a generic solution to all kinds of customized searching. What if users wants to use fuzzy searching here? Shall we add a configure like { fuzzy: true }? IMHO, we could not cover all scenario by just adding configuration.
Hope this helps. :p
BTW, no core team member of ng-zorro-antd is native English speaker. So we will appreciate it very much if you could please optimize our doc.

I understand, and I don't mind helping with the documentation.
I work with this library in my everyday, so I should be able to contribute in terms of documentation and language improvement for English speakers.
Thanks for updating the example above, it makes a lot more sense now.
It also helped to look at the implementation here of the default filter.


chore(master): release 1.1.239

:robot: I have created a release beep boop
1.1.239 (2023-11-10)
Miscellaneous

deps: update dependency @tailwindcss/forms to v0.5.7 (117112d)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/wayofdev/keywind/releases/tag/v1.1.239 :sunflower:


ArxivLoader incorrect results

System Info
Latest pip versions
Who can help?
@eyurtsev
Information

[X] The official example notebooks/scripts
[ ] My own modified scripts

Related Components

[ ] LLMs/Chat Models
[ ] Embedding Models
[ ] Prompts / Prompt Templates / Prompt Selectors
[ ] Output Parsers
[X] Document Loaders
[ ] Vector Stores / Retrievers
[ ] Memory
[ ] Agents / Agent Executors
[ ] Tools / Toolkits
[ ] Chains
[ ] Callbacks/Tracing
[ ] Async

Reproduction
I tried searching by exact title in the following way:
docs = ArxivLoader(query="MetaGPT: Meta Programming for Multi-Agent Collaborative Framework", load_max_docs=1).load()

But the result is incorrect. The search works properly on the arxiv website.
Expected behavior
Correct paper returned

cc @leo-gan

Sure:
docs = ArxivLoader(query="MetaGPT: Meta Programming for Multi-Agent Collaborative Framework", load_max_docs=1).load()
docs[0].metadata['Title']

The above code returns:

'Impact of International Cooperation for Sustaining Space-Science Programs'

Whereas I expect it to return:

'MetaGPT: Meta Programming for Multi-Agent Collaborative Framework'

@austinmw Thank you for reporting this issue! Much appreciated!
It happens that ":" character somehow messed with the search. It works without problem in the arxiv UI but not in the arxiv API.


Enet integration works beautifully only 1 error message (2024).

Hello Magnus,
I am very grateful to you for the effort you have put into the Enet integration for Home assistant. I am very happy with this.
I have Home Assistant running together with the Homey Pro 2023 and so I can control the whole house.
Everything works 100% I only have this error message in Home Assistant.
By now, I am also more experienced with HA, below is a picture of the error message and a piece from the debug log, hopefully you can get more information from this as to why an error message is displayed.

2024-11-17 21:16:34.778 DEBUG (MainThread) [custom_components.enet] EnetCoordinator.init()
2024-11-17 21:16:34.778 DEBUG (MainThread) [custom_components.enet.aioenet] Requesting https://my.enet-smarthome.com/nl/my-enet/jsonrpc/management userLogin
2024-11-17 21:16:34.844 WARNING (MainThread) [custom_components.enet.aioenet] Request to https://my.enet-smarthome.com/nl/my-enet/jsonrpc/management failed with status 405
2024-11-17 21:16:34.845 DEBUG (MainThread) [custom_components.enet.aioenet] Requesting https://my.enet-smarthome.com/nl/my-enet/jsonrpc/management setClientRole
2024-11-17 21:16:34.851 WARNING (MainThread) [custom_components.enet.aioenet] Request to https://my.enet-smarthome.com/nl/my-enet/jsonrpc/management failed with status 405
2024-11-17 21:16:34.851 DEBUG (MainThread) [custom_components.enet.aioenet] Requesting https://my.enet-smarthome.com/nl/my-enet/jsonrpc/visualization getLocations
2024-11-17 21:16:34.861 WARNING (MainThread) [custom_components.enet.aioenet] Request to https://my.enet-smarthome.com/nl/my-enet/jsonrpc/visualization failed with status 405
2024-11-17 21:16:34.861 ERROR (MainThread) [homeassistant.config_entries] Error setting up entry Enet Smart Home for enet
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/config_entries.py", line 635, in __async_setup_with_context
result = await component.async_setup_entry(hass, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/enet/init.py", line 38, in async_setup_entry
hub.devices = await hub.get_devices()
^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/enet/aioenet.py", line 139, in get_devices
device_locations = await self.get_device_locations()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/enet/aioenet.py", line 189, in get_device_locations
locations = await self.get_locations()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/enet/aioenet.py", line 163, in get_locations
return result["locations"]
~~~~~~^^^^^^^^^^^^^
TypeError: 'ClientResponse' object is not subscriptable
Thank you in advance for your response.
Best Regards,
Robert

It looks like you are not connecting to your local enet server. Has this ever worked?

Hello Magnus,
I am connected to the local eNet server, this has worked very well from the beginning (and still does).
Also I have the error message from the beginning that eNet is working, could it be that connection is being sought to the eNet Cloud and this connection cannot be made and is causing the error message?