OutOfMemory exception with low system memory usage

We're struggling to diagnose an issue where we are instantiating several Index objects (built on the same shared directory) in asynchronous coroutines, and setting the following stack trace:
ValueError: Failed to open file for read: 'IoError { io_error: Os { code: 12, kind: OutOfMemory, message: "Cannot allocate memory" }, filepath: "<index file path>" }

We have a relatively small index (~5Gb), and even with several instantiated Index objects, the system RAM usage is low (on a machine with 60Gb of memory, we're only seeing ~3Gb used). But when running with 4 concurrent coroutines we see the above issue.
In trying to reproduce, I made a tiny index with ~100 files, and ran 100 simultaneous coroutines, each of which with their own Index object, and was not able to reproduce the error.
Do you have any suggestions on what to investigate next?

We're running with thousands of different indexes on a shared box, but we're using process pools instead of your async handling. We don't see memory growth in these long lived processes (that access different indexes over time), but to be fair we have not yet upgrade to 0.22.
As for suggestions, this might be a good application of the FIL memory profiler: https://pythonspeed.com/fil/docs/index.html
In essence, you can run your application in a staging environment with a copy of your "real" index, do what you need to do to make memory grow, and then when memory is high, kill the process. The memory profiler will dump a file showing where allocations were held.  You get a flamegraph view:
https://pythonspeed.com/fil/docs/interpreting-output.html

It would also be good document more specifics about your situation, i.e., package version, platform, etc.

@mskarlin is this still an issue for you? Without a reproducer there isn't much we can do I think.

Hi @cjrh yes it is still an issue (I work with @mskarlin on https://github.com/Future-House/paper-qa), sorry for the delayed response.
I hit it with tantivy.Index.open on https://github.com/Future-House/paper-qa/blob/81f8c5f1395aef58d1d5317c478ce5b24949f88a/paperqa/agents/search.py#L176:
ValueError: Failed to open file for read: 'IoError { io_error: Os { code: 12, kind: OutOfMemory, message: "Cannot allocate memory" }, filepath:
"/path/to/indexes/index_name/index/abc123.fieldnorm" }'

I also seem to get it on .fast and .term. I think it's related to https://github.com/quickwit-oss/tantivy/issues/1563. Here's my system info:
> python --version
Python 3.12.7
> pip list | grep tantivy
tantivy                   0.22.0
> cat /etc/os-release | grep VERSION
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
> ulimit -a | grep virtual
virtual memory              (kbytes, -v) unlimited
> cat /proc/meminfo | grep Vm
VmallocTotal:<PHONE_NUMBER>7199 kB
VmallocUsed:      390844 kB
VmallocChunk:          0 kB

Using this answer, I run: strace -e trace=open,mmap,brk,mmap2,munmap -f python a.py 2>strace.log, and in my strace.log I see:
[pid  8370] mmap(NULL, 4443, PROT_READ, MAP_SHARED, 25, 0) = -1 ENOMEM (Cannot allocate memory)

One detail that may matter is our index was built on a macOS machine, and is being used on Ubuntu, might that affect things?

Thanks for the detail. Unfortunately, I can't think of an obvious solution for this.

One detail that may matter is our index was built on a macOS machine, and is being used on Ubuntu, might that affect things?

Are you able to run Index.open on the macOS machine to check whether it produces the same error?

Hmm, actually, I am slowly remembering something we saw in prod a while back.  We also saw this OutOfMemory error on one of our larger indexes that had a very large number of segments.   For example, each of these clusters
8b6d10f00a4a4363b6dbbbdaf034679b.term
8db1b848a5164252843cd706ae8c804a.fast
8db1b848a5164252843cd706ae8c804a.fieldnorm
8db1b848a5164252843cd706ae8c804a.idx
8db1b848a5164252843cd706ae8c804a.pos
8db1b848a5164252843cd706ae8c804a.store
8db1b848a5164252843cd706ae8c804a.term
935c493b48364e1d8cd8cb571b59a513.fast
935c493b48364e1d8cd8cb571b59a513.fieldnorm
935c493b48364e1d8cd8cb571b59a513.idx
935c493b48364e1d8cd8cb571b59a513.pos
935c493b48364e1d8cd8cb571b59a513.store
935c493b48364e1d8cd8cb571b59a513.term
9f9a01da2e7f441ab08528aaf44634f9.fast
9f9a01da2e7f441ab08528aaf44634f9.fieldnorm
9f9a01da2e7f441ab08528aaf44634f9.idx
9f9a01da2e7f441ab08528aaf44634f9.pos
9f9a01da2e7f441ab08528aaf44634f9.store
9f9a01da2e7f441ab08528aaf44634f9.term
d8968eb9e0c844deb1f0cdae38a6ee2a.fast
d8968eb9e0c844deb1f0cdae38a6ee2a.fieldnorm
d8968eb9e0c844deb1f0cdae38a6ee2a.idx
d8968eb9e0c844deb1f0cdae38a6ee2a.pos
d8968eb9e0c844deb1f0cdae38a6ee2a.store
d8968eb9e0c844deb1f0cdae38a6ee2a.term

That share a UUID name is a segment.   So what we figured was happening might be that each segment's files are separately being mmap'ed, and eventually we just run out of virtual address space causing the OutOfMemory error.
So what we did, which worked, was we recreated this tantivy index with a much larger memory limit on the writer.  Basically (handwavy explanation), the writer accumulates data in memory over multiple document additions, and when the specified memory limit is reached, it writes out that accumulated data as a segment.  A higher memory limit means fewer segments.
This resolved our problem, but we didn't do a deeper dive to see precisely why the original memory error was being raised in the first place. The virt mem thing was just a hunch.
So perhaps the best option for you to try is to regenerate your problematic tantivy index with a much larger memory limit on the writer.

I just checked, we're currently using 1GB as the heap setting for the writer.  I am wondering whether I should set this as the default in this package.  It might be better to default to bigger (avoiding the hard-to-debug OutOfMemory error), rather than the current default.

Thank you so much for the replies @cjrh I am following what you're saying.
For reference my index is built from about 5.5k PDFs and goes to this many files:
> ls /path/to/index | wc -l
21001
> ls /path/to/index | head -n 8
000a60b7148d4e8285cd78737474a3da.fast
000a60b7148d4e8285cd78737474a3da.fieldnorm
000a60b7148d4e8285cd78737474a3da.idx
000a60b7148d4e8285cd78737474a3da.pos
000a60b7148d4e8285cd78737474a3da.store
000a60b7148d4e8285cd78737474a3da.term
000d73f244864638833d735c8a76ebd7.fast
000d73f244864638833d735c8a76ebd7.fieldnorm

Are you able to run Index.open on the macOS machine to check whether it produces the same error?

Last night I rebuilt the index on the Ubuntu machine itself. What happens is:

Index built on macOS laptop, running on macOS laptop: OutOfMemory errors don't happen
Index built on Ubuntu server, running on Ubuntu server: OutOfMemory errors happen
Index built on Ubuntu server, running on Ubuntu server: OutOfMemory errors happen

So actually I don't think it's an "index across OS's compatibility" issue any more.

So perhaps the best option for you to try is to regenerate your problematic tantivy index with a much larger memory limit on the writer

This sounds quite reasonable, let's do this. Can you give me some rough guidance on how I can regenerate the index using a much larger memory limit?
We instantiate our writer like this here:
index: tantivy.Index
writer: tantivy.IndexWriter = index.writer()

Should I configure the writer's heap_size argument? https://github.com/quickwit-oss/tantivy-py/blob/0.22.0/tantivy/tantivy.pyi#L342

Firstly I want to say again @cjrh thank you again for your help here, basically your help is very awesome :)

So perhaps the best option for you to try is to regenerate your problematic tantivy index with a much larger memory limit on the writer

I decided to be proactive here and took a guess at how to do this, making the below change and rebuilding the index:
index: tantivy.Index
--- writer: tantivy.IndexWriter = index.writer()
+++ writer: tantivy.IndexWriter = index.writer(heap_size=1_000_000_000)

Then looking at the new index's size:
> ls /path/to/index | wc -l
19273

And that did not resolve the issue, I still get the OutOfMemory error:
           ValueError: Failed to open file for read: 'IoError { io_error: Os { code: 12, kind: OutOfMemory, message: "Cannot allocate memory" }, filepath:
           "/path/to/index/14a007c93a624cd4ade83137b518b779.fieldnorm" }'

How can I correctly adjust the writer's memory limit?

Okay thank you @wallies, I tried that just now (both position arg and keyword arg variations of heap_size) and the error persists :/
Can you suggest me more things to try?

So I am actually pretty stalled out due to this error, so sharing a thought

So what we figured was happening might be that each segment's files are separately being mmap'ed, and eventually we just run out of virtual address space causing the OutOfMemory error.

Perhaps there is some way tantivy IndexWriter.commit can validate against this error type? Like during the build tanvity can check how much space is available and how much it has assigned. Just an idea.

Do you think it could be arm64 (my Mac where everything works) vs amd64 (server getting the IoError)?

@jamesbraza

Do you think it could be arm64

I don't know.  But In my case, we're all on amd64.

Then looking at the new index's size:
ls /path/to/index | wc -l
19273
And that did not resolve the issue, I still get the OutOfMemory error:

So something is wrong here. Increasing the writer memory threshold should definitely result in fewer segments.  Unless there is some other reason your segments are small. How often are you calling commit?

Can you show a snippet of code that shows the loop where the index being created (docs being added to a writer)?

Can you also show the output of ulimit -a on the problem amd64 machine pls.  On my dev machine it looks like this:
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 124938
max locked memory           (kbytes, -l) 4007836
max memory size             (kbytes, -m) unlimited
open files                          (-n) 65536
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 124938
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

Here is a small test file I made to generate indexes with a few of the settings configurable as command line args:
import argparse
import random
import string
from tantivy import SchemaBuilder, Index, Document

def run(path, heap_size=15_000_000, doc_count=1000, string_length=10, num_text_fields=10) -> Index:
    text_fieldnames = [f"text{i}" for i in range(num_text_fields)]
    schema = (
        SchemaBuilder()
        .add_unsigned_field("order", fast=True)
        .add_text_field("text", stored=True)
        .build()
    )

    schema_b = (
        SchemaBuilder()
        .add_unsigned_field("order", fast=True)
    )

    for fieldname in text_fieldnames:
        schema_b.add_text_field(fieldname, stored=True)

    schema = schema_b.build()

    index = Index(schema, path)
    writer = index.writer(heap_size)

    for i in range(doc_count):
        doc = Document()
        doc.add_unsigned("order", i)
        for fieldname in text_fieldnames:
            doc.add_text(fieldname, "".join(random.choices(string.ascii_lowercase, k=string_length)))
        writer.add_document(doc)

    writer.commit()
    writer.wait_merging_threads()
    index.reload()
    return index

def main(args):
    import os
    import tempfile
    with tempfile.TemporaryDirectory() as path:
        index = run(path, args.heap, args.count, args.length, args.fields)

        # Print out some diagnostics
        items = os.listdir(path)
        print(len(items))
        print(items[:10])

        # Check that we can open the index
        index = Index.open(path)
        print(index)

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Tantivy Python segment test")
    parser.add_argument("--heap", type=int, default=15_000_000, help="Heap size")
    parser.add_argument("--count", type=int, default=1000, help="Document count")
    parser.add_argument("--length", type=int, default=10, help="String length")
    parser.add_argument("--fields", type=int, default=10, help="Number of text fields")
    args = parser.parse_args()
    main(args)

Running this with these settings, for example (python 3.12, tantivy 0.22.0):
$ python main.py --count 1_000_000 --length 1000

Takes a while to run, but produces a large index, nearly 10 GB.  The important thing to note is that when it completes, there are 70 files in the index folder, not tens of thousands.
During the indexing, the number of files goes up and down, because tantivy also merges segments together as part of the indexing process.  This is me listing out the status of the tmp index folder a few times during the indexing (the tmp dir was tmpisktkptr):
$ du -d0 -h tmpisktkptr/
1,4G	tmpisktkptr/
(venv) /tmp  [venv://h/c/t/t/v:3.12.3]
$ ls -l tmpisktkptr/ | wc -l
38
(venv) /tmp  [venv://h/c/t/t/v:3.12.3]
$ ls -l tmpisktkptr/ | wc -l
20
(venv) /tmp  [venv://h/c/t/t/v:3.12.3]
$ ls -l tmpisktkptr/ | wc -l
20
(venv) /tmp  [venv://h/c/t/t/v:3.12.3]
$ du -d0 -h tmpisktkptr/
6,2G	tmpisktkptr/
(venv) /tmp  [venv://h/c/t/t/v:3.12.3]
$ ls -l tmpisktkptr/ | wc -l
44
(venv) /tmp  [venv://h/c/t/t/v:3.12.3]
$ ls -l tmpisktkptr/ | wc -l
56
(venv) /tmp  [venv://h/c/t/t/v:3.12.3]
$ du -d0 -h tmpisktkptr/
6,9G	tmpisktkptr/
(venv) /tmp  [venv://h/c/t/t/v:3.12.3]
$ ls -l tmpisktkptr/ | wc -l
26
(venv) /tmp  [venv://h/c/t/t/v:3.12.3]
$ du -d0 -h tmpisktkptr/
7,4G	tmpisktkptr/

So I definitely think that your problem is due to ending up with a very large number of files in your tantivy index.
Perhaps you can run the above script on the mac and see whether you end up with a large number of files or not. And if not, compare what the script is doing to what our live code is doing.

If the same invocation of the script produces a large number of files on mac, but not on amd64, then that suggests there is a bug in the mac build somehow, and we can escalate to the experts in the tantivy project 😄

How often are you calling commit?

Once per input PDF, there are about 5.5k of them.

Can you show a snippet of code that shows the loop where the index being created (docs being added to a writer)?

Yes it's here: https://github.com/Future-House/paper-qa/blob/v5.3.0/paperqa/agents/search.py#L404-L481
commit gets called within add_document, which is invoked near the bottom of that snippet.

Can you also show the output of ulimit -a on the problem amd64 machine pls.

Here is the Ubuntu server, which hits the IoError issue:
> ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 512038
max locked memory           (kbytes, -l) 16413240
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 512038
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited
> uname -a
Linux sn4622119851 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
> python -c 'import multiprocessing as mp; print(mp.cpu_count())'
48

Here is my Mac, which does not hit the IoError issue:
> ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8176
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       6000
-n: file descriptors                256
> uname -a
Darwin Jamess-MacBook-Pro.local 23.5.0 Darwin Kernel Version 23.5.0: Wed May  1 20:13:18 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6030 arm64
> python -c 'import multiprocessing as mp; print(mp.cpu_count())'
11

I just rebuilt the indexes using now writer(heap_size=int(15e9)) on both my Mac and the Ubuntu server. Per this comment:

Increasing the writer memory threshold should definitely result in fewer segments.

It seems the trend does hold, but just not as dramatically as you stated:
> ls /path/to/normal-index/index/ | wc -l
21001
> ls /path/to/1GBheap-index/index/ | wc -l
19273
> ls /path/to/15GBheap-index/index/ | wc -l
15343

Even with a 15 GB heap size, the error persists:
           Starting paper search for 'proton antiport serotonin transporter essential residues'.
           Starting paper search for 'serotonin transporter proton antiport mechanism'.
           Encountered exception during tool call for tool paper_search: ValueError('Failed to open file for read: \'IoError { io_error: Os { code: 12, kind: OutOfMemory, message: "Cannot
           allocate memory" }, filepath: "/path/to/15GBheap-index/index/259ec6cdcc6a49388d2e301fc6d3cfe8.term" }\'')
           Encountered exception during tool call for tool paper_search: ValueError('Failed to open file for read: \'IoError { io_error: Os { code: 12, kind: OutOfMemory, message: "Cannot
           allocate memory" }, filepath: "/path/to/15GBheap-index/index/259ec6cdcc6a49388d2e301fc6d3cfe8.term" }\'')
           Starting paper search for 'HER2 positive breast cancer cells upregulated genes SREBF1 SREBP1c NR1H2 ACLY FASN ABCA1'.
           Starting paper search for 'HER2 positive breast cancer gene expression'.
           Encountered exception during tool call for tool paper_search: ValueError('Failed to open file for read: \'IoError { io_error: Os { code: 12, kind: OutOfMemory, message: "Cannot
           allocate memory" }, filepath: "/path/to/15GBheap-index/index/259ec6cdcc6a49388d2e301fc6d3cfe8.term" }\'')
           Encountered exception during tool call for tool paper_search: ValueError('Failed to open file for read: \'IoError { io_error: Os { code: 12, kind: OutOfMemory, message: "Cannot
           allocate memory" }, filepath: "/path/to/15GBheap-index/index/259ec6cdcc6a49388d2e301fc6d3cfe8.term" }\'')
           Encountered exception during tool call for tool paper_search: ValueError('Failed to open file for read: \'IoError { io_error: Os { code: 12, kind: OutOfMemory, message: "Cannot
           allocate memory" }, filepath: "/path/to/15GBheap-index/index/259ec6cdcc6a49388d2e301fc6d3cfe8.term" }\'')
           Encountered exception during tool call for tool paper_search: ValueError('Failed to open file for read: \'IoError { io_error: Os { code: 12, kind: OutOfMemory, message: "Cannot
           allocate memory" }, filepath: "/path/to/15GBheap-index/index/259ec6cdcc6a49388d2e301fc6d3cfe8.term" }\'')
           Encountered exception during tool call for tool paper_search: ValueError('Failed to open file for read: \'IoError { io_error: Os { code: 12, kind: OutOfMemory, message: "Cannot
           allocate memory" }, filepath: "/path/to/15GBheap-index/index/329655a70d9648ab8dd88ad3fba4ff5e.fieldnorm" }\'')
[15:14:40] Encountered exception during tool call for tool paper_search: ValueError('Failed to open file for read: \'IoError { io_error: Os { code: 12, kind: OutOfMemory, message: "Cannot
           allocate memory" }, filepath: "/path/to/15GBheap-index/index/329655a70d9648ab8dd88ad3fba4ff5e.term" }\'')
           Starting paper search for 'mucin-degrading bacteria enriched in mouse gut microbiome fiber-free conditions'.
           Starting paper search for 'B. intestinihominis fiber-free conditions'.
           Starting paper search for 'C. symbiosum fiber-free conditions'.
           Starting paper search for 'B. thetaiotaomicron fiber-free conditions'.
           Starting paper search for 'A. muciniphila fiber-free conditions'.
           Encountered exception during tool call for tool paper_search: ValueError('Failed to open file for read: \'IoError { io_error: Os { code: 12, kind: OutOfMemory, message: "Cannot
           allocate memory" }, filepath: "/path/to/15GBheap-index/index/86532e330292440e9d50c41993060b56.store" }\'')
           Encountered exception during tool call for tool paper_search: ValueError('Failed to open file for read: \'IoError { io_error: Os { code: 12, kind: OutOfMemory, message: "Cannot
           allocate memory" }, filepath: "/path/to/15GBheap-index/index/86532e330292440e9d50c41993060b56.store" }\'')
           Encountered exception during tool call for tool paper_search: ValueError('Failed to open file for read: \'IoError { io_error: Os { code: 12, kind: OutOfMemory, message: "Cannot
           allocate memory" }, filepath: "/path/to/15GBheap-index/index/9fa60161320944e0bbf0ec0a382f4bd7.fast" }\'')
           Encountered exception during tool call for tool paper_search: ValueError('Failed to open file for read: \'IoError { io_error: Os { code: 12, kind: OutOfMemory, message: "Cannot
           allocate memory" }, filepath: "/path/to/15GBheap-index/index/9fa60161320944e0bbf0ec0a382f4bd7.fast" }\'')
           Starting paper search for 'ZBTB16 effect on PAX6 promoters'.
           Encountered exception during tool call for tool paper_search: ValueError('Failed to open file for read: \'IoError { io_error: Os { code: 12, kind: OutOfMemory, message: "Cannot
           allocate memory" }, filepath: "/path/to/15GBheap-index/index/9fa60161320944e0bbf0ec0a382f4bd7.fast" }\'')
           Starting paper search for 'drosophila melanogaster embryos heat shock gene expression'.
           Encountered exception during tool call for tool paper_search: ValueError('Failed to open file for read: \'IoError { io_error: Os { code: 12, kind: OutOfMemory, message: "Cannot
           allocate memory" }, filepath: "/path/to/15GBheap-index/index/9fa60161320944e0bbf0ec0a382f4bd7.idx" }\'')
           Encountered exception during tool call for tool paper_search: ValueError('Failed to open file for read: \'IoError { io_error: Os { code: 12, kind: OutOfMemory, message: "Cannot
           allocate memory" }, filepath: "/path/to/15GBheap-index/index/9fa60161320944e0bbf0ec0a382f4bd7.idx" }\'')

Please note that it only seems to be a subset of index files lead to the IoError, it's not all of them. To share, here's me running:
> du -h /path/to/15GBheap-index/index/9fa60161320944e0bbf0ec0a382f4bd7.idx
4.0K    /path/to/15GBheap-index/index/9fa60161320944e0bbf0ec0a382f4bd7.idx

I know this is a lot to take in, but can you please give some guidance on what to try next?
Also, I want to restate your efforts are very much appreciated 🙏

Do you think we should not be calling commit so much? It seems you:
writer.commit()
writer.wait_merging_threads()
index.reload()

We don't do wait_merging_threads or reload, not sure if that matters or not

Okay I was talking in-person at FutureHouse and I want to highlight something to help make it clear:

We build successfully, and do not see the IoError during build, regardless of heap_size
Using the built Index at "inference time" (when we use the index for search) when calling Index.open:

On my Mac: no IoError issues
On the Ubuntu server: hits the IoError issues

And with this new mental clarity, I decided to play around and discovered this (with an index built on the 5.5k papers using the default heap_size):
In [1]: from paperqa.agents.search import SearchDocumentStorage, SearchIndex
In [2]: index_1 = await SearchIndex(index_name="normal-index").index
In [3]: index_2 = await SearchIndex(index_name="normal-index").index
In [4]: index_3 = await SearchIndex(index_name="normal-index").index
In [5]: index_4 = await SearchIndex(index_name="normal-index").index
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[5], line 1
----> 1 index_4 = await SearchIndex(index_name="normal-index").index

File /path/to/.venv/lib/python3.12/site-packages/paperqa/agents/search.py:176, in SearchIndex.index(self)
    174 index_path = await self.index_filename
    175 if await (index_path / "meta.json").exists():
--> 176     self._index = Index.open(path=str(index_path))  # type: ignore[assignment]
    177     print("success", self._index)
    178 else:
    179     # NOTE: this creates the above meta.json file

ValueError: Failed to open file for read: 'IoError { io_error: Os { code: 12, kind: OutOfMemory, message: "Cannot allocate memory" }, filepath: "/path/to/normal-index/index/9ac6cd3a75304ff794ef4377ae3ba1b2.store" }'

At this point, running free -h we see there's still lots of RAM left:
> free -h
               total        used        free      shared  buff/cache   available
Mem:           125Gi       7.5Gi        60Gi        12Mi        57Gi       116Gi
Swap:            9Gi       307Mi       9.7Gi

So, seeing this, can you help me figure out what is going wrong?

So, seeing this, can you help me figure out what is going wrong?

No

At this point, running free -h we see there's still lots of RAM left:

Yes, my strong suspicion is that you're hitting some kind of virtual memory limit from mmap calls being made for too many segments. So the physical memory in your machine doesn't matter.

It seems the trend does hold, but just not as dramatically as you stated:
ls /path/to/normal-index/index/ | wc -l
21001
ls /path/to/1GBheap-index/index/ | wc -l
19273
ls /path/to/15GBheap-index/index/ | wc -l
15343

These are still too many I think.
Either you should try to run my test script that I posted above, and check whether you get a similar number of files as I do, or you will have to give me a script that I can easily run on my machine to verify that I get the large number of files that you see.  If I can reproduce the issue on my computer it is very likely I will figure out what is going on. But failing that, it's very difficult. A very simple reproducer that both you and I can run on our machines and compare results would be very helpful.

We build successfully, and do not see the IoError during build, regardless of heap_size
On the Ubuntu server with concurrency of 20: hits the IoError issues

Out of interest, have you tried building the index on the Ubuntu server, and when doing so how many segments (files) to you end up with in the index directory when doing so? Is it similar to what you end up with on the mac?

Do you think we should not be calling commit so much?
We don't do wait_merging_threads or reload, not sure if that matters or not

You must call wait_merging_threads, and after the commit just like the sequence I showed.  These threads are combining segments so that you end up with fewer segments. This is important and could be directly related to ending up with so many segments. You should add this line.
The frequency of calling commit shouldn't really matter for how many segments you end up with, because segments get merged together periodically anyway. However, calling commit too often will make things slow. So if you have a lot of data ready to index it's better to call commit less frequently; although, below some frequency it won't matter any more.

Allllright, coming from https://github.com/Future-House/paper-qa/pull/628#issuecomment-2438512212 on adding in use of wait_merge_threads:

Index file count: 1000X smaller (21004 to 22 files)
Index size: 40% smaller (650 to 389 MiB)
Index build time: ~2X longer (anecdotally)
(Runtime) tantivy.Searcher.search performance: unchanged
(Runtime) tantivy.Searcher.search speed: definitely faster (but didn't quantify this)

So it turns out by not calling wait_merging_threads, we were missing out on tons of compaction.

It turns out wait_merging_threads really matters.
I re-ran the original code that lead to the IoError, and it no longer occurs 👍. So this case is resolved.

I guess, I am left wondering, how can tantivy-py make this easier for "the next guy"?
Is there some way we can change the Python bindings to make this more seamless, or obvious that wait_merging_threads is important?

For sure, the first port of call should be the documentation. The very first examples seen by a new user should include all the parts, including the merging threads. I can add this to the docs over the next few days for any that are missing.
In terms of changing the code to make it easier for users, one of the design goals we have is to mimic the tantivy (upstream crate) interface as much as possible, so that knowledge of how one works can be transferred over to the other. So I wouldn't want to change the existing calls themselves, for example I wouldn't want to call wait_merging_threads inside the commit wrapper in the python code.
However, we should be able to add new methods, perhaps in a new python module namespace that automates some of these typical state management steps. For example, we could expose a new context manager (which rust doesn't have) that lets the user add docs and then in the finalizer section of the context manager we can call commit and wait_merging_threads.  This can be explained as some kind of pythonic extension of the raw tantivy interface.  I'll think about it more.
For now, I am happy to close this issue if you are, since the original issue is now solved.

Good to hear 😄
Yes when I do my next pass over the docs I'll make a special callout about the wait_merging_threads call.


Routes are not removed when DEA is evacuated

It often happens that routing records are not deleted while DEA is evacuated and disappear only within 120-150 seconds. This causes problems for POST requests (gorouter returns 502) for cf-release prior to v237.
How to reproduce:

Run a loop to check a particular route every second from gorouter

while true; do date; curl http://router-status:PASS@<IP_ADDRESS>:8080/routes 2>/dev/null | jq '.["myapp.example.net"]'; sleep 1; echo; done

Mon Dec 26 08:05:50 UTC 2016
[
  {
    "address": "<IP_ADDRESS>:61182",
    "ttl": 0
  }
]

...

Evacuate a DEA running one of instances associated with the 'myapp.example.net' by bosh stop dea/0

When the instance stops and starts on another dea the routing record for this instance disappears immediately however in a few seconds it comes back and disappears later within timeout mentioned above:
Mon Dec 26 08:05:51 UTC 2016
[
  {
    "address": "<IP_ADDRESS>:61155",
    "ttl": 0
  },
  {
    "address": "<IP_ADDRESS>:61182",
    "ttl": 0
  }
]

It seems to be happening here: https://github.com/cloudfoundry/dea_ng/blob/c8d1f7a622a4b856c040f92fc721cfd02214f452/lib/dea/bootstrap.rb#L404
Why does it try to register routes while evacuating?

Please use a more current release. Many of these issues have been fixed. The gorouter has also been fixed to retry when connects fail which should also occur.

I know about gorouter issue being fixed (and mentioned it above) but I did not manage to find anything about fixing the original issue. The related code has not being changed. This link points to the latest commit of dea_ng, there are: https://github.com/cloudfoundry/dea_ng/blob/c8d1f7a622a4b856c040f92fc721cfd02214f452/lib/dea/bootstrap.rb#L404

The evacuating instance lives until the DEA is told the new instance is running on another DEA. The instance is removed at this point.

The evacuating instance lives until the DEA is told the new instance is running on another DEA. The instance is removed at this point.

This is how it should work however it works differently. The new instance starts and receives traffic. The old instance is already deleted (moreover DEA itself is already stopped completely) however routing table on gorouter still has both old and new records which should not be.

This does never happens if I change code like this:
--- /var/vcap/data/packages/dea_next/lib/dea/bootstrap.rb.orig      2016-03-16 16:33:20.000000000 +0000
+++ /var/vcap/data/packages/dea_next/lib/dea/bootstrap.rb   2016-12-26 14:06:27.223234727 +0000
@@ -363,7 +363,7 @@

     def register_routes
       instance_registry.each do |instance|
-        next if !(instance.running? || instance.evacuating?) || instance.application_uris.empty?
+        next if !instance.running? || instance.application_uris.empty?
         router_client.register_instance(instance)
       end

Any good reason to register routes while evacuating? It looks like a mistake.


$gt(e), $lt(e) should also accept string as a value (types)

the title says it all. if I have id fields with a type of string that I want to compare (e.g. ULID) then typescript gives an error that gt(e) and lt(e) accept only numbers.

I think it should support string | Date | number

Good suggestion 👍🏻. Fixed and published to NPM.


I submited my current configuration showcasing my current customized Tmux statusline together with the Gotham colorschemes on the unix porn subredit !!
I hope it helps more people find this awesome colorscheme.

:tada: thanks so much for the kind words and for the visibility :yum:


Declare Python version requirement

Codecov Report

Merging #43 (6e023e0) into master (e5fe5b3) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #43   +/-   ##
=======================================
  Coverage   82.88%   82.88%
=======================================
  Files          28       28
  Lines        1385     1385
  Branches      140      131    -9
=======================================
  Hits         1148     1148
  Misses        189      189
  Partials       48       48

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e5fe5b3...6e023e0. Read the comment docs.


Add loading progress bar

Add a loading progress bar to classroom, which is particularly important on first application load.

https://github.com/chieffancypants/angular-loading-bar


Consider using tox-gh

Given that PR #22 caused some pain, has the dev team considered using the tox team supported https://github.com/tox-dev/tox-gh project?
This would just mean replacing
https://github.com/AbigailMcGovern/iterseg/blob/ba22adeb6558c28d4fb9f0b31b963ed387e87933/.github/workflows/test-and-deploy.yml#L53
with
python -m pip install setuptools tox tox-gh

and then tox should be able to discover the proper setup from the GitHub Actions matrix (I think. Full disclosure, I don't use this package in any of my workflows).

Full disclosure, I don't use this package in any of my workflows

LOL 😂
Anyway, I like this idea, thanks for making an issue. It might take us a while to get to it but it is appreciated! 😊 🙏


nodeContent redraws all nodes each time a node is collapsed/uncollapsed

Hello,
I would like to thank you above all for the enormous effort you are making.
Describe the bug
nodeContent method redraws all nodes each time a node is collapsed/uncollapsed (folded/unfolded).
this consumes a lot of performance, especially for multi-level trees.
imagine if you have 1000 entries, and all of them are visible, if you change one of them, all the others will be redrawn.
in my case, I have nodes that contain VueJs components, imagine the impact on Ram and CPU, every time you have to recreate VueJs instances of each component and link them to the DOM.
Also, we don't have the node destruction event with the old rendering reference, which causes a memory leak, because the components are not removed from the virtual DOM.
Expected behavior

Redraw only affected nodes
Save the Html rendering of the nodes, to avoid recalculating it each time
Add destruction (event/callback)

thank you again..

I will keep this in mind for v3 and will see if I can do some optimizations

Hey @bumbeishvili, is there anything I can do to temp optimize the performance until a better fix comes along?
I am rendering the node content using react, it's expensive and in my usecase nodes will easily be more than 500.

@akshay-developer-2021 can you take a video recording and post how it behaves? Are your 500 nodes always visible? Or do you just mean, they are in the tree but are hidden?
v2 had a nonoptimal loop when looping over deep-level hierarchies (>10), which was fixed in v3.
Usually redrawing does not have as much impact on performance, as transition animation for example, or adding drop shadow in the nodes

Hello, @bumbeishvili
First of all, thank you very much for your contribution to this project
I'm using vue3. The number of nodes is now over 1500, and the dom updates when nodes are expanded or closed are causing a huge impact on my Ram and CPU
I was wondering if we could throw a method to determine if the dom needs to be updated
Something like that
     isUpdate: (d) => true, // Callback for node update

And then
    restyleForeignObjectElements() {
        const attrs = this.getChartState();
        attrs.svg
            .selectAll(".node-foreign-object")
            .attr("width", ({ width }) => width)
            .attr("height", ({ height }) => height)
            .attr("x", ({ width }) => 0)
            .attr("y", ({ height }) => 0);
        attrs.svg
            .selectAll(".node-foreign-object-div")
            .style("width", ({ width }) => `${width}px`)
            .style("height", ({ height }) => `${height}px`)
            .each(function (d, i) {
                if (attrs.isUpdate(d)) {
                    d3.select(this)
                        .html(function (d, i, arr) {
                            if (d.data._pagingButton) {
                                return `

${attrs.pagingButton(
                                    d,
                                    i,
                                    arr,
                                    attrs
                                )}

`;
                            }
                            return attrs.nodeContent.bind(this)(d, i, arr, attrs);
                        });
                }
            })
    }

This reduces the overhead of updating the dom
const renderChart = () => {
  chart.value = new OrgChart<GenPersonWithId>()
    .parentNodeId((item) => {
      // @ts-ignore
      const id = item.parent_id ? item.parent_id : item.data.parent_id
      if (id == -1) return ''
      return id
    })
    .nodeWidth(() => 105)
    .nodeHeight(() => 165)
    .container('#content')
    .layout('left')
    .compact(false)
    .onExpandOrCollapse(handleButtonClick)
    .nodeContent(function (node) {
      return `<div id=${node.data.id} style="width:100%;height:100%;"></div>`
    })
    .nodeUpdate((node) => {
      const dom = document.getElementById(node.data.id)
      if (dom && dom.getAttribute('complete') !== 'true') {
        dom.setAttribute('complete', 'true')
        render(
          h(Cover, {
            node: node.data,
            showFirstName,
            showRank,
            showSex,
            choose,
            onNodeClick: onNodeClick,
          }),
          dom
        )
      }
    })
    // @ts-ignore
    .isUpdate((node) => {
      if (ids.value.includes(node.data.id) || !chart.value) {
        return true
      } else {
        return false
      }
    })
    .data(dataList.value)
    .render()
    .expandAll()
}

Hi, most probably the transition of nodes is causing issues, not the redrawing part. If you can prepare some kind of sample with the issue reproduced I can look into it deeper.

Hello @sunlho
Your issue is not laying inside the Library, but in your code.
Whenever you do any update e.g. collapse / expand the new dom (in out case) will be created which will lead to creating new Vue component instance which means dom.setAttribute('complete', 'true') is not efficient enough. With 1500 nodes considering you have all of them made via Vue components on each interaction you are adding the Vue component instance and depending of the "weight" of it the memory and perf is going to drop very fast.
I encourage you to have a look at Vue explanation and inside nodeUpdate you need to have some sort of cache that is keeping track on vNode instances and with combination of MutationObserver u can indicate whether there was new instance created or not - if the prev. DOM is de-attached you need to call
render(null, event.target)

inside MutationObserver callback.
Let me know if it's all clear, I may add fully working example which retains the Vue components - it's more like creating your own "mechanism" for keeping a track of the instances and making sure they are properly removed.


fetchFromPypi: fix api lookup & improve prediction

Done:

Improve predicted URL for wheel files by embedding the python version into the URL
Fix pypi api lookup for wheel files by using the version parsed from the filename instead of the one provided by the user
add more tests

I'm tempted to drop fetchFromPypi entirely and only use fetchFromLegacy instead.


Configure pool neofs-nodes

Consider support configuration more than one neofs-node in repository string
neofs:grpc://s01.neofs.devenv:8080/BQuLfh5yPSWLRsoRGtvqg8itXRREN17dgmHDsw1yLJgb

For example we can use previous one with the 1 priority. And add other endpoint in options with the 2 priority (but in this case we should be sure restic tries make another requests when fails the first one (otherwise pool will never use the endpoints with 2 priority))

Add example of different configurations in this issue.

I think we can use the following configuration:
neofs:BQuLfh5yPSWLRsoRGtvqg8itXRREN17dgmHDsw1yLJgb \
 -o neofs.endpoint=grpc://s01.neofs.devenv:8080,1,1 \
 -o neofs.endpoint=grpc://s02.neofs.devenv:8080,2,1 \
 -o neofs.endpoint=grpc://s03.neofs.devenv:8080,3,9

The neofs.endpoint will have the format <endpoint>,<priority>,<weight>  and be required (or we can have some default endpoint to testnet/mainnet for example).

Are these going to be correct configurations?
1. neofs:[cid] -o neofs.endpoints="grpc://s01.neofs.devenv:8080"
2. neofs:[cid] -o neofs.endpoints="grpc://s01.neofs.devenv:8080 1; grpc://s02.neofs.devenv:8080 2"
3. neofs:[cid] -o neofs.endpoints="grpc://s01.neofs.devenv:8080 1 1; grpc://s02.neofs.devenv:8080 2 1; grpc://s02.neofs.devenv:8080 2 9"

Are these going to be correct configurations?

I suppose the third configuration has typo (the third endpoint should be s03). If so then such configurations are going to be correct.

Done in #15


[AKS] az aks create/update: Add LicenseType support for Windows

Description

This will allow cx to create/update AKS Clusters to enable Azure Hybrid User Benefit (AHUB) for all AKS Windows nodes.
Testing Guide

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change.
[Component Name 2] az command b: Add some customer-facing feature.

This checklist is used to make sure that common guidelines for a pull request are followed.

[x] The PR title and description has followed the guideline in Submitting Pull Requests.

[x] I adhere to the Command Guidelines.

AKS

Hi @AbelHu , the code is good to me, could you please help fix the CI error ?

Yes. Will update it soon.

@arrownj Is it ok to merge this PR? We expect that it can be included in next azure-cli official release. Thanks.

@arrownj Do you know when this can be included in the azure-cli official release?


Refactor to use one similar logic to call plot backends

This PR proposes:

Remove koalas_plotting_backends. We don't currently have such mechanism like https://github.com/pandas-dev/pandas/commit/e9a60bbc4f70820630f12666c273d14bcf83f1a7
Load and use plotting backend in the same way:

If a plot module has plot, we use it after converting it to pandas instance.
if a plot module has plot_koalas (Koalas' matplotlib and plotly modules for example), we just pass Koalas instances to plot.

Now databricks.koalas.plot.plotly and databricks.koalas.plot.matplotlib modules work like external plotting backends.

Codecov Report

Merging #2005 (38404eb) into master (3ce2d87) will decrease coverage by 0.87%.
The diff coverage is 79.41%.

@@            Coverage Diff             @@
##           master    #2005      +/-   ##
==========================================
- Coverage   94.55%   93.67%   -0.88%
==========================================
  Files          50       50
  Lines       11086    11085       -1
==========================================
- Hits        10482    10384      -98
- Misses        604      701      +97

Impacted Files
Coverage Δ

databricks/koalas/plot/core.py
92.06% <69.23%> (+1.24%)
:arrow_up:

databricks/koalas/plot/matplotlib.py
92.07% <82.35%> (-0.56%)
:arrow_down:

databricks/koalas/plot/plotly.py
97.72% <100.00%> (+0.05%)
:arrow_up:

databricks/koalas/usage_logging/__init__.py
24.78% <0.00%> (-67.53%)
:arrow_down:

databricks/koalas/usage_logging/usage_logger.py
47.82% <0.00%> (-52.18%)
:arrow_down:

databricks/conftest.py
94.20% <0.00%> (-5.80%)
:arrow_down:

databricks/koalas/__init__.py
85.93% <0.00%> (-4.69%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3ce2d87...38404eb. Read the comment docs.

Codecov Report

Merging #2005 (38404eb) into master (3ce2d87) will decrease coverage by 0.87%.
The diff coverage is 79.41%.

@@            Coverage Diff             @@
##           master    #2005      +/-   ##
==========================================
- Coverage   94.55%   93.67%   -0.88%
==========================================
  Files          50       50
  Lines       11086    11085       -1
==========================================
- Hits        10482    10384      -98
- Misses        604      701      +97

Impacted Files
Coverage Δ

databricks/koalas/plot/core.py
92.06% <69.23%> (+1.24%)
:arrow_up:

databricks/koalas/plot/matplotlib.py
92.07% <82.35%> (-0.56%)
:arrow_down:

databricks/koalas/plot/plotly.py
97.72% <100.00%> (+0.05%)
:arrow_up:

databricks/koalas/usage_logging/__init__.py
24.78% <0.00%> (-67.53%)
:arrow_down:

databricks/koalas/usage_logging/usage_logger.py
47.82% <0.00%> (-52.18%)
:arrow_down:

databricks/conftest.py
94.20% <0.00%> (-5.80%)
:arrow_down:

databricks/koalas/__init__.py
85.93% <0.00%> (-4.69%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3ce2d87...38404eb. Read the comment docs.

Thanks guys!

Thanks guys!

Nice!

Nice!


Fixes broken link to Kindle version

I believe I have linked to the correct link - http://www.amazon.com/Pro-Git-Scott-Chacon-ebook/dp/B00LPDVAX2/ref=mt_kindle?_encoding=UTF8&me= .
Feel free to change it accordingly if I have not.

Why not leave it short?
Using defalut "Share" button i got https://amzn.com/B00LPDVAX2
It links to some Amazon product code (the same as in both old and new links) directly.

I think we want to leave the links mostly as-is, because they contain affiliate links for Software Freedom Conservancy (so if somebody does buy it, Amazon gives a percentage to Conservancy).
There are also two links (the cover picture is a link), and they both need adjusted.
I opened #820 to fix this.


Add support for NetworkConfigProxy v0 and v1 api

This PR adds support to ncproxy service for both the NetworkConfigProxy grpc v0 and v1 api. This is done by converting the v0 api to the v1 api before calling the underlying implementation of the various ncproxy calls and converting from the v1 responses to the v0 responses before returning the results. This PR additionally adds tests for the v0 api.
Currently with the NetworkConfigProxy v1 api, ncproxy supports two types of networking: hcn networking and ncproxy networking. NetworkConfigProxy grpc v0 api notably does not have support for the ncproxy networking types, dual stack, or the additional hcn policies that are present in the v1 api.
Note: v0 support is intended to be temporary, so this code will likely be removed in the near future.

proto is failing cause of formatting in the generated go code, which is think is difference with how go1.19 and 1.20 gofmt runs.
you could manually change the files, or use go 1.19 locally, or upgrade the pipeline to 1.20

Isn't this a pretty major breaking change, since anyone currently using the v1 API will have their code break suddenly?
Can we keep the original v1 name, but instead call the old package v1legacy or something like that?

Yes, this is a breaking change, however, afaik, no one is using the v2 api or even has the v2 api file pulled in, so we should be safe to change that.

Isn't this a pretty major breaking change, since anyone currently using the v1 API will have their code break suddenly?
Can we keep the original v1 name, but instead call the old package v1legacy or something like that?

Yes, this is a breaking change, however, afaik, no one is using the v2 api or even has the v2 api file pulled in, so we should be safe to change that.

I'd suggest we keep the latest proto file unchanged, and just add the old one in a package dir named something like v0/legacy/beta/etc. That way it's not a breaking change to anyone.

I'd suggest we keep the latest proto file unchanged, and just add the old one in a package dir named something like v0/legacy/beta/etc. That way it's not a breaking change to anyone.

@kevpar and @helsaawy updated to use v0 for the old api and kept the current api as v1.

I think PR title/description need to be updated.

You cannot run the v0 implementation at the same time as the v1 implementation since the v0 implementation will return an error if any "ncproxy networks" or "ncproxy endpoints" are present.

I'm confused by this comment. Isn't running v0 and v1 at the same time precisely what this PR is doing?

You cannot run the v0 implementation at the same time as the v1 implementation since the v0 implementation will return an error if any "ncproxy networks" or "ncproxy endpoints" are present.

I'm confused by this comment. Isn't running v0 and v1 at the same time precisely what this PR is doing?

@kevpar This PR adds the ability to use v0 or v1 within the same executable for ncproxy. This was more of an implementation decision that I made. Since the code works by converting the v0 api to the v1 api and back for the response, if both apis are being used simultaneously, it's possible there are "ncproxy networks" or "ncproxy endpoints" that were created from a call to the v1 apis. In that scenario, a v0 agent making a call to something like "GetNetworks" would get an error since there are "ncproxy networks" that ncproxy is tracking which cannot be encoded in the response to the v0 "GetNetworks" api. We could instead just ignore "ncproxy networks" or "ncproxy endpoints" and add a log on our side, but I was concerned about doing that since we wouldn't be giving the node network agent the true picture of what networks or endpoints have been created through calls to ncproxy.
So basically the expectation is that if node network agent wants to use the v1 api, it must fully move all calls to ncproxy over to the v1 api, not just a subset.

@kevpar This PR adds the ability to use v0 or v1 within the same executable for ncproxy. This was more of an implementation decision that I made. Since the code works by converting the v0 api to the v1 api and back for the response, if both apis are being used simultaneously, it's possible there are "ncproxy networks" or "ncproxy endpoints" that were created from a call to the v1 apis. In that scenario, a v0 agent making a call to something like "GetNetworks" would get an error since there are "ncproxy networks" that ncproxy is tracking which cannot be encoded in the response to the v0 "GetNetworks" api. We could instead just ignore "ncproxy networks" or "ncproxy endpoints" and add a log on our side, but I was concerned about doing that since we wouldn't be giving the node network agent the true picture of what networks or endpoints have been created through calls to ncproxy.
So basically the expectation is that if node network agent wants to use the v1 api, it must fully move all calls to ncproxy over to the v1 api, not just a subset. This will hopefully additionally prevent some bugs that could show up if they only move a subset of the calls to the newer api.

Okay, it's just that callers can't use both the v0 and v1 API at the same time with a single ncproxy instance? The phrasing made it sound like we can't host both APIs at the same time.

Okay, it's just that callers can't use both the v0 and v1 API at the same time with a single ncproxy instance? The phrasing made it sound like we can't host both APIs at the same time.
Edit: I think it would probably be better if we can just show a limited view in the v0 API, rather than returning an error. For instance, just filter out the "ncproxy networks" in v0 GetNetworks call. This allows the v0 API to continue working, and a v0 caller doesn't need to know about "ncproxy networks", since it was never designed to care about them.

@kevpar Sure, if that's what's preferred, I'll make those changes now.

@kevpar code now allows for running both v0 and v1 api simultaneously. IOW, no erroring if "ncproxy networks" or "ncproxy endpoints" are present.

Fixed a linter error in the naming of the new error variables I made.

what was the decision on log warnings in all the v0 calls that the API is marked for deprecation and users should transition to v1?

I added a warning log in the server setup to indicate that the v0 api is deprecated. Unfortunately, as far as I could find, there isn't a nice way to indicate that a field or api call is deprecated without adding deprecated tags to the proto files and having the node network agent take that updated proto file for v0 api. We could add a log statement to all the functions for v0, but that wouldn't communicate to the node network agent that the field is deprecated, that would just add a log in our own ncproxy logs, which I'm not sure would be useful.

@helsaawy and @kevpar I added an additional option in the v0 protobuf file to indicate that it's deprecated. Based on the language spec https://protobuf.dev/programming-guides/proto3/, this typically just produces a warning when the file is compiled, so should be fine to add.

@helsaawy and @kevpar I added an additional option in the v0 protobuf file to indicate that it's deprecated. Based on the language spec https://protobuf.dev/programming-guides/proto3/, this typically just produces a warning when the file is compiled, so should be fine to add.

The protobuf docs seem to indicate that this option is only valid at field scope, rather than file. However, both protoc-gen-gogo and protoc-gen-go seem to generate a comment based on it being set at file-scope, so I guess it's fine?

The protobuf docs seem to indicate that this option is only valid at field scope, rather than file. However, both protoc-gen-gogo and protoc-gen-go seem to generate a comment based on it being set at file-scope, so I guess it's fine?

Yeah, I saw a reference to using the option at a file level here. I chose to do this to avoid needing to add the option to every type in the proto, but I'm fine adding that if that's the level of verbosity we'd prefer here.

The protobuf docs seem to indicate that this option is only valid at field scope, rather than file. However, both protoc-gen-gogo and protoc-gen-go seem to generate a comment based on it being set at file-scope, so I guess it's fine?

Yeah, I saw a reference to using the option at a file level here. I chose to do this to avoid needing to add the option to every type in the proto, but I'm fine adding that if that's the level of verbosity we'd prefer here.

The current approach is fine with me.


Add AcceptDialog::remove_button method

Add AcceptDialog::remove_button method which allows to remove a Button added with AcceptDialog::add_button or AcceptDialog::add_cancel_button method.
Doesn't allow to remove OK button.
Doesn't prevent removing Cancel button not added by the user, e.g. from the ConfirmationDialog (which derives from AcceptDialog) which could break things up. However, currently user can manually remove such button anyway so it shouldn't be an issue.
Fixes #50192.

Thanks!


Crash in Variable Explorer

Description
variable explorers list view crashes on opening nested xml elements
What steps will reproduce the problem?

extract content.xml from *.odt, load it into xml.dom.minidom object
variable2View = dom.getElementsByTagName(...) becomes a dom.minicompat.NodeList
open variable2View, and try to open one of the contained cells

What is the expected output? What do you see instead?
expected to see content of the xml node, got a crash instead
Please provide any additional information below
File "/usr/local/lib/python3.5/dist-packages/spyder/widgets/variableexplorer/collectionseditor.py", line 513, in createEditor
readonly=readonly)
File "/usr/local/lib/python3.5/dist-packages/spyder/widgets/variableexplorer/collectionseditor.py", line 1308, in setup
readonly=readonly, remote=remote)
File "/usr/local/lib/python3.5/dist-packages/spyder/widgets/variableexplorer/collectionseditor.py", line 1260, in init
title)
File "/usr/local/lib/python3.5/dist-packages/spyder/widgets/variableexplorer/collectionseditor.py", line 1145, in init
minmax=minmax)
File "/usr/local/lib/python3.5/dist-packages/spyder/widgets/variableexplorer/collectionseditor.py", line 110, in init
self.set_data(data)
File "/usr/local/lib/python3.5/dist-packages/spyder/widgets/variableexplorer/collectionseditor.py", line 158, in set_data
self.set_size_and_type()
File "/usr/local/lib/python3.5/dist-packages/spyder/widgets/variableexplorer/collectionseditor.py", line 178, in set_size_and_type
for index in range(start, stop) ]
File "/usr/local/lib/python3.5/dist-packages/spyder/widgets/variableexplorer/collectionseditor.py", line 178, in
for index in range(start, stop) ]
File "/usr/local/lib/python3.5/dist-packages/spyder/widgets/variableexplorer/collectionseditor.py", line 78, in getitem
return getattr(self.obj, key)
AttributeError: _localName
Version and main components

Spyder Version: 3.2.4
Python Version: 3.5.2
Qt Versions: 5.9.1, PyQt5 5.9 on Linux

Dependencies
pyflakes >=0.6.0 :  1.6.0 (OK)
pycodestyle >=2.3:  2.3.1 (OK)
pygments >=2.0   :  2.2.0 (OK)
pandas >=0.13.1  :  0.19.2 (OK)
numpy >=1.7      :  1.13.1 (OK)
sphinx >=0.6.6   :  1.6.5 (OK)
rope >=0.9.4     :  0.10.7 (OK)
jedi >=0.9.0     :  0.11.0 (OK)
psutil >=0.3     :  5.4.0 (OK)
nbconvert >=4.0  :  5.3.1 (OK)
sympy >=0.7.3    :  None (NOK)
cython >=0.21    :  None (NOK)
qtconsole >=4.2.0:  4.3.1 (OK)
IPython >=4.0    :  6.2.1 (OK)
pylint >=0.25    :  1.7.4 (OK)

Thanks for reporting, but unfortunately your description is not complete. Please post the Python code you used to get variable2View along with a simple odt file we can use in our end to reproduce this problem.
Thanks!

Thanks for getting back to me, and my apologies for the incomplete information.
Instead of the odt file, I attached the content.xml extracted from the odt and had to rename it the txt...
content226_uifz_block6.txt
`
file = "content226_uifz_block6.txt"
if name == 'main':
with open(file) as f:
read_data = f.read()
content = xml.dom.minidom.parseString(read_data)
tables = content.getElementsByTagName("table:table")
# the crash happens when opening the tables variable with VariableExplorer right after the above line

`
Generally speaking, the script does what I want it to do, extracting strings from the cells within tables[0] if I don't stop it. And sorry for dumping tables into writer documents, I know it is odd ;-)

I've reproduced and fixed this by just catching the error where it occurs, and you are now able to view the dom.minidom.Element just fine. PR incoming shortly. More complete code to reproduce:
from xml.dom.minidom import parseString

file = "Data/content226_uifz_block6.txt"
with open(file) as f:
    read_data = f.read()

    content = parseString(read_data)
    tables = content.getElementsByTagName("table:table")


Playing recorded event without recording new getevent.

Hi! Thanks for sharing the code!
I run the code on cmd.exe and its working fine, but I want to tweak the code so I can play the recorded event without recording a new getevent. Is there a way to do this?? Thanks in advance.

Of course :)
´´´subprocess.run(
[pla.adb_path, "-s", pla.device_serial, "shell"],
bufsize=0,
input=results["adbcommand"],  # just write down the string that you get here: results["adbcommand"] - something like su -c "...." and you can use it as much as you want
)´´´

But what if I want to play the recorded event (the bin file) that has been stored in the preferred location, which I specify in tempfolder_hdd=rf"the location"?
Sorry if I used the wrong terminology and confused you

You can the delete the files from your hdd, because they are on your device too (/sdcard/)

If I run the whole script without incorporating the changes you mentioned earlier, I receive a list of bin files that I can refer to as "old record".  However, running the entire script will create "new record" file and overwriting the existing "old record" file. Is there a way to prevent the recording function and just play the "old record" file?

Don't run the script again! It is only for creating the script. This is enough afterward:
import subprocess
subprocess.run(
    [pla.adb_path, "-s", pla.device_serial, "shell"],
    bufsize=0,
    input=YOUR COMMAND STRING,  # just write down the string that you get from: results["adbcommand"] - something like su -c "...." and you can use it as much as you want
)

Thank you for your patience and always responding, now I understand what you mean and the code works! :)
Here's the code if anyone wants to do the same thing
import subprocess
adb_path=                    #adb.exe address
device_serial=               #your emu/andro ip:port
subprocess.run(
[adb_path, "-s", device_serial, "shell"],
bufsize=0,
input= (your string command) ,  #do print(results["adbcommand"]) copy & paste the result
)

Perfect!! I am happy that it worked out! Are you using an emulator or a real rooted device?

an emulator

Awesome! I haven't had a chance yet to test it on a real device. I would like to know if it works. I am going to close this here, ok?


Raspberry PI freezes after LDK shutdown

LDK shutdown is successful and the response is received by the frontend, but then the RPI becomes unresponsive:

all API calls no longer work
SSH connection freezes
cannot do anything except unplug the device.

We need to try to trace what exact operation causes it.

The LDK node does stop successfully :white_check_mark:
The service successfully completes the StopLNClient function :white_check_mark:
The HTTP service returns OK from the API call and this is received by the browser :white_check_mark:
On Desktop Everything is then operational and you can start the node by entering the unlock password :white_check_mark:
On RPI all further HTTP requests get stuck, the RPI is at 100% CPU usage, ssh connection gets stuck, the only option is to physically unplug the RPI :red_circle:


Use with android.app.Fragment

Hi,
I would really like to use this library, but i'm using android.app.Fragment instead of the android.support.v4.app.Fragment used by this library. Is it possible to use this library with android.app.Fragment?

Hi @Tripwire999, thanks for your reply.
We refactored library and now there are two options:

extends your tutorial fragment from TutorialFragment, which is the successor of android.app.Fragment.
extends your tutorial fragment from TutorialSupportFragment, which is the successor of android.support.v4.app.Fragment.

In your case you should inherit from the TutorialFragment](https://github.com/Cleveroad/SlidingTutorial-Android/blob/master/lib/src/main/java/com/cleveroad/slidingtutorial/TutorialFragment.java).
That's all.


ci(workflows): reusable workflow implementation

:tada: This PR is included in version 8.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Jenkins / pytest integration

Hi,
I would like to use pyresttest in our CI. is there currently a suuport for xml test results generation?
Thanks

Hi,
PyRestTest offers a couple features to assist with CI integration.  One, upon completion, the process returns a status code equal to the number of test failures.  Two, you can also set the logging level on command line execution (i.e. --log=ERROR) so it only shows log lines in case of error.
Unfortunately, there is no support for custom output formats yet; perhaps you could assist with defining what you'd expect output to look like?  I've considered adding XML/JSON/YAML output but get hung up on defining the output structure.   (It will take a considerable amount of effort to implement this, however -- the python logging facility is not designed to do this, so it will require rewriting all logging components.)
Cheers,
Sam

Hi all,
@svanoort I guess it will probably be difficult for you to come up with an output structure that would fit the needs of the various clients. Maybe pytest should expose an "interface" for the clients to implement, thus having their own custom format. Of course, one may argue that it's now wise to push this problem to the client, though - some sort of a plugin design may do the trick.
//Marius

Thanks for the response. I will follow up in case I will come up with a solution.
Best,
Nir

@mariusmagureanu Yes, that is a good idea and along the lines I was thinking -- one notion is to use a listener API that is invoked at each step of the test execution/benchmark pipline.  This is somewhat linked to a longer-term goal of refactoring test/benchmark components into a set of composable, pluggable steps.
I'm also toying the idea of offering some sort of Jenkins plugin for integration in the future -- due to job changes, I expect to have a great deal of experience with this in the near future.

Logging output to an x-unit format would help with Jenkins integration without having to write a plugin, it is somewhat of an informal standard for test reporting.  My current employer (and several before it) leveraged this formatting, even for non-java projects.  Sometimes through test runner output format changes, sometimes just transforming the default output format to something x-unit-like, essentially something sufficient enough for Jenkins' reporting plugins to pick up on.
Here is a JUnit Schema for Apache Ant's JUnit, in case you haven't seen it before:
https://github.com/windyroad/JUnit-Schema
I feel like this format, in particular, will help drive broader adoption of this admittedly cool tool.
This issue post over at TravisCI (https://github.com/travis-ci/travis-ci/issues/239) lists several CI systems that expect this sort of format, presented as evidence for my argument.  However, TravisCI doesn't currently support it, from what I can tell with a few minutes of searching.  In fact, there is a bounty open for JUnit style build artifacts that are parsed into a simple report:  https://www.bountysource.com/issues/848108-handle-ant-s-junit-xml-formatted-output-from-test-spec-runs.
None of this is to say a Jenkins plugin wouldn't be completely awesome, but don't forget about all the other CI systems that would get free reporting integration if you were to just add an xunit-style output file (as ci_reporter, NodeUnit, Jasmine, Clojure.test, PyUnit, SimpleTest, JUnit, NUnit have done).  Then you have Hudson/Jenkins, Bamboo, and apparently at some point in the foreseeable future, TravisCI as painless integrators of this framework.
At any rate, to close a long post, we will likely build something internally, either hack at a fork or just transform the output data to a minimally acceptable junit-format for our Jenkins integration, as we have a large and varied body of testing assets spanning languages and testing types that all either already conform to xunit-style outputs or have tooling around them to produce such output from the default output formats.

@parametrization  With this in mind, I think this is a pretty reasonable output format, and maps well to what is currently output, to be honest.
I like the suggestion from @mariusmagureanu to do this by providing hooks for different output formats.
I think this will be dependent to some extent on changes planned for v2:  https://github.com/svanoort/pyresttest/issues/45
Specifically, the more flexible config structure and the stepwise decomposition of test/benchmark execution components.

  PyRestTest supports features that *Unit doesn't really, and vice versa, so we're aiming to provide coverage for the most common case
Here's proposed mapping.
TestSets <--> TestSuites in Junit
TestSet <--> TestSuite in JUnit
ALL macro-level elements: (Test/Benchmark) <--> Test element in JUnit
Configuration/bindings/etc: --> No mapping to output?   Just map context vars?  Undefined so far.
Undefined: system out, etc?
For each step, they're going to call a method in resttest.py to perform logging/reporting (potentially using configuration variables to define what logs).
I don't have a formal ETA for integration here, because of work commitments, but it's getting bumped up in priority quite a bit (for Jenkins Integration use), and I think is higher priority than implicit templating.

@ksreddy543 As you had asked, there is not an expected ETA behind this since I'm tying to a larger architectural change/extension to avoid having to rework both completely, specificially https://github.com/svanoort/pyresttest/issues/45 .
Since I am heavily committed with employment at CloudBees (limiting free time), and this is an uncompensated, outside-of-working-hours side project, my guess is this is probably some months out.

Adding a note for later use:  there's a library to assist with this -- https://pypi.python.org/pypi/junit-xml

This is officially a v2 featured, tied to the step/macro system:  https://github.com/svanoort/pyresttest/issues/92

Alternate option that removes blocking dependence on the major refactors:
Provide unittest-extending wrappers for the testtest, test, and benchmark elements, plus testconfig.
Tie into existing unittest JUnit compatible wrappers, such as: https://pypi.python.org/pypi/unittest-xml-reporting

Optimistically moving a rough (alpha-quality) implementation this to the 1.8.0 milestone because it's in high demand... if it can be readily done using convenient wrappers.   Otherwise it'll get pushed back again until rearchitecting supports it better.

This can probably run successfully using a callback-based methodology, and moving execution logic into the test/benchmark libraries.  This is also a big step toward PyRestTest v2


Allow AnyIO 4

This bumps the asyncer lib used to run some code asynchronously to latest version so AnyIO >= 4 is supported. This is currently not possible and can be a major issue when trying to integrate with other libraries such as langfuse.

Thank you for opening this PR! I updated asyncer in another PR, closing this one.


Parent Child on Lookup Field Does not seem to return data

Not sure if this is a limitation of the library but if I try and use
SELECT Account.Technical_Solution_Engineer__r.FirstName FROM Opportunity

This reference Account.Technical_Solution_Engineer__r.FirstName, which is a lookup to the User table.
Returns nothing in the library response but execute in salesforce workbench

@gaberger Thank you for opening an issue!
I'll take a look at this when I can, but in the meantime, does this query work if you use the REST explorer in workbench? Or by using Postman? The endpoint should be something like /services/data/vXX.X/query?q=query
Also, double check that the user executing the auth flow has permissions for that field.

@k-capehart thanks for the quick response.
If I issue a request to
/services/data/v58.0/query?q=SELECT%20Account.Technical_Solution_Engineer__r.FirstName%20FROM%20Opportunity

I get back a list of records like
[Item 3]
   attributes
      Account
        attributes
           Technical_Solution_Engineer__r
               attributes
                  FirstName: Gary

@gaberger Thanks for checking.
This is how I wrote my code to access a field on a custom Lookup relationship. The key is to append "__r" in your golang struct field for a custom relationship. There might be opportunity to improve this (maybe with the go-soql package?).
type Contact struct {
  Id       string
  LastName string
}

type Account struct {
  Id                 string
  Primary_Contact__r Contact
}

type Opportunity struct {
  Id      string
  Account Account
}

opps := []Opportunity{}
err = sf.Query("SELECT Id, Account.Primary_Contact__r.LastName FROM Opportunity WHERE Account.Primary_Contact__c != null", &opps)
if err != nil {
  panic(err)
}
fmt.Println(opps)

Result in my test org:
➜  go-testing go run main
[{006Dn00000EkTNFIA3 { { Stark}}}]

Let me know if that works, and if it doesn't, then please share some code snippets so I can further debug.

@gaberger Were you able to resolve this?


autoRequire for windows users fails for backslash paths

In phoenix, we have reports of windows users' application js not loading https://github.com/phoenixframework/phoenix/issues/1319 , we've tracked it down to here:
https://github.com/brunch/brunch/blob/master/lib/fs_utils/generate.js#L172
our brunch-config looks like this:
  modules: {
    autoRequire: {
      "js/app.js": ["web/static/js/app"]
    }
  },

But for windows users joinKey is "js\app.js" (note backslash)
The result is the autoRequire feature is ignored. I'm not familiar enough with node to know if it should be normalizing paths, but I would be happy to send a fix if you could point me in the right direction. Thanks!

The most obvious solutions to this within brunch could turn out to be problematic for someone else's use case, such as a project developed only by Windows users that already uses backslashes in the path keys.
It's not all that pretty, but a potentially immediate solution to this for a cross-compatible project is
  modules: {
    autoRequire: {
      "js/app.js": ["web/static/js/app"]
      "js\\app.js": ["web/static/js/app"]
    }
  },

Hi,
For now, I don't have a windows machine to check that; but could be this related to the slash config replacement?
https://github.com/brunch/brunch/blob/master/lib/helpers.js#L150
By the way, why do we replace '/' by '' on windows ? According to the node documentation, node handles '/' on windows:
https://nodejs.org/api/path.html#path_path_isabsolute_path
Should not we replace all backslashes by slashes on windows?
(Hoping to not be off topic)

@ghostd that's a good catch actually. That helper was added to address the equivalent issue in other parts of the config. Having it do the same thing for everything within autoRequire should be a good fix.

Should not we replace all backslashes by slashes on windows?

Then we'd have to do the opposite operation in case users use backslashes in their configs. There's more to it than just whether it would work in a node fs or path operation, it needs to match up to the paths that are actually read from the file system by chokidar, which come with backslashes.
Yes, we could probably solve it the opposite way by switching to require('path').POSIX.normalize at https://github.com/brunch/brunch/blob/master/lib/fs_utils/file_list.js#L4 and using it in several more places throughout that file, but I'm not sure if we gain a lot of advantage by making that switch at this point. Maybe it would be better though, preventing any future replaceConfigSlashes whack-a-mole. @paulmillr wdyt?

yeah

@es128 imho, it's more natural for "brunch" dev beginners (like me) to handle paths with slashes than the opposite. A couple of days ago, i wanted to write some unit tests and broke the 'replaceConfigSlashes()' spec with my little refactoring. I needed some minutes to understand brunch converted the path into a non posix way.
But, maybe it's too risky/invasive to make this switch now.

Or maybe you've correctly pointed out that the old approach was a mistake. The next release will be a major version bump, it's a good time to attempt switching it to work the other way.

Fixed.


feat(plugins): implement Alicloud IAM plugin

Hi pak @rahmatrhd, I have already done working with the comments, please kindly help to review again.
Thanks 🙏


[bug] No notifications, but still the (1) badge on app

Hello, after receiving the last notification about wallet migration, the notification badge won't leave.

I tried to reboot the phone, my os is the latest, app version is the latest. iPhone 13 Pro, iOS 17.0.3,  App 2.0.6, nothing worked to make the notification badge disappear.

The badge appears when there's an iOS system notification. Check in the iOS Notification Center for any missed messages. You can also make system notifications more prominent in iOS Settings > Apps > Phoenix > Notifications, and select the Banner options.

I think you misunderstood, it has been days since the notification badge on the phoenix wallet app wont leave, this notification was for the wallet upgrade thing on the last update, and now even after more than a week, multiple reboot, the notification badge is still here


[question] global options

Hi @cesarrew, thanks for this, saved my day!
How can we set global options for an entire app instead of going with the below config for each and every input?
[options]="{ prefix: 'R$ ', thousands: '.', decimal: ','}"

It's not possible. What I do in similar situation is create a service with global objects ("options" in this case) and inject it in every necessary component.


Schema prefix not fetched from query

We were using Scrivener.Ecto version 2.2.0 until today, and passing schema prefix on the query was working like:
query
|> Map.put(:prefix, tenant_name)
|> Repo.paginate(page: page_no)

But with the update to the latest version (2.6.0 as of today), the library has stopped respecting the prefix on the query, which is one of the methods used by Ecto to specify the DB schema of a query.
This is the expected behaviour if we look at #91 , but is this correct? Shouldn't it respect the prefix on the query if no one is passed on the paginate method options?
I can work in a patch for this, but I would like to be sure of the expected behaviour before.

Please do. The new update broke my application. 😞

Apologies, this should now be fixed in 2.8.0.

Thanks @drewolson! <3


rune exec: how to exec new process using docker with rune?

Hi, I am using the docker with rune and WAMR as the enclave runtime. I can run the container successfully, but when i exec the new process in container, it failed:
~/enclave/inclavare-containers ‹master*› » sudo docker ps                                                                                126 ↵
CONTAINER ID   IMAGE          COMMAND           CREATED       STATUS       PORTS     NAMES
13111117dd50   wamr-sgx-app   "/run/rune/app"   3 hours ago   Up 3 hours             cool_sinoussi
f246e9bda16a   ff5fee98b32c   "bash"            2 days ago    Up 2 days              focused_wozniak
~/enclave/inclavare-containers ‹master*› » sudo docker exec -it 13111117dd50 /run/rune/app
OCI runtime exec failed: exec failed: container_linux.go:279: creating new parent process caused "container_linux.go:408: creating child agent pipe caused \"dial unix /var/run/docker/runtime-runc/moby/13111117dd5063b6dfdaad625bbcf4722bab1a9141fc0f887e74a52609baf811/agent.sock: connect: invalid argument\"": unknown

Can you help me with how to exec new precess in the running container?

@lifeng68 This issue is fixed and thus automatically closed. If you have any problem, plz reopen this issue.


test: update OpenProperty with SelectEntityByName

This exercise intents to replace OpenPropertyPane() js helper method with ts helper SelectEntityByName() and other methods on use case basis

/ok-to-test

/ok-to-test

/ok-to-test

/ok-to-test

/ok-to-test

/ok-to-test

/ok-to-test

/ok-to-test

/ok-to-test

/ok-to-test

/ok-to-test


Final ux-writers-edits

Reviewed docs in apiary and made a few final updates

Thank you @madeline-brennan !


My radar screen looks muted

Hi - module looks great, thank you!  one question; the image looks muted/faded on the screen.  I see an opacity setting, but it does not seems to impact the presentation.  Is there another option/method?

comment out the opacity: 0.5; line in MMM-RAIN-RADAR.css.


Version required to install pre-release tools

We found that we needed to specify the 'version' in order to install the report generator in our pipeline
So;
arguments: install --tool-path . dotnet-reportgenerator-globaltool
becomes
arguments: install dotnet-reportgenerator-globaltool --tool-path tools --version 4.4.7

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ec716c61-248c-75c9-2bc7-262749cc7421
Version Independent ID: 45f47edc-e800-2c42-416a-92cd93c9e892
Content: Build, test, and deploy .NET Core apps - Azure Pipelines
Content Source: docs/pipelines/ecosystems/dotnet-core.md
Product: devops
Technology: devops-cicd
GitHub Login: @juliakm
Microsoft Alias: jukullam

Here's the build log error.
"C:\Program Files\dotnet\dotnet.exe" tool install dotnet-reportgenerator-globaltool --tool-path tools
C:\Program Files\dotnet\sdk\3.1.101\NuGet.targets(123,5): error : Failed to retrieve information about 'dotnet-reportgenerator-globaltool' from remote source 'https://pkgs.dev.azure.com/LabCh/7f6101fd-4c9c-472f-b550-518907ae6aab/_packaging/e1f24fbc-156d-432a-b92e-ee33e4f64c0e/nuget/v3/flat2/dotnet-reportgenerator-globaltool/index.json'. [C:\Users\VssAdministrator\AppData\Local\Temp\rwrqzds1.gyh\restore.csproj]
C:\Program Files\dotnet\sdk\3.1.101\NuGet.targets(123,5): error :   Response status code does not indicate success: 401 (Unauthorized). [C:\Users\VssAdministrator\AppData\Local\Temp\rwrqzds1.gyh\restore.csproj]
The tool package could not be restored.
Tool 'dotnet-reportgenerator-globaltool' failed to install. This failure may have been caused by:

* You are attempting to install a preview release and did not use the --version option to specify the version.
* A package by this name was found, but it was not a .NET Core tool.
* The required NuGet feed cannot be accessed, perhaps because of an Internet connection problem.
* You mistyped the name of the tool.

We've updated this article to use DotNetCoreCLI@2 since this issue was filed. Closing issue for now.


PHP 8.2 and 8.3 support

PHP 8.2 and 8.3 support

I have independently updated the package.
Pinging @douglasduarte for awareness.


qa_crowbarsetup: Use m1.tiny when booting testvm instance for docker

It seems that the docker backend doesn't allow disk overcommit.

Looks good, running testsetup now...

So, I failed to execute full testsetup (because I patched wrong qa_crowbarsetup), but I can confirm that nova boot --image cirros --flavor m1.tiny actually works while it fails with --flavor m1.smaller

+1

+1

+1


Some fix typo error

Fixed some typos found when playing the game.

@fv3rdugo, nice finds. Thanks for the corrections!


Ship aperture as binary

This will fix #19 (hopefully) 😄..
The npm package now includes "only" the aperture-compiled-swift binary and the index.js.
Beside of the stuff which is always packed with npm pack like README and license.
I've tested it like this:

run npm pack
uploaded the tgz to github releases
Used these url as dependency in another project:

  "dependencies": {
    "aperture.js": "https://github.com/StefMa/aperture.js/releases/download/v0.1.0/aperture.js-0.1.0.tgz"
  },

run yarn/npm install
run my app which can record the screen with aperture \o/

Rebased and removed the build.sh file... 👍

Thank you :)


It is not possible to save the default value of a CharField in a child object

If a page has a child object with a CharField and that Charfield has choices and a default, it is not possible to save a page with a child object with the default value set as the value of the Charfield.
Tested on 1.0rc2
Steps to reproduce

Add a child model similar to the one below to a page.

class TestModel(models.Model):
    page = ParentalKey('demo.BlogPage', related_name='topics')

    topic = models.CharField(
        max_length=50,
        blank=False,
        default='foo',
        choices=(('foo', 'foo'), ('bar', 'bar'), ('baz', 'baz'))
    )

    panels = [
        FieldPanel('topic')
    ]

Create an instance of the page in the Wagtail admin and add one of the child models. Do not change the default value in the dropdown. Save the page.

View the page in the Wagtail admin again.

-> The child object has not been saved.

This is standard django behaviour, it setups a with default and compares to what you post, in this case of 'foo' this is the same and it tags the form as non changed.
For a standard ModelForm one could override has_changed to always return True

I think it makes sense for Wagtail to override Django's standard behaviour here. This behaviour is necessary for Django's ordinary (non-javascript-enhanced) formsets, where leaving a form unchanged is the only way to denote "don't add it as a child object"; this situation doesn't arise in Wagtail, where you can add/delete forms at will (so it's safe to assume that any forms the editor did submit are ones they wanted to be there).

Reviewed today – assigning to myself to check where else we’ve had that discussion and where it’s at (something something saving unedited InlinePanel entries, client project).


For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f2f86a70f86 in /home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f2f86a1fd10 in /home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f2f86b4bf08 in /home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f2f87d683e6 in /home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7f2f87d6d600 in /home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7f2f87d742ba in /home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f2f87d766fc in /home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7:  + 0xdbbf4 (0x7f2fd550fbf4 in /home/ahs/anaconda3/envs/py310/bin/../lib/libstdc++.so.6)
frame #8:  + 0x76db (0x7f2fd75486db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #9: clone + 0x3f (0x7f2fd6acc61f in /lib/x86_64-linux-gnu/libc.so.6)
[rank3]:[E1030 19:36:07.515488087 ProcessGroupNCCL.cpp:1515] [PG 0 (default_pg) Rank 3] Process group watchdog thread terminated with exception: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f6bcb2adf86 in /home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f6bcb25cd10 in /home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f6bcb388f08 in /home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f6bcc5a53e6 in /home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7f6bcc5aa600 in /home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7f6bcc5b12ba in /home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f6bcc5b36fc in /home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7:  + 0xdbbf4 (0x7f6c19d4cbf4 in /home/ahs/anaconda3/envs/py310/bin/../lib/libstdc++.so.6)
frame #8:  + 0x76db (0x7f6c1bd856db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #9: clone + 0x3f (0x7f6c1b30961f in /lib/x86_64-linux-gnu/libc.so.6)
W1030 19:36:08.253000<PHONE_NUMBER>22304 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 96054 closing signal SIGTERM
E1030 19:36:08.468000<PHONE_NUMBER>22304 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: -6) local_rank: 0 (pid: 96052) of binary: /home/ahs/anaconda3/envs/py310/bin/python
Traceback (most recent call last):
File "/home/ahs/anaconda3/envs/py310/bin/torchrun", line 8, in
sys.exit(main())
File "/home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 348, in wrapper
return f(*args, **kwargs)
File "/home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/distributed/run.py", line 901, in main
run(args)
File "/home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in run
elastic_launch(
File "/home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 133, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
train.py FAILED
Failures:
[1]:
time      : 2024-10-30_19:36:08
host      : ahs-SYS-4029GP-TRTC-ZY001
rank      : 1 (local_rank: 1)
exitcode  : -6 (pid: 96053)
error_file: <N/A>
traceback : Signal 6 (SIGABRT) received by PID 96053
[2]:
time      : 2024-10-30_19:36:08
host      : ahs-SYS-4029GP-TRTC-ZY001
rank      : 3 (local_rank: 3)
exitcode  : -6 (pid: 96055)
error_file: <N/A>
traceback : Signal 6 (SIGABRT) received by PID 96055
Root Cause (first observed failure):
[0]:
time      : 2024-10-30_19:36:08
host      : ahs-SYS-4029GP-TRTC-ZY001
rank      : 0 (local_rank: 0)
exitcode  : -6 (pid: 96052)
error_file: <N/A>
traceback : Signal 6 (SIGABRT) received by PID 96052

There are many possible causes for this error, such as discrepancies between the specified and actual number of GPUs, or an out-of-bounds index somewhere in the code. Without more information, I can't pinpoint the issue. The best solution would be to add the following lines to train.py:
import os
os.environ['CUDA_LAUNCH_BLOCKING'] = '1'

Alternatively, use debug mode to identify the specific location of the problem.

Try not to paste the log directly, this will cause # references all other issues
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f6bcb2adf86 in /home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/lib/libc10.so)
frame https://github.com/Peterande/D-FINE/pull/1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f6bcb25cd10 in /home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/lib/libc10.so)
frame https://github.com/Peterande/D-FINE/issues/2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f6bcb388f08 in /home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame https://github.com/Peterande/D-FINE/issues/3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f6bcc5a53e6 in /home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame https://github.com/Peterande/D-FINE/pull/4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7f6bcc5aa600 in /home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame https://github.com/Peterande/D-FINE/issues/5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7f6bcc5b12ba in /home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame https://github.com/Peterande/D-FINE/issues/6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f6bcc5b36fc in /home/ahs/anaconda3/envs/py310/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame https://github.com/Peterande/D-FINE/issues/7: + 0xdbbf4 (0x7f6c19d4cbf4 in /home/ahs/anaconda3/envs/py310/bin/../lib/libstdc++.so.6)
frame https://github.com/Peterande/D-FINE/issues/8: + 0x76db (0x7f6c1bd856db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame https://github.com/Peterande/D-FINE/issues/9: clone + 0x3f (0x7f6c1b30961f in /lib/x86_64-linux-gnu/libc.so.6)

我也遇到了这个问题，我的原因是因为我自定义的数据集的类别数量在配置文件中写少了，不匹配导致索引超出，然后报了一样的错误

我也遇到了这个问题，我的原因是因为我自定义的数据集的类别数量在配置文件中写少了，不匹配导致索引超出，然后报了一样的错误

感谢告知

There are many possible causes for this error, such as discrepancies between the specified and actual number of GPUs, or an out-of-bounds index somewhere in the code. Without more information, I can't pinpoint the issue. The best solution would be to add the following lines to train.py:
import os
os.environ['CUDA_LAUNCH_BLOCKING'] = '1'

Alternatively, use debug mode to identify the specific location of the problem.

src/data/datasetset/coco_dataset.py:
category2label = kwargs.get('category2label', None)
if category2label is not None:
labels = [category2label[obj["category_id"]] for obj in anno]
else:
labels = [obj["category_id"] for obj in anno]
这里代码建议修改：
category2label = kwargs.get('category2label', None)
if category2label is not None:
labels = [category2label[obj["category_id"]] for obj in anno]
else:
labels = [obj["category_id"] -  for obj in anno]
使用coco格式的数据集，"category_id"自动从1开始的，而labels值需要从0开始，上述报错就是这个原因。

D-FINE/configs/dataset/obj365_detection.yml:
task: detection
evaluator:
type: CocoEvaluator
iou_types: ['bbox', ]
num_classes: 366   # 上面bug就是因为这个类别数设置的问题，应该改成clsnum + 1
remap_mscoco_category: False


Hidden forms do not respond correctly

I am providing a wizard to the user, using forms and tabs. I do not want the user to move from one tab to the next until they correctly complete the first tab. So each tab has its own form. The first form, because it is not hidden works correctly. All of the hidden forms behave in a very weird way.

The submit button is not disabled.
The fields seem to respond correctly. If I blank out a required field it will correct display as an error.

I have attached a simple example
newhtml.zip

Actually, this is probably the same issue as the it not working in a bootstrap Modal window. Hopefully what you do to fix that will also fix this.
I will just have to initialize form2 as part of showing it. If I give the user the ability to move back and forth between the 2 forms. Should I be doing anything special or will re-initializing use the existing instance? I don't want to consume too much browser memory.

I noticed the same issues and i solved it by simply moving the forms to be hidden off the canvas.
This way all the forms are available to the script but not visible to the user.
CSS is along the ways shown below:
#FormContainerToBeHidden { 	left:0; 	position:relative; } .moveAway { 	left:-9999px !important; 	position: absolute !important; }
Hope this helps...

Hidden form fields are no longer excluded in v0.11.0


Add --nocache_test_results to overrideableBazelFlags

Sometimes you're working on an e2e test and you want it to run all the time. The --nocache_test_results flag lets you rerun tests after each rebuild.

@googlebot I signed it!

Thanks for sending in the PR! If you need this as part of a release please ping this thread and I'll do one for you.

Sure, a release would be convenient. Thanks.

I just sent out a release as v0.11.1 thanks for sending in your PR!


Including the methods get_access_control() & set_access_control()

https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/storage/azure-storage-file-datalake/samples/datalake_samples_access_control.py

Docs Build status updates of commit 9dbc86c:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

docs-ref-autogen/azure-storage-file-datalake/azure.storage.filedatalake.DataLakeDirectoryClient.yml
:x:Error

Details

docs-ref-autogen/azure-storage-file-datalake/azure.storage.filedatalake.DataLakeDirectoryClient.yml

Line 103, Column 19: [Error-yaml-syntax-error] While parsing a block mapping, did not find expected key.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

@navba I really appreciate the effort in adding these two missing functions.
Unfortunately, this PR makes changes in an autogenerated file. The very next time that the pipeline that generates everything under docs-ref-autogen/ runs, your changes will be erased. For that reason I won't be merging this PR.
There is a major version update of the yml generator that we currently use. Last I saw it was pretty much complete. Docs team is working on rollout.