[Remote Store] Root cause deleted segment files during remote uploads

Describe the bug
During remote store uploads,  we do GatedCloseable<SegmentInfos> segmentInfosGatedCloseable = indexShard.getSegmentInfosSnapshot() which makes sure that the files we are uploading are not deleted/merged away.
However we have seen in async flow , where in multiple refreshes can happen and the remote store refresh listener is uploading older files . When it tries to update the tracker, it sees that the file has been deleted from local
Stack Trace
[2023-10-30T19:13:39,289][ERROR][o.o.i.s.RemoteDirectory  ] [29713b453747eb14e6f0eb2ae1c4367f] Exception in segment postUpload for file [_12r.si]
java.lang.NullPointerException: Cannot invoke "java.lang.Long.longValue()" because the return value of "java.util.Map.get(Object)" is null
	at org.opensearch.index.shard.RemoteStoreRefreshListener$2.onFailure(RemoteStoreRefreshListener.java:551)
	at org.opensearch.index.shard.RemoteStoreRefreshListener.lambda$uploadNewSegments$5(RemoteStoreRefreshListener.java:400)
	at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90)
	at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:84)
	at org.opensearch.index.store.RemoteDirectory.lambda$uploadBlob$3(RemoteDirectory.java:369)
	at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
	at org.opensearch.core.action.ActionListener$6.onResponse(ActionListener.java:301)
	at org.opensearch.repositories.s3.S3BlobContainer.lambda$asyncBlobUpload$5(S3BlobContainer.java:235)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at org.opensearch.repositories.s3.async.AsyncTransferManager.lambda$uploadInOneChunk$21(AsyncTransferManager.java:362)
	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.lambda$execute$0(AsyncApiCallMetricCollectionStage.java:56)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.lambda$execute$2(AsyncApiCallTimeoutTrackingStage.java:69)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:177)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$null$0(MakeAsyncHttpRequestStage.java:105)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:163)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:858)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
The locked files should not get deleted

The underlying map is backed by concurrent hashmap. If the both the executions are happening simultaneously, it should still be idempotent and should not lead to any entry go missing from the map.

The underlying map is backed by concurrent hashmap. If both the executions are happening simultaneously, it should still be idempotent and should not lead to any entry go missing from the map.

Agree, but the modifications to the map is not happening concurrently, but some millis apart. The problem is the upload is still in progress for the previous refresh segment files, which are removed from the map due to newer refresh flow has already called updateLatestLocalFileNameLengthMap .
ConcurrentHashMap would ensure no two threads modify it at the same time, but we are actually modifying the map few millis apart.
Checkout the logs from integ test where i saw this issue - https://github.com/opensearch-project/OpenSearch/issues/9774#issuecomment-1892192591

The underlying map is backed by concurrent hashmap. If both the executions are happening simultaneously, it should still be idempotent and should not lead to any entry go missing from the map.

Agree, but the modifications to the map is not happening concurrently, but some millis apart. The problem is the upload is still in progress for the previous refresh segment files, which are removed from the map due to newer refresh flow has already called updateLatestLocalFileNameLengthMap .
ConcurrentHashMap would ensure no two threads modify it at the same time, but we are actually modifying the map few millis apart.
Checkout the logs from integ test where i saw this issue - #9774 (comment)

The case you mentioned can happen if there is a retry ongoing coinciding with a refresh/flush. On the solution part, lets not introduce more synchronisations or locks usage, we can deep clone the segment tracker and use it throughout the method's lifecycle.

Ideally refreshes and the afterRefresh listeners are not supposed to be executed in concurrently, due to refreshLock in ReferenceManager.java
https://github.com/apache/lucene/blob/c746bea233fd14fe81d9805633fcce0f7b9681b6/lucene/core/src/java/org/apache/lucene/search/ReferenceManager.java#L161-L186
But with ReleasableRetryableRefreshListener we allow retries to be executed separately from this flow.
https://github.com/opensearch-project/OpenSearch/blob/c132db950548ee99127bc92be8d0f388cc0f4a51/server/src/main/java/org/opensearch/index/shard/ReleasableRetryableRefreshListener.java#L124-L131

Even after upload is completed, another place we are using the local files reference from segmentTracker which could provide incorrect results due to allowed concurrency with retries.
https://github.com/opensearch-project/OpenSearch/blob/c132db950548ee99127bc92be8d0f388cc0f4a51/server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java#L293-L303

The underlying map is backed by concurrent hashmap. If both the executions are happening simultaneously, it should still be idempotent and should not lead to any entry go missing from the map.

Agree, but the modifications to the map is not happening concurrently, but some millis apart. The problem is the upload is still in progress for the previous refresh segment files, which are removed from the map due to newer refresh flow has already called updateLatestLocalFileNameLengthMap .
ConcurrentHashMap would ensure no two threads modify it at the same time, but we are actually modifying the map few millis apart.
Checkout the logs from integ test where i saw this issue - #9774 (comment)

The case you mentioned can happen if there is a retry ongoing coinciding with a refresh/flush. On the solution part, lets not introduce more synchronisations or locks usage, we can deep clone the segment tracker and use it throughout the method's lifecycle.

yeah, not going introduce any more locks as they are not needed. But need to understand if we can skip the the call to updateLocalSizeMapAndTracker in runAfterRefreshExactlyOnce
https://github.com/opensearch-project/OpenSearch/blob/c132db950548ee99127bc92be8d0f388cc0f4a51/server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java#L123-L140
We anyways update this map in syncSegments. Only caveat i can see is the bytes lag computation would be delayed a bit. is there any other use of this call?
The deep clone solution would complicate the code further, ideally the segmenttracker should support addition of new files and removal of files as they are uploaded, instead of just supporting a replacement of local files map.

We anyways update this map in syncSegments. Only caveat i can see is the bytes lag computation would be delayed a bit. is there any other use of this call?

If there is any upload taking a lot of time, the lag would be incorrect and difficult to debug issues. We need this at both the places to ensure that the local refreshes show up asap and if there is retry happening, the local state again reflects correctly if there has been any refreshes performed since the original retry.

The deep clone solution would complicate the code further, ideally the segmenttracker should support addition of new files and removal of files as they are uploaded, instead of just supporting a replacement of local files map.

The local map contains the state as on local. On the approach you mentioned about removal of files once uploaded can lead to memory leak situations as we have seen that segments file name consist of file names that are not present locally in certain circumstances.

@linuxpi  @gbbafna  - the issue mentioned in the description is different from what we have discussed above. The issue that @gbbafna has mentioned happens when the segmentinfos returns a segment file that does not exists locally. This will lead to 2 problems - 1) During the upload, the segment file is missing and the upload fails. 2) After the upload fails, we try to update the segment tracker with the failed upload details and they do not exist.
We still need to handle the above 2 cases - just fyi.

i think its the same issue which @gbbafna has mentioned. The files would still not be deleted which are being uploaded, the only issue was that local segments map in segmentTracker was getting updated prematurely

During the upload, the segment file is missing and the upload fails.
We should have a separate issue for this i believe as GatedClosable is not able to guarantee file wont be removed

After the upload fails, we try to update the segment tracker with the failed upload details and they do not exist.
This is due the same issue we are trying to solve here. If the local files map would have had the file that were being uploaded, it would not have failed.

We anyways update this map in syncSegments. Only caveat i can see is the bytes lag computation would be delayed a bit. is there any other use of this call?

If there is any upload taking a lot of time, the lag would be incorrect and difficult to debug issues. We need this at both the places to ensure that the local refreshes show up asap and if there is retry happening, the local state again reflects correctly if there has been any refreshes performed since the original retry.

The deep clone solution would complicate the code further, ideally the segmenttracker should support addition of new files and removal of files as they are uploaded, instead of just supporting a replacement of local files map.

The local map contains the state as on local. On the approach you mentioned about removal of files once uploaded can lead to memory leak situations as we have seen that segments file name consist of file names that are not present locally in certain circumstances. As discussed, we can have a clone created in the method scope to ensure that the segment tracker is populated correctly.

sounds good. raised a draft PR https://github.com/opensearch-project/OpenSearch/pull/11896

@gbbafna @ashking94 @sachinpkale  Looks like the issue still persists. Re-opening
[2024-02-09T08:36:34,456][WARN ][o.o.i.r.RemoteSegmentTransferTracker] [109734d9693e4e20ca5986a72e60da3c] [grab_185][1] Exception while reading the fileLength of file=segments_j
java.nio.file.NoSuchFileException: /hdd1/mnt/env/root/<REDACTED>var/es/data/nodes/0/indices/00Ftf6TCIORAuST1uEfyBg8A/1/index/segments_j
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
        at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:148)
        at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
        at java.base/java.nio.file.Files.readAttributes(Files.java:1851)
        at java.base/java.nio.file.Files.size(Files.java:2468)
        at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:208)
        at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:70)
        at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:70)
        at org.opensearch.index.remote.RemoteSegmentTransferTracker.lambda$updateLatestLocalFileNameLengthMap$3(RemoteSegmentTransferTracker.java:323)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
        at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
        at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
        at java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1707)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
        at org.opensearch.index.remote.RemoteSegmentTransferTracker.updateLatestLocalFileNameLengthMap(RemoteSegmentTransferTracker.java:320)
        at org.opensearch.index.shard.RemoteStoreRefreshListener.updateLocalSizeMapAndTracker(RemoteStoreRefreshListener.java:482)
        at org.opensearch.index.shard.RemoteStoreRefreshListener.syncSegments(RemoteStoreRefreshListener.java:244)
        at org.opensearch.index.shard.RemoteStoreRefreshListener.performAfterRefreshWithPermit(RemoteStoreRefreshListener.java:153)
        at org.opensearch.index.shard.ReleasableRetryableRefreshListener.runAfterRefreshWithPermit(ReleasableRetryableRefreshListener.java:160)
        at org.opensearch.index.shard.ReleasableRetryableRefreshListener.lambda$scheduleRetry$2(ReleasableRetryableRefreshListener.java:126)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:858)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
[2024-02-09T08:36:34,457][WARN ][o.o.i.s.RemoteStoreRefreshListener] [109734d9693e4e20ca5986a72e60da3c] [grab_185][1] Exception: [java.nio.file.NoSuchFileException: /hdd1/mnt/env/root/<REDACTED>/var/es/data/nodes/0/indices/00Ftf6TCIORAuST1uEfyBg8A/1/index/segments_j] while uploading segment files
java.nio.file.NoSuchFileException: /hdd1/mnt/env/root/<REDACTED>/var/es/data/nodes/0/indices/00Ftf6TCIORAuST1uEfyBg8A/1/index/segments_j
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:181)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:298)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:357)
        at org.apache.lucene.store.MappedByteBufferIndexInputProvider.openInput(MappedByteBufferIndexInputProvider.java:74)
        at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:323)
        at org.opensearch.index.store.FsDirectoryFactory$HybridDirectory.openInput(FsDirectoryFactory.java:181)
        at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
        at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
        at org.opensearch.index.store.RemoteDirectory.calculateChecksumOfChecksum(RemoteDirectory.java:403)
        at org.opensearch.index.store.RemoteDirectory.uploadBlob(RemoteDirectory.java:347)
        at org.opensearch.index.store.RemoteDirectory.copyFrom(RemoteDirectory.java:330)
        at org.opensearch.index.store.RemoteSegmentStoreDirectory.copyFrom(RemoteSegmentStoreDirectory.java:452)
        at org.opensearch.index.shard.RemoteStoreRefreshListener.uploadNewSegments(RemoteStoreRefreshListener.java:434)
        at org.opensearch.index.shard.RemoteStoreRefreshListener.syncSegments(RemoteStoreRefreshListener.java:280)
        at org.opensearch.index.shard.RemoteStoreRefreshListener.performAfterRefreshWithPermit(RemoteStoreRefreshListener.java:153)
        at org.opensearch.index.shard.ReleasableRetryableRefreshListener.runAfterRefreshWithPermit(ReleasableRetryableRefreshListener.java:160)
        at org.opensearch.index.shard.ReleasableRetryableRefreshListener.lambda$scheduleRetry$2(ReleasableRetryableRefreshListener.java:126)


compute_chunk_sizes could be more efficient for arrays with >1 dimension

MCVE: https://colab.research.google.com/drive/1KNlfPiZECPcCestEh3gzpDrB9mvbbN2H?usp=sharing
In the above example, after indexing a 3d dask array with a 1d dask array via da.compress(), I would like to then compute the chunk sizes. Running compute_chunk_sizes() takes 44 s. Manually computing the chunk sizes requires ~200 ms.
Looking at the source code for compute_chunk_sizes() I suspect the inefficiency comes because it is mapping a function across all blocks in the array, whereas it only needs to compute the chunk sizes along the margin of each dimension. I.e., a lot of work is being duplicated.
Btw this is a usability issue that's come up in working with large genomic datasets. When there are multiple large n-d arrays that need to be indexed and have their chunk sizes computed, it is inconvenient to have to wait minutes for that operation to complete. This came up in the course of investigating https://github.com/pydata/xarray/issues/4663.
Environment:

Dask version: 2021.3.0
Python version: 3.6
Operating System: Ubuntu
Install method (conda, pip, source): pip

Thanks for bringing this up @alimanfoo! I agree that the current implementation could be improved by just computing chunksize across the margins of each dimension. FWIW we do a similar optimization for computing the length of a Dask DataFrame (i.e. we only compute the length of a single column).

cc @gjoseph92 as this issue may interest you

Thanks for pointing this out @alimanfoo. I agree that there's probably a more efficient way to compute the chunksizes. A couple notes:

I might be wrong here, but I think we actually are already only computing along the margin/basis of each dimension: yes, we map_blocks, but then for each axis, we slice out the chunk shapes at index 0 of all other axes (key here is the [0] (I also find this code very confusing to read FYI)):
https://github.com/dask/dask/blob/b288de7d448ba8f05a93ea5cee19807740b86249/dask/array/core.py#L1264-L1270
I think this means we only end up referencing the blocks along the "edges" of the chunk_shapes, not the "center", so when we cull the graph in compute, unnecessary keys of the chunk_shapes array—and therefore the keys of the array we're calling compute_chunk_sizes on—won't be computed.
Your approach is clever, but because you're computing over the indexer, versus the array that has NaNs in its chunks, I think it would be hard for dask to use in generality. I really like the approach of only needing to compute the indexer, but just given some random array with NaNs in its chunks, you can't necessarily go back and find the thing that indexed it, compute it, then propagate those known chunks forward though whatever other operations came subsequently (without, at least, some very complex and likely impossible graph introspection).

Basically, if you need to have known chunksizes, arr[idx.compute()] is probably always going to be faster than arr[idx].compute_chunk_sizes(), since you have some special information here that dask can't determine: idx is the only thing you need to compute to determine the final chunk sizes. This may be worth considering for xarray, while unknown shapes still aren't supported.
Still, there's room to improve: we're computing chunk sizes even for axes where we already know them. In fact, even when there are no NaNs at all in the chunks, this method will still compute everything. The docstring says "This is especially useful when the chunk sizes are unknown" as though there are other situations in which you'd want to call it; perhaps what we actually want is "This fills in NaNs in chunksizes, if there are any".
Another little thought: once compute_chunk_sizes has gone to the trouble of computing all this, it would be kinda nice if it could cull blocks from the Array that we now know aren't used. For example:
nothing = x[da.full(x.shape[:1], False)]
# determine that there are going to be 0 elements in `nothing`
nothing.compute_chunk_sizes()
# yet this still requires computing all of `x` again, just to toss it out
nothing.compute()

Thanks Gabe, appreciate there is some complexity here.
Just wanted to mention that arr[idx.compute()] is not an option for me
unfortunately. Because idx is of the order of 50MB computing it then using
as an argument causes a blow up of memory usage on the scheduler. That is
in fact what I was originally doing, then found I was running out of
memory, then started exploring indexing with the dask array, then hit this
issue :-)
On Fri, 19 Mar 2021, 23:01 Gabe Joseph, @.***> wrote:

Thanks for pointing this out @alimanfoo https://github.com/alimanfoo. I
agree that there's probably a more efficient way to compute the chunksizes.
A couple notes:

I might be wrong here, but I think we actually are already only
computing along the margin/basis of each dimension: yes, we map_blocks,
but then for each axis, we slice out the chunk shapes at index 0 of all
other axes (key here is the [0] (I also find this code very confusing
to read FYI)):

https://github.com/dask/dask/blob/b288de7d448ba8f05a93ea5cee19807740b86249/dask/array/core.py#L1264-L1270
I think this means we only end up referencing the blocks along the
"edges" of the chunk_shapes, not the "center", so when we cull the
graph in compute, unnecessary keys of the chunk_shapes array—and
therefore the keys of the array we're calling compute_chunk_sizes
on—won't be computed.

Your approach is clever, but because you're computing over the
indexer, versus the array that has NaNs in its chunks, I think it
would be hard for dask to use in generality. I really like the approach of
only needing to compute the indexer, but just given some random array with
NaNs in its chunks, you can't necessarily go back and find the thing that
indexed it, compute it, then propagate those known chunks forward though
whatever other operations came subsequently (without, at least, some very
complex and likely impossible graph introspection).

Basically, if you need to have known chunksizes, arr[idx.compute()] is
probably always going to be faster than arr[idx].compute_chunk_sizes(),
since you have some special information here that dask can't determine:
idx is the only thing you need to compute to determine the final chunk
sizes. This may be worth considering for xarray, while unknown shapes still
aren't supported.
Still, there's room to improve: we're computing chunk sizes even for axes
where we already know them. In fact, even when there are no NaNs at all in
the chunks, this method will still compute everything. The docstring says
"This is especially useful when the chunk sizes are unknown" as though
there are other situations in which you'd want to call it; perhaps what we
actually want is "This fills in NaNs in chunksizes, if there are any".
Another little thought: once compute_chunk_sizes has gone to the trouble
of computing all this, it would be kinda nice if it could cull blocks from
the Array that we now know aren't used. For example:
nothing = x[da.full(x.shape[:1], False)]
determine that there are going to be 0 elements in nothing
nothing.compute_chunk_sizes()
yet this still requires computing all of x again, just to toss it out
nothing.compute()
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/7416#issuecomment-803180836, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFLYQTD7NAIPEG7MFUFOG3TEPJVXANCNFSM4ZOQYEAQ
.

Because idx is of the order of 50MB computing it then using as an argument causes a blow up of memory usage on the scheduler

Interesting. 50MB is big, but it's not that big. I wonder we should instead focus our efforts on fancy indexing (take) performance with large indexers.
It's interesting to me that the memory is blowing up on the scheduler specifically, but neither on your client process (as it generates the graph for arr[idx.compute()]), nor on the workers as they actually compute it? Currently, there's no high-level graph for slicing, so invoking arr[idx.compute()] on the client will generate the whole graph up-front—basically, the memory usage of that should be the same on both the client and the scheduler. Does your scheduler have less memory than the client machine?
This is a bit of a shot in the dark, but I'm curious if you see the same memory blow-up with:
computed_idx = idx.compute()
with dask.config.set(**{'array.slicing.split_large_chunks': False}):
    sliced = arr[computed_idx]


Deploy new version with logger and box score bye bug

Codecov Report

Merging #67 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #67   +/-   ##
=======================================
  Coverage   95.84%   95.84%
=======================================
  Files          13       13
  Lines         577      577
=======================================
  Hits          553      553
  Misses         24       24

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8c2ac9f...52693e4. Read the comment docs.


docs.rs build doc failed

Bug Summary

docs.rs failed to build ibc-0.44.1
Please check the [build logs](https://docs.rs/crate/ibc/0.44.1/builds) for more information.
See [Builds](https://docs.rs/about/builds) for ideas on how to fix a failed build, or [Metadata](https://docs.rs/about/metadata) for how to configure docs.rs builds.
If you believe this is docs.rs' fault, [open an issue](https://github.com/rust-lang/docs.rs/issues/new/choose).

https://docs.rs/crate/ibc/0.44.1
Details

Version

Interesting, looking at it quickly, I can't reproduce the issue.

This is resolved through this PR: https://github.com/cosmos/ibc-rs/pull/876


[Go][CI] Use apache/arrow-go for integration test

Describe the enhancement requested
After apache/arrow-go works.
We should use apache/arrow-go like apache/arrow-rs: https://github.com/apache/arrow/blob/58415d1fac50cb829b3dcf08526033d6db8c30db/.github/workflows/integration.yml#L72-L76
Component(s)
Continuous Integration, Go

Issue resolved by pull request 44142
https://github.com/apache/arrow/pull/44142


Add gte/lte functionality to job-results created and completed timestamps.

Environment

Python version: 3.7
Nautobot version: v1.0.3

Proposed Functionality
Implement lookup expressions for greater-than and less-than for the created and completed fields in the JobResultFilterSet.
Something similar to the implementation that was done on CreatedUpdatedFilterSet
Use Case
As Albert the API user, I would like to be able to query job results for the last x number of days or hours.
I would know this is complete when I can query something similar to /extras/job-results?created__gte=2021-10-29
Database Changes
None
External Dependencies
None

Looking at the code more closely, it may be that changing the filter definition to use MultiValueDateTimeFilter instead of DateTimeFilter for these fields may make this "just work". See BaseFilterSet in nautobot/utilities/filters.py.

This was implemented some time ago.


Inconsistent behaviour when merging maps

I'm pasting a test case that shows inconsistent behaviour when merging maps. I hope it's clear enough. This problem appeared with commit 65847eaa22876897dd35728eb47c4009972e1b17  that fixed issue #23.
type typeA struct {
    A *typeB
}

type typeB struct {
    B map[string]typeC
}

type typeC struct{ C string }

func TestInconsistentMergeBehaviourOnMaps(t *testing.T) {
    dst := typeA{
        A: &typeB{
            B: map[string]typeC{"C": typeC{"value"}},
        },
    }
    src := typeA{
        A: &typeB{
            B: map[string]typeC{},
        },
    }
    MergeWithOverwrite(&dst.A.B, src.A.B)
    if dst.A.B["C"].C != "value" {
        t.Error("Bad merge (1)") // No problem here.
    }
    MergeWithOverwrite(&dst, src)
    if dst.A.B["C"].C != "value" {
        t.Error("Bad merge (2)") // Problem here.
    }
}

I'll check this during next week, as I'm out of office until Monday.

First of all, sorry again. Second, I will take care of this this next week.

I guess it doesn't matter anymore but I checked this today, almost two years later.
This behavior is confusing but expected. I wrote a test to show what happens with structs, as we try to keep merging behaviors coherent between types.
type typeD struct {
        A *typeE
}

type typeE struct {
        B typeC
}

func TestMergeBehaviourOnStruct(t *testing.T) {
        dst := typeD{
                A: &typeE{
                        B: typeC{"value"},
                },
        }
        src := typeD{
                A: &typeE{
                        B: typeC{},
                },
        }
        Merge(&dst.A.B, src.A.B, WithOverride)
        if dst.A.B.C != "value" {
                t.Error("Bad merge (1)") // No problem here.
        }
        Merge(&dst, src, WithOverride)
        if dst.A.B.C != "value" {
                t.Error("Bad merge (2)") // Problem here.
        }
}

This fails in the second mergo too, as dst.A.B.C is "". The main point is the merging scope (undocumented concept, I'm just creating it now). When you merge a map or a struct, we compare each key/field value. If it's non-zero, we override the destination key/field if it's zero. We override it anyway if we use MergeWithOverwrite or Merge with param WithOverride.
When you merge a struct which contains a map or a struct, the comparison is made at field level, the one that holds the map or the struct. If the stated condition is true, we overwrite it as a whole.
If it worked in a previous version I wouldn't assume it was ok. Mergo is very tricky and two years ago was far from complete. Also, the pointed commit isn't related to map or struct merging.
So, as it would make even more trickier to handle this in a more fine-grained way and it works in a consistent in both maps and struct, I'm closing this issue.


RequireJS + Phonegap Developer App problem

Hello!
I'm trying to use RequireJS in Phonegap Application. It works great on real device with applicaiton built using Phonegap cloud building, but sometimes does not work using Phonegap Developer App.
(tested with iOS and Android 5).
That is simple project I created:
https://drive.google.com/file/d/0B2GGYFr2TYMNcmI0MUE5R3dfb0U/view?usp=sharing
Could you please help me with this problem?
Thanks in advance!

I found the same issue this morning using iphone 6.
Any solutions for this problem?
Thanks in advance!

Hi.
We investigated this problem a bit and found out that it's caused by script socket.io.js which is appended to body by phonegap serve.
So it's not strictly phonegap-app-developer issue.
It uses require.js when available and that causes "Mismatched anonymous define()" error. http://requirejs.org/docs/errors.html#mismatch
Solution is described here: http://stackoverflow.com/questions/15371918/mismatched-anonymous-define-module
"load the non-require.js standalone bundles in script tags before require.js is loaded"

@EugeneBakin thanks for tracking this down. I can likely fix this issue if you need to have socket.io.js loaded before the other scripts.

@simar88 Another possible way to work around it - is to namespace your require related functions: http://requirejs.org/docs/faq-advanced.html#rename

@mwbrooks
It should be executed before require.js defines its functions.
If not it would use define and would not expose itself as a global variable.

The bug is still present but i managed to fix it using setTimeout()'s and script injection.
Also you should extend the console object instead of overwriting it with socket.io wrappers.
I relied on console.groupCollapsed() and since you removed it my code broke.
For everyone else having these problems:
You can find the code that monkey-patches my App below.
Add it as the first script to your <head> and it should just work™
function injectRequirejs() {
    var h = document.getElementsByTagName('head')[0];
    var s = document.createElement('script');
    s.type = 'text/javascript';
    s.src = 'bower_components/requirejs/require.js';
    s.setAttribute('data-main', 'js/Main.js');
    h.appendChild(s);
}

function fixConsole(callback) {
    if (window.console.log.toString().match(/.*socket.*/gi) !== null) {
        window.console.log = function (msg, format) {
            __console__.log(msg, format);
        };

        window.console.groupCollapsed = function (msg, template) {
            __console__.groupCollapsed(msg, template);
        };

        window.console.groupEnd = function () {
            __console__.groupEnd();
        };

        callback();
    } else {
        setTimeout(function () {
            fixConsole();
        }, 500);
    }
}

(function () {
    window.__console__ = window.console;

    setTimeout(function () {
        if (typeof window.phonegap !== typeof undefined) {
            console.log('Monkey-patching phonegap... (This may take some time)');
            fixConsole(injectRequirejs);
        } else {
            injectRequirejs();
        }
    }, 1000);
})();


 All of the options are driven by command-line flags. These updates are backward-compatible and the default behavior of the nightly.php script is not changed.
The new options are:

--tab : provides the output in tab-delimited text format
--report-inits : lets the user define which initiatives to report on using the name of the initiative, e.g.:

--report-inits='Head Counts'
--report-inits='Head Counts','Reference Transactions'

--report-date : select an individual date for reporting (default is still 'yesterday'), e.g.:

'2017-12-03'
'1 year ago'
any other date parseable by strtotime()

--omit-header : omit the column headers and initiative labels from the output
--prepend-date : prepends the report date to the beginning of each line

Many of these additions are envisioned for using together, e.g.:
php nightly.php --prepend-date --omit-header --report-init="Head Counts" --tab
Will provide tab-delimited output for a single initiative on a single date with the date as the first column in the output. Suitable for concatenating several days outputs, when combined with, for instance, a shell script that dynamically requests several individual dates in series.
In addition to these new features, this pull request also fixes issue #74, eliminating a PHP Notice when the --hide-zeros option was invoked.

Thanks for this pull request Ken! I'm out of the office for a few days, but will try to review this late next week or early the week after.

Great – thanks
Ken
From: Bret Davidson<EMAIL_ADDRESS>Sent: Thursday, March 08, 2018 3:00 PM
To: suma-project/Suma<EMAIL_ADDRESS>Cc: Ken Irwin<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [suma-project/Suma] More report options (nightly.php) (#75)
Thanks for this pull request Ken! I'm out of the office for a few days, but will try to review this late next week or early the week after.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/suma-project/Suma/pull/75#issuecomment-371605979, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABzooUoA7Z7M-bT_oR9a55mdPVwyQ2v3ks5tcY2lgaJpZM4SjQC1.


ImportError: cannot import name collections_abc

Hi, i install for 3.0.3 and this is the error when test the wsgidav,
Traceback (most recent call last):
  File "/usr/local/bin/wsgidav", line 5, in <module>
    from wsgidav.server.server_cli import run
  File "/usr/local/lib/python2.7/dist-packages/wsgidav/server/server_cli.py", line 38, in <module>
    from wsgidav import __version__, util
  File "/usr/local/lib/python2.7/dist-packages/wsgidav/util.py", line 12, in <module>
    from wsgidav import compat
  File "/usr/local/lib/python2.7/dist-packages/wsgidav/compat.py", line 15, in <module>
    from six.moves import (
ImportError: cannot import name collections_abc

Not: I test install 3.0.2 and use #172 to resolve the error and run, but when i reinstall with 3.0.3, the above error print.
Here is other version:
python3 -m pip show six
Name: six
Version: 1.14.0
Summary: Python 2 and 3 compatibility utilities
Home-page: https://github.com/benjaminp/six
Author: Benjamin Peterson
Author-email<EMAIL_ADDRESS>License: MIT
Location: /usr/local/lib/python3.7/dist-packages
Requires:
Required-by: WsgiDAV, cheroot

3.0.3 is not yet released, did you try the latest code from master, e.g.
pip install git+https://github.com/mar10/wsgidav.git@master

Itest it just now，still not work.
wsgidav --version -v Traceback (most recent call last):   File "/home/pi/.local/bin/wsgidav", line 5, in <module>     from wsgidav.server.server_cli import run   File "/home/pi/.local/lib/python2.7/site-packages/wsgidav/server/server_cli.py", line 38, in <module>     from wsgidav import __version__, util   File "/home/pi/.local/lib/python2.7/site-packages/wsgidav/util.py", line 12, in <module>     from wsgidav import compat   File "/home/pi/.local/lib/python2.7/site-packages/wsgidav/compat.py", line 15, in <module>     from six.moves import ( ImportError: cannot import name collections_abc

I assume you installed to a macOS system Python?
In this case the system's six is not updated to 1.13+. You can either do this manually or install it to a virtual env instead, as described in the docs.
Anyway: I added a fix that removes the dependency, so v3.0.3 it should work now with a six version 1.12.
Until 3.0.3 is released, this could work for you:
sudo pip install git+https://github.com/mar10/wsgidav -U

I install it on a rsapberry, and the six is 1.14.0.

Can you share more information like the output of your installation command and pip list? What does which wsgidav print?

pip install cheroot DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support Defaulting to user installation because normal site-packages is not writeable Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple Collecting cheroot   Downloading cheroot-8.3.0-py2.py3-none-any.whl (86 kB)      |███▉                            | 10 kB 7.6 MB/s eta 0     |███████▋                        | 20 kB 338 kB/s eta 0     |███████████▍                    | 30 kB 489 kB/s eta 0     |███████████████▏                | 40 kB 342 kB/s eta 0     |███████████████████             | 51 kB 420 kB/s eta 0     |██████████████████████▉         | 61 kB 500 kB/s eta 0     |██████████████████████████▋     | 71 kB 574 kB/s eta 0     |██████████████████████████████▍ | 81 kB 648 kB/s eta 0     |████████████████████████████████| 86 kB 214 kB/s Requirement already satisfied: backports.functools-lru-cache; python_version < "3.3" in /usr/lib/python2.7/dist-packages (from cheroot) (1.5) Collecting jaraco.functools   Downloading https://www.piwheels.org/simple/jaraco-functools/jaraco.functools-3.0.0-py2.py3-none-any.whl (6.7 kB) ERROR: Package 'jaraco.functools' requires a different Python: 2.7.16 not in '>=3.6'

`pi@raspberrypi:~ $ pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package                       Version

arandr                        0.1.9
asn1crypto                    0.24.0
automationhat                 0.2.0
backports.functools-lru-cache 1.5
beautifulsoup4                4.7.1
blinker                       1.4
blinkt                        0.1.2
buttonshim                    0.0.2
Cap1xxx                       0.1.3
certifi                       2018.8.24
chardet                       3.0.4
Click                         7.0
colorama                      0.3.7
colorzero                     1.1
configparser                  3.5.0b2
cookies                       2.2.1
cryptography                  2.6.1
defusedxml                    0.6.0
dnspython                     1.16.0
drumhat                       0.1.0
entrypoints                   0.3
enum34                        1.1.6
envirophat                    1.0.0
ExplorerHAT                   0.4.2
Flask                         1.0.2
fourletterphat                0.1.0
funcsigs                      1.0.2
gpg                           1.12.0
gpiozero                      1.5.1
html5lib                      1.0.1
idna                          2.6
ipaddress                     1.0.17
itsdangerous                  0.24
Jinja2                        2.10
json5                         0.9.1
keyring                       17.1.1
keyrings.alt                  3.1.1
lxml                          4.3.2
MarkupSafe                    1.1.0
microdotphat                  0.2.1
mock                          2.0.0
mote                          0.0.4
motephat                      0.0.2
numpy                         1.16.2
oauthlib                      2.1.0
olefile                       0.46
pantilthat                    0.0.7
pbr                           4.2.0
phatbeat                      0.1.1
pianohat                      0.1.0
picamera                      1.13
picraft                       1.0
piglow                        1.2.5
pigpio                        1.44
Pillow                        5.4.1
pip                           20.0.2
psutil                        5.5.1
pycairo                       1.16.2
pycrypto                      2.6.1
pygame                        1.9.4.post1
PyGObject                     3.30.4
pyinotify                     0.9.6
PyJWT                         1.7.0
pyOpenSSL                     19.0.0
pyserial                      3.4
pyxdg                         0.25
PyYAML                        5.3
rainbowhat                    0.1.0
requests                      2.21.0
requests-oauthlib             1.0.0
responses                     0.9.0
RPi.GPIO                      0.7.0
RTIMULib                      7.2.1
scrollphat                    0.0.7
scrollphathd                  1.2.1
SecretStorage                 2.3.1
sense-hat                     2.2.0
setuptools                    40.8.0
simplejson                    3.16.0
six                           1.12.0
skywriter                     0.0.7
sn3218                        1.2.7
soupsieve                     1.8
spidev                        3.4
touchphat                     0.0.1
twython                       3.7.0
unicornhathd                  0.0.4
urllib3                       1.24.1
webencodings                  0.5.1
Werkzeug                      0.14.1
wheel                         0.32.3
WsgiDAV                       3.0.2`

sudo pip install git+https://github.com/mar10/wsgidav -U

?

pi@raspberrypi:~ $ sudo pip install git+https://github.com/mar10/wsgidav -U DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple Collecting git+https://github.com/mar10/wsgidav   Cloning https://github.com/mar10/wsgidav to /tmp/pip-req-build-CKzzXG   Running command git clone -q https://github.com/mar10/wsgidav /tmp/pip-req-build-CKzzXG   Installing build dependencies ... done   Getting requirements to build wheel ... done   Installing backend dependencies ... done     Preparing wheel metadata ... done Requirement already satisfied, skipping upgrade: PyYAML in /usr/local/lib/python2.7/dist-packages (from WsgiDAV==3.0.3) (5.3) Requirem
After this，i use pip list，the wsgidav is still 3.0.2

Seems your output is incomplete, also please use three ticks instead of one to format the output in this comments.

also: what does which wsgidav print?

'''
pi@raspberrypi:~ $ sudo pip install git+https://github.com/mar10/wsgidav -U
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting git+https://github.com/mar10/wsgidav
Cloning https://github.com/mar10/wsgidav to /tmp/pip-req-build-CKzzXG
Running command git clone -q https://github.com/mar10/wsgidav /tmp/pip-req-build-CKzzXG
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing wheel metadata ... done
Requirement already satisfied, skipping upgrade: PyYAML in /usr/local/lib/python2.7/dist-packages (from WsgiDAV==3.0.3) (5.3)
Requirement already satisfied, skipping upgrade: Jinja2 in /usr/lib/python2.7/dist-packages (from WsgiDAV==3.0.3) (2.10)Requirement already satisfied, skipping upgrade: defusedxml in /usr/local/lib/python2.7/dist-packages (from WsgiDAV==3.0.3) (0.6.0)
Requirement already satisfied, skipping upgrade: json5 in /usr/local/lib/python2.7/dist-packages (from WsgiDAV==3.0.3) (0.9.1)
Requirement already satisfied, skipping upgrade: six in /usr/lib/python2.7/dist-packages (from WsgiDAV==3.0.3) (1.12.0)
Building wheels for collected packages: WsgiDAV
Building wheel for WsgiDAV (PEP 517) ... done
Created wheel for WsgiDAV: filename=WsgiDAV-3.0.3-py2.py3-none-any.whl size=169532 sha256=7c205a804c14b3cee095a8141b05277a7b7ff6cb86cc0f29302dd464437f5722
Stored in directory: /tmp/pip-ephem-wheel-cache-s1jv5K/wheels/45/bb/eb/f41d9d9a9cfaa155728c48076e2ca8422765f68b4ec64ed8d0
Successfully built WsgiDAV
Installing collected packages: WsgiDAV
Attempting uninstall: WsgiDAV
Found existing installation: WsgiDAV 3.0.3
Uninstalling WsgiDAV-3.0.3:
Successfully uninstalled WsgiDAV-3.0.3
Successfully installed WsgiDAV-3.0.3
pi@raspberrypi:~ $ wsgidav --version -v                     Traceback (most recent call last):
File "/home/pi/.local/bin/wsgidav", line 5, in
from wsgidav.server.server_cli import run
File "/home/pi/.local/lib/python2.7/site-packages/wsgidav/server/server_cli.py", line 38, in
from wsgidav import version, util
File "/home/pi/.local/lib/python2.7/site-packages/wsgidav/util.py", line 12, in
from wsgidav import compat
File "/home/pi/.local/lib/python2.7/site-packages/wsgidav/compat.py", line 15, in
from six.moves import (
ImportError: cannot import name collections_abc
'''

pi@raspberrypi:~ $ which wsgidav /home/pi/.local/bin/wsgidav

It seems that you install to /home/pi/.local/bin/wsgidav but the libs are searched in /usr/local/lib/python2.7/dist-packages
Could you try
sudo pip install -U six

and share the output?

pi@raspberrypi:~ $ sudo pip install -U six DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple Collecting six   Using cached six-1.14.0-py2.py3-none-any.whl (10 kB) Installing collected packages: six   Attempting uninstall: six     Found existing installation: six 1.12.0     Uninstalling six-1.12.0:       Successfully uninstalled six-1.12.0 Successfully installed six-1.14.0


Adding original name and ID to society detail page

Current coverage is 91.18%

Merging #278 into master will not affect coverage as of f17bc74

@@            master    #278   diff @@
======================================
  Files           35      35
  Stmts         1826    1826
  Branches         0       0
  Methods          0       0
======================================
  Hit           1665    1665
  Partial          0       0
  Missed         161     161

Review entire Coverage Diff as of f17bc74

Powered by Codecov. Updated on successful CI builds.


Java 8 compatibility of 1.5.0

Describe the bug
Using datafaker 1.5.0 with java 8 throws java.lang.UnsupportedClassVersionError: net/datafaker/Faker has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
To Reproduce
Use net.datafaker:datafaker:1.5.0 with Java 8
Expected behavior
That the library can be used with Java 8 as that is stated on the Readme: This library is a modern port of java-faker, built on Java 8, with up to date libraries and several newly added Fake Generators.
Versions:

OS: Linux
JDK 1.8
Faker Version: 1.5.0

Additional context
Maybe release a 1.5.1 version that is compiled with target Java 8.

I think that's my bad, I probably released it locally from a Java 11 machine instead of using the github action.
Thanks for reporting it, I'll address it soon.

Also in need of that compatibility. Thanks for bringing it up @matrei

Can you please try the 1.6.0-SNAPSHOT version, and let me know if that one is 1.8 compatible? (It should be)

I have now tested 1.6.0-SNAPSHOT and this version is 1.8 compatible.

Thanks for testing. Normally I release the versions through github actions, but for 1.5 I released it from my local machine using the maven release plugin, but I think I used 11 by accident. Thanks for testing it!

It should be fixed now, please use version 1.6.0, released today.


Set value inside ng-repeat

Hi, when i try to output a slider outside the looping div, its outputting the correct value.
But when i tried to put it inside ng-repeat, the blue circle always stay in the zero value. But when you click on it, the circle will move to the specified value. I hope you can help me on this one.

I am also having this issue

Could you please provide a fiddle showing this issue?
You can try to force redraw with rzSliderForceRender event:
$scope.$broadcast('rzSliderForceRender');


Find user by birthday

As a user, I can find users by birthdays so I know whose birthday is on that month.

so I know whose birthday is on that month.

Does that mean they do something like find 1, find 2 for people born in January and February respectively?

It would either be first three letters of the month or a number from 1-12. Which do you think will be better?

Also, I think for /find commands we could start to use prefixes as well so we can identify what are we actually looking for. E.g. find t/ | find b/ | find n/

i think can be both cause its not like theres many way to represent a month. also dates are stored in a fixed format already i.e. 01 02 03....12 s
maybe can have a case thing for each month e.g case 1: if (input == 2 || input == 02 || input == "feb" || input == "february") return all the feb babies
SHould have 12 cases plus the base case that is invalid input.

I agree that if we allow a user to find something other than names, we will have to add prefixes. e.g

find t/friend
find n/David
find b/January

Or we can be even more complex:

find n/David t/friend

that returns person that has "David" in their name and with a "friend" tag. What do you think?


Add configuration option to control what host is allowed to use FakeAuth

Use case
Building end-to-end testing such that writing e2e tests via Cypress (e.g.) I wouldn't have to instrument logging in via Azure AD, plus I wouldn't have to administer a testing-only AD account (e.g.) in order to validate that my app works correctly when logged in as a certain kind of user. The intent is to control it similar to this so that FakeAuth is only in play when it's needed:
var authBuilder = app.AddAuthentication();
var fakeAuthHost = Configuration["FakeAuthHost"];
if(!string.IsNullOrEmpty(fakeAuthHost)
{
    authBuilder.AddFakeAuth(opts => {
        opts.AllowedHost = fakeAuthHost;
    });
}

I'll likely use something like backend for the FakeAuthHost setting to match what I've configured in, say, a docker-compose.yml file.
I changed the claims serialization as well so that we can set the claims in a textual way instead of the binary serialization scheme that it was using.

that build failure makes no sense - aren't I supposed to target Develop?

"That build failure makes no sense"
.. agreed. I need to refactor the Workflows so that regression tests run, but right now it forces a fail if it's not actually on the develop branch. My GH Action -fu is weak sauce.
I like the idea of including a list of hosts, other than local. Much better than an on / off flag. I'm trying to think of scenarios where wild cards would be helpful in this scenario.. or if that opens up FA to too many opportunities for abuse.
I'll have time to take look more closely on this later today.

Fix the GH action thing too 😉

@floyd-may I think this CI build to Nuget was published from the PR, it should only be published when a PR is merged to Develop. 😳 https://www.nuget.org/packages/FakeAuth/1.2.0-ci-61

@calebjenkins this ought to be ready to go. Also performed necessary incantations to get the ci actions workflow to only push to NuGet on the Develop branch.


why the ResNet backbone not frozen completely？

Thanks for your great work.
When I read your code, I find the network optimize the backbone's Layer 2 and Layer 3. Why you do it?
Looking forward to your reply.

Thanks for your great work. When I read your code, I find the network optimize the backbone's Layer 2 and Layer 3. Why you do it? Looking forward to your reply.

The SSP model only has a backbone which has learnable parameters. If you freeze it, there is no learnable parameters.


"strict mode" doesn't get enabled for nodejs script when runing through forever

I have a script that uses strict mode, it's an executable script with a "#!/usr/bin/node --harmony" shebang and runs fine when I directly call it from the terminal. I use this line at the start of the script to enable strict mode:
#!/usr/bin/node --harmony
"use strict";

However, if I run it through forever (forever start script_name) I obtain the following error in the logs that causes the script to abort over and over:
		let report = {
        ^^^
SyntaxError: Unexpected strict mode reserved word
  at Module._compile (module.js:439:25)
  at Object.Module._extensions..js (module.js:474:10)
  at Module.load (module.js:356:32)
  at Function.Module._load (module.js:312:12)
  at Function.Module.runMain (module.js:497:10)
  at startup (node.js:119:16)
  at node.js:906:3
error: Forever detected script exited with code: 8

Is there some way for me to indicate to forever that I want to run the nodejs script in strict mode? Or how can I specity commandline arguments for the node binary when running forever?
EDIT: I also tried doing this:
if(process.argv[2] == "start") {
    require("forever").start(__filename, {
           'uid': 'script_name',
           'minUptime': 500,
           'spinSleepTime': 0.5 * 60 * 1000,
           'command': 'node',
           'args': [ '--use_strict', '--harmony' ]
      });
}

This didn't work either, I get the same result.
EDIT2: Funnily enough, I found it works when I use 'perl' as the command interpreter...
if(process.argv[2] == "start") {
    require("forever").start(__filename, {
           'uid': 'script_name',
           'minUptime': 500,
           'spinSleepTime': 0.5 * 60 * 1000,
           'command': 'perl'
      });
}

Looks like perl executes the file correctly after reading the shebang.

Found my way here trying to do the same thing; specifically, I want to pass --use_strict --harmony_arrow_functions to the node process.

@Ferk this appears to do what we want:
"start": "node --use_strict --harmony_arrow_functions ./node_modules/.bin/forever ./bin/www"

However, forever itself appears to shit the bed when it's run in strict-mode :person_frowning:

Yes. Either that or just do what most folks I know do and put "use strict" at the top of every one of your source code files.

"use strict" is not enough, like it's shown in the description the string was there and it was running fine when executed through the terminal, but the problem with 'let' was still happening when running though forever. At least last time I tried.

I figured out the following incantation to make things work; This is working in production for the Heroku dashboard w/iojs:
"scripts": {
  "start": "node ./node_modules/.bin/forever -c 'node --use_strict --harmony_arrow_functions' --spinSleepTime 1 --minUptime 1000 ./bin/www"
}


Execute multi-line select sql failed

Using 1.0.0.M6 execute bellow multi-line select sql failed:
select id from T
  where id in ($1, $2)
  order by id asc

r2dbc-postgres error:
Suppressed: java.lang.UnsupportedOperationException: Binding parameters is not supported for the statement ...

r2dbc-h2 error:
Suppressed: org.h2.message.DbException: Method is not allowed for a query. Use execute or executeQuery instead of executeUpdate [90001-197]

But execute success if change to a single line sql such as 'select id from T where id in ($1, $2) order by id asc'.

Have you retested the scenario with 1.0.0.M7?

The problem fixed after upgrade to 1.0.0.M7.

Thanks for retrying and the update.


Allow VMRC connections with VMs not in the local zone

Description
When having a federation with OpenNebula, it will be useful to allow VMRC connections through not local zone machines.
This should also be considered in #5422
Use case
Improve VM connectivity and usability.

Progress Status

[x] Branch created
[ ] Code committed to development branch
[ ] Testing - QA
[ ] Documentation
[ ] Release notes - resolved issues, compatibility, known issues
[ ] Code committed to upstream release/hotfix branches
[ ] Documentation committed to upstream release/hotfix branches

wontfix, covered in 5763 for KVM


Question about netG_train_loss in code

Hello,
I have read the code. Some questions confuse me.
First, In TF.py line 194, "netG_train_loss = tf.reduce_mean(netD_netG_train_output1_1) - tf.reduce_mean(netD_netG_train_output2_2)". It seems different from the calculation of netG_train_loss in paper Page 6 Eq.(9), which is sum of two netD results rather than minus.
Besides, In TF.py line 247, "netG_tr_psnr2 = tf.constant(0, dtype=FLAGS['data_compute_dtype'])". I don't know why netG_tr_psnr2 is set to 0.
Thank you!

netD_train_loss = (-tf.reduce_mean(netD_train_output1_1) + tf.reduce_mean(netD_train_output2_1)) + (-tf.reduce_mean(netD_train_output1_2) + tf.reduce_mean(netD_train_output2_2))
The netD_train_loss seems also different from Eq.  7 and Eq. 8.
I think it should be netD_train_loss = (tf.reduce_mean(netD_train_output1_1) + (-tf.reduce_mean(netD_train_output2_1))) + (-tf.reduce_mean(netD_train_output1_2) + tf.reduce_mean(netD_train_output2_2))

The netD_train_loss maybe reasonable. Eq. 12 in the paper is max. The code minimizes the opposite of Eq. 12. Such as this line, "netD_loss = netD_train_loss + gradient_penalty_1 + gradient_penalty_2", Sign of the gradient_penalty is "+", but in Eq. 12, it is "-". So I think the code is opposite of Eq. 12.

I didn't refer to the sign of gradient_penalty of netD_loss . I only talk about netD_train_loss , which refers to AD in the paper. It is E(DX)-E(DX')+E(DY)-E(DY') in the paper. But in the code, I think the netD_train_loss corresponds to -E(DX)+E(DX')+E(DY)-E(DY')

@aduqiuc

You can define the score of the good image to be positive or negative. It is actually the same.
We will not have ground truth for HDR images.
@qsy777
refer to 1)

It should be "netG_train_loss = tf.reduce_mean(netD_netG_train_output1_1) +tf.reduce_mean(netD_netG_train_output2_2)",isnot it?


Update iot-hub-devguide-security.md

Correct the print statement as the lack of parenthesis will throw off pylint as a syntax error

@AndyCross : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#sign-off


live stream support

could you be more specific? level itself does not support live streams

right, but level-live-stream wouldn't work with multileveldown right?

maybe a first step would be the ability to subscribe to pre/post op events


Boolean without default value is not set in form value on submit

Boolean without default value is not set in form value on submit
But when I check it and uncheck it everything is ok. There is this first initialization missing I think.
For third option state we need to create different control for formly, so I think this third state when nothing is set is not used.
Is this as expected?
Expected behaviour
As I expect boolean has two values true/false
So when default value is not set then we have falsy values in boolean fields => field is set to false
Minimal Reproduction
https://stackblitz.com/edit/angular-boolean-example?file=src/app/app.component.ts
Environment

Angular version: any
Formly version: any

I think that's expected so either mark the field as required or use the default value


Support JWK public key VM processing into Key

Closes #1289
Decided to use askar-crypto for processing JWK in simplest way possible, did not want to try process the keys myself, and couldn't find another smaller crate that was suitable.
For now I've made this feature optional via jwk feature-flag which a consumer can optionally enable. Didn't want to force a requirement on askar-crypto for a relatively small crate (public_key).

Apparently there's some minor conflicts present on this one


Failed to open VapourSynth script ver <IP_ADDRESS>

Hi Revan: I noticed that the stack on failure has the following which I assume is part of the problem when running VS..  For some reason it is using the following home directory:
C:\Users\Revan\Desktop\staxrip-<IP_ADDRESS>\
at StaxRip.AVIFile..ctor(String path) in C:\Users\Revan\Desktop\staxrip-<IP_ADDRESS>\General\AVIFile.vb:line 50
at StaxRip.VideoScript.Synchronize(Boolean convertToRGB, Boolean comparePath) in C:\Users\Revan\Desktop\staxrip-<IP_ADDRESS>\General\VideoScript.vb:line 202
at StaxRip.MainForm.OpenVideoSourceFiles(IEnumerable`1 files, Boolean isEncoding) in C:\Users\Revan\Desktop\staxrip-<IP_ADDRESS>\Forms\MainForm.vb:line 2082

Without a Log I can't really do anything. What you posted is just where code triggers.

Sorry man here you go. I just ran it again to replicate.
#28

Where? You just linked to this thread.

Hugo (2011)_staxrip.log

Sorry Revan,  kind of new and this drag and drop didnt work I guess.  Hopefully now you can see it with the manual add.

You will notice it says plugin folder not found but it is there so I looked at the permissions and they have full access for my id and admin.

It's your Python Search Path. You also have to make sure your using Python 3.6 with R44.

oh ok.  Yes I had R45 and Python 3.7.1 installed.  I will clean deinstall and then install 3.6 with R44 and see how it goes.  Thank you.

the r should be right before the string source. It's all correct, It still your Search Path, When you install Python, make sure you check Include python in the search path.
It seems you are also missing: Plugin directory is missing!
I would also check make sure you have read, write & execute access to StaxRip. It's why the Docs says not to install it in ProgramFiles folder.


[BUG] Non-anonymized queries can be added to OpenSearch logs

QueryDataAnonymizer fails to anonymize queries that are  only supported by the v2 engine.
When that happens, the query including parameter values is added to OpenSearch log.
###Root Cause
QueryDataAnonymizer uses v1 parser to determine what elements in the AST need to be anonymized. When a query fails v1 engine, anonymization is skipped and the query is logged as-is.

Relates to #787

Completed via #1665


Restart events do not work for Dev Box machines

Dev Home version
No response
Windows build number
No response
Other software
No response
Steps to reproduce the bug

Trigger a restart event in Windows Customization or in Environments flow
Click the restart button on the banner in environments OR decline the restart on the modal popup in Windows Customization and then click the restart button on the banner

Expected result
Restart flow is initiated.
Actual result
Nothing happens -- there is no way to tell that the restart action has not worked. In this case I have an access denied error.
Included System Information
No response
Included Extensions Information
No response

@nieubank @bbonaby for visibility. I believe this only affects Windows Customization and Environments for the time-being.

hmm, its probably the use of shutdown.exe on some computers. @joadoumie, can you try opening command prompt or a terminal window (non-admin) on that machine and running the following command.
shutdown.exe -r -t 0
If you have permission the machine should restart.
if you get an access denied message can you tell me if you see your username or the Users group in this group policy?
Type in "group policy" the start menu and click the "Edit Group Policy" app -> In the app select "Local Computer Policy" -> Computer Configuration -> Windows Settings -> Security Settings -> Local Policies -> User Rights Assignment

Just looked in my Dev Box and yea, looks like the Users group isn't in this policy by default
I guess there are 3 things we can either do.

When we enable the optional features we also add the user account to this group. Issue here is that they will keep this permission/right even after the reboot unless they take themselves out. Also, maybe some enterprises prevent them from even updating the policies so this might not work.
Find some other way to invoke a reboot if possible.
Check if the user is in the group. if they are, then show the reboot button. If they aren't then just show an info bar telling them they need to reboot.

Moving to P3 as this is a niche issue and the most that can be done here is show an additional message saying that the restart wasn't successful.


docs: Fix typo

@mattklein123 There we go. I accidentally force-pushed over my changes (GitHub reusing branch names...) but I managed to recreate some of it.


Do not swallow error from native call

Hi, I am not sure about implementation here but at least error from native call should be propagated somehow.

@bossek yes! This is the intended behavior! I just forgot to implement it. :D Thank you so much!

@bossek yes! This is the intended behavior! I just forgot to implement it. :D Thank you so much!

@rexim no problem, I thank you for very interesting and entertaining streams!

@rexim no problem, I thank you for very interesting and entertaining streams!


Visibility enhancements for partners

Description

Closes #3737 and Closes #3733. This adds a new visibility setting of applicant partners. This addition turned out to be a bit unwieldy because of how situation it is. The main constraints I was designing for was in the case of a lab/fund that was using the partner role for vendors. I was trying to also account for the possibility that multiple vendors will be assigned to a single submission (which I believe is an edge case).
The main concern was applicants/vendors inputting sensitive data into the communications tab and having each other see such. For example: if a vendor uploads banking information an applicant (or other vendor) should not be able to see this information and vice versa.
Also tried to include a few docstring additions in lieu of the documentation updates.
Test Steps

Setup

[ ] As an Applicant submit a new application to a fund/lab.
[ ] Leave two comments with the visibilities of Staff only & Partners
[ ] As a Staff Admin add a Partner account (that you can access) to the submission
[ ] Leave three comments with the visibilities of: Applicant, Partners, & Applicant & Partners.
[ ] As the previous mentioned Partner account, navigate to the comms. tab of the submission.
[ ] Leave three comments with the visibilities of: Applicant, Partners, & Staff only.

Testing Partner Visibility

[ ] As the previous mentioned Partner account, navigate to the comms tab of the submission with these comments.
[ ] Ensure the only visible comments are the ones that were left with the visibility of Partners & Applicant & Partners
[ ] Ensure the comment the current account left with the visibility of Staff only is also visible

Testing Applicant Visibility

[ ] As the Applicant of the submission, navigate to the comms tab.
[ ] Ensure the only visible comments are the ones that were left with the visibility of Applicant & Applicant & Partners

Testing Staff Visibility

[ ] As Staff Admin navigate to the comms tab.
[ ] Ensure all previously made comments comments are visible.

Testing all label (#3733)

[ ] As Staff Admin navigate to the comms tab.
[ ] Make a comment with visibility of All
[ ] Ensure there is a visibility indicator with an eye icon labeled all (see below for example).

This is what I see when logged in as a partner. The option "Partner" I assume makes it possible for the applicant to see the message as well. I think this should be made more clear.

That's really odd, this should be what you see:

are you any other roles aside from partner? and is the partner assigned to this application?

Yes, the user was applicant + partner, when I uncheck applicant I to see this.

But having partner and applicant role is not that uncommon I think. As soon as you apply for anything you get the applicant role.
I think that if you have the partner roles and are assigned as a partner to a submission you should be treated as a partner. If you have a applicant role should not effect in this case. Sounds logical?

@frjo Yeah that makes total sense and would likely simplify things a bit for staff too. That last commit disallows applicants to be partners on their own application and checks if a user is a listed partner on the application before showing the partner visibility. I think this might be more inline with that logic.


commander: battery failsafe action back to warning

This reverts a previous change because we're not entirely sure about all implications on various airframes. It makes sense to change this default after the failsafe state machines have been consolidated and the various failsafe behaviours are more predictable for all airframes (not just
multicopter).
This was brought up by @bkueng after #13295.

I agree this is the right default. We almost need an additional level of defaults for "properly" setup and sanity checked setups.


refactor: config Postcss preset and plugins together

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Description

TODO:

[ ] Consider how to config postcss.plugins and attach defaults

Fix #3944
Checklist:

[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly. (PR: #)
[ ] I have added tests to cover my changes (if not applicable, please state why)
[ ] All new and existing tests passed.

Codecov Report

Merging #3927 into dev will decrease coverage by 0.7%.
The diff coverage is 54.54%.

@@            Coverage Diff             @@
##              dev    #3927      +/-   ##
==========================================
- Coverage   97.35%   96.65%   -0.71%
==========================================
  Files          18       18
  Lines        1248     1254       +6
  Branches      345      349       +4
==========================================
- Hits         1215     1212       -3
- Misses         32       39       +7
- Partials        1        3       +2

Impacted Files
Coverage Δ

lib/builder/webpack/utils/postcss.js
76.74% <54.54%> (-23.26%)
:arrow_down:

lib/core/nuxt.js
95% <0%> (+0.83%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6a88e06...9bba077. Read the comment docs.


fix: merged some PRs from the original repo + some other minor fixes.

Merged PR.
https://github.com/josephschmitt/Clamp.js/pull/18 closes https://github.com/josephschmitt/Clamp.js/issues/5
https://github.com/josephschmitt/Clamp.js/pull/20
https://github.com/josephschmitt/Clamp.js/pull/44
https://github.com/josephschmitt/Clamp.js/pull/50/files (Partial merge)
https://github.com/josephschmitt/Clamp.js/pull/54
https://github.com/josephschmitt/Clamp.js/pull/66 (Partial merge)

:tada: This PR is included in version 1.1.1 :tada:
The release is available on:

GitHub release
npm package (@latest dist-tag)

Your semantic-release bot :package::rocket:


Add $router.resolve support in programmatic mode.

What problem does this feature solve?
It would be helpful to access $router.resolve instance when using nuxt as a middleware. Check route existence before performing nuxt.render and do page redirect or pass to other routes like /api.
What does the proposed changes look like?
let routeExist = nuxt.resolveRoute('/some-path')

This feature request is available on Nuxt community (#c9248)

+1

I guess this is still relevant today for Nuxt 3, as there is no way of doing a useRouter().resolve on server-side. (Or is it?)

I think that should be possible!

We are approaching the Nuxt 2 EOL date (June 30, 2024) - see this article for more information. I'm closing this issue as it's marked as a Nuxt 2 related enhancement and it's not a critical issue.
That doesn't mean it might not be relevant for Nuxt 3. If it is, please feel free to open a new issue (or just comment, and I can reopen it). 🙏
Thank you for your understanding and for your contribution to Nuxt! 🎉


Fix implicit conversions of dynamic array to bool

Here is the warning message.
rdmd.d(79): Warning: implicit conversion of dynamic arrays to bool can be ambiguous and will be deprecated. Use one of: exe !is null, exe.length, or exe.ptr instead
...

I just replaced exe with exe.length.

Sooooooooooooooo ugly.

It does clarify intent though.  I don't know the code well enough to know if arr != null would be more appropriate.

BTW, changes to DustMite need to be made upstream, otherwise the next update will overwrite them.

It does clarify intent though. I don't know the code well enough to know if arr != null would be more appropriate.

You mean !is null, surely? And yes, at least some of the DustMite changes need to be !is null or (.ptr). Implicit array-to-bool conversion checks .ptr so if we're talking about mass conversions, that one would be the correct one.

Yeah, I meant !is null.  != null is the same as .length.  !is null is not the same as .ptr for all cases, because it also checks length is zero.

Thank you for your comments.
I will send a request to upstream using arr !is null instead of arr.length.

I will send a request to upstream using arr !is null instead of arr.length.

This is probably fine, but...

arr.ptr is the same a arr in all cases, so you should use this if you don't know what the code originally indented to check for.
arr.length or !arr.empty or arr != null is usually what was indented, except for some cases that specifically want to handle null arrays differently to empty arrays.
arr !is null is like arr != null except it handles invalid (null but non-empty) arrays differently.  It's very unlikely this is what you want.

Thank you for clarifying the difference but I have a question.
What is null arrays? Is it different from null or arrays with 0 elements?

null arrays have ptr == null and length == 0.
Empty arrays have length == 0, but ptr may not be null.  == will just check the lengths and elements match, but is will check ptr and length match.

Yes, but we have to fix it in order to reintroduce it.
if (arr)
    writeln(arr[0]);

Yes, but we have to fix it in order to reintroduce it.

I don't understand, could you elaborate?

I don't understand, could you elaborate?

After this is completely deprecated + some extra waiting time, we change the semantics of if (arr) to mean if (arr.length != 0), i.e. non-empty array.

Is that the ultimate goal for all this breakage? Because it seems unwarranted to me...

Seriously, as far as good reasons to break the language go, this isn't anywhere near the top of my list. And the breakage is significant - just look at the size of this patch! Did @andralex greenlight this?

Please read the discussion in the bugzilla issue if you want to know more.

Link please?

Found it: https://issues.dlang.org/show_bug.cgi?id=4733
I don't see anything insightful there.

Making this change will find broken code.

Please come up with a valid example.

Please look at this patch. How many bugs have been uncovered here?

Please look at this patch. How many bugs have been uncovered here?

I don't know the code well enough to know if you were intentionally differentiating between null and empty arrays.  Were you?
This warning hasn't found any bugs in my code, but it would have prevented bugs that I've introduced and fixed in the past.
The breakage is annoying, but trivial to fix.  It's also just a warning, so you're free to ignore it for the next six months if you want.  The current semantics do not make any sense and should be fixed.

I don't know the code well enough to know if you were intentionally differentiating between null and empty arrays. Were you?

Yep.

This warning hasn't found any bugs in my code, but it would have prevented bugs that I've introduced and fixed in the past.

Not to get all internet lawyer in here but this is not a quantifiable argument.

The breakage is annoying, but trivial to fix. It's also just a warning, so you're free to ignore it for the next six months if you want.

Unfortunately in practice this means that I'll have to disable all warnings altogether. And I don't see the point in delaying the inevitable.

The current semantics do not make any sense and should be fixed.

I don't know, they make sense to me. The difference between empty and null is as intuitive to me as that between an empty bag and no bag.
Anyway, I would like to hear Walter Bright and Andrei Alexandrescu's definitive decision on this before we expend more effort on this (both coding and arguing).

Sure, let's push that discussion back to https://github.com/D-Programming-Language/dmd/pull/2885 or the newsgroup. Those are hard to make tradeoffs, but it's important that we make a definite decision here.

Is that the ultimate goal for all this breakage? Because it seems unwarranted to me...

No. The point is to correct a very common misconception that if(arr) meant "if an arr has elements." Most often, an array's import is its contents, not where it lives. The reintroduction of if(arr) later is irrelevant to the fix, but clearly can be done after some time. I'm not concerned at all whether that gets reintroduced. The goal was to find broken code and let the user know about it.

Please look at this patch. How many bugs have been uncovered here?

Obviously, in some cases if(arr) really was meant as if(arr.ptr), and fixing those cases is busywork. Note that some of the patches to Phobos found by this new warning were bug fixes.

It's still controversial enough and we might decide to live with that wart.
Can you open a discussion on the newsgroup @schveiguy?

I sent a request to DustMite upstream (CyberShadow/DustMite#30).
I will remove the changes to DustMite in this request.
Anyway, which is better to replace if (arr) with if (arr.length) or if (!arr.empty)?
I am not sure there is a consensus about it in the newsgroup (http://forum.dlang.org/thread/mg6p6k$2j7u$1@digitalmars.com).

Anyway, which is better to replace if (arr) with if (arr.length) or if (!arr.empty)?

It's more of a style choice, so do what you want, though arr.empty also works with ranges.

Thank you for your advice.
I update to use !arr.empty instead of arr.length.

Sorry for changing this request so many times...

Currently, we cannot build this repository using make due to this issue.
As mentioned by yebblies above, there are several ways to fix it but only using arr.ptr is applicable for all cases.
IMO it is better to use the same way to fix it in tools and DustMite upstream repositories.

I will choose the conservative way, i.e. using arr.ptr.
Although using !arr.empty or others may be more appropriate, we can sure that using arr.ptr does not break anything and enables us to build this repository without errors or warnings.

I think we should close this request, since it will be overwritten, as @CyberShadow says.
Please us arr.ptr or arr !is null as those are equivalent to original code. There may be cases where the original code was wrong, but the safe consistent change is to do what was already being done. Looking at the changes you made, I think all of them can be a comparison of the pointer. The few I looked at that might be better as checking length were correctly using the pointer and not the length.

I think we should close this request, since it will be overwritten, as @CyberShadow says.

Not sure what you mean here.

Not sure what you mean here.

I was referring to this:

BTW, changes to DustMite need to be made upstream, otherwise the next update will overwrite them.

Seems to suggest the correct path for changing DustMite is to change the original project, and then pull the changes into this repository.

@tom-tan has already removed the DustMite changes from this pull, it only contains rdmd etc. patches.

duh! sorry. I thought this was specifically just for DustMite! Carry on and ignore my ignorance :)

And forget about my comment on using !arr.ptr, I may be wrong on that too.

I replaced arr with arr.ptr.

Thanks!

@MartinNowak I thought we agreed to wait on an OK from leaders before merging. Also this pull needed a rebase, it's 4 commits that make and undo/redo changes.

There hasn't been a single concern in<EMAIL_ADDRESS>I also use null arrays quite a lot in my code and will definitely miss if (auto s = get()), but fixing a constant source of bugs is worthwhile.


Access token API issue :https://docs.microsoft.com/en-us/azure/azure-video-analyzer/video-analyzer-for-media-docs/video-indexer-use-apis

[Enter feedback here]
The code snippet gives the below error while fetching the token
ErrorType:ARM_ACCOUNT_MUST_BE_MANAGED_BY_ARM,Message:You can only use the ARM management API for management calls on accounts created on ARM. Learn more. Trace id: '31b41e53-7e55-4f1c-9913-245410129d63'.}"

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c4116ee9-db60-920f-0eba-cd766e974580
Version Independent ID: 3f6e2214-afd7-0bad-f7e4-bf014255b2d2
Content: Use the Azure Video Analyzer for Media (formerly Video Indexer) API
Content Source: articles/azure-video-analyzer/video-analyzer-for-media-docs/video-indexer-use-apis.md
Service: azure-video-analyzer
Sub-service: azure-video-analyzer-media
GitHub Login: @Juliako
Microsoft Alias: Juliako

@AparnaJid Thank you for the detailed feedback. This is being assigned to the content author to evaluate and update as appropriate.

@AnuragSharma-MSFT , will ping a PM to look into the issue

seems like are following an old classic accounts documentation.
Will update the doc with a note that this docs is for classic accounts and make sure there is no ARM account sample there. Thank you. #please-close


Unable to detect --API

I installed it according to the documentation, but the API was also correct, but there was a situation where I tried many methods but it didn't work.
Automatic1111 is runnning，but you forgot to add --api flag to the webui command flags

I have the same problem in windows 10

can you include which method you used to install Auto1111?

I solved the problem by finding where to add "--api" in my case in the file launch.py ..... i really do not remember well the install, I followed a tutorial from the aitrepreneur youtubber...

Where in launch.py did you put this? Can you copy that here?


Link a new page be it night or day

If you login with an account that had chosen doctor as profession,it takes you to a doctor page which contains an appointments tab.
I have already liked a css file and html file for it in static and templates folders. Link those to the to the appointments option by seeing previous examples for signup and login pages. Also the doctor page has a dummy light and dark mode toggler, remove that and replace it with one that works.

can you assign me this

!assign @Lalith2907

!assign @Lalith2907 60

@Lalith2907 this has assigned you 60 minutes you can request for more by pinging

can i get assigned

@rohancyriac029 going for dinner rn

can i get assigned to this

@Lalith2907 you want me to extend time for this?

!extend 30

umm no actually u can unassign me

You are automatically unassigned as time ran out

!deassign

can i get this

yes

!assign @asmitdesai  60

@asmitdesai you have already been assigned to another repo

i dont have anything showing up on git

Asmit i will try to assign it to you once more

can i be assigned to it

can i be assigned to it

there are already people working on it

hey can i get assigned to this??

!deassign

!assign @asmitdesai 30

what is the error

@asmitdesai can you tell me the sem hall you are in

can you desassign me

ok

!deassign

!assign @sherlockhome12 60

@sherlockhome12 you have already been assigned to another repo

does it tell which repo?

Can you please assign this to me

!assign @H-Mudliar 60

does it tell which repo?

sorry i dont have access to that

can i be assigned to it

!deassign

!assign @Git-Cat-21 60

!assign @Git-Cat-21 60

can i please get assigned to this

!assign @Udaya-G 60

youve been assigned 60 mins
note that the current toggle is just a theme and not an actual button you need to replace that as well and then make it functional

!assign @Udaya-G 60

sorry not working on it  because of the events

!assign @Udaya-G 60

sorry not working on it because of the events

do you want to be deassigned or an extension?

!deassign

can i be assigned this ?

!assign @Nightwing000 60

@Nightwing000 the toggle key right now is just a theme and is not a functioning button you need to convert that to a dark and light mode theme changer by makeing it a button first doesnt matter how it looks

thanks

please assign me this

so can i not use the already provided theme and just switch the color of the text and background?

hey can i be assigned on this? i m not working on any other repo

@Nightwing000 do you require any extension?

so can i not use the already provided theme and just switch the color of the text and background?

that theme is not interactable as it is a span element you will get to know this if you do inspect element on the website that is if you run the code and try it out first

I know but I thought we had to use it as a template for how the dark mode should look
I've made the button and stuff but I'm getting a strict MIME error
can you extend for 30 minutes I"ll try to get past it

yea sure

!deassign

!assign @Nightwing000 30

!assign @Nightwing000 40

@Nightwing000 ive added 40 mins more

thanks

can you make a branch for me to push into?

just make pr to main

@Nightwing000 ill check and let you know in a bit

yeah thanks

@Nightwing000 can you just check the comment on your pr

!deassign


[examples/demos/integration/ideas]

RAG from the street with your AR device
POC with
https://brilliant.xyz/
use case:

i walk in the street and want to find some book part i was reading on my laptop
i wear brilliant frames and tap once on the frame and say "yo remind me this quote from Charlie Munger about patience please, check my laptop screen text fyi"
and AI stream an answer in audio in my airpods

would do a dirty tunnel with ngrok on my mac so i can query it all the time (with some api key to avoid the world hacking into my digital memory)
plus ios app on phone that do bridge between chatgpt (or local llm) + frame + laptop
Other ideas
TODO

There is a way
Help CocktailPeanut or his team make this work on Pinokio.computer
(mostly helping with the script to install it for any machine)
Then you can use his Feature (Cloudflare backend) to access it from anywhere in the world

ai tiktok video based on what you do
i like https://www.revid.ai/ even tho im not tiktok user at all
just shipping AI tiktok videos in < 5 min and i see now they have an api
so i could automatically push tiktok video at the end of my day based on stuff i do

Maybe a walkthrough for people how to use it with Ollama (LM studio is also viable) in the Readme files, so more non-programmer folks can get started with it easier and you get more people to test and break things (hopefully they report the bugs here ^^)
like so they know where to put the below when running with python
client = OpenAI(
base_url = 'http://localhost:11434/v1',
api_key='ollama', # required, but unused
)
maybe also add the base url into the environment file
when using your vercel file
Vercel AI SDK
The Vercel AI SDK is an open-source library for building conversational streaming applications. To get started, use create-next-app to clone the example repo:
npx create-next-app --example https://github.com/vercel/ai/tree/main/examples/next-openai example
cd example
Then make the following two edits in app/api/chat/route.ts to update the chat example to use Ollama:
const openai = new OpenAI({
baseURL: 'http://localhost:11434/v1',
apiKey: 'ollama',
});
const response = await openai.chat.completions.create({
model: 'llama2',
stream: true,
messages,
});
Next, run the app:
npm run dev
Finally, open the example app in your browser at http://localhost:3000/:

@AllyourBaseBelongToUs thanks! we're going to release a version for nontechnical users that include latest/best local LLM by default (and people can easily switch to openai if they want)
make sure to join the waitlist for nontechnical users here:
https://screenpi.pe/
what use case of screenpipe excite you?

@AllyourBaseBelongToUs thanks! we're going to release a version for nontechnical users that include latest/best local LLM by default (and people can easily switch to openai if they want)
make sure to join the waitlist for nontechnical users here: https://screenpi.pe/
what use case of screenpipe excite you?

Alright thank you very much Louis
use case:
Mostly the tracking feature, and not sure if It's not already there yet.
The capability to RAG based on my date, making it learn from my experiences
Similar to Quivr Quant from Stan Girard and the Second Brain inspired by @BrainRoemmele on Twitter


vmware: configdrive ISO not attached to VM deployed with deployasis template

ISSUE TYPE

Bug Report

COMPONENT NAME

VMware hypervisor

CLOUDSTACK VERSION

4.15.0

CONFIGURATION

OS / ENVIRONMENT

N/A
SUMMARY

When a VM is deployed with a deployasis template for L2 network with config drive, the configdrive ISO is not attached to the VM.
For VM with older/non-deployasis templates, configdrive ISO is found attached.
STEPS TO REPRODUCE

1. Register a new template if there is no template other than default template or templates before upgrading to 4.15.0. This will register the template with deployasis as true.
2. Deploy a VM with the template.
3. On vCenter, check if CD/DVD drive is shown connected in the VM hardware. vCenter will show disconnected.

EXPECTED RESULTS

ISO is shown connected in the Center as  CD/DVD drive.

ACTUAL RESULTS

VM hardware show no CD/DVD drive

VM with template having deployasis as false. CD/DVD drive shown connected

VM with template having deployasis as true. CD/DVD drive shown as disconnected

@nvazquez can you have a look at this? Related to deploy-as-is/vmware.

@rhtyd we've discussed this issue with @Pearl1594 and she'll provide a fix

cc @vladimirpetrov

Fixed in https://github.com/apache/cloudstack/pull/4907


add dashboard site to app

guys, for the app - we don't want to include the whole upper row - it is for the web only. Please see the mock in figma for this screen. Also, dark mode should be picked up from the app settings.

@avive ui is already fixed, top screenshot showed initial state


Is there a way to install browser extensions declaratively?

Is there a way to install browser (Chrome and Firefox) extensions (for example "ublock") declaratively (via configuration.nix) ?

I think this feature was added for chromium just a few days ago to nixpkgs master.

Yes there is a possibility install extensions in chromium: https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/programs/chromium.nix#L25

That's cool, besides Chromium is the only browser to be compiled locally. But it makes trivial to do the same for the other browsers (and perhaps faster than Chromium compilation finishes).
Thank you!

This configuration is only for NixOS. Is it possible to apply to the chromium package that can be installed on non-NixOS environments?


Schnorr Multisig AA

adds contracts repository of Schnorr MultiSig

Hey, could you please create 1 PR and add all 3 projects instead of creating PR for each? Thank you!

Hi @arddluma , will do Thank you.
The reason for multiple PRs is that I was following contribution document that states Make an individual pull request for each suggestion.


Conflicts with existing global highcharts instance

In my case, Highcharts has been included as global instance, and "import VueHighcharts from 'vue-highcharts'" throws highcharts error 16
https://www.highcharts.com/errors/16
Is it supported that vue-highcharts works with global highcharts together?

If you include Highcharts as a global variable and want vue-highcharts to use this, you should include vue-highcharts as global too.
<script src="/path/to/highcharts.js"></script>
<script src="/path/to/vue-highcharts.js"></script>

import Vue from 'vue';
Vue.use(window.VueHighcharts);
// or specify the `Highcharts` clearly
Vue.use(window.VueHighcharts, { Highcharts: window.Highcharts });

I removed the dependency from vue-highcharts to highcharts, passing the global Highcharts object and it works.
https://github.com/scaret/vue-highcharts/commit/eb0be06b30bc0fa115846e71a1084e0c820b8af4
I think the problem is that Highcharts package injects something on the global, which is not so necessary.