Debugging output for example to give full context

Sorry if this already exists, but I honestly wasn't sure how to search for it.  I was actually a bit surprised when I went to create this issue because the most recent #2615 addressed it from a different angle.
Also, this is mainly to get an understanding of what you as the maintainers think of the idea.  If it's something you're interested in I could potentially help with a PR.
Basically my thought was that people often have trouble debugging RSpec examples because the context for the example (the befores and the lets and the subjects etc...) may be in other parts of the file.  Also, if you use nested describe/context blocks, it can get harder to figure out which ones apply.
All of this can be managed to some extent by careful design of the tests and only extracting logic from the example itself some of the time.  Some people advocate for DAMP / having the full setup/exercise/expectation/teardown inside of the example.  I can see the advantages of this, but it can also be harder to scan and maintain those tests.
My thought was: For any given example RSpec is keeping track of what context applies for an example (again before / let / subject / etc...).  I was wondering if it would be difficult to have all of those lines outputted in a sort of "debug" format mode.  This would preserve the advantages of having somewhat DRYer setup with the ability to see the context of each example along with the examples in the output.
Is this crazy?  Does it already exist?  If not, do you think it would be difficult?  Thank you for your time and thanks for rspec.

From the perspective of #2615 we could automatically add into that contextual information lets that have been accessed, that wouldn't be too hard as the memoization would provide a key to whats where.
In terms of providing the code used around an example, that may be somewhat challenging, as we don't actually track where hooks are defined, nor lets. We also let Ruby handle the ordering of lets, but I guess we might be able to get at it via source locations?
We have some code in rspec-support for extracting example snippets, this would be the place to look to see how to handle this.
If you want to tackle this I'd happily help with reviewing it and advise, but I'm a bit busy atm to work on it directly.

It seems to me a very good idea. I would see this as a configure option and as an arg when running RSpec. For the display, it can be challenging.
Don't you think we should stick to https://github.com/rspec/rspec-core/issues/2615 ?

Thanks @JonRowe !  I'd definitely like to try to take a stab at this, but won't get a chance to for at least a week or two.
@benoittgt Yeah, I think it could definitely make sense as a configuration option, though it's the sort of thing that I would expect to want to jump into when I need it, so maybe and/or a feature flag like --fail-fast.  Or if it were a formatter it would be easy enough to specify the formatter as a command line option as well.
I think I still like the idea of outputting the lines of the spec file better than outputting the values as suggested by #2615 because it can be hard to return to where in the file the variable / subject / whatever was defined if you just have the value, especially if there are overlapping lets.  Also, by outputting appropriate lines we would ideally be shown things like before blocks which also give context as side-effects which are often just as important as let variables.

Are you still up to hack on this @cheerfulstoic ?

I want to, but probably not any time soon 😢

Closing due to inactivity rather than migrating to the monrorepo I don't think theres anything directly actionable? But feel free to open any PRs on the monorepo.


Fix min ada issue for outputs

fix: update lovelace amount in output if current amount does not satisfy min ada

Thanks @nikhils9 for looking into it! Much appreciated!


Add RFC for Layer Intersections.

For #

Background
Propose an RFC for layer intersection operations.

Change List

Add RFC for Layer Intersections.

"Intersection" implies a commutative operation, but this feature is not. I'd call it a "mask" layer after this.


Dialog for DAO group creation does not close.

Describe the bug
Dialog for DAO group creation does not close.
To Reproduce
Steps to reproduce the behavior:

Go to DAOs
Click on 'Register a voter' for any group
A dialog shows up
Press cancel button
It does not close the dialog

Screenshots

Closing this issue because is fixed and merged


Cannot enter values into the Owner 1, Owner 2 and Members fields

Hi, When creating a request, I cannot enter or search for names within the 3 above fields.  The fields appear to be blocked out. Any help would be much appreciated

I have the same problem. Is there a resolution for this yet as the app is pretty unusable at this time for me.

Same issue, fields aren't doing anything. Anyone solved this issue yet?

Set "isSearchabel" to ' true ' for all of the boxes. If they already on true, set it to 'false' and then back to 'true'


Commandlineoptions

#416 Document the commandline options that ServiceBusExplorer supports
if run with serviceBusExplorer.exe --help:

And adds a toolbar option.

utilizes https://github.com/commandlineparser (https://www.nuget.org/packages/CommandLineParser/)

Thanks @rajrao for the great addition


merge from template-arlac77-github

package.json

BREAKING CHANGE: add >=20.16.0 remove >=20.15.1 (engines.node)
fix(engines): add >=20.16.0 remove >=20.15.1 (engines.node)
.github/workflows/ci.yml

chore(deps): add 20.16.0 remove 20.15.1 (jobs.test-node.strategy.matrix.node-version)

Pull Request Test Coverage Report for Build<PHONE_NUMBER>9
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 75.519%

Totals

Change from base Build<PHONE_NUMBER>2:
0.0%

Covered Lines:
165

Relevant Lines:
209

💛  - Coveralls


Improve feature bounds

I am attaching a screen shot from a recent data anlaysis.  When we have some noisy MS1 we don't properly bound the features.  Any advice would be appreciated.  Also can the MS1 intensity array be smoothed?

Hi,
Sorry, I've missed that issue. If that kind of problem is still relevant by any chance, we can discuss it in more details.
Regards,
Mark Ivanov


[SPARK-6667] [PySpark] retry after timeout to accept

This is a small chance that the socket server will timed out after 3 seconds, but client had connected to it, then the Python process will hang.
This PR will retry once after timeout, it also add a timeout at client side.

cc @JoshRosen

Test build #29587 has started for   PR 5324 at commit b838f35.

After testing for a while, it seems that the retry does not work, but the timeout on client side can help:
15/04/01 22:42:14 WARN PythonRDD: Timed out after 4 seconds, retry once
15/04/01 22:42:14 ERROR PythonRDD: Error while sending iterator
java.net.SocketTimeoutException: Accept timed out
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
	at java.net.ServerSocket.implAccept(ServerSocket.java:530)
	at java.net.ServerSocket.accept(ServerSocket.java:498)
	at org.apache.spark.api.python.PythonRDD$$anon$2.run(PythonRDD.scala:624)
**********************************************************************
File "/Users/davies/work/spark/python/pyspark/rdd.py", line 1090, in __main__.RDD.variance
Failed example:
    sc.parallelize([1, 2, 3]).variance()
Exception raised:
    Traceback (most recent call last):
      File "//anaconda/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest __main__.RDD.variance[0]>", line 1, in <module>
        sc.parallelize([1, 2, 3]).variance()
      File "/Users/davies/work/spark/python/pyspark/rdd.py", line 1093, in variance
        return self.stats().variance()
      File "/Users/davies/work/spark/python/pyspark/rdd.py", line 948, in stats
        return self.mapPartitions(lambda i: [StatCounter(i)]).reduce(redFunc)
      File "/Users/davies/work/spark/python/pyspark/rdd.py", line 745, in reduce
        vals = self.mapPartitions(func).collect()
      File "/Users/davies/work/spark/python/pyspark/rdd.py", line 720, in collect
        return list(_load_from_socket(port, self._jrdd_deserializer))
      File "/Users/davies/work/spark/python/pyspark/rdd.py", line 120, in _load_from_socket
        for item in serializer.load_stream(rf):
      File "/Users/davies/work/spark/python/pyspark/serializers.py", line 131, in load_stream
        yield self._read_with_length(stream)
      File "/Users/davies/work/spark/python/pyspark/serializers.py", line 148, in _read_with_length
        length = read_int(stream)
      File "/Users/davies/work/spark/python/pyspark/serializers.py", line 526, in read_int
        length = stream.read(4)
      File "//anaconda/lib/python2.7/socket.py", line 380, in read
        data = self._sock.recv(left)
    timeout: timed out
**********************************************************************

Test build #29589 has started for   PR 5324 at commit 7977c2f.

Test build #29592 has started for   PR 5324 at commit e5a51a2.

Test build #29587 has finished for   PR 5324 at commit b838f35.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

class MessageWithHeader extends AbstractReferenceCounted implements FileRegion

This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29587/
Test PASSed.

Test build #29589 has finished for   PR 5324 at commit 7977c2f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29589/
Test PASSed.

Test build #29592 has finished for   PR 5324 at commit e5a51a2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29592/
Test PASSed.

I've merged this to master (1.4.0), branch-1.3 (1.3.1), and branch-1.2 (1.2.2).  Thanks!


OData '$expand' attribute doesn't work

Disabling/Enabling in Program.cs seem to have no effect
.AddOData(options =>
    {
        options.Filter()
        .Select()
        .OrderBy()
        //.Expand()
        .SetMaxTop(5000);
    });

Hello cigircii, did you solve this issue? I'm facing the same problem

Hello cigircii, did you solve this issue? I'm facing the same problem

Hey @MagnusMutai, unfortunately, I did not, didn't pursue it either.
However, this is project is written in .NET 6 so.. you might find different results with some of the newer versions.

Okay thank you, I think I did.


Poor iGPU performance on Stable Diffusion

It seems like the iGPU was not fully utilized, and part of the pipeline is kinda blocking or locking.

It seems like the iGPU was not fully utilized, and part of the pipeline is kinda blocking or locking.

Maybe you don't configure multi-device but tell me how you configure iGPU with your Intel Iris Xe graphics.
Show me the ouput of this command :
from openvino.runtime import Core

ie = Core()

devices = ie.available_devices

for device in devices:
    device_name = ie.get_property(device, "FULL_DEVICE_NAME")
    print(f"{device}: {device_name}")

@bouachalazhar it was about running inference of StableDiffusion on GPU only - to show how the GPU gets utilized; this lead to other optimizations as bottlenecks were found.
Considering batch, num-of-inference-requests, async-mode and making use of all available "accelerators" - and the overall throughput will look differently.


Template develop

Template Gvec class and Wave_functions class

@manhin321 What is the use case for templated G-vector class? From what I see, it's only the Cartesian components and length are affected.

@manhin321 What is the use case for templated G-vector class? From what I see, it's only the Cartesian components and length are affected.

It was originally intended to template everything inside the Wave_functions class to allow full fp32 calculation including the unit cell matrix storage. This change would indeed create overhead for copying when doing mix-precision calculation.
We can still allow the templated version of Gvec class but always use Gvec to see the difference later

Still, I want to argue that wave-functions in FP32 can co-exist with the G-vector Cartesian coordinates in FP64. You will use the G^2/2 as a kinetic energy term, and this can be contained inside Local_operator class.

I agree with that. I just take reference from the Julia dftk code to propose the change. Otherwise, I try to roll back for non templated Gvec class version

I agree with that. I just take reference from the Julia dftk code to propose the change. Otherwise, I try to roll back for non templated Gvec class version

Yes, Gvec should be rolled back to it's original non-templated version.


CosmosOperationCanceledException: Adds serializable functionality

Pull Request Template
Description
Code changes to make the CosmosOperationCanceledException Exception serializable. It should be noted that CosmosDiagnostics are not serializable and are left out of sterilization and upon deserialization the original CosmosDiagnotics are replaced with a CosmosTraceDiagnostics with a NoOpTrace. lazyMessage and toStringMessage for deserialized CosmosOperationCanceledExceptions are also updated to inform users that CosmosDiagnostics cannot be deserialized. Additionally CosmosOperationCanceledTests.cs was also created to test that the relevant information can be serialized and recovered.
Type of change
Please delete options that are not relevant.

[] New feature (non-breaking change which adds functionality)
[] This change requires a documentation update

Closing issues
To automatically close an issue: closes #3402

@NaluTripician See the test failures:
 Failed ContractChanges [179 ms]
  Error Message:
   Assert.Fail failed. Public API has changed. If this is expected, then run (EnlistmentRoot)\UpdateContracts.ps1 . To see the differences run the update script and use git diff.
  Stack Trace:

Because you are modifying a public type, this generates a change in the API contract, please run the Powershell the test indicates that should be in the root of your clone.


Add an example to demonstrate the use of API:Links module to identify red links on a page

More in response to the question asked here: https://discourse-mediawiki.wmflabs.org/t/data-returned-by-the-query-api-does-not-say-if-a-link-is-a-red-link-or-not/1043.
It would be a cool use case of how API:Links module can also be used as a generator.

Hi @srish I'd like to work on this issue.

Hey @srish can I work on this issue ?

Hello, @srish  I have made a pull request for this issue. Kindly review for corrections. Thanks!

@srish Hello, I am an outreachy applicant. And I wish to work on some issues. But as this issue is taken up by @edyasikpo , I would like to work on different issue.

@srish, I have made a Pull request for this issue. Please review. Thanks.!

#50

hey @srish  can i work on this issue despite other people already submitting their pull requests??

Thanks all for your interest in working on this issue! As @edyasikpo and @SupriyaSupugade are already working on this, please wait till we have other tickets available. In a couple of days, I will be opening more issues that you could look into! Check back again in a few days. Thanks for your patience!
In the future, when there will be more tasks, it might be ideal if you assign yourself to a task first that you are interested in, and then start working on it. That way others know, that the task is already taken up.

@srish isn't it the case that once you comment on the issue you wish to work on and then you start working on it only after it is assigned to you? As I don't see that practice being followed here. Please have a look at it as I commented on this issue as I wanted to work on but others just sent a pull request without the task being assigned to them.

@edyasikpo @SupriyaSupugade @brendajerop I've reviewed your pull requests! I don't like this approach, but I will only be able to accept one person's contribution. Whenever you are done incorporating the feedback, I will take a second look. But, don't worry, I will add more microtasks today.
@hasinifoss Yes, that's exactly how you would do it!
@yatri1609 that's exactly what I'm referring to in https://github.com/wikimedia/MediaWiki-Action-API-Code-Samples/issues/37#issuecomment-466203063.

@srish Thank you! I will send the PR with the above described implementation.

Okay @srish Thanks for the feedback!

@srish Okay. Apologies for the uncouth behaviour

@srish Okay. Apologies for the uncouth behaviour

@hasinifoss As others are already working on this issue, I encourage you to take a look at other microtasks now available: https://phabricator.wikimedia.org/T215682.

@srish, I am Racheal  an outreachy applicant and I would love to contribute on issues you posted 2 days ego here https://phabricator.wikimedia.org/T215682.  but as i can see all tasks have been booked. Is it okay if i pick the one am interested in and contribute or ? how should go about this? am really confused. Thanks

Thanks @brendajerop for your work on this! I've made some corrections in your Sandbox. You can now copy paste Example 2 from https://www.mediawiki.org/wiki/User:Jeropbrenda/Sandbox/API:Links#Example_2:_Fetch_missing_links and add it as a second example here https://www.mediawiki.org/wiki/API:Links!


🐛 BUG: Not injecting D1 Database for 'DB' as this version of Miniflare only supports D1 beta bindings

Which Cloudflare product(s) does this pertain to?
D1
What version of Wrangler are you using?
2.15.0
What operating system are you using?
macOS
Describe the Bug
When I run miniflare --modules -w build/index.ts I get this message :
Not injecting D1 Database for 'DB' as this version of Miniflare only supports D1 beta bindings. Upgrade Wrangler and/or Miniflare and try again.

My wrangler.toml:
name = "limey-custom-domains"
compatibility_date = "2023-03-14"
account_id = "<REDACTED>"
main = "build/index.ts"

compatibility_flags = [
	"streams_enable_constructors"
]

[vars]
ZONE_ID = "<REDACTED>"
WORKER_NAME = "limey-custom-domains"

[build]
command = "npm run build"

[[ d1_databases ]]
binding = "DB"
database_name = "limey"
database_id = "<REDACTED>"

Related:

#2402 (That issue is about integration tests)

Hello @probablykasper, if you upgrade to the latest version of wrangler v3 do you still have this issue? This should be resolved there

Seems to work now, yes!


[v1.x] provide a faster PrefetchedDataLoader

Description
there already exists some faster dataloader in mxnet 2.0, but in v1.x, the exist dataloader is slower and could be improved by changing its prefetch behavior as what 2.0 have done.
test:
$ cat iternew.py && python iternew.py
import mxnet as mx
from mxnet.gluon.data import DataLoader,ArrayDataset
from time import sleep,perf_counter_ns
train_data=ArrayDataset(mx.nd.array([[i] for i in range(50000)]),mx.nd.array([[99-i] for i in range(50000)]))
test_data=ArrayDataset(mx.nd.array([[i] for i in range(10000)]),mx.nd.array([[99-i] for i in range(10000)]))
def transform_train(sample):
  sleep(0.0016)
  return sample

def transform_test(sample):
  sleep(0.0008)
  return sample

train_iter=DataLoader(train_data.transform_first(transform_train),batch_size=500,num_workers=10)
test_iter =DataLoader(test_data .transform_first(transform_test ),batch_size=500,num_workers=10)
if True:
  tic=perf_counter_ns()
  for epoch in range(10):
    print("epoch"+str(epoch)+" start at "+str(round((perf_counter_ns()-tic)*1e-9,2))+"s")
    for i in train_iter:
      sleep(0.1)
    print("       finished train phase at "+str(round((perf_counter_ns()-tic)*1e-9,2))+"s")
    for i in test_iter:
      sleep(0.05)
    print("        finished test phase at "+str(round((perf_counter_ns()-tic)*1e-9,2))+"s")
  print("cost="+str((perf_counter_ns()-tic)*1e-9)+"s")
epoch0 start at 0.0s
       finished train phase at 11.28s
        finished test phase at 12.35s
epoch1 start at 12.35s
       finished train phase at 22.73s
        finished test phase at 23.79s
epoch2 start at 23.79s
       finished train phase at 34.15s
        finished test phase at 35.21s
epoch3 start at 35.22s
       finished train phase at 45.59s
        finished test phase at 46.66s
epoch4 start at 46.66s
       finished train phase at 57.01s
        finished test phase at 58.07s
epoch5 start at 58.07s
       finished train phase at 68.43s
        finished test phase at 69.5s
epoch6 start at 69.5s
       finished train phase at 79.87s
        finished test phase at 80.93s
epoch7 start at 80.93s
       finished train phase at 91.3s
        finished test phase at 92.37s
epoch8 start at 92.37s
       finished train phase at 102.74s
        finished test phase at 103.8s
epoch9 start at 103.8s
       finished train phase at 114.17s
        finished test phase at 115.23s
cost=115.23376344s

(cost ~129.67192333600002s if we are using Dataloader rather than PrefetchedDataLoader)
(test is done using v1.7.0, the newest v1.x is on my GPU server and running, I do not want to bother my GPU server.)
Checklist
Essentials

[X] PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
[X] Changes are complete (i.e. I finished coding on this PR)
[x] All changes have test coverage
[X] Code is well-documented

Changes

[X] add a auto_reload flag for DataLoader, which loads data faster in the first several batch compare to the default Dataloader.
(Don't know whether the test case is appropriate.)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here
Now, the default behavior of DataLoader is prefetch immediately after it is created, rather than wait for its __iter__() is called.
This behavior is compatible to MXNet 2.0's dataloader with nopython mode, since it really speeds up the program (~5% faster with my handwriting autoaugment transform function when training CIFAR-100 with RTX 3090, batch_size=250, wide resnet 16-4 and deep mutual learning technique.) and provide almost no difference(all the parameter create the prefetched dataloader is private and should not be modified by any other code), I switch the default behavior to prefetch.

find a better solution.

find a better solution.

@mxnet-bot run ci [unix-gpu]

@mxnet-bot run ci [unix-gpu]

[2021-01-15T11:17:46.702Z] test_operator_gpu.test_kernel_error_checking ... Process SpawnProcess-1:

[2021-01-15T11:17:46.702Z] Traceback (most recent call last):

[2021-01-15T11:17:46.702Z]   File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap

[2021-01-15T11:17:46.702Z]     self.run()

[2021-01-15T11:17:46.702Z]   File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run

[2021-01-15T11:17:46.702Z]     self._target(*self._args, **self._kwargs)

[2021-01-15T11:17:46.702Z]   File "/work/mxnet/tests/python/gpu/test_operator_gpu.py", line 2238, in kernel_error_check_imperative

[2021-01-15T11:17:46.702Z]     c = (a / b).asnumpy()

[2021-01-15T11:17:46.702Z]   File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__

[2021-01-15T11:17:46.702Z]     self.gen.throw(type, value, traceback)

[2021-01-15T11:17:46.702Z]   File "/work/mxnet/tests/python/unittest/../../../python/mxnet/test_utils.py", line 2523, in environment

[2021-01-15T11:17:46.702Z]     yield

[2021-01-15T11:17:46.702Z]   File "/work/mxnet/tests/python/gpu/test_operator_gpu.py", line 2238, in kernel_error_check_imperative

[2021-01-15T11:17:46.702Z]     c = (a / b).asnumpy()

[2021-01-15T11:17:46.702Z]   File "/work/mxnet/tests/python/unittest/../../../python/mxnet/ndarray/ndarray.py", line 354, in __truediv__

[2021-01-15T11:17:46.702Z]     return divide(self, other)

[2021-01-15T11:17:46.702Z]   File "/work/mxnet/tests/python/unittest/../../../python/mxnet/ndarray/ndarray.py", line 3820, in divide

[2021-01-15T11:17:46.702Z]     _internal._rdiv_scalar)

[2021-01-15T11:17:46.702Z]   File "/work/mxnet/tests/python/unittest/../../../python/mxnet/ndarray/ndarray.py", line 3576, in _ufunc_helper

[2021-01-15T11:17:46.702Z]     return fn_array(lhs, rhs)

[2021-01-15T11:17:46.702Z]   File "<string>", line 52, in broadcast_div

[2021-01-15T11:17:46.702Z]   File "/work/mxnet/tests/python/unittest/../../../python/mxnet/_ctypes/ndarray.py", line 91, in _imperative_invoke

[2021-01-15T11:17:46.702Z]     ctypes.byref(out_stypes)))

[2021-01-15T11:17:46.702Z]   File "/work/mxnet/tests/python/unittest/../../../python/mxnet/base.py", line 246, in check_call

[2021-01-15T11:17:46.702Z]     raise get_last_ffi_error()

[2021-01-15T11:17:46.702Z] mxnet.base.MXNetError: Traceback (most recent call last):

[2021-01-15T11:17:46.702Z]   [bt] (9) /usr/lib/python3.5/lib-dynload/_ctypes.cpython-35m-x86_64-linux-gnu.so(+0x9fcb) [0x7f5b336f8fcb]

[2021-01-15T11:17:46.702Z]   [bt] (8) /usr/lib/python3.5/lib-dynload/_ctypes.cpython-35m-x86_64-linux-gnu.so(_ctypes_callproc+0x49a) [0x7f5b3370501a]

[2021-01-15T11:17:46.702Z]   [bt] (7) /usr/lib/python3.5/lib-dynload/_ctypes.cpython-35m-x86_64-linux-gnu.so(ffi_call+0x2eb) [0x7f5b3370a88b]

[2021-01-15T11:17:46.702Z]   [bt] (6) /usr/lib/python3.5/lib-dynload/_ctypes.cpython-35m-x86_64-linux-gnu.so(ffi_call_unix64+0x4c) [0x7f5b3370ae20]

[2021-01-15T11:17:46.702Z]   [bt] (5) /work/mxnet/python/mxnet/../../lib/libmxnet.so(MXImperativeInvokeEx+0x5b) [0x7f5ad94b34bb]

[2021-01-15T11:17:46.702Z]   [bt] (4) /work/mxnet/python/mxnet/../../lib/libmxnet.so(MXImperativeInvokeImpl(void*, int, void**, int*, void***, int, char const**, char const**)+0x650) [0x7f5ad94b1e10]

[2021-01-15T11:17:46.702Z]   [bt] (3) /work/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::Imperative::Invoke(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&)+0x35c) [0x7f5ad8bdb1dc]

[2021-01-15T11:17:46.702Z]   [bt] (2) /work/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::imperative::SetShapeType(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, mxnet::DispatchMode*)+0x23a5) [0x7f5ad8bd4de5]

[2021-01-15T11:17:46.702Z]   [bt] (1) /work/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::op::BinaryBroadcastShape(nnvm::NodeAttrs const&, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*)+0x3c2) [0x7f5ad3e89d42]

[2021-01-15T11:17:46.702Z]   [bt] (0) /work/mxnet/python/mxnet/../../lib/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x61) [0x7f5ad3bd3ef1]

[2021-01-15T11:17:46.702Z]   File "src/operator/numpy/linalg/./../../tensor/elemwise_binary_broadcast_op.h", line 68

[2021-01-15T11:17:46.702Z] MXNetError: Check failed: l == 1 || r == 1: operands could not be broadcast together with shapes [3] [0]

[2021-01-15T11:17:51.940Z] Process SpawnProcess-2:

[2021-01-15T11:17:51.940Z] Traceback (most recent call last):

[2021-01-15T11:17:51.940Z]   File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap

[2021-01-15T11:17:51.940Z]     self.run()

[2021-01-15T11:17:51.940Z]   File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run

[2021-01-15T11:17:51.940Z]     self._target(*self._args, **self._kwargs)

[2021-01-15T11:17:51.940Z]   File "/work/mxnet/tests/python/gpu/test_operator_gpu.py", line 2249, in kernel_error_check_symbolic

[2021-01-15T11:17:51.940Z]     g = f.outputs[0].asnumpy()

[2021-01-15T11:17:51.940Z]   File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__

[2021-01-15T11:17:51.940Z]     self.gen.throw(type, value, traceback)

[2021-01-15T11:17:51.940Z]   File "/work/mxnet/tests/python/unittest/../../../python/mxnet/test_utils.py", line 2523, in environment

[2021-01-15T11:17:51.940Z]     yield

[2021-01-15T11:17:51.940Z]   File "/work/mxnet/tests/python/gpu/test_operator_gpu.py", line 2247, in kernel_error_check_symbolic

[2021-01-15T11:17:51.940Z]     'b':mx.nd.array([],ctx=mx.gpu(0))})

[2021-01-15T11:17:51.940Z]   File "/work/mxnet/tests/python/unittest/../../../python/mxnet/symbol/symbol.py", line 2119, in bind

[2021-01-15T11:17:51.940Z]     ctypes.byref(handle)))

[2021-01-15T11:17:51.940Z]   File "/work/mxnet/tests/python/unittest/../../../python/mxnet/base.py", line 246, in check_call

[2021-01-15T11:17:51.940Z]     raise get_last_ffi_error()

[2021-01-15T11:17:51.940Z] mxnet.base.MXNetError: Traceback (most recent call last):

[2021-01-15T11:17:51.940Z]   [bt] (9) /work/mxnet/python/mxnet/../../lib/libmxnet.so(MXExecutorBindEX+0xc0a) [0x7f8286b4442a]

[2021-01-15T11:17:51.940Z]   [bt] (8) /work/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::Executor::Bind(nnvm::Symbol, mxnet::Context const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, mxnet::Context, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, mxnet::Context> > > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, mxnet::Executor*)+0x204) [0x7f8286c5d364]

[2021-01-15T11:17:51.940Z]   [bt] (7) /work/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::exec::GraphExecutor::Init(nnvm::Symbol, mxnet::Context const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, mxnet::Context, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, mxnet::Context> > > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, mxnet::Executor*, std::unordered_map<nnvm::NodeEntry, mxnet::NDArray, nnvm::NodeEntryHash, nnvm::NodeEntryEqual, std::allocator<std::pair<nnvm::NodeEntry const, mxnet::NDArray> > > const&)+0x11eb) [0x7f8286c5140b]

[2021-01-15T11:17:51.940Z]   [bt] (6) /work/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::exec::InferShape(nnvm::Graph&&, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x83) [0x7f8286c2aeb3]

[2021-01-15T11:17:51.940Z]   [bt] (5) /work/mxnet/python/mxnet/../../lib/libmxnet.so(+0x7b9437e) [0x7f8286c2837e]

[2021-01-15T11:17:51.940Z]   [bt] (4) /work/mxnet/python/mxnet/../../lib/libmxnet.so(+0x7b903c7) [0x7f8286c243c7]

[2021-01-15T11:17:51.940Z]   [bt] (3) /work/mxnet/python/mxnet/../../lib/libmxnet.so(bool mxnet::op::ElemwiseShape<2, 1>(nnvm::NodeAttrs const&, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*)+0x229) [0x7f8281ed7ce9]

[2021-01-15T11:17:51.940Z]   [bt] (2) /work/mxnet/python/mxnet/../../lib/libmxnet.so(bool mxnet::op::ElemwiseAttrHelper<mxnet::TShape, &mxnet::op::shape_is_none, &mxnet::op::shape_assign, true, &mxnet::op::shape_string[abi:cxx11], -1, -1>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*, mxnet::TShape const&)+0x215) [0x7f828132c4e5]

[2021-01-15T11:17:51.940Z]   [bt] (1) /work/mxnet/python/mxnet/../../lib/libmxnet.so(bool mxnet::op::ElemwiseAttrHelper<mxnet::TShape, &mxnet::op::shape_is_none, &mxnet::op::shape_assign, true, &mxnet::op::shape_string[abi:cxx11], -1, -1>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*, mxnet::TShape const&)::{lambda(std::vector<mxnet::TShape, std::allocator<mxnet::TShape> > const&, unsigned long, char const*)#1}::operator()(std::vector<mxnet::TShape, std::allocator<mxnet::TShape> > const&, unsigned long, char const*) const+0x10a8) [0x7f828132acf8]

[2021-01-15T11:17:51.940Z]   [bt] (0) /work/mxnet/python/mxnet/../../lib/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x61) [0x7f82812b5ef1]

[2021-01-15T11:17:51.940Z] MXNetError: Error in operator _div0: [11:17:47] src/operator/numpy/linalg/./../../tensor/../elemwise_op_common.h:135: Check failed: assign(&dattr, vec.at(i)): Incompatible attr in node _div0 at 1-th input: expected [3], got [0]

[2021-01-15T11:17:46.702Z] test_operator_gpu.test_kernel_error_checking ... Process SpawnProcess-1:

[2021-01-15T11:17:46.702Z] Traceback (most recent call last):

[2021-01-15T11:17:46.702Z]   File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap

[2021-01-15T11:17:46.702Z]     self.run()

[2021-01-15T11:17:46.702Z]   File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run

[2021-01-15T11:17:46.702Z]     self._target(*self._args, **self._kwargs)

[2021-01-15T11:17:46.702Z]   File "/work/mxnet/tests/python/gpu/test_operator_gpu.py", line 2238, in kernel_error_check_imperative

[2021-01-15T11:17:46.702Z]     c = (a / b).asnumpy()

[2021-01-15T11:17:46.702Z]   File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__

[2021-01-15T11:17:46.702Z]     self.gen.throw(type, value, traceback)

[2021-01-15T11:17:46.702Z]   File "/work/mxnet/tests/python/unittest/../../../python/mxnet/test_utils.py", line 2523, in environment

[2021-01-15T11:17:46.702Z]     yield

[2021-01-15T11:17:46.702Z]   File "/work/mxnet/tests/python/gpu/test_operator_gpu.py", line 2238, in kernel_error_check_imperative

[2021-01-15T11:17:46.702Z]     c = (a / b).asnumpy()

[2021-01-15T11:17:46.702Z]   File "/work/mxnet/tests/python/unittest/../../../python/mxnet/ndarray/ndarray.py", line 354, in __truediv__

[2021-01-15T11:17:46.702Z]     return divide(self, other)

[2021-01-15T11:17:46.702Z]   File "/work/mxnet/tests/python/unittest/../../../python/mxnet/ndarray/ndarray.py", line 3820, in divide

[2021-01-15T11:17:46.702Z]     _internal._rdiv_scalar)

[2021-01-15T11:17:46.702Z]   File "/work/mxnet/tests/python/unittest/../../../python/mxnet/ndarray/ndarray.py", line 3576, in _ufunc_helper

[2021-01-15T11:17:46.702Z]     return fn_array(lhs, rhs)

[2021-01-15T11:17:46.702Z]   File "<string>", line 52, in broadcast_div

[2021-01-15T11:17:46.702Z]   File "/work/mxnet/tests/python/unittest/../../../python/mxnet/_ctypes/ndarray.py", line 91, in _imperative_invoke

[2021-01-15T11:17:46.702Z]     ctypes.byref(out_stypes)))

[2021-01-15T11:17:46.702Z]   File "/work/mxnet/tests/python/unittest/../../../python/mxnet/base.py", line 246, in check_call

[2021-01-15T11:17:46.702Z]     raise get_last_ffi_error()

[2021-01-15T11:17:46.702Z] mxnet.base.MXNetError: Traceback (most recent call last):

[2021-01-15T11:17:46.702Z]   [bt] (9) /usr/lib/python3.5/lib-dynload/_ctypes.cpython-35m-x86_64-linux-gnu.so(+0x9fcb) [0x7f5b336f8fcb]

[2021-01-15T11:17:46.702Z]   [bt] (8) /usr/lib/python3.5/lib-dynload/_ctypes.cpython-35m-x86_64-linux-gnu.so(_ctypes_callproc+0x49a) [0x7f5b3370501a]

[2021-01-15T11:17:46.702Z]   [bt] (7) /usr/lib/python3.5/lib-dynload/_ctypes.cpython-35m-x86_64-linux-gnu.so(ffi_call+0x2eb) [0x7f5b3370a88b]

[2021-01-15T11:17:46.702Z]   [bt] (6) /usr/lib/python3.5/lib-dynload/_ctypes.cpython-35m-x86_64-linux-gnu.so(ffi_call_unix64+0x4c) [0x7f5b3370ae20]

[2021-01-15T11:17:46.702Z]   [bt] (5) /work/mxnet/python/mxnet/../../lib/libmxnet.so(MXImperativeInvokeEx+0x5b) [0x7f5ad94b34bb]

[2021-01-15T11:17:46.702Z]   [bt] (4) /work/mxnet/python/mxnet/../../lib/libmxnet.so(MXImperativeInvokeImpl(void*, int, void**, int*, void***, int, char const**, char const**)+0x650) [0x7f5ad94b1e10]

[2021-01-15T11:17:46.702Z]   [bt] (3) /work/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::Imperative::Invoke(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&)+0x35c) [0x7f5ad8bdb1dc]

[2021-01-15T11:17:46.702Z]   [bt] (2) /work/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::imperative::SetShapeType(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, mxnet::DispatchMode*)+0x23a5) [0x7f5ad8bd4de5]

[2021-01-15T11:17:46.702Z]   [bt] (1) /work/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::op::BinaryBroadcastShape(nnvm::NodeAttrs const&, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*)+0x3c2) [0x7f5ad3e89d42]

[2021-01-15T11:17:46.702Z]   [bt] (0) /work/mxnet/python/mxnet/../../lib/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x61) [0x7f5ad3bd3ef1]

[2021-01-15T11:17:46.702Z]   File "src/operator/numpy/linalg/./../../tensor/elemwise_binary_broadcast_op.h", line 68

[2021-01-15T11:17:46.702Z] MXNetError: Check failed: l == 1 || r == 1: operands could not be broadcast together with shapes [3] [0]

[2021-01-15T11:17:51.940Z] Process SpawnProcess-2:

[2021-01-15T11:17:51.940Z] Traceback (most recent call last):

[2021-01-15T11:17:51.940Z]   File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap

[2021-01-15T11:17:51.940Z]     self.run()

[2021-01-15T11:17:51.940Z]   File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run

[2021-01-15T11:17:51.940Z]     self._target(*self._args, **self._kwargs)

[2021-01-15T11:17:51.940Z]   File "/work/mxnet/tests/python/gpu/test_operator_gpu.py", line 2249, in kernel_error_check_symbolic

[2021-01-15T11:17:51.940Z]     g = f.outputs[0].asnumpy()

[2021-01-15T11:17:51.940Z]   File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__

[2021-01-15T11:17:51.940Z]     self.gen.throw(type, value, traceback)

[2021-01-15T11:17:51.940Z]   File "/work/mxnet/tests/python/unittest/../../../python/mxnet/test_utils.py", line 2523, in environment

[2021-01-15T11:17:51.940Z]     yield

[2021-01-15T11:17:51.940Z]   File "/work/mxnet/tests/python/gpu/test_operator_gpu.py", line 2247, in kernel_error_check_symbolic

[2021-01-15T11:17:51.940Z]     'b':mx.nd.array([],ctx=mx.gpu(0))})

[2021-01-15T11:17:51.940Z]   File "/work/mxnet/tests/python/unittest/../../../python/mxnet/symbol/symbol.py", line 2119, in bind

[2021-01-15T11:17:51.940Z]     ctypes.byref(handle)))

[2021-01-15T11:17:51.940Z]   File "/work/mxnet/tests/python/unittest/../../../python/mxnet/base.py", line 246, in check_call

[2021-01-15T11:17:51.940Z]     raise get_last_ffi_error()

[2021-01-15T11:17:51.940Z] mxnet.base.MXNetError: Traceback (most recent call last):

[2021-01-15T11:17:51.940Z]   [bt] (9) /work/mxnet/python/mxnet/../../lib/libmxnet.so(MXExecutorBindEX+0xc0a) [0x7f8286b4442a]

[2021-01-15T11:17:51.940Z]   [bt] (8) /work/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::Executor::Bind(nnvm::Symbol, mxnet::Context const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, mxnet::Context, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, mxnet::Context> > > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, mxnet::Executor*)+0x204) [0x7f8286c5d364]

[2021-01-15T11:17:51.940Z]   [bt] (7) /work/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::exec::GraphExecutor::Init(nnvm::Symbol, mxnet::Context const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, mxnet::Context, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, mxnet::Context> > > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, mxnet::Executor*, std::unordered_map<nnvm::NodeEntry, mxnet::NDArray, nnvm::NodeEntryHash, nnvm::NodeEntryEqual, std::allocator<std::pair<nnvm::NodeEntry const, mxnet::NDArray> > > const&)+0x11eb) [0x7f8286c5140b]

[2021-01-15T11:17:51.940Z]   [bt] (6) /work/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::exec::InferShape(nnvm::Graph&&, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x83) [0x7f8286c2aeb3]

[2021-01-15T11:17:51.940Z]   [bt] (5) /work/mxnet/python/mxnet/../../lib/libmxnet.so(+0x7b9437e) [0x7f8286c2837e]

[2021-01-15T11:17:51.940Z]   [bt] (4) /work/mxnet/python/mxnet/../../lib/libmxnet.so(+0x7b903c7) [0x7f8286c243c7]

[2021-01-15T11:17:51.940Z]   [bt] (3) /work/mxnet/python/mxnet/../../lib/libmxnet.so(bool mxnet::op::ElemwiseShape<2, 1>(nnvm::NodeAttrs const&, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*)+0x229) [0x7f8281ed7ce9]

[2021-01-15T11:17:51.940Z]   [bt] (2) /work/mxnet/python/mxnet/../../lib/libmxnet.so(bool mxnet::op::ElemwiseAttrHelper<mxnet::TShape, &mxnet::op::shape_is_none, &mxnet::op::shape_assign, true, &mxnet::op::shape_string[abi:cxx11], -1, -1>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*, mxnet::TShape const&)+0x215) [0x7f828132c4e5]

[2021-01-15T11:17:51.940Z]   [bt] (1) /work/mxnet/python/mxnet/../../lib/libmxnet.so(bool mxnet::op::ElemwiseAttrHelper<mxnet::TShape, &mxnet::op::shape_is_none, &mxnet::op::shape_assign, true, &mxnet::op::shape_string[abi:cxx11], -1, -1>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*, mxnet::TShape const&)::{lambda(std::vector<mxnet::TShape, std::allocator<mxnet::TShape> > const&, unsigned long, char const*)#1}::operator()(std::vector<mxnet::TShape, std::allocator<mxnet::TShape> > const&, unsigned long, char const*) const+0x10a8) [0x7f828132acf8]

[2021-01-15T11:17:51.940Z]   [bt] (0) /work/mxnet/python/mxnet/../../lib/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x61) [0x7f82812b5ef1]

[2021-01-15T11:17:51.940Z] MXNetError: Error in operator _div0: [11:17:47] src/operator/numpy/linalg/./../../tensor/../elemwise_op_common.h:135: Check failed: assign(&dattr, vec.at(i)): Incompatible attr in node _div0 at 1-th input: expected [3], got [0]

@mxnet-bot run ci [unix-gpu]

@mxnet-bot run ci [unix-gpu]

What's the advantage of this PR compared to use explicit iter(train_iter) in the auto_reload=False case? As your example code shows, the auto_reload=True case makes it much harder to understand what is going on with the Dataloader, especially if users create multiple iters of the same Dataloader at the same time.. Without your PR, user can just call iter multiple times explicitly and obtain the same behavior. "Explicit is better than implicit." (Zen of Python)

if the users create multiple iters of the same Dataloader at the same time, with this PR, user can also call iter multiple times.
the first time, Dataloader generate the prefetched iter,
and the rest of the time, dataloader return an iter and start to fetch data for the iter.
if you do not modify the Dataloader(e.g., modify self._dataset), all the result except the performance are the same for the iters.
The reason why using Dataloader with auto_reload is:

MXNet 2.0's DataLoader with the default nopython mode prefetch data by default.
There is only one iter for a DataLoader in most of the cases.(Thus only one prefetched iter is generated.)
if we call iter explicitly, we should call it twice (one right after the define of the DataLoader, and another one after the previous iter is consumed).
(maybe we should not using with ag.record(): since "Explicit is better than implicit." (Zen of Python))

Actually, I do not think there is something that was not considered in my Example, I created 4 iters, the only differences among them is the prefetching behavior:
    >>> from mxnet.gluon.data import DataLoader, ArrayDataset
    >>> train_data = ArrayDataset([i for i in range(10)],[9-i for i in range(10)])
    >>> def transform_train(sample):
    ...   if sample == 0 : print('(pre)fetching data here')
    ...   return sample
    ...
    >>> train_iter = DataLoader(train_data.transform_first(transform_train),
    ...                         auto_reload=False, batch_size=1,num_workers=1)
    >>> # no prefetch is performed, the prefetch & autoload start after
    >>> # train_iter.__iter__() is called.
    >>> for i in train_iter:pass
    (pre)fetching data here
    >>> train_iter = DataLoader(train_data.transform_first(transform_train),
    ...                         batch_size=1,num_workers=1)
    (pre)fetching data here
    >>> it = iter(train_iter) # nothing is generated since lazy-evaluation occurs
    >>> it2 = iter(train_iter)
    >>> it3 = iter(train_iter)
    >>> it4 = iter(train_iter)
    >>> _ = next(it2) # the first iter we are using is the prefetched iter.
    >>> _ = next(it) # since the prefetched iter is cconsumed, we have to fetch data for `it`.
    (pre)fetching data here
    >>> _ = [None for _ in it3]
    (pre)fetching data here
    (pre)fetching data here
    >>> # Here, 2 prefetches are triggered, one is fetching the first batch of `it3` and
    >>> # another is when `it3` yield its last item, a prefetch is automatically performed.
    >>> _ = [None for _ in it]
    >>> # no prefetch is happened since train_loader has already prefetch data.
    >>> _ = next(it4)
    >>> # since the prefetch is performed, it4 become the prefetched iter.
    >>>
    >>> test_data = ArrayDataset([i for i in range(10)],[9-i for i in range(10)])
    >>> test_iter = PrefetchedDataLoader(test_data,
    ...                                  batch_size=1,num_workers=1)
    >>> for epoch in range(200):
    ...   # there is almost no difference between it and the default DataLoader
    ...   for data, label in train_iter:
    ...     # training...
    ...   for data, label in test_iter:
    ...     # testing...

What's the advantage of this PR compared to use explicit iter(train_iter) in the auto_reload=False case? As your example code shows, the auto_reload=True case makes it much harder to understand what is going on with the Dataloader, especially if users create multiple iters of the same Dataloader at the same time.. Without your PR, user can just call iter multiple times explicitly and obtain the same behavior. "Explicit is better than implicit." (Zen of Python)

if the users create multiple iters of the same Dataloader at the same time, with this PR, user can also call iter multiple times.
the first time, Dataloader generate the prefetched iter,
and the rest of the time, dataloader return an iter and start to fetch data for the iter.
if you do not modify the Dataloader(e.g., modify self._dataset), all the result except the performance are the same for the iters.
The reason why using Dataloader with auto_reload is:

MXNet 2.0's DataLoader with the default nopython mode prefetch data by default.
There is only one iter for a DataLoader in most of the cases.(Thus only one prefetched iter is generated.)
if we call iter explicitly, we should call it twice (one right after the define of the DataLoader, and another one after the previous iter is consumed).
(maybe we should not using with ag.record(): since "Explicit is better than implicit." (Zen of Python))

Actually, I do not think there is something that was not considered in my Example, I created 4 iters, the only differences among them is the prefetching behavior:
    >>> from mxnet.gluon.data import DataLoader, ArrayDataset
    >>> train_data = ArrayDataset([i for i in range(10)],[9-i for i in range(10)])
    >>> def transform_train(sample):
    ...   if sample == 0 : print('(pre)fetching data here')
    ...   return sample
    ...
    >>> train_iter = DataLoader(train_data.transform_first(transform_train),
    ...                         auto_reload=False, batch_size=1,num_workers=1)
    >>> # no prefetch is performed, the prefetch & autoload start after
    >>> # train_iter.__iter__() is called.
    >>> for i in train_iter:pass
    (pre)fetching data here
    >>> train_iter = DataLoader(train_data.transform_first(transform_train),
    ...                         batch_size=1,num_workers=1)
    (pre)fetching data here
    >>> it = iter(train_iter) # nothing is generated since lazy-evaluation occurs
    >>> it2 = iter(train_iter)
    >>> it3 = iter(train_iter)
    >>> it4 = iter(train_iter)
    >>> _ = next(it2) # the first iter we are using is the prefetched iter.
    >>> _ = next(it) # since the prefetched iter is cconsumed, we have to fetch data for `it`.
    (pre)fetching data here
    >>> _ = [None for _ in it3]
    (pre)fetching data here
    (pre)fetching data here
    >>> # Here, 2 prefetches are triggered, one is fetching the first batch of `it3` and
    >>> # another is when `it3` yield its last item, a prefetch is automatically performed.
    >>> _ = [None for _ in it]
    >>> # no prefetch is happened since train_loader has already prefetch data.
    >>> _ = next(it4)
    >>> # since the prefetch is performed, it4 become the prefetched iter.
    >>>
    >>> test_data = ArrayDataset([i for i in range(10)],[9-i for i in range(10)])
    >>> test_iter = PrefetchedDataLoader(test_data,
    ...                                  batch_size=1,num_workers=1)
    >>> for epoch in range(200):
    ...   # there is almost no difference between it and the default DataLoader
    ...   for data, label in train_iter:
    ...     # training...
    ...   for data, label in test_iter:
    ...     # testing...

The reason why using Dataloader with auto_reload is:
MXNet 2.0's DataLoader with the default nopython mode prefetch data by default.

MXNet 2 uses version number 2 because it breaks APIs. MXNet uses https://semver.org/ and we must not introduce backward incompatible changes in the v1.x branch. (Changing defaults with major impact is backwards incompatible). It's fine to add new features in v1.x.

There is only one iter for a DataLoader in most of the cases.(Thus only one prefetched iter is generated.)
if we call iter explicitly, we should call it twice (one right after the define of the DataLoader, and another one after the previous iter is consumed).

So what's the problem here? Currently I'm not convinced your code / documentation is correct. For example:
    >>> train_iter = DataLoader(train_data.transform_first(transform_train),
    ...                         batch_size=1,num_workers=1)
    (pre)fetching data here
    >>> it = iter(train_iter) # nothing is generated since lazy-evaluation occurs
    >>> it2 = iter(train_iter)
    >>> it3 = iter(train_iter)
    >>> it4 = iter(train_iter)
    >>> _ = next(it2) # the first iter we are using is the prefetched iter.
    >>> _ = next(it) # since the prefetched iter is cconsumed, we have to fetch data for `it`.

However, looking at your implementation, actually 4 prefetched iters are created and the comments in the last two lines are wrong. Please correct me if you disagree.

(maybe we should not using with ag.record(): since "Explicit is better than implicit." (Zen of Python))

What's the relation to the current discussion?

The reason why using Dataloader with auto_reload is:
MXNet 2.0's DataLoader with the default nopython mode prefetch data by default.

MXNet 2 uses version number 2 because it breaks APIs. MXNet uses https://semver.org/ and we must not introduce backward incompatible changes in the v1.x branch. (Changing defaults with major impact is backwards incompatible). It's fine to add new features in v1.x.

There is only one iter for a DataLoader in most of the cases.(Thus only one prefetched iter is generated.)
if we call iter explicitly, we should call it twice (one right after the define of the DataLoader, and another one after the previous iter is consumed).

So what's the problem here? Currently I'm not convinced your code / documentation is correct. For example:
    >>> train_iter = DataLoader(train_data.transform_first(transform_train),
    ...                         batch_size=1,num_workers=1)
    (pre)fetching data here
    >>> it = iter(train_iter) # nothing is generated since lazy-evaluation occurs
    >>> it2 = iter(train_iter)
    >>> it3 = iter(train_iter)
    >>> it4 = iter(train_iter)
    >>> _ = next(it2) # the first iter we are using is the prefetched iter.
    >>> _ = next(it) # since the prefetched iter is cconsumed, we have to fetch data for `it`.

However, looking at your implementation, actually 4 prefetched iters are created and the comments in the last two lines are wrong. Please correct me if you disagree.

(maybe we should not using with ag.record(): since "Explicit is better than implicit." (Zen of Python))

What's the relation to the current discussion?

The reason why using Dataloader with auto_reload is:
MXNet 2.0's DataLoader with the default nopython mode prefetch data by default.

MXNet 2 uses version number 2 because it breaks APIs. MXNet uses https://semver.org/ and we must not introduce backward incompatible changes in the v1.x branch. (Changing defaults with major impact is backwards incompatible). It's fine to add new features in v1.x.

most of the behavior is not changed since it is only prefetch data rather than modify data.

There is only one iter for a DataLoader in most of the cases.(Thus only one prefetched iter is generated.)
if we call iter explicitly, we should call it twice (one right after the define of the DataLoader, and another one after the previous iter is consumed).

So what's the problem here? Currently I'm not convinced your code / documentation is correct. For example:
    >>> train_iter = DataLoader(train_data.transform_first(transform_train),
    ...                         batch_size=1,num_workers=1)
    (pre)fetching data here
    >>> it = iter(train_iter) # nothing is generated since lazy-evaluation occurs
    >>> it2 = iter(train_iter)
    >>> it3 = iter(train_iter)
    >>> it4 = iter(train_iter)
    >>> _ = next(it2) # the first iter we are using is the prefetched iter.
    >>> _ = next(it) # since the prefetched iter is cconsumed, we have to fetch data for `it`.

However, looking at your implementation, actually 4 prefetched iters are created and the comments in the last two lines are wrong. Please correct me if you disagree.

due to the lazy evaluation, the iter will not call self.refresh/self.clean until the first __next__() is called, thus we have 4 iters, but only the first iter we use (it2 here) is the prefetched iter.
what's more, for a regular training procedure:
    >>> train_data = ArrayDataset([i for i in range(10)],[9-i for i in range(10)])
    >>> def transform_train(sample):
    ...   if sample == 0 : print('(pre)fetching data here')
    ...   return sample
    ...
    >>> train_iter = DataLoader(train_data.transform_first(transform_train),
    ...                         auto_reload=False, batch_size=1,num_workers=1)
    >>> test_data = ArrayDataset([i for i in range(10)],[9-i for i in range(10)])
    >>> test_iter = DataLoader(test_data, batch_size=1,num_workers=1)
    >>> for epoch in range(200):
    ...   # there is almost no difference between it and the default DataLoader
    ...   for data, label in train_iter:
    ...     # training...
    ...   for data, label in test_iter:
    ...     # testing...

there is only one iter per DataLoader each time. Most of the times, users will not consider what happened under the dataloader.

(maybe we should not using with ag.record(): since "Explicit is better than implicit." (Zen of Python))

What's the relation to the current discussion?

Here we implicit modify something help for calculate the gradient of the network.
I say it just for that, it is fine for us to using some implicit operations to simplify the execution of the program.

The reason why using Dataloader with auto_reload is:
MXNet 2.0's DataLoader with the default nopython mode prefetch data by default.

MXNet 2 uses version number 2 because it breaks APIs. MXNet uses https://semver.org/ and we must not introduce backward incompatible changes in the v1.x branch. (Changing defaults with major impact is backwards incompatible). It's fine to add new features in v1.x.

most of the behavior is not changed since it is only prefetch data rather than modify data.

There is only one iter for a DataLoader in most of the cases.(Thus only one prefetched iter is generated.)
if we call iter explicitly, we should call it twice (one right after the define of the DataLoader, and another one after the previous iter is consumed).

So what's the problem here? Currently I'm not convinced your code / documentation is correct. For example:
    >>> train_iter = DataLoader(train_data.transform_first(transform_train),
    ...                         batch_size=1,num_workers=1)
    (pre)fetching data here
    >>> it = iter(train_iter) # nothing is generated since lazy-evaluation occurs
    >>> it2 = iter(train_iter)
    >>> it3 = iter(train_iter)
    >>> it4 = iter(train_iter)
    >>> _ = next(it2) # the first iter we are using is the prefetched iter.
    >>> _ = next(it) # since the prefetched iter is cconsumed, we have to fetch data for `it`.

However, looking at your implementation, actually 4 prefetched iters are created and the comments in the last two lines are wrong. Please correct me if you disagree.

due to the lazy evaluation, the iter will not call self.refresh/self.clean until the first __next__() is called, thus we have 4 iters, but only the first iter we use (it2 here) is the prefetched iter.
what's more, for a regular training procedure:
    >>> train_data = ArrayDataset([i for i in range(10)],[9-i for i in range(10)])
    >>> def transform_train(sample):
    ...   if sample == 0 : print('(pre)fetching data here')
    ...   return sample
    ...
    >>> train_iter = DataLoader(train_data.transform_first(transform_train),
    ...                         auto_reload=False, batch_size=1,num_workers=1)
    >>> test_data = ArrayDataset([i for i in range(10)],[9-i for i in range(10)])
    >>> test_iter = DataLoader(test_data, batch_size=1,num_workers=1)
    >>> for epoch in range(200):
    ...   # there is almost no difference between it and the default DataLoader
    ...   for data, label in train_iter:
    ...     # training...
    ...   for data, label in test_iter:
    ...     # testing...

there is only one iter per DataLoader each time. Most of the times, users will not consider what happened under the dataloader.

(maybe we should not using with ag.record(): since "Explicit is better than implicit." (Zen of Python))

What's the relation to the current discussion?

Here we implicit modify something help for calculate the gradient of the network.
I say it just for that, it is fine for us to using some implicit operations to simplify the execution of the program.

@mxnet-bot run ci [centos-cpu]

@mxnet-bot run ci [centos-cpu]


Automator: update ztunnel@master in istio/istio@master

Generated by Automator - 2024-03-07T15:34:07+00:00

@istio-testing: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-ambient_istio
46415b8b2df34ed4fcda5e9f4cd0010ae981409f
link
true
/test integ-ambient

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Issue with https://github.com/istio/ztunnel/pull/820?

/retest

https://github.com/istio/ztunnel/pull/768  is the only pr related to cert

@istio-testing: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-ambient_istio
46415b8b2df34ed4fcda5e9f4cd0010ae981409f
link
true
/test integ-ambient

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

cc @MorrisLaw

@istio-testing: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:
Test name	Commit	Details	Required	Rerun command
integ-ambient_istio	b893636 link	true	/test integ-ambient

Looking into this. I think the issue is with the more dynamic retry time for refreshing certs. Used to be 60 seconds, the retry duration now starts at 500ms and gets up to as high as 2.5 minutes.

@istio-testing: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-ambient_istio
6ba72572ee3451a68cb62e439368237fbfa4aee4
link
true
/test integ-ambient

unit-tests-arm64_istio
6ba72572ee3451a68cb62e439368237fbfa4aee4
link
true
/test unit-tests-arm64

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-ambient_istio
e67f54cfd208f1441415a9497c2f6dd84676d7d0
link
true
/test integ-ambient

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-ambient_istio
c62852d9d9d145e4a6bb9e1a6ee890e0a3f5d616
link
true
/test integ-ambient

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-ambient_istio
eb5eaed5a2ec275015b94654f7a0cddd5f5d5332
link
true
/test integ-ambient

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-ambient_istio
d6a0d9fe35a6ddecbabea919705b390e96a0e8cb
link
true
/test integ-ambient

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-ambient_istio
0ac14d1bd0b85ed8ee7fb9ac142476dd7dbbdc1a
link
true
/test integ-ambient

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-ambient_istio
0ac14d1bd0b85ed8ee7fb9ac142476dd7dbbdc1a
link
true
/test integ-ambient

integ-security-multicluster_istio
0ac14d1bd0b85ed8ee7fb9ac142476dd7dbbdc1a
link
true
/test integ-security-multicluster

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-security-multicluster_istio
0ac14d1bd0b85ed8ee7fb9ac142476dd7dbbdc1a
link
true
/test integ-security-multicluster

integ-ambient_istio
a1b9c4e991cddc0ef219421b89bdeeb108edaf12
link
true
/test integ-ambient

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-security-multicluster_istio
0ac14d1bd0b85ed8ee7fb9ac142476dd7dbbdc1a
link
true
/test integ-security-multicluster

integ-ambient_istio
a1b9c4e991cddc0ef219421b89bdeeb108edaf12
link
true
/test integ-ambient

integ-telemetry_istio
a1b9c4e991cddc0ef219421b89bdeeb108edaf12
link
true
/test integ-telemetry

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-security-multicluster_istio
0ac14d1bd0b85ed8ee7fb9ac142476dd7dbbdc1a
link
true
/test integ-security-multicluster

integ-ambient_istio
a1b9c4e991cddc0ef219421b89bdeeb108edaf12
link
true
/test integ-ambient

integ-telemetry_istio
a1b9c4e991cddc0ef219421b89bdeeb108edaf12
link
true
/test integ-telemetry

integ-telemetry-istiodremote_istio
a1b9c4e991cddc0ef219421b89bdeeb108edaf12
link
true
/test integ-telemetry-istiodremote

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-telemetry_istio
a1b9c4e991cddc0ef219421b89bdeeb108edaf12
link
true
/test integ-telemetry

integ-telemetry-istiodremote_istio
a1b9c4e991cddc0ef219421b89bdeeb108edaf12
link
true
/test integ-telemetry-istiodremote

integ-ambient_istio
69106f550ad8b65f1ae3750b2ad7afb7c55a6b9b
link
true
/test integ-ambient

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-ambient_istio
5370712f80bd1faf28db9b99206bee7b20e489eb
link
true
/test integ-ambient

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-ambient_istio
5370712f80bd1faf28db9b99206bee7b20e489eb
link
true
/test integ-ambient

integ-telemetry_istio
5370712f80bd1faf28db9b99206bee7b20e489eb
link
true
/test integ-telemetry

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-telemetry_istio
5370712f80bd1faf28db9b99206bee7b20e489eb
link
true
/test integ-telemetry

integ-ambient_istio
98217a88ed1321fca872cc97a7266b493ab69b33
link
true
/test integ-ambient

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-ambient_istio
446dcae0f7aa25d16e79a9e80806fa5b1570720a
link
true
/test integ-ambient

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-ambient_istio
3c2396cecf4a6be49413d027827715bba25d3c2d
link
true
/test integ-ambient

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-ambient_istio
3c2396cecf4a6be49413d027827715bba25d3c2d
link
true
/test integ-ambient

integ-pilot-multicluster_istio
3c2396cecf4a6be49413d027827715bba25d3c2d
link
true
/test integ-pilot-multicluster

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-pilot-multicluster_istio
3c2396cecf4a6be49413d027827715bba25d3c2d
link
true
/test integ-pilot-multicluster

integ-ambient_istio
3c5236c8ea83bdb865e1047fb13d55373228c99c
link
true
/test integ-ambient

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-pilot-multicluster_istio
3c2396cecf4a6be49413d027827715bba25d3c2d
link
true
/test integ-pilot-multicluster

integ-ambient_istio
3c5236c8ea83bdb865e1047fb13d55373228c99c
link
true
/test integ-ambient

unit-tests-arm64_istio
3c5236c8ea83bdb865e1047fb13d55373228c99c
link
true
/test unit-tests-arm64

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-ambient_istio
d3448fee8895b2af577648393d4d3edd4e55b254
link
true
/test integ-ambient

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-ambient_istio
dbfa73bbc7147339d06f95b239ea88046f4917e3
link
true
/test integ-ambient

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-ambient_istio
0bd534783d7b6b61498d69db21fd9b7b8706adeb
link
true
/test integ-ambient

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


ACR Build doesn't honor included directories in dockeringore

Describe the bug
A directory is excluded by a .dockerignore file, and a single directory within that directory is included. az acr build doesn't seem to understand this.
To Reproduce
Have a Dockerfile that exludes a directory called database/, but includes the directory within that database/directory-to-include
Like this:
# Exclude the entire directory
**/database

# Include this (override the exclusions)
!/database/directory-to-include

az acr build will throw an error because the contents of /database/directory-to-include won't be included in the build context.
Expected behavior
That it works. It works fine with docker build for the same Dockerfile.

Minimal repro:
Create a file under:
/ignoredfolder/includedsubfolder/file.txt
Dockerfile:
 FROM node:12-alpine
 WORKDIR /app
 COPY ["ignoredfolder/includedsubfolder/file.txt", "src"]
 COPY . .
 ENTRYPOINT ["sh"]

.dockerignore:
**/ignoredfolder
!/ignoredfolder/includedsubfolder

Then run:
az acr build --image some-name  --registry myregistry.azurecr.io --file .\Dockerfile .

route to appropriate team

I am having a similar issue but without directories. It seems az acr build and docker build don't interpret the .dockerignore file the same way.
.dockerignore file:
# Ignore everything
**

# Allow files and directories
!/start.sh

run az acr build ends with an error:
Step 9/28 : COPY ./start.sh .
COPY failed: stat /var/lib/docker/tmp/docker-builder021336173/start.sh: no such file or directory

Whereas docker build runs successfully.

+@northtyphoon to investigate

Problem is here: https://github.com/Azure/azure-cli/blob/29767d75d850ddc1c24cc85bd46d861b61d77a47/src/azure-cli/azure/cli/command_modules/acr/_archive_utils.py#L87
It only do matching to ignore_items.

To @CarolineNB for investigation. Thanks!

This still doesn't seem to work...
Example dockerignore...
https://github.com/dracan/danclarkeblog/blob/7fa215bb19f15358492c4d725d0aa513a42b0465/.dockerignore
And failing build...
https://github.com/dracan/danclarkeblog/runs/8072733424?check_suite_focus=true

Hi @dracan, thank you for reaching out. We are looking into it.

Hi @dracan, try removing the trailing '/'s in your .dockerignore file and let me know if that works.
This will be fixed in the next CLI release, but I don't want you to have to wait until then.
Thanks!


S3 writes & local s3 test server for CI

Implemented basic implementation for writing  HTTP/S3 files. The files are converted to buffers on opening to allow writes. On closing the file we upload the buffer synchronously with a single HTTP PUT request. For large uploads we should look into multipart uploads. I can look into that after implementing s3 writes in DuckDB as well.
I also added a local test s3 server using S3rver. It seems to work quite well, but as discussed with @ankoh, we should keep an eye on its stability and possibly change it for something more solid such as [minio]{https://min.io/}. switching to minio should be quite simple but complicate the local dev
Note that s3rver only really seems to validate the credentials and does not actually check the authorization signature, therefore I added a testcase to the manual validation of the auth header to make this a bit more solid.

Awesome, thx!

For large uploads we should look into multipart uploads. I can look into that after implementing s3 writes in DuckDB as well.

Hi @samansmink, @ankoh -- not sure if this is helpful, but we developed and used this a ways back to upload large video files (1TB) to s3 using threads with webworkers: https://github.com/cinely/mule-uploader. Though I'm sure there are lots of similar things that you could use but see if that might work for this.

Hey @david542542 thanks for the suggestion! That definitely looks interesting for us!


Multithreading with TBB  Module in AMD processor is significantly slower

This is regarding the TBB mulithreading enhancement.
While working on ITKRANSAC module I found that a locally build package (AMD system)
without TBB module  support is around 10 times faster than the pip package obtained from Github repo.
The difference is the presence of TBB module.
On building the module locally with TBB support I see similar slow results.
Is there a recommended way to handle this such that the optimized way to perform multithreading is selected?
I am using MultiThreaderBase as following:
itk::MultiThreaderBase::SetGlobalDefaultNumberOfThreads(this->numberOfThreads);
itk::MultiThreaderBase::Pointer threader = itk::MultiThreaderBase::New();

As per my reading, I found that the TBB module is independent of the processor but I am getting slow results on my system.
Another observation I found is that even with a lesser number of threads I see all the CPU cores getting utilized when using the non-TBB package. And while using the TBB module I see the correct number of cores getting used.
Is there some optimization happening under the hood in AMD system?

Description

Impact analysis

Expected behavior

Actual behavior

Versions

Environment
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.3 LTS
Release:	20.04
Codename:	focal

AMD Ryzen 7 5800 8-Core Processor

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   48 bits physical, 48 bits virtual
CPU(s):                          16
On-line CPU(s) list:             0-15
Thread(s) per core:              2
Core(s) per socket:              8

Additional Information

I have AMD system on my main Windows workstation. I ran extensive benchmarks of ITKMontage, both with and without TBB. I found it to be slightly better than other multi-threaders in ITK. One interesting thing I noticed when using TBB, CPU use was sometimes not recorded as 100%, despite using maximum threading and not being slower than other multi-threaders. My CPU is AMD Ryzen 7 Pro 1700 8-core.

@PranjalSahu do you get the same results for the RANSAC tests on an Intel system? You could try blaster.kitware.com.


OLS-160: Definition of Done

Description
Definition of Done
Type of change

[ ] Refactor
[ ] New feature
[ ] Bug fix
[ ] CVE fix
[ ] Optimization
[x] Documentation Update
[ ] Configuration Update
[ ] Bump-up dependent library

Related Tickets & Documents

Related Issue #OLS-160: Definition of Done

It's there to be discussed first!

/approve
/lgtm


WIP Update PPP

Update PPP

Pull Request Test Coverage Report for Build 794

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 94.364%

Totals

Change from base Build 775:
0.0%

Covered Lines:
2478

Relevant Lines:
2626

💛  - Coveralls


Correct indent eslint rule config

Q
A

Bug fix?
no

Breaking change?
no

New feature?
no

Deprecations?
no

Spec compliancy?
no

Tests added/pass?
no

Fixed tickets
comma-separated list of tickets fixed by the PR, if any

License
MIT

This enables the eslint indent rule the way it should be and removes all the disables of this rule from the js files.
refs: babel/eslint-config-babel#10

Current coverage is 97.47% (diff: 100%)

Merging #276 into master will not change coverage

@@             master       #276   diff @@
==========================================
  Files            21         21
  Lines          3996       3996
  Methods         476        486    +10
  Messages          0          0
  Branches       1178       1168    -10
==========================================
  Hits           3895       3895
  Misses           44         44
  Partials         57         57

Powered by Codecov. Last update cd9aaf2...f0403ff


no need to override the UIWindow methods if you're in release mode

very minor change.
moving the prerocessor macro to encapsulate the method overrides.
it might prevent a collision with other shake gesture recognizer in release mode.

thank you


[Transcripts] Revamp Gong oAuth to use a default workspace configuration

Description
Revamps Gong's transcripts to use a default connection for the entire workspace while making sure users only receive transcripts of calls they participated in.

First user to setup the Gong connection will be redirected to Nango (they need to be a technical admin in Gong for the whole thing to go through)
Other users see the same connection button, but clicking Connect just creates a configuration with an empty connectionId
Workflow keep running on each user's connection to make sure we only send transcripts to users that participated in specific calls
All Gong API calls are made using the workspace default config

Risk

Deploy Plan

@albandum let me know once addressed the comments from our live discussion 🙃.

@albandum I believe you had a chat with @Duncid on this? What was the outcome? Do we have a firm plan here?

@albandum I believe you had a chat with @Duncid on this? What was the outcome? Do we have a firm plan here?

The chat was about another PR (moving the Beta menu somewhere else) - the outcome was that we'll have to build a User menu somewhere someday.