fix fcos gt_areas calculation

I fix the gt_areas calculation in the FCOS model.

Hi @datumbox , can you take a look of this?

@simonJJJ LGTM, thanks a lot for patching this. This is definitely a typo which remained undetected.
@xiaohu2015 @zhiqwang @jdsgomes I assume this ended up having little effect on the overall accuracy. Do we want to retrain and confirm?

yes, I think it maybe has little effect on the overall accuracy, becuase here we don't have many situations which one position match multiple GTs. But I think we should retrain the model if possible.

I fix the gt_areas calculation in the FCOS model.

Thanks a lot

Thanks for the fix @simonJJJ . I am running the training just to make sure the impact is not significant.

Thanks for the fix @simonJJJ . I am running the training just to make sure the impact is not significant.

I have re-run the training job and the performance difference does not seem significant.
With the fix:
IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.390
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.578
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.418
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.218
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.423
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.513
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.326
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.530
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.565
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.358
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.602
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.731
Training time 5:50:18
submitit INFO (2022-04-21 01:28:12,411) - Job completed successfully

Original results:
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.392
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.582
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.420
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.222
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.423
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.513
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.324
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.529
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.562
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.350
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.609
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.728

So I don't think there is a need to update the pre-trained weights or take any further action.
In any case thank you @simonJJJ   again for making the fix!

Agreed. I don't think it's worth releasing a new pair of weights for 0.2 mAP.
Soon we can try to retraining FCOS after #5410 is completed to really push its accuracy similarly to what we did with FasterRCNN and RetinaNet. This should give us a material push on the accuracy of several mAP points.


would like to be able to monitor LOCAL_DEVELOPMENT server

I would like to be able to set a http_port for the embedded local development server.
Would be nice to be able to run nats-top.

Done in https://github.com/nats-io/natscli/pull/748


Slicing data using the square brackets

Hello!
Would that makes sense to allow slicing using text, instead of requiring to access via isel?
For instance, a["x", "marker1", :] would call in the background return a.isel(...)
I think it would make things much easier for someone who is not accustomed to xarray. If I recall well, we discussed this already, but I don't recall the reason not to do it. Recently I helped someone who did not know how to use xarrays precisely for the reason (and I must admit, I had to browse on internet myself quite a lot to perform what feels to be very simple tasks such a slicing data).
In terms of changes, I think it would be very minimal, but would greatly help the community!

It is already possible to do this: markers.loc["x", "markers1", :].
Here, loc refers to a selection by label.
xarray already allows many indexing routines that cover most use cases, see here.
These can be summarized here :

Dimension lookup
Index lookup
Syntax

Positional
By integer
d[:, 0]

Positional
By label
d.loc[:, 'IA']

By name
By integer
d.isel(space=0)

By name
By label
d.sel(space='IA')


pasted code block

Description
What steps will reproduce the problem?

pasted code intoo scriptpasted code intoo scriptpasted code intoo scriptpasted code intoo scriptpasted code intoo scriptpasted code intoo scriptpasted code intoo script
pasted code intoo scriptpasted code intoo scriptpasted code intoo scriptpasted code intoo scriptpasted code intoo script
Traceback
  File "/Users/hidemiasakura/anaconda/envs/py37/lib/python3.7/site-packages/spyder/plugins/editor/panels/codefolding.py", line 256, in paintEvent

Traceback (most recent call last):
  File "/Users/hidemiasakura/anaconda/envs/py37/lib/python3.7/site-packages/spyder/plugins/editor/panels/codefolding.py", line 256, in paintEvent
    self._draw_fold_region_background(block, painter)
  File "/Users/hidemiasakura/anaconda/envs/py37/lib/python3.7/site-packages/spyder/plugins/editor/panels/codefolding.py", line 300, in _draw_fold_region_background
    end = self.folding_regions[start]
KeyError: 135

QPainter::setCompositionMode: Painter not active
Traceback (most recent call last):
  File "/Users/hidemiasakura/anaconda/envs/py37/lib/python3.7/site-packages/spyder/plugins/editor/panels/codefolding.py", line 256, in paintEvent
    self._draw_fold_region_background(block, painter)
  File "/Users/hidemiasakura/anaconda/envs/py37/lib/python3.7/site-packages/spyder/plugins/editor/panels/codefolding.py", line 300, in _draw_fold_region_background
    end = self.folding_regions[start]
KeyError: 135

QPainter::setCompositionMode: Painter not active
Traceback (most recent call last):
  File "/Users/hidemiasakura/anaconda/envs/py37/lib/python3.7/site-packages/spyder/plugins/editor/panels/codefolding.py", line 256, in paintEvent
    self._draw_fold_region_background(block, painter)
  File "/Users/hidemiasakura/anaconda/envs/py37/lib/python3.7/site-packages/spyder/plugins/editor/panels/codefolding.py", line 300, in _draw_fold_region_background
    end = self.folding_regions[start]
KeyError: 135

QPainter::setCompositionMode: Painter not active
Traceback (most recent call last):
  File "/Users/hidemiasakura/anaconda/envs/py37/lib/python3.7/site-packages/spyder/plugins/editor/panels/codefolding.py", line 256, in paintEvent
    self._draw_fold_region_background(block, painter)
  File "/Users/hidemiasakura/anaconda/envs/py37/lib/python3.7/site-packages/spyder/plugins/editor/panels/codefolding.py", line 300, in _draw_fold_region_background
    end = self.folding_regions[start]
KeyError: 135

QPainter::setCompositionMode: Painter not active
Traceback (most recent call last):
  File "/Users/hidemiasakura/anaconda/envs/py37/lib/python3.7/site-packages/spyder/plugins/editor/panels/codefolding.py", line 256, in paintEvent
QPainter::begin: A paint device can only be painted by one painter at a time.
QPainter::setCompositionMode: Painter not active
QBackingStore::endPaint() called with active painter on backingstore paint device
    self._draw_fold_region_background(block, painter)QPainter::begin: A paint device can only be painted by one painter at a time.
QPainter::setCompositionMode: Painter not active
QBackingStore::endPaint() called with active painter on backingstore paint device

  File "/Users/hidemiasakura/anaconda/envs/py37/lib/python3.7/site-packages/spyder/plugins/editor/panels/codefolding.py", line 300, in _draw_fold_region_background

QPainter::setCompositionMode: Painter not active
QWidget::repaint: Recursive repaint detected

Traceback (most recent call last):
  File "/Users/hidemiasakura/anaconda/envs/py37/lib/python3.7/site-packages/spyder/plugins/editor/panels/codefolding.py", line 256, in paintEvent
    self._draw_fold_region_background(block, painter)
  File "/Users/hidemiasakura/anaconda/envs/py37/lib/python3.7/site-packages/spyder/plugins/editor/panels/codefolding.py", line 300, in _draw_fold_region_background
    end = self.folding_regions[start]
KeyError: 135

Versions

Spyder version: 4.0.0rc3
Python version: 3.7.0
Qt version: 5.9.6
PyQt5 version: 5.9.2
Operating System: Darwin 17.7.0

Dependencies
cloudpickle >=0.5.0          :  1.2.2 (OK)
pygments >=2.0               :  2.4.2 (OK)
qtconsole >=4.6.0            :  4.6.0 (OK)
nbconvert >=4.0              :  5.6.0 (OK)
sphinx >=0.6.6               :  2.2.0 (OK)
pylint >=0.25                :  2.4.2 (OK)
psutil >=0.3                 :  5.6.3 (OK)
qtawesome >=0.5.7            :  0.6.0 (OK)
qtpy >=1.5.0                 :  1.9.0 (OK)
pickleshare >=0.4            :  0.7.5 (OK)
zmq >=17                     :  18.1.0 (OK)
chardet >=2.0.0              :  3.0.4 (OK)
numpydoc >=0.6.0             :  0.9.1 (OK)
spyder_kernels >=1.8.0;<2.0.0:  1.8.0 (OK)
qdarkstyle >=2.7             :  2.7 (OK)
atomicwrites >=1.2.0         :  1.3.0 (OK)
diff_match_patch >=20181111  :  20181111 (OK)
intervaltree                 :  None (OK)
watchdog                     :  None (OK)
keyring                      :  None (OK)
pexpect >=4.4.0              :  4.7.0 (OK)
pympler                      :  None (OK)
sympy >=0.7.3                :  None (NOK)
cython >=0.21                :  0.29.13 (OK)
IPython >=4.0                :  7.8.0 (OK)
matplotlib >=2.0.0           :  3.1.1 (OK)
pandas >=0.13.1              :  0.25.2 (OK)
numpy >=1.7                  :  1.17.2 (OK)
scipy >=0.17.0               :  1.3.1 (OK)
pyls >=0.31.1;<0.32.0        :  0.31.1 (OK)
applaunchservices >=0.1.7    :  0.2.1 (OK)
rtree >=0.8.3                :  0.8.3 (OK)

Duplicate of #10889

Please don't open more issues about the same problem.

Also, please paste on issue #10889 the code you pasted in the editor that generated this error.

was so tired, i was clicking close and the report window just kept on
pooping out, wouldn't close, even if i clicked on close button, so i tried
to submit it to see if it would close so i can just save my script, but
that didnt work either.  So i had to quit spyder to make it go away.
thanks for your patience.
On Sat, Nov 30, 2019 at 12:32 PM Carlos Cordoba<EMAIL_ADDRESS>wrote:

Also, please paste on issue #10889
https://github.com/spyder-ide/spyder/issues/10889 the code you pasted
in the editor that generated this error.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/spyder-ide/spyder/issues/10888?email_source=notifications&email_token=ACM6IWJ3C6NABVAJH4U4SPLQWHUHDA5CNFSM4JTE3YT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFPZF3A#issuecomment-559911660,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACM6IWJ2HCAJXZJGJK5BPQ3QWHUHDANCNFSM4JTE3YTQ
.

I dont have the code that I pasted, but I do have what was printed in the
console multiple times.
QPainter::begin: A paint device can only be painted by one painter at a
time.
QPainter::setCompositionMode: Painter not active
QPainter::begin: A paint device can only be painted by one painter at a
time.
QPainter::setCompositionMode: Painter not active
QWidget::repaint: Recursive repaint detected
QBackingStore::endPaint() called with active painter on backingstore paint
device
QPainter::begin: A paint device can only be painted by one painter at a
time.
QPainter::setCompositionMode: Painter not active
QBackingStore::endPaint() called with active painter on backingstore paint
device
On Sat, Nov 30, 2019 at 1:20 PM Hidemi Asakura<EMAIL_ADDRESS>wrote:

was so tired, i was clicking close and the report window just kept on
pooping out, wouldn't close, even if i clicked on close button, so i tried
to submit it to see if it would close so i can just save my script, but
that didnt work either.  So i had to quit spyder to make it go away.
thanks for your patience.
On Sat, Nov 30, 2019 at 12:32 PM Carlos Cordoba<EMAIL_ADDRESS>wrote:

Also, please paste on issue #10889
https://github.com/spyder-ide/spyder/issues/10889 the code you pasted
in the editor that generated this error.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/spyder-ide/spyder/issues/10888?email_source=notifications&email_token=ACM6IWJ3C6NABVAJH4U4SPLQWHUHDA5CNFSM4JTE3YT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFPZF3A#issuecomment-559911660,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACM6IWJ2HCAJXZJGJK5BPQ3QWHUHDANCNFSM4JTE3YTQ
.

was so tired, i was clicking close and the report window just kept on
pooping out, wouldn't close, even if i clicked on close button

Ok, no problem. I understand this error can be very annoying.
If you see this error again, please post the code that generated it on issue #10889, as I advised you above.

@hasakura511 Was there a reason you didn't just click the Don't show again checkbox in the error dialog? That's exactly what its there for, so the error dialog won't keep popping up again and again.

im not sure that worked either ,  don’t remember exactly however

On Dec 2, 2019, at 3:14 AM, CAM Gerlach<EMAIL_ADDRESS>wrote:

@hasakura511 Was there a reason you didn't just click the Don't show again checkbox in the error dialog? That's exactly what its there for, so the error dialog won't keep popping up again and again.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

@hasakura511 If that didn't work either, then its another bug that would need to be fixed. At least running Spyder 4.0.0rc3 and master, I could not reproduce this issue; if I manually triggered an error in the Internal Console, checking the box would make the dialog stop re-appearing. If you can reliably reproduce it and its not already reported here, please file an issue with all the details. Thanks.


Include non Bower file

Is there any way to include non bower plugin?
I want to escape creating separate concat task for some plugins which are not available trough Bower.

If the plugin has a public repository, fork it and add a bower.json to it, then pull from your own repo.
Here's the one I just did for flot, which I needed but had an obsolete bower specification:
{
	"name": "flot",
	"version": "0.8.3",
	"main": "jquery.flot.js",
	"dependencies": {
		"jquery": ">= 1.2.6"
	}
}

Then, I just added this to the "dependencies" in the bower.json for my main project:
"flot": "git@github.com:ssteinerx/flot.git#^0.8.3"


Update ExtendedMath: Change ToFixed() to return a string instead of a number

Change the ToFixed() function to return a string to match the javascript function that it emulates (and uses internally).
Previously, this function returned a number which meant that whole numbers would not display the right number of zeroes after the decimal point.
This update could break games that use this function AND manually update this extension.

It seems good to me.
What do you thin about "Format" instead of "Convert"?

I'm unsure why this has to be a breaking change, not wanting to have a number exceed a certain amount of digits after the period is a perfectly valid use and what it must have been used for so far. Let's just make a separate function for formatting it as a string with the extra zeroes for those who want this other use case.

I'm unsure why this has to be a breaking change, not wanting to have a number exceed a certain amount of digits after the period is a perfectly valid use and what it must have been used for so far. Let's just make a separate function for formatting it as a string with the extra zeroes for those who want this other use case.

There is a reason why the JS method return a string:

JS numbers have no explicit 0
JS numbers are binary float not decimal float so it's not possible to round them in the decimal sense. 0.1 is not a rational binary number, printing it with the default format will give a lot of digits.

I'm unsure why this has to be a breaking change, not wanting to have a number exceed a certain amount of digits after the period is a perfectly valid use and what it must have been used for so far. Let's just make a separate function for formatting it as a string with the extra zeroes for those who want this other use case.

Doesn't the "trunc()" function already perform what you are describing?

There is a reason why the JS method return a string

No doubt that there's a reason why you would or should prefer a string there, though that's not the problem, the problem is that this PR is a breaking change. Therefore, I think that this should be made an extra function to not break people's games when updating. A breaking change must only be made of the tradeoffs are worth it, and I don't think that having some zeroes or specific numbers displayed, which is not even something we can be sure current users of this function may want or care about, is worth breaking all of their games if they decide to update, especially when GDevelop doesn't (yet) have a way to warn about breaking changes.

There is a reason why the JS method return a string

No doubt that there's a reason why you would or should prefer a string there, though that's not the problem, the problem is that this PR is a breaking change. Therefore, I think that this should be made an extra function to not break people's games when updating. A breaking change must only be made of the tradeoffs are worth it, and I don't think that having some zeroes or specific numbers displayed, which is not even something we can be sure current users of this function may want or care about, is worth breaking all of their games if they decide to update, especially when GDevelop doesn't (yet) have a way to warn about breaking changes.

In another hand, this expression never did what it says it does and can not possibility do it.
Should we keep a bugged expression that seems to work at first glance but will sometimes do graphical bugs in games?

I added some release notes to the description and incremented a major version to honor SemVer.
Version 2.0.0
- "ToFixed()" now returns a string instead of a number.  This matches the javascript function and is helpful because whole numbers will now show the right number of zeroes.
- NOTE: This is a breaking change.  If you upgrade to this version, make sure to update every expression that uses "ToFixed()".

I really think that "format" is a better choice than "convert".

"convert" usually imply a operation that can be done both way. This is not the case here.
"format" is widely used to describe a way to present a number with text.
This not only a technical term, even Calc uses it. This "format" documentation has a "leading zeroes" section: https://help.libreoffice.org/latest/en-US/text/shared/01/05020300.html

Since both of you agree, I will change the wording to "format".
Just for fun:
Mozilla uses "format"
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/toFixed
W3 uses "convert"
https://www.w3schools.com/jsref/jsref_tofixed.asp

this expression never did what it says it does and can not possibility do it.

I disagree, the expression says it rounds the number to up to a specific amount of digits. That's what it does, it never said it assured a fixed number of decimals. Sure, the name ToFixed might be confusing, and not correspond to JavaScript, but at no point does this extension or GDevelop assure a 1:1 mapping of the JavaScript API.
Either way, I think we should keep that function, bugged or not. More people will get issues by upgrading than because of graphical bugs due to one or two decimals that are 0 not being displayed.
The community keeps on complaining more and more about breaking changes. By allowing such changes because it doesn't fit your idea of what that function is supposed to do is not in my opinion for the best. Users use the function as it is and rely on it, and that change means suddenly breaking parts of their game, and while I see why you would want a different behavior for this expressions, this shouldn't be forced onto users if it may break their games.
Note also that keeping it doesn't mean we are recommending it in any way, we can make it private so that it doesn't show up anymore in autocomplete or the expression builder. It just means that we are not ruthlessly breaking players games.

Note also that keeping it doesn't mean we are recommending it in any way, we can make it private so that it doesn't show up anymore in autocomplete or the expression builder. It just means that we are not ruthlessly breaking players games.

Ok, what name should we choose? "FormatToFixed"?
We could add a warning in the description of the old expression like: "This expression is deprecated. Prefer using FormatToFixed to set values in text objects. This avoids to sometimes display a lot more digits than expected."

I disagree, the expression says it rounds the number to up to a specific amount of digits. That's what it does, it never said it assured a fixed number of decimals. Sure, the name ToFixed might be confusing, and not correspond to JavaScript, but at no point does this extension or GDevelop assure a 1:1 mapping of the JavaScript API.

You can't decimally round a float. That is just not possible, many values doesn't exist.

Ok, what name should we choose? "FormatToFixed"?

I reverted the breaking change and added the ToFixedString() function.

Ok

We could add a warning in the description of the old expression like: "This expression is deprecated. Prefer using FormatToFixed to set values in text objects. This avoids to sometimes display a lot more digits than expected."

I think you might be confused about the problem. Yes, the slider extension has the bug where the Value can have dozens of decimals for certain values (I wonder if we should fix that by inserting the ToFixed() javascript function into that extension...)
The "ToFixed" problem is that extra zeros at the end are not preserved, so ToFixed(5.000,3) becomes "5".

Indeed, JS numbers has enough precision to round display numbers with few digit correctly. So the fact that the returned number is not right doesn't show.
Yes, we can tell about the trailing zeroes. Something like: "This expression is deprecated. Prefer using ToFixedString to set values in text objects. This allows to keep the trailing zeroes."

The ToFixed() function doesn't round, it truncates.

No, the difference between rounding and truncating is:

round(1.127, 2) = 1.13
truncate(1.127, 2) = 1.12
round(1.124, 2) = 1.12
truncate(1.124, 2) = 1.12

I think I've made all the changes people wanted.  Does anyone want to change from yellow to green?

I think I've made all the changes people wanted. Does anyone want to change from yellow to green?

What about making "ToFixed" hidden and adding this: "This expression is deprecated. Prefer using ToFixedString to set values in text objects. This allows to keep the trailing zeroes." to its description?

What about making "ToFixed" hidden and adding this: "This expression is deprecated. Prefer using ToFixedString to set values in text objects. This allows to keep the trailing zeroes." to its description?

Great ideas!  I am submitting these changes soon...

@Bouh I guess your change requests have been answered. Do you confirm?

@Bouh I guess your change requests have been answered. Do you confirm?

I confirm :)
Merged, thank you everyone for this update! 👍


GPU memory leak when using gluon.data.DataLoader with num_workers>0

Description
GPU memory leak when using gluon.data.DataLoader after upgrading Cuda-11.1/Cudnn-8.2.x (also tested with latest Cuda11.5+CuDnn8.3.x but still leaking). Minimal code to repro attached below.
No memory leak with older Cuda version (Cuda-10.1 + CuDnn-7.6.5).
Error Message
gpu memory keeps increasing during training.
To Reproduce
(If you developed your own code, please provide a short script that reproduces the error. For existing examples, please provide link.)
import mxnet.gluon as gl
import mxnet as mx
import gc

if __name__ == "__main__":
    gpu_ctx = mx.gpu()
    model = gl.nn.Embedding(10, 5)
    model.initialize(ctx=gpu_ctx)
    X = mx.random.uniform(shape=(1000, 3))
    dataset = mx.gluon.data.dataset.ArrayDataset(X)
    num_workers_list = [0, 4, 8]
    for num_workers in num_workers_list:

        for epoch in range(5):
            dataset = mx.gluon.data.dataset.ArrayDataset(X)
            data_loader = gl.data.DataLoader(
                dataset,
                batch_size=1,
                num_workers=num_workers,
            )
            for batch in data_loader:
                # move data to gpu
                data_gpu = batch.copyto(mx.gpu())
                # forward
                l = model(data_gpu)
                # force immediate compute
                l.asnumpy()
            # gc & gpu_ctx.empty_cache
            mx.nd.waitall()
            del dataset
            del data_loader
            gc.collect()
            gpu_ctx.empty_cache()
            mx.nd.waitall()

            a, b = mx.context.gpu_memory_info(0)
            print(f"num_workers: {num_workers} epoch {epoch}: "
                  f"current gpu memory {(b - a) / (1024 * 1024 * 1024)} GB, "
                  f"Total gpu memory {b / (1024 * 1024 * 1024)} GB.")

Steps to reproduce
(Paste the commands you ran that produced the error.)
### Output with MXNet-1.9 built with Cuda11.1 CuDnn 8.2.0 (Memory leak when `num_workers` > 0)
  (also tested with the latest Cuda11.5+CuDnn8.3.x)

num_workers: 0 epoch 0: current memory 1.381591796875 GB, Total memory 15.78173828125 GB.
num_workers: 0 epoch 1: current memory 1.381591796875 GB, Total memory 15.78173828125 GB.
num_workers: 0 epoch 2: current memory 1.381591796875 GB, Total memory 15.78173828125 GB.
num_workers: 0 epoch 3: current memory 1.381591796875 GB, Total memory 15.78173828125 GB.
num_workers: 0 epoch 4: current memory 1.381591796875 GB, Total memory 15.78173828125 GB.
num_workers: 4 epoch 0: current memory 1.483154296875 GB, Total memory 15.78173828125 GB.
num_workers: 4 epoch 1: current memory 1.582763671875 GB, Total memory 15.78173828125 GB.
num_workers: 4 epoch 2: current memory 1.683349609375 GB, Total memory 15.78173828125 GB.
num_workers: 4 epoch 3: current memory 1.782958984375 GB, Total memory 15.78173828125 GB.
num_workers: 4 epoch 4: current memory 1.880615234375 GB, Total memory 15.78173828125 GB.
num_workers: 8 epoch 0: current memory 1.980224609375 GB, Total memory 15.78173828125 GB.
num_workers: 8 epoch 1: current memory 2.080810546875 GB, Total memory 15.78173828125 GB.
num_workers: 8 epoch 2: current memory 2.180419921875 GB, Total memory 15.78173828125 GB.
num_workers: 8 epoch 3: current memory 2.281982421875 GB, Total memory 15.78173828125 GB.
num_workers: 8 epoch 4: current memory 2.380615234375 GB, Total memory 15.78173828125 GB.

### Output with MXNet-1.9 built with Cuda10.1 CuDnn 7.6.5 (No memory leak)

num_workers: 0 epoch 0: current memory 1.301513671875 GB, Total memory 15.78173828125 GB.
num_workers: 0 epoch 1: current memory 1.301513671875 GB, Total memory 15.78173828125 GB.
num_workers: 0 epoch 2: current memory 1.301513671875 GB, Total memory 15.78173828125 GB.
num_workers: 0 epoch 3: current memory 1.301513671875 GB, Total memory 15.78173828125 GB.
num_workers: 0 epoch 4: current memory 1.301513671875 GB, Total memory 15.78173828125 GB.
num_workers: 4 epoch 0: current memory 1.301513671875 GB, Total memory 15.78173828125 GB.
num_workers: 4 epoch 1: current memory 1.301513671875 GB, Total memory 15.78173828125 GB.
num_workers: 4 epoch 2: current memory 1.301513671875 GB, Total memory 15.78173828125 GB.
num_workers: 4 epoch 3: current memory 1.301513671875 GB, Total memory 15.78173828125 GB.
num_workers: 4 epoch 4: current memory 1.301513671875 GB, Total memory 15.78173828125 GB.
num_workers: 8 epoch 0: current memory 1.301513671875 GB, Total memory 15.78173828125 GB.
num_workers: 8 epoch 1: current memory 1.301513671875 GB, Total memory 15.78173828125 GB.
num_workers: 8 epoch 2: current memory 1.301513671875 GB, Total memory 15.78173828125 GB.
num_workers: 8 epoch 3: current memory 1.301513671875 GB, Total memory 15.78173828125 GB.
num_workers: 8 epoch 4: current memory 1.301513671875 GB, Total memory 15.78173828125 GB.

What have you tried to solve it?

python gc clean doesn't help
upgrade cuda/cudnn to least version doesn't help

Environment
We recommend using our script for collecting the diagnostic information with the following command
curl --retry 10 -s https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py | python3

Environment Information
# Paste the diagnose.py command output here

----------Python Info----------
Version      : 3.6.14
Compiler     : GCC 7.5.0
Build        : ('default', 'Feb 19 2022 10:06:15')
Arch         : ('64bit', 'ELF')
------------Pip Info-----------
No corresponding pip install for current python.
----------MXNet Info-----------
Version      : 1.9.0
Directory    : /efs-storage/debug_log/test-runtime/lib/python3.6/site-packages/mxnet
Commit hash file "/efs-storage/debug_log/test-runtime/lib/python3.6/site-packages/mxnet/COMMIT_HASH" not found. Not installed from pre-built package or built from source.
Library      : ['/efs-storage/debug_log/test-runtime//lib/libmxnet.so']
Build features:
✔ CUDA
✔ CUDNN
✖ NCCL
✖ CUDA_RTC
✖ TENSORRT
✔ CPU_SSE
✔ CPU_SSE2
✔ CPU_SSE3
✖ CPU_SSE4_1
✖ CPU_SSE4_2
✖ CPU_SSE4A
✖ CPU_AVX
✖ CPU_AVX2
✔ OPENMP
✖ SSE
✖ F16C
✖ JEMALLOC
✔ BLAS_OPEN
✖ BLAS_ATLAS
✖ BLAS_MKL
✖ BLAS_APPLE
✔ LAPACK
✔ MKLDNN
✔ OPENCV
✖ CAFFE
✖ PROFILER
✖ DIST_KVSTORE
✖ CXX14
✖ INT64_TENSOR_SIZE
✔ SIGNAL_HANDLER
✖ DEBUG
✖ TVM_OP
----------System Info----------
Platform     : Linux-4.14.232-177.418.amzn2.x86_64-x86_64-with
system       : Linux
node         : ip-10-0-10-233.ec2.internal
release      : 4.14.232-177.418.amzn2.x86_64
version      : #1 SMP Tue Jun 15 20:57:50 UTC 2021
----------Hardware Info----------
machine      : x86_64
processor    : x86_64
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              64
On-line CPU(s) list: 0-63
Thread(s) per core:  2
Core(s) per socket:  16
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               79
Model name:          Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
Stepping:            1
CPU MHz:             2630.103
CPU max MHz:         3000.0000
CPU min MHz:         1200.0000
BogoMIPS:            4600.04
Hypervisor vendor:   Xen
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            46080K
NUMA node0 CPU(s):   0-15,32-47
NUMA node1 CPU(s):   16-31,48-63
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq monitor est ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx xsaveopt ida

Some additional resources I've found:

This is a similar issue for CPU memory leak with the MultiWorker setup in DataLoader. The solution was to add the python gc to clean up the memory, however this solution doesn't work for GPU.
The Cudnn release note mentions a new buffer management that might affect the Cuda>=10.2, which seems to be related. And the issue only surfaces after I upgrade Cuda version (tested with Cuda10.2/Cuda11.1/Cuda11.5, and all 3 have memory leak issue).

@TristonC

one additional finding is that the memory leak happens with the default thread_pool option set as False (a.k.a leak when using the multiprocessing.Pool), if I switch to use ThreadPool, there is no memory leak any more! This could be a good indicate for the issue in shared memory.

@ptrendx not that its your fault, but you had the most recent commit in the dataloader code: https://github.com/apache/incubator-mxnet/blob/v1.9.x/python/mxnet/gluon/data/dataloader.py#L654-L657 do you have any thoughts here about this memory leak issue?

We are checking this issue. Thanks for the feedback @ann-qin-lu. It does look like more of multiprocessing package related.

one more data point I've gathered is that if I remove the logic of using the shared memory (a.k.a the global _worker_dataset), it also resolves the memory leak issue. Most like the multiprocess + shared memory implementation is left behind some staled references, which are holding the gpu memory with the latest Cuda implementation.

@ann-qin-lu  I did repro the same behavior as you mentioned in your comments. We don't have the conclusion now. If you need the workaround, use the thread_pool might be a good choice. Thanks.

Hi @TristonC, thanks a ton for looking into the issue. I tried with thread_pool option, and it did work without memory leak. However, since the thread_pool option is slow in preparing the data, I do observe the increased E2E latency (mostly increased during validation time). My production use cases are very sensitive to the training time, and we'd still like to explore the option for multiprocess.Pool (assume the memory leak issue can be resolved soon).
Do you have any hunch about what changes in Cuda/Cudnn that might lead to this issue?

@ann-qin-lu
Do you have any hunch about what changes in Cuda/Cudnn that might lead to this issue?
Not that I have been aware of. I will seek help from related NVIDIA teams. Stay tuned.

Tested the above script on two mxnet nightly builds:

mxnet_cu112-1.9.0b20220227-py3-none-manylinux2014_x86_64.whl - no memory issue.
mxnet_cu112-1.9.0b20220301-py3-none-manylinux2014_x86_64.whl - has memory issue.

This may indicate that the issue was introduced by the commits in between 02/27 to 03/01. Possibly: https://github.com/apache/incubator-mxnet/commit/8041c0da75cd146ebd578c7bee13af25dc231a98

After more deep dives, this issue is actually not caused by cuda upgrade from 10 to 11, but introduced by this specific commit: Remove cleanup on side threads, which skips the cuda deinitialization when destructing engine. I've confirmed that after reverting this commit, the memory leak issue is gone.
I'll work with MXNet team to see if this commit should be reverted in both MxNet master and 1.9 branch. (actually another user reported similar memory issue when using the multiprocessing and tried to revert this commit). Here is the open issue for better handling the engine destruction, which needs to be addressed first if the above workaround will be reverted.

@TristonC False alarm on the Cuda version. Thanks a lot for your help!

I wonder you might be interested in digger little deeper @ann-qin-lu. The it seems current gluon dataloader using fork to start a worker process in multiprocessing.Pool(..) function call (as it is default in Unix-like system). It might be a problem for this issue as the child process inherit everything from its parent process. It might be a good idea to use spawn instead of using fork this function. Unfortunately, I ran into a issue that blocks my test of multiprocessing.get_context('spawn').Pool(...) .
Traceback (most recent call last):                                                                                File "<string>", line 1, in <module>                                                                                File "/usr/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main                                                                                exitcode = _main(fd, parent_sentinel)                                                                                File "/usr/lib/python3.8/multiprocessing/spawn.py", line 126, in _main                                                                                self = reduction.pickle.load(from_parent)                                                                                File "/opt/mxnet/python/mxnet/gluon/data/dataloader.py", line 58, in rebuild_ndarray                                                                                return nd.NDArray(nd.ndarray._new_from_shared_mem(pid, fd, shape, dtype))                                                                                File "/opt/mxnet/python/mxnet/ndarray/ndarray.py", line 193, in _new_from_shared_mem                                                                                check_call(_LIB.MXNDArrayCreateFromSharedMemEx(                                                                                File "/opt/mxnet/python/mxnet/base.py", line 246, in check_call                                                                                raise get_last_ffi_error()                                                                                mxnet.base.MXNetError: Traceback (most recent call last):                                                                                File "../src/storage/./cpu_shared_storage_manager.h", line 179                                                                                MXNetError: Check failed: ptr != ((void *) -1) (0xffffffffffffffff vs. 0xffffffffffffffff) : Failed to map shared memory. mmap failed with error Permission denied

It looks like the memory leak in the above script is because we instantiate multiple dataloader objects in the for loop. Having one dataloader object seems to mitigate the issue:
import mxnet.gluon as gl
import mxnet as mx
import gc

if __name__ == "__main__":
    gpu_ctx = mx.gpu()
    model = gl.nn.Embedding(10, 5)
    model.initialize(ctx=gpu_ctx)
    X = mx.random.uniform(shape=(1000, 3))
    dataset = mx.gluon.data.dataset.ArrayDataset(X)
    num_workers = 8
    data_loader = gl.data.DataLoader(
                dataset,
                batch_size=1,
                num_workers=num_workers,
            )

    for epoch in range(5):
        for batch in data_loader:
            # move data to gpu
            data_gpu = batch.copyto(mx.gpu())
            # forward
            l = model(data_gpu)
            # force immediate compute
            l.asnumpy()

        mx.nd.waitall()

        a, b = mx.context.gpu_memory_info(0)
        print(f"num_workers: {num_workers} epoch {epoch}: "
              f"current gpu memory {(b - a) / (1024 * 1024 * 1024)} GB, "
              f"Total gpu memory {b / (1024 * 1024 * 1024)} GB.")
        data_loader.refresh()

num_workers: 8 epoch 0: current gpu memory 1.43017578125 GB, Total gpu memory 15.78192138671875 GB.
num_workers: 8 epoch 1: current gpu memory 1.43017578125 GB, Total gpu memory 15.78192138671875 GB.
num_workers: 8 epoch 2: current gpu memory 1.43017578125 GB, Total gpu memory 15.78192138671875 GB.
num_workers: 8 epoch 3: current gpu memory 1.43017578125 GB, Total gpu memory 15.78192138671875 GB.
num_workers: 8 epoch 4: current gpu memory 1.43017578125 GB, Total gpu memory 15.78192138671875 GB.

Seems previously we had mshadow::DeleteStream<gpu>(stream) to clean up the GPU memory by the life cycle of dataloader object, but it had a segfault issue. In the workaround PR, we removed mshadow::DeleteStream<gpu>(stream) and relied on the OS to clean up memory at the end of the program. That may explain why we see memory leak when creating multiple dataloaders in the program.

Hi @waytrue17, thanks for sharing above info. Yep, by skipping recreating dataloader for each epoch does prevent such issue, but in my use case, I need to shard the big dataset into smaller ones in each epoch, and therefore data loader needs to be created multiple times.
I've seen a few comments (e.g. issue 1, issue 2) that mentioned memory error with this workaround commit. Reverting this commit does resolve my accumulated gpu memory issue.
Side question: Could you share more insights about how this workaround commit, which skips the clean up gpu memory in Naive Engine, affects the usage pattern of dataloader?

@TristonC I think your error is due to the fact that Dataloader uses shared memory to hold the dataset. I am not sure if using spawn would require copying shared memory or not. If yes, I am assuming this approach going to increase the total memory usage?

The workaround skips the clean up for all engines, not just the NaiveEngine.
So, the general problem here is that when you create the dataloader, it creates a pool of workers by forking the main process, which creates a copy of everything, including the engine and the resources held by it. Then the forked process destroys this copy of the engine to become a much leaner dataloader worker. This would normally destroy the stream engine uses, but with the workaround commit in place, the destruction of the stream does not happen. Now, the problem is that CUDA does not in fact survive forking and the fact that it seems to work is just a lucky coincidence. That is why the spawn method should be used to fix the dataloader - with that the worker processes do not inherit anything from the parent and start from a clean state - with nothing copied to destroy.
In principle in the end it should work the same way as currently, via shared memory so there should be no visible differences compared to the current way of things (if anything, it should actually work slightly faster, since it would not need to spend the time to destroy the copied engine during the dataloader construction). I guess the error that @TristonC encounters means that there is some additional issue in the dataloader that it somehow depends on some copied variable from the parent process in order to initiate the communication channel with the parent.

Hi @ptrendx, thanks a lot for the explanation! Now I get a much clear picture of what's going wrong. If the actually RC is that "CUDA does not in fact survive forking", does it mean multiprocessing with fork method should be avoided from the very beginning?
Just a quick summary for the two approaches we discussed:

with the workaround that skips the clean up for all engines, it has the issue of lingering gpu resources held by engine, whenever the multiprocess fork method is used. Proposed solution is to use spawn in Gluon.DataLoader. @waytrue17 if you can help?
if we revert the workaround, we will see the non-deterministic segfault issue at exit. This segfault could be resolved if this open Open issue for Better handling of the engine destruction can be resolved first.

We recommend the first solution that @ann-qin-lu proposed.

with the workaround that skips the clean up for all engines, it has the issue of lingering gpu resources held by engine, whenever the multiprocess fork method is used. Proposed solution is to use spawn in Gluon.DataLoader.

@DickJC123 will help to follow up this issue.


UX when JavaScript is disabled

When JavaScript is disabled and the user visits the site for the first time, images on the index page don't load properly (the placeholder image stays on). This is weird, because images on other pages seem to load without issues. To me it looks like images are handled the same way on all pages. What's special about the index page?
Note: for testing, it's best to disable JavaScript via browser settings and then open a new incognito windo to test. (Do not disable via developer tools, not via browser extensions -- or you will encounter additional issues)

Solved by using  instead of : https://github.com/baobabKoodaa/blog/commit/12cf6b89926b78295f6da4e6bbf7cee396210183

Solved by using <Picture> instead of <Image>: https://github.com/baobabKoodaa/blog/commit/12cf6b89926b78295f6da4e6bbf7cee396210183


Link "pre-production" text in banner to explanation

Original concern and Donald's reply:

The new website is "pre-production”.

This is mostly just a warning at this point that the UI might return funky results, some things might not be great yet, and to not start throwing the full force of people’s production traffic at it. The current hosting arrangement can’t handle the full amount of traffic we get from pip install … (but uploads are fine!) and we’re working on building on the new infrastructure with our future hosting arrangement that can handle the full production load.

IMO, it would help to link the "pre-production" text in the banner to a page on Warehouse that explains what about Warehouse is pre-production.

We also should add a section to the help page with said explanation.

@di Where do you think would be the best place to add it?

@blackHatMonkey Somewhere on the help page would be ideal.

This is effectively done with the PR at https://github.com/pypa/warehouse/pull/2376, the only thing left is for me to edit the banner image.

This is unnecessary if we do #2518 first.


Put some thought into best way of implementing GA4GH write API

@briandoconnor FYI

Working on specification at https://wiki.oicr.on.ca/display/SOFTENG/Dockstore+Write+API+(Paste-bin+service)+Specification

Pull request created at https://github.com/ga4gh/tool-registry-schemas/pull/19


Dynamically updating a CR based on responsse of an API request

I have a scenrio in which my operator ussing one CR creaates a pod, deployment , service and route for an API sservice we have. This API service connects to a DB on OpenShift annd adds data to the DB with subsequent HTTPS requests to my API.
Now I want that based on the data added to the DB( or the response we get for these HTTPS requests), another CR is updated and the controller for this second CR does further tasks.
I am using operator SDK v1
How do you suggest I do it? would be a great help.

Hi @anusood,
In the PR https://github.com/operator-framework/operator-sdk/pull/4021, I am proposing some changes in the tutorial guide which shows answer your question. See;

Understanding API’s

The goal of this command is to create Custom Resource (CR) and Custom Resource Definition (CRD) resources for the  Memcached Kind. This command is creating the API with the group cache.example.com, and version v1alpha1  which uniquely identifies the new CRD of the Memcached Kind. For more info see Extend the Kubernetes API with CustomResourceDefinitions.
Consequently, by using the Operator SDK tool, we can create our APIs and objects that will represent our solutions on these  platforms. Here will adds only a single kind of resource; however, it could have as many Kinds as needed (1…N). Basically, the CRDs are a definition of our customized Objects, and the CRs are an instance of it.

Getting a better idea

Let’s think about the classic scenario where the goal is to have an application and its database running on the platform with Kubernetes. Then, one object could represent the App, and another one could represent the DB. By having one CRD to describe the App and another one for the DB, we will not be hurting concepts such as encapsulation, the single responsibility principle, and cohesion. Damaging these concepts could cause unexpected side effects, such as difficulty in extending, reuse, or maintenance, just to mention a few.

In conclusion, the App CRD will have as its controller the DB CRD. Imagine, that a Deployment and Service are required for the application run so that the App’s Controller will provide these resources in this example. Similarly, the DB’s controller will have the business logic implementation of its objects.

In this way, for each CRD, one controller should be produced according to the design set by the controller-runtime.

Also, you can see my personal project which is in the legacy layout yet which requires manipulated one CRD as a secundary resource in a controller. See; https://github.com/dev4devs-com/postgresql-operator
See that in the Backup controller is required to get the Database CR as well. E.g https://github.com/dev4devs-com/postgresql-operator/blob/master/pkg/controller/backup/controller.go#L130-L135

can you direct me in a way where I can manipulate one of the CRs of the operator from inside the Java code of the API pod/service i am running.

Hi @anusood,

can you direct me in a way where I can manipulate one of the CRs of the operator from inside the Java code of the API pod/service i am running.

It shows out of the scope of SDK  tool domain since it would be outside of the operator project build with. However, if you would like to do requests to the K8S API outside of the operator project, as for example, from another application you need to use the client for that. The controller already provides it for you which allows you the client.GET/CREATE/DELETE/UPDATE and etc from the reconcile func.
The Go client is here. You also can look for a Java k8s client such as https://github.com/kubernetes-client/java. The k8s API is a rest service like any other, you could do the requests as well. However, I'd recommend you use the client instead of trying o re-implement what the Client does.
Closing it as solved. However, please feel free to ping us if you need to re-open this one.


Added WebXR Platform

XRTK - Mixed Reality Toolkit Pull Request
Overview

Adds WebXR platform support.

This seems good to merge @StephenHodgson to just add the platform?

Lots of work to be done here still. It's far from ready.


Use asynchronous Promise when checking address input

Use Promise.allSettled so failed promises can be handled within isAddressSafe instead of having the caller handle it.
Reference link: https://github.com/ScopeLift/umbra-protocol/pull/371#discussion_r940266355

I can take this on

@alexkeating Do you still want to handle this one or should we unassign for now?

I can take this going to shift back to Umbra work now


Breaking change in 7.2, when frame response is 4xx, 5xx OR if frame is missing

On updating from Turbo 7.1.x to to 7.2.x, via turbo-rails, we were quite surprised to see frames suddenly doing full-page redirects on failures, ie. for a 4xx or 5xx response.
I'm aware of hotwired/turbo#445, and the reasoning behind the turbo:frame-missing event, quite useful!
However, the changes introduced in hotwired/turbo#693, where Turbo now transforms 4xx and 5xx responses to full-page Visits, feel like a breaking change. The minor version bump from 7.1 to 7.2 certainly broke our app 😅
I've read up on the discussion in hotwired/turbo#670, and noticed this sentiment by @seanpdoyle :

Scenarios like this make me nervous about breaking the https://github.com/hotwired/turbo/issues/94#issuecomment-756968205 that a navigation within a frame will stay within a frame.

I believe the changes introduced in 7.2 specifically break that “promise”: a navigation within a frame will stay within a frame
Are apps now required to add additional/custom JS by default, to handle this scenario?
Or am I simply misunderstanding something? (quite possible!)

Related:

Original issue raised on hotwired/turbo-rails#407, was told to post here instead
Issue #404
A minimal test app that reproduces my issue: https://github.com/cabgfx/turbo72-repro

I should have a PR ready for that in the next couple of days.

Awesome! Appreciate your work on this! If I can assist with testing it out or anything, please let me know

... only paths from an allowlist, like redirects to /sign_in, will break out of the frame, and other 4xx and 5xx responses will instead stay in the frame, but fill it with an error message.

This is a great plan!  🎉
Will it be possible to get the details of what went wrong in the console?

Just for breadcrumbs' sake, the allow-list concept ended up getting scrapped in https://github.com/hotwired/turbo/pull/864 to instead create a flag (of sorts) to put on the incoming page which forces Turbo to full-visit the page, even if the request originated from within a frame

And with 7.3 I think this issue is closable 😁

Thanks for the heads-up, @jon-sully!


Link Color 'forgotten' after tiling map

This occurs in the dev branch and is readily demonstrated with both the demo data set and the KCDC/COVID-19 data set. Please follow these steps to reproduce:
1.) Load a network with edge and node list. The edge list must have a categorical type column. The node list must have geocoordinates (either ZIP or lat-lon will suffice).
2.) Color the links according to a categorical 'type' variable
3.) Next, generate a map in the map view. You will see that, as expected, the colors have propagated to the map view.
4.) Finally, tile the map view adjacent to the network view. You will see that the link colors have been set back to default for both map and network views. This is also reflected in the global settings menu.

initialize linkColorVariable based on session state


[WIP] update MiqReport::Search to include all columns

Ok, backdrop: report.yml file.
Columns for the screen? Use col_order.
Associated models? Use includes_for_find.
The foreign_key needed to bring back those associated models?
That is tricky, I guess you could declare all the col_order again in the cols attribute.
rails 5.0
We were bringing back data for all models in a single query (sometimes to our detriment).
We did not need the extra associated models' foreign_key ids because we had all the associated models' data to create the associated AR records.
rails 5.1, Before PR:
We were bring back associated models in additional queries, but the foreign_key necessary to bring them back was not present. This caused and exception / 500 error.
rails 5.1, after PR:
We are now bringing back more columns, specifically the foreign_key.
The associated models able to be queried.
Concerns
It appears the rules as to what models are brought back by the primary query changed again. It used to require includes(:model2), then it required 'includes(:model2).references(:model2)` in rails 4.x.
Now there appears to be another rule.
There was a bug in 5.0 where has_many associations would be part of the primary query. Which brought back many many rows. So this could be the driver of the 5.1 change. But that is speculation
But what ever the reason, these associated foreign_key ids should be brought back, for all versions of rails.

@kbrock I think I follow the change you make here but it's still unclear what exactly changed in rails 5.1 to break us.  How did the foreign_key get included in the select in 5.0 and not 5.1?

WIP status - I'm looking through rails searching for the actual issue

not going this route. modifying the way includes is supported under the covers.
Core rails also fails when you select(:name).includes(:model).references(:model)


Error converting chara_4_eflame_only_01.png 162x162 to bntx

hey sorry if Im not supposed to start a new issue. I wasn't sure since you closed the last one but now the newest release it unfortunately still doesnt work but shows an error message this time.
Error converting chara_4_eflame_only_01.png: error compressing surface: surface dimensions 162 x 162 x 1 are not divisibly by the block dimensions 4 x 4
Is this something thats fixable?

This is fixable, but the changes will be more involved on my end. I don't have any estimate on when I'll release 0.2.3 for Ultimate Tex.
You can technically still use 162x162 textures on the current version as long as you use DDS as the input. It's only the conversions from uncompressed formats like  png or jpeg to formats like dds, bntx, or nutexb that assume the input dimensions are divisible. If you're on windows, the easiest for now is to just use something like paint.net or photoshop to export your edits as dds and use Ultimate Tex to convert to bntx.

unfortunately dds came with the same problem no matter what I tried. Are there any other options?

It should work as long as the output image format matches the input image format for the dds file. Ultimate Tex can detect this case and avoid doing any conversions. You'll want to save the file as BC7Srgb or BC7Unorm in an image editor depending on how you want to handle gamma and use Ultimate Tex to convert that dds to bntx.

unfortunately thats not working either.

It definitely works as long as the format doesn't change. Here's the settings I'm using.


[Accessibility] [Supporting Platform] Fix large text to be displayed properly in the Log panel

Fixes ADO Issue: #64526
Describe the issue
If the user applies large text and activity update section content information does not visible properly, so it would be difficult for the user to understand content information.
Actual behavior:
After applying large text, the Activity Update section content information does not visible properly and because of this, the user is unable to understand content information.
Expected behavior:
After applying large text, the Activity Update section content information should be visible properly, so those users get proper information.
Repro Steps:

Open Bot Framework Emulator installed on the system.
Bot Framework Emulator Application with Welcome screen opens.
Navigate on the welcome screen and select the Open Bot button.
Open A bot Dialog opens, navigate on the dialog and enter inputs in edit fields and select Connect button.
Bot Gets connected and a new Tab opens, navigate to the Type a Message field and enter any chat message.
Bot Response as per text entered appears and changes occur in Activity Section.
Navigate to the Activity section.
Verify whether large text content is visible properly or not.

Changes included in the PR

Added a CSS style to break the large text and display the content properly in the log panel.

Screenshots
Test cases executed successfully

Log panel

Coverage decreased (-0.04%) to 67.806% when pulling 292f415a25a9eaf9e0e4607084f9f46386e77834 on southworks:external/southworks/fix/log-panel/fix-large-text-issue into 694023cc1f3cf74d34784fb1d749e0a7be918082 on microsoft:main.


Paging3 uses the MainThread RPC calling that block the ui thread

I wrap the RecyclerView with a SwipeRefreshLayout, when first enter the list page,  set the SwipeRefreshLayout to loading progress, then do the paging refresh call like this,
 @ExperimentalCoroutinesApi
    private fun searchHistoryData() {
        swipe_container.isRefreshing = true

        searchJob?.cancel()
        searchJob = lifecycleScope.launch {
            viewModel?.refreshGameHistory()?.collect {
                swipe_container.isRefreshing = false
                adapter.submitData(it)
            }
        }
    }

Found the loading components not showing expected. profile the process via Android Studio,  the RPC call executed after the adapter.submitData() calling, and it's running upon the main thread.
Since the PRC logics are in PagingSource load() suspend function.  I can't decide the load function execute in which thread. my PRC function is a GRPC function,  If I convert it to an async function, it will not convenient to handle the result.
In addition, the same as loading views, the footer view LoadState also been blocked.
And how should I do about this issue?  If can consider adding an IO thread way to support it?
dependencies:
implementation "androidx.paging:paging-runtime:3.0.0-alpha01"
thanks!

This is not an issue with the codelab.
Please use the official issue tracker to report issues with the library.
@dlam fyi

This is not an issue with the codelab.
Please use the official issue tracker to report issues with the library.
@dlam fyi

suspend functions are async. You can switch the executing context using withContext(Dispatchers.IO) { ... } in any suspend function. However, the right place to do that is probably to make the grpc call itself suspend and then run in the correct context there, rather than at every place it's called (such as in .load)

suspend functions are async. You can switch the executing context using withContext(Dispatchers.IO) { ... } in any suspend function. However, the right place to do that is probably to make the grpc call itself suspend and then run in the correct context there, rather than at every place it's called (such as in .load)


Create example application

[ ] Move sources files into Sources folder
[ ] Create example project inside xcodeproj, including a mp4 video asset
[ ] Add tests using mp4 video asset
[ ] Change podspec to only target source files in Sources folder
[ ] Investigate changes necessary to Carthage

Yes we need this so we can test while developing lol.


add master type node with bridge network supported for external access

The default VagrantConfig-1m-1a-1p.yaml configuration makes master node only can be accessed by the Host server and VMs, cannot access from the LAN co-located with Host server. Therefore, add the options in .yaml file for let master node can expose itself for the other LAN PCs accessing.

dcos-vagrant is deprecated and the recommended replacement is dcos-e2e.


Prune deleted and unresolvable configurations from single lock file

The topic of non longer existing configurations has always been a discussion in the context of dependency locking.
Because of its potential for dynamic addition of configurations, Gradle cannot really automate cleaning up stale lock state for configurations that no longer exist in the build.
With one lockfile per configuration per project, manually deleting stale files was not such a big issue.
However with the single lockfile format, introduced in #11881, it becomes more complicated as manual edits of the file are not recommended.
So Gradle should provide a system for pruning stale configurations from that lock file.

Configurations that still exist but are no longer resolvable are also candidates for removal.
e.g., in 7.6, the default configuration was considered resolvable (but deprecated) and may have appeared in lock files.
In 8.0, the default configuration is no longer resolvable, but still exists.


[8.x] Add dd() and dump() to the request object

This PR adds dump() and dd() to the Request class.
Usage
// use Illuminate\Http\Request;
public function index(Request $request)
{
    // instead of
    dd($request->all());

    // we can use
    $request->dd();

    $request->dd(['name', 'age']); // print only the keys from the array

    $request->dd('name', 'age'); // pass them as separate arguments
}

dump() can be used the same way, and it will be useful for debugging AJAX requests.
Real world example
// use Illuminate\Http\Request;
public function index(Request $request)
{
    $request->dd()->validate([
        'name' => 'required'
    ]);
}

Quickly inspect the request params before passing them to the validator.
When using dd() or dump() without params the all() method is used, and when there are params passed in, the only() method is used.

I realise it's a bit late to share opinion on this, but FWIW I do think this is an unnecessary and confusing addition.
Firstly, the it shouldn't be the Request's responsibility to dump itself, it just goes against the idea of what a class is.
Secondly, and more importantly, magically dumping $request->all() instead of $request could and probably will cause more confusion than just making people use dd($request->all()). I wouldn't expect dd($request) and $request->dd() to dump different things, all for the sake of saving 5 characters and losing a whole lot of clarity.
Debugging is a situation where you want to be really sure what you're looking at, and not a place to throw in magic for little return.

@danabrey you don't need to use this if you don't want to.

I realise that. I'm just pointing out the possible confusion for users.

@danabrey I completely agree with your comment above. But this is just a helper function given my "real world" example above that will save you couple of keystrokes.
And I doubt that it will bring confusion as this is not a documented function, having said that in contrary I think that it will help a lot of inexperienced developers to debug their request data better.
I've been active on SO and Laracasts for the last couple of years, and I can't count the times I've asked developers to debug their request for what it contains, which usually ends in dd($request->all());.
And talking for myself, I've never used dd($request), but always reach for any of the methods on the $request object.

I realise it's a bit late to share opinion on this, but FWIW I do think this is an unnecessary and confusing addition.
Firstly, the it shouldn't be the Request's responsibility to dump itself.
Secondly, and more importantly, magically dumping $request->all() instead of $request could and probably will cause more confusion than just making people use dd($request->all()). I wouldn't expect dd($request) and $request->dd() to dump different things, all for the sake of saving 5 characters and losing a whole lot of clarity.
Debugging is a situation where you want to be really sure what you're looking at, and not a place to throw in magic for little return.

I am pretty sure that someone who is used to coding in Laravel, would always understand what does dd($content) refer to. It is not like - we only have one way to dump contents of request. We can do it in multiple ways if we forget one way to do it.
Up to you - to use it or not.

Up to you - to use it or not.

Is this the slogan of Laravel now?

I doubt that it will bring confusion as this is not a documented function

What's less confusing than undocumented functions that do things people don't expect? :)

Up to you - to use it or not.

Is this the slogan of Laravel now?
I think the slogan was "Batteries included but replaceable". It surely implies that it is up to to use it or not. The same about request validation - you use special request validation class or do it "inline" in the method. That's the beauty of this framework.

No one will get hurt if this gets introduced.

I'm not going to keep making the same point. Adding end-user confusion is a negative that should be considered - "you don't have to use it" isn't some magic sentence that removes any debate.
Peace out.

I would tend to agree with @danabrey here, we all understand we have the choice to use or not use these methods, we're trying to have a debate about the value of an addition. Let's take the argument ad absurdum for a second. What if I got the following code merged into the Request class:
public function nuke()
{
    DB::statement("DROP TABLE `users`");
}

I can imagine someone commenting that this doesn't belong on the Request object, and is super dangerous because of what it does. Can I just respond, "Don't use it if you don't want to"?  No, we should debate the cost/benefit of the addition. This change obviously has a very high cost, and a very low benefit.
In regards to this PR, the proposed benefit is apparently that it saves you a couple of keystrokes. From a purely character count aspect, it looks like we're saving about 5 characters. And with any type of modern IDE and autocompletion, saved characters don't necessarily translate directly to saved keystrokes. Some people may still view this as a benefit, but I'm hard pressed to see it that way.
As to the cost of this addition, we now have an additional 36 lines of code to maintain in the codebase, a little more visual debt on the class, and (possibly) slightly more memory when the object is used. The weight you assign to these costs is obviously subjective. I personally probably assign a little higher cost than others.
Every line of code is always a trade off, with a cost and a benefit. We need to be willing to look at both when having a discussion about PRs.
This is not a hill I have any interest in dying on. At the end of the day, yes, this PR is relatively low impact, and people can just avoid using it if they don't like it. But I don't think we should shut the conversation down immediately. We're all working towards the same goal, and those of us who raise concerns are just doing so to make sure we acknowledge the short term and long term effects of these changes.


FirebaseAuthProvider.CreateUserWithEmailAndPasswordAsync throws null Exception

FirebaseAuthProvider.CreateUserWithEmailAndPasswordAsync throws an Exception that is null and this, contains no message about the exception.
However, the acccount is indeed created.

Could you please provide code for people to see where things may have gone wrong?  👍


Live555 won't download

all ok

Hi I'm getting errors building with
vcpkg install live555:x86-windows --head
Shouldn't
http://www.live555.com/live.2019.03.06.tar.gz
be
http://live555.com/liveMedia/public/live.2019.03.06.tar.gz
I was also reading somewhere where someone was saying using the --head flag should make it use live.latest.tar.gz is that right?
The following packages will be built and installed:
live555[core]:x86-windows (from HEAD)
Starting package 1/1: live555:x86-windows
Building package live555[core]:x86-windows from HEAD...
live555 cannot currently be built dynamically. Building static instead.
-- Downloading http://www.live555.com/live.2019.03.06.tar.gz...
-- Downloading http://www.live555.com/live.2019.03.06.tar.gz... Failed. Status: 22;"HTTP response code said error"
CMake Error at scripts/cmake/vcpkg_download_distfile.cmake:172 (message):
  Failed to download file.
  If you use a proxy, please set the HTTPS_PROXY and HTTP_PROXY environment
  variables to "https://user:password@your-proxy-ip-address:port/".
  Otherwise, please submit an issue at https://github.com/Microsoft/vcpkg/issues

Call Stack (most recent call first):
ports/live555/portfile.cmake:11 (vcpkg_download_distfile)
scripts/ports.cmake:71 (include)
Error: Building package live555:x86-windows failed with: BUILD_FAILED
Please ensure you're using the latest portfiles with .\vcpkg update, then
submit an issue at https://github.com/Microsoft/vcpkg/issues including:
Package: live555:x86-windows
Vcpkg version: 2018.11.23-nohash
Additionally, attach any relevant sections from the log files above.

Thanks for report this issue. It has been fixed by PR:https://github.com/Microsoft/vcpkg/pull/6176 .


Timeout when running hugging face LLMs with Langchain (Streamlit)

Describe the bug
ValueError: Error raised by inference API: Model google/flan-t5-xl time out

Specifically on my case, when using langchain with t5-xl, I am getting time-out errors. When using smaller size models, errors seems to vanish. I wonder if this is caused by the HF Hub?
Adding max length parameter usually sorted out timeout errors. For some reason, max_length does not make difference with t5-xl time-out.
llm_hf = HuggingFaceHub(
        repo_id="google/flan-t5-xxl",
        model_kwargs={"temperature":0.9,"max_length":64 }
)

For more reference, see this issue - https://github.com/hwchase17/langchain/issues/3275
Reproduction
No response
Logs
No response
System info
- huggingface_hub version: 0.14.1
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.11
- Running in iPython ?: Yes
- iPython shell: Shell
- Running in notebook ?: Yes
- Running in Google Colab ?: Yes
- Token path ?: /root/.cache/huggingface/token
- Has saved token ?: False
- Configured git credential helpers:
- FastAI: 2.7.12
- Tensorflow: 2.12.0
- Torch: 2.0.0+cu118
- Jinja2: 3.1.2
- Graphviz: 0.20.1
- Pydot: 1.4.2
- Pillow: 8.4.0
- hf_transfer: N/A
- gradio: N/A
- ENDPOINT: https://huggingface.co
- HUGGINGFACE_HUB_CACHE: /root/.cache/huggingface/hub
- HUGGINGFACE_ASSETS_CACHE: /root/.cache/huggingface/assets
- HF_TOKEN_PATH: /root/.cache/huggingface/token
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False

{'huggingface_hub version': '0.14.1',
 'Platform': 'Linux-5.15.107+-x86_64-with-glibc2.31',
 'Python version': '3.10.11',
 'Running in iPython ?': 'Yes',
 'iPython shell': 'Shell',
 'Running in notebook ?': 'Yes',
 'Running in Google Colab ?': 'Yes',
 'Token path ?': PosixPath('/root/.cache/huggingface/token'),
 'Has saved token ?': False,
 'Configured git credential helpers': '',
 'FastAI': '2.7.12',
 'Tensorflow': '2.12.0',
 'Torch': '2.0.0+cu118',
 'Jinja2': '3.1.2',
 'Graphviz': '0.20.1',
 'Pydot': '1.4.2',
 'Pillow': '8.4.0',
 'hf_transfer': 'N/A',
 'gradio': 'N/A',
 'ENDPOINT': 'https://huggingface.co',
 'HUGGINGFACE_HUB_CACHE': '/root/.cache/huggingface/hub',
 'HUGGINGFACE_ASSETS_CACHE': '/root/.cache/huggingface/assets',
 'HF_TOKEN_PATH': '/root/.cache/huggingface/token',
 'HF_HUB_OFFLINE': False,
 'HF_HUB_DISABLE_TELEMETRY': False,
 'HF_HUB_DISABLE_PROGRESS_BARS': None,
 'HF_HUB_DISABLE_SYMLINKS_WARNING': False,
 'HF_HUB_DISABLE_EXPERIMENTAL_WARNING': False,
 'HF_HUB_DISABLE_IMPLICIT_TOKEN': False,
 'HF_HUB_ENABLE_HF_TRANSFER': False}

Hi @elvinagam, I'm sorry you're experiencing those issues. For now I can't really say what happened. Could you provide the full error traceback? Either the model really timed out in our infra or a too low timeout value is set in Python's requests. In general, I would not be so surprised given the size of google/flan-t5-xxl model.
As a reminder, :hugs: Inference API is a service allowing you to test models easily and for free but it is not meant to be production ready. If you need a bullet-proof solution, I would recommend you the Inference Endpoints product.

Otherwise you can also have a look at the text-generation-client guide: https://github.com/huggingface/text-generation-inference/tree/main/clients/python. This is a client specifically designed if you want to do text-generation on the Hub. In particular, you can list the models that are currently deployed using our optimized "text-generation-inference" infra (including google/flan-t5-xxl) by doing:
from text_generation.inference_api import deployed_models

print(deployed_models())

To avoid timeouts you might want to look into the .generate_stream method of the client to get an output "token by token"

I too am facing similar issue on following code and os.environ["HUGGINGFACEHUB_API_TOKEN"] set to my API token:
from langchain import HuggingFaceHub
llm = HuggingFaceHub(repo_id="google/flan-t5-xl", model_kwargs={"temperature":1, "max_length":256})
llm("translate English to German: How are you doing?")

In my case I am running this in google colab notebook. Above could throws error ValueError: Error raised by inference API: Model google/flan-t5-xl time out and stack trace is as follows:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-13-db9d961b91b0>](https://localhost:8080/#) in <cell line: 4>()
      2 llm = HuggingFaceHub(repo_id="google/flan-t5-xl", model_kwargs={"temperature":1, "max_length":256})
      3
----> 4 llm("translate English to German: How are you doing?")

4 frames
[/usr/local/lib/python3.10/dist-packages/langchain/llms/base.py](https://localhost:8080/#) in __call__(self, prompt, stop, callbacks)
    295             )
    296         return (
--> 297             self.generate([prompt], stop=stop, callbacks=callbacks)
    298             .generations[0][0]
    299             .text

[/usr/local/lib/python3.10/dist-packages/langchain/llms/base.py](https://localhost:8080/#) in generate(self, prompts, stop, callbacks)
    189             except (KeyboardInterrupt, Exception) as e:
    190                 run_manager.on_llm_error(e)
--> 191                 raise e
    192             run_manager.on_llm_end(output)
    193             return output

[/usr/local/lib/python3.10/dist-packages/langchain/llms/base.py](https://localhost:8080/#) in generate(self, prompts, stop, callbacks)
    183             try:
    184                 output = (
--> 185                     self._generate(prompts, stop=stop, run_manager=run_manager)
    186                     if new_arg_supported
    187                     else self._generate(prompts, stop=stop)

[/usr/local/lib/python3.10/dist-packages/langchain/llms/base.py](https://localhost:8080/#) in _generate(self, prompts, stop, run_manager)
    434         for prompt in prompts:
    435             text = (
--> 436                 self._call(prompt, stop=stop, run_manager=run_manager)
    437                 if new_arg_supported
    438                 else self._call(prompt, stop=stop)

[/usr/local/lib/python3.10/dist-packages/langchain/llms/huggingface_hub.py](https://localhost:8080/#) in _call(self, prompt, stop, run_manager)
    110         response = self.client(inputs=prompt, params=_model_kwargs)
    111         if "error" in response:
--> 112             raise ValueError(f"Error raised by inference API: {response['error']}")
    113         if self.client.task == "text-generation":
    114             # Text generation return includes the starter text.

Closing this issue as this has to be fixed in langchain directly. For more info, I commented on https://github.com/langchain-ai/langchain/issues/3275#issuecomment-1630855785.


sorting arrays

Sorting operations like sort, sortBy & sortWith in List can't be found in the Array module.
I miss them the most when working with arrays so I think these would be worth adding in a future release to supply a default way of sorting arrays.

Hi, this would be a great feature. It seems to me, that the functionality is already there, since the array seems to be used for sorting lists. It only needs to be exposed also to the array module. I think something like this could also be relevant to create more performant statistics libraries ...
https://github.com/elm/core/blob/84f38891468e8e153fc85a9b63bdafd81b24664e/src/Elm/Kernel/List.js#L73-L86


Correction to Windows ARM64 installer location

The enumerator for the ARM64 TARGETDIR was set to ProgramFilesARMFolder, which doesn't seem to exist in Wix (3/4/5) any longer (see here for details)
Also, see this document for a discussion on where applications for different architectures should be installed on Windows.
So, for the OpenJDK Windows installers, the following folders will be used:

x86-32: C:\Program Files (x86)\<vendor>\jdk-x
x64: C:\Program Files\<vendor>\jdk-x
ARM64: C:\Program Files\<vendor>\jdk-x

Fixes #858

/merge

/approve

/approve


Cannot read property 'oneOf' of undefined in 0.3.7 in React Native

I tried bringing your Toolbar component into a React Native 0.49.5 project, and brought in your other files needed to remove error messages.  I traced the error in the title to a cryptic node module, and I never explicitly use oneOf.

I also got same issue. unable to find PropTypes in react lib.

I have the same error.

Remove import {PropTypes} from 'react-native' and place import PropTypes from 'prop-types'.  Do it manually

Error is still there.

Also getting this error

I am also getting this error, any work around?

I've get this error after installing react-native-multiple-select-list. Solved this error by installing prop-types library, because this lib used to use react-native-vector-icons lib which imports PropTypes from 'prop-types' lib and not from 'react' lib.

I've get this error after installing react-native-multiple-select-list. Solved this error by installing prop-types library, because this lib used to use react-native-vector-icons lib which imports PropTypes from 'prop-types' lib and not from 'react' lib.

we can't do manually in so many files it's like stupidity. is there way to resolve this


Data profiling tool for dataset insights

Why are these changes needed?
This tool generates an output with different metrics that can help a user quickly understand insights from the data (code and text) and see if the data is fit for their use.

Data profiler can be usefulfor personas like Developers, Business Users, Data Scientists for use cases like,

Understand the data before starting any data prep
Understand data issues found during data prep
Profile validation sets to understand where the model does well and where it does not do well
Help customer build the right validation set based on their needs

Features:

Works on both code files and instruction pairs
Data stats to help the user make recommendations during model tuning, eg * padding sizes
Program Analysis based stats for data coverage
Data prep-based stats on issues found in the data

Related issue number (if any).

Thank you for your contribution! Can we rename this to code-profiler and add it in the code folder  ?

Thank you for your contribution! Can we rename this to code-profiler and add it in the code folder  ?

I kept it generic so that in future we can some profiling models for the natural language data.

Couple of thoughts

Can this be made into a transform instead of a tool.  This will help scalability.
We currently have a "profiler" transform that only counts words.  Would it be possible to extend this profiler transform with the capabilities here?

Agree on making it a transform. I would prefer that we keep the code and natural language profilers separate, and keep them under natural language and code folders, so it is easy to find.

Agree on making it a transform. I would prefer that we keep the code and natural language profilers separate, and keep them under natural language and code folders, so it is easy to find.

@Bytes-Explorer
There could be a case where a common transform could be applied for both the code and text data. I created the folder hierarchy of data-profiler as per current folder convention defined in the data-prep-kit's transforms, where the transforms are categorized into three types code, text, and universal.
transform's hierarchy:

data-profiler hierarchy:

Data-profiler is not transforming (changing/modifying) the data but instead providing insights into the dataset based on some metrics in the form of JSON metadata file. In this case is it still okay to call it a transform?
Current PR has support for Python runtime, in future PRs we are planning to add support for more runtimes to make it scalable to run.

@pankajskku We have taken a call that everything will be now under transforms, so will be good if we can add it accordingly. My understanding is that this code does not have any profiling information for NLP, so if it is just code, lets add it as a separate module under code folder. As and when we get NLP based profiling ready, we can extend the current available module that David pointed to. Does this work for you?

In that case, it works great for me too. I'll move the code to transforms and update the same PR with a modified structure. Till then I'll request @shivdeep-singh-ibm to pause the code review.

As to language profiler, we currently have the doc_quality profiler/transform under language, see below.  What is the difference between these two?
https://github.ibm.com/ai-models-data/data-prep-kit-inner/tree/multimodal/transforms/language/doc_quality/ray


Issue 46: deleteBy...And... is not deleting records in Dynamo

✔  deleteById (works already as expected)
✔  deleteByIdAndName - Does a SCAN followed by the actual delete

Codecov Report

Merging #143 into master will decrease coverage by 0.18%.
The diff coverage is 46.15%.

@@             Coverage Diff              @@
##             master     #143      +/-   ##
============================================
- Coverage     64.48%   64.29%   -0.19%
- Complexity      548      549       +1
============================================
  Files            71       71
  Lines          1875     1882       +7
  Branches        353      354       +1
============================================
+ Hits           1209     1210       +1
- Misses          538      543       +5
- Partials        128      129       +1

Impacted Files
Coverage Δ
Complexity Δ

...in/spring/data/dynamodb/core/DynamoDBTemplate.java
64.55% <100%> (ø)
18 <0> (ø)
:arrow_down:

...namodb/repository/query/PartTreeDynamoDBQuery.java
85.71% <100%> (+0.71%)
10 <1> (+1)
:arrow_up:

...namodb/repository/query/AbstractDynamoDBQuery.java
20.72% <30%> (-1.19%)
7 <0> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 19c7828...2129cf8. Read the comment docs.


Add geometry column to sql server adapter

Differentiate between geometry and geography columns in sql server

This should be put into 0.13 as it is a breaking change in behavior where anyone using SQLServer and PHINX_TYPE_GEOMETRY will need to replace it with PHINX_TYPE_GEOGRAPHY.

Can you check if I can merge this? As the rebase doesn't seem to retrigger any tests

Test suite passes fine (except for codecov), though there are no tests around geometry / geography in sqlserver to begin with.


Promoted reasons

Add in promoted reasons checkboxes
Fixes #4496

Just confirming, so once a Reportback Item is promoted, the whole Reportback is considered promoted? If an editor makes a mistake, will this remove the promoted flag upon changing the Reportback Item status?

@aaronschachter yeah, it works the same way flagged does, the promoted value will toggle between 0/1

:+1:

@angaither  - Random thought around this came to mind. What if a reportback has two items we want to promote, but for different reasons?
Once a reportback item is flagged, the reportback is ignored entirely, so there would never be the opportunity to update the flagged reason upon adding additional litems.  With this its different... wouldnt' the last promoted item always overwrite the reason?


Can't login via Twitter

Method not found: 'System.Security.Principal.IIdentity Microsoft.Owin.Security.Forms.FormsResponseSignInContext.get_Identity()'.
Went to login tonight via Twitter and received this message.

Still an issue for me; anything I'm missing?


TextIOTest: shrink size of test data

We have assertions that the files are still big enough, so this is safe. Speeds up TextIOTest
2x.

Coverage decreased (-0.003%) to 69.678% when pulling 3301d6fe082605e138a11df3308b6fbd2b9cc56d on dhalperi:tiny-write into 1a265e34f31a32300515c454c619a26d61ba2482 on apache:master.


alejandra: init at unstable 2022-01-30

Motivation for this change
Add alejandra to nixpkgs.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 157380 run on x86_64-linux 1

1 package built:

alejandra

Result of nixpkgs-review pr 157380 run on aarch64-darwin 1

1 package built:

alejandra

I think we can merge as is, thank you @0x4A6F !


[BEAM-354]: Modify DatastoreIO to use the v1beta3 API

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[ ] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[ ] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[ ] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ ] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

Use v1beta3 datastore API and remove all v1beta2 dependencies
Modify tests to pass.

Hi @dhalperi @davorbonaci, can you take a look?

Fixed Dan's review comments.

R: @dhalperi

Can you please post a verbose mvn dependency:tree -Dverbose diff from before this change, as well as post the entire tree (without -Dverbose) after it?

Attached the files. I looked at the dependencies brought it by beta3 but all of them were something we already used and with same versions.
dependencies.txt
mvndepDiff.txt

LGTM; merged


[ALLUXIO-1897] use Multipart to upload large file to S3

https://alluxio.atlassian.net/browse/ALLUXIO-1897
Upload large file to S3 using Multipart: https://aws.amazon.com/blogs/aws/amazon-s3-multipart-upload/. The threshold can probably be configurable..

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/9540/
Test PASSed.

Merged build finished. Test PASSed.

LGTM, thank you for the improvement!

@siyangy thanks for the improvement! Is there a way to add a test for it?

@yupeng9, it seems hard to add a test for this. All the current S3 test is mock based, but since this change mostly relies on the jets3t lib, it's not so useful to mock test it again. Otherwise if testing this in integration test, it requires a huge file which doesn't make much sense either.

yeah, that makes sense to me. LGTM


Fix React example in README

There were some typos in the React example. This fixes those.

oops! thank you!

@acao: Turns out the URLs were also outdated. Apologies I didn't get catch that at the same time.
Scripts should be:
<script crossorigin src="https://unpkg.com/react/umd/react.production.min.js"></script>
  <script crossorigin src="https://unpkg.com/react-dom/umd/react-dom.production.min.js"></script>

Want me to make a new PR?

yes thank you, the more the merrier!


Update samples and test sites to new formats

:watch:

:up::date:

Set content root
Make sure web.config and *.cshtml files are published
Add missing Razor tooling in a few cases

Did I miss anything? If that's the story, LGTM.

:shipit:

Yep, that's the idea, will rebase and merge.


Menu badges not rendering

// MenuBuilderSubscriber.php
        $forms = new MenuItemModel('forms', 'Forms', null, [], 'fab fa-wpforms');
        $forms->setBadge('1');
        $forms->setBadgeColor('red');

but the red badge isn't rendering

Fixed by #40, thanks for the report!


--rollback-to- slot does not clear stake addresses

See #238 for background.
The rollback function and delete functions in Delete.hs (https://github.com/input-output-hk/cardano-db-sync/blob/master/cardano-db/src/Cardano/Db/Delete.hs) do not clear stake addresses as they do not have any dependencies to the block table that would be caught by a cascade.
Probably a minor issue.

Probably a minor issue.

I think this is a little more important, especially for debugging and reproducing issues.


Json.Schema.KeywordExtensions is not thread safe with custom keywords

Nuget Package
JsonSchema.Net
Package Version
5.3.1
Operating System
None
.Net Target (if relevant, please specify the version in the description)
None
A clear and concise description of the bug.
_keywordNames is updated on first occurrence of custom keywords. This operation uses a plain Dictionary instead of a ConcurrentDictionary and is therefore not thread safe.
What did you expect?
Using Json.Schema should be thread safe.
Please add test code or steps to reproduce the behavior.
This is a race condition. A reproduction case that is guaranteed to trigger this error is not trivial to create.
The fix is fairly straightforward, though. Use ConcurrentDictionary for caches that may be updated in code paths called during schema building (already done in SchemaKeywordRegistry, for example).
Is there any other information you'd like to share regarding this bug?
This was mentioned in #503, but rejected due to not having a repro case. If desired, I can create a PR with suggested changes. Let me know.
Code of Conduct & Contributor Guidelines

[X] I agree to follow this project's Code of Conduct and Contribution Guidelines.

I think this is resolved with v7.  I've reworked the keyword registry, and it's now using a custom MultiLookupConcurrentDictionary<T> which is backed by ConcurrentDictionary<TKey, TValue>.


"TypeError: 'range' object does not support item assignment"

I made an error in line 1167 of ml_opt.py when I surfaced this paper， the error message showed  "TypeError: 'range' object does not support item assignment". I would like to ask  this question is because of my python's version or other reasons.

I found that the problem was that the range function in the code had to be converted to a list,for example, range(0,n) must write as list(range(0,n)).But I don't know why.

if config.use_attention_embedding and num_states > config.batch_size:
config.batch_size = 1000
May I ask why using less than 1000 states？

I also have this issue. Have you already solved it?

No. I just learn their method but doesn't solve it. I find some code can't be 100% realized. It is a pity.
Are you also interested in RL for CO？
---Original---
From: @.>
Date: Sun, Nov 5, 2023 13:13 PM
To: @.>;
Cc<EMAIL_ADDRESS>Subject: Re: [rlopt/l2i] "TypeError: 'range' object does not support itemassignment" (#6)
I also have this issue. Have you already solved it?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: @.***>


To try out the latest state of this promising tool for EG-2

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@2b05203). Click here to learn what that means.
The diff coverage is n/a.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
@@            Coverage Diff            @@
##             master       #1   +/-   ##
=========================================
  Coverage          ?   54.34%
=========================================
  Files             ?      188
  Lines             ?    12070
  Branches          ?        0
=========================================
  Hits              ?     6560
  Misses            ?     4951
  Partials          ?      559

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


I'd like to be able to set the editor font under settings

I'd like to be able to set the webchuck editor font; the obvious place for this would be under the settings menu.
If no one else gets to it, I may one day submit a PR for this if it's likely to be accepted.

That sounds great! Let me know if there's anything I can do to help!


Unread count is not updating properly in conversation list

https://github.com/chatwoot/chatwoot/issues/4376

I would like to have a meeting regarding this. Working fine, If the agent is not seen the chat. Not working for the user is not seen the chat.
Code implemented
let messageModels = conversationsModel.messages.filter { messageModel in
    return messageModel.createdAt * 1000 > conversationsModel.agentLastSeenAt * 1000 &&
    messageModel.messageType == 0 &&
    messageModel.isPrivate != true
}
return messageModels.count

inbox_name not found in the /conversations API.


Render list descriptions everywhere exactly as the user typed them

We started representing the opportunity and solutions we are currently committed to in stand-up, and now feel this has gone outside of the annotation user experience and "I don't feel I have the expertise / I don't feel qualified to share my thoughts publicly". So we'll pragmatically close this ticket to lower the cost of delay on the next opportunity.


MB-15277: add requested move date for NTS-release shipments

Jira ticket for this change
Summary
The requested move date column now accounts for shipment information related to NTS-release shipments.
Setup to Run Your Code

💻 You will need to use two separate terminals to test this locally.
Terminal 1
Start the UI locally.
make client_run

Terminal 2
Start the Go server locally.
make server_run

Additional steps

Access the office app as a TOO user
Verify that the dates are populating based on the least date

Messages

:book:
:link: MB-15277

  Generated by :no_entry_sign: dangerJS against 3ef9970fc651914137f9686cb9b873c111a82292

Just calling out that I looked over the latest changes (test fix) and my approval stands. Nice work as usual @ronaktruss