Test performance on multiple sample rate and buffer size

Hi it would be useful to test multiple sample rate (nam at 96khz) and buffer size and compare the two.
An additional 96Khz nam model from tonehub should be added to the rep
AUDIO_BUFFER_SIZE x
SAMPLE_RATE y (get from the NAM json)
AUDIO_CHANNELS z (1 - mono or 2 - stereo)
size_t numBuffers = (y / x) * z; (number of buffers in 1 second)
and get the performance  (loop on the numBuffers value)  to see the realtime performance (it should be < 1sec) assuming buffer_size will always be the same.
That should be after prewarming ....(a one shot operation after loading a model)
I normally use y/x/z as 48000/128/1 or 96000/128/1 in live envs.
Thx a lot

Testing with models trained at different sample rates won't really be meaningful... the RTNeural implementation (at least in its current form) does not do any re-sampling, so the operations being performed by the RTNeural implementation will be exactly the same regardless of the sample rate (and presumably the performance will be the same as well).
Similarly I'm not sure if it's worth testing mono and stereo separately... again I imagine the exact same operations will be performed on each channel?
Testing at multiple buffer sizes would be nice! If you feel comfortable implementing that, please go ahead.

Probably my fault (i am not an expert) but different sample rates is good to test realtime performance with higher quality envs. It is just to verify that NAMRTNeural can process 96000 sample/sec (the total duration for 96000 samples should be < 1 sec) in addition to 48000 sample/sec. I am assuming no resampling (DAW, Audio card and plugin operates at 96khz).
Mono and stereo are the same and I agree with you...just double the computation in the same time window.
I have seen you are changing a lot the main.cpp should I wait for a more stable version or should I try to implement on the current version ?

As long as RT-Neural can interpret 96khz models correctly I think that is fine right? I would set the sample rate to 96khz before RT-Neural in HISE for instance, so that the sample rate always matches what the model expects.


Compatible with Google maps version web

First congrats for this project!
I don't know if this is in the project scope, but I think this would be a very nice feature.
If you are using cordova-plugin-googlemaps on web, this only be a wrapper of Google maps version web. I think it is comfortable for develop and especially if you are using the same code for apps and web version.
What do you think about this functionality?

I have no plan to wrap the web version.


JAMES-3726 Allow relaxing CassandraMailRepository LWT usage

Jira https://issues.apache.org/jira/browse/JAMES-3726

BTW @ottoka this might interest you a lot to avoid bad surprises upon your error handling.

rebase master & squash fixup


NETTY-3 Use weighted score to match request URI to destination method

JIRA: https://issues.cask.co/browse/NETTY-3
Build: http://builds.cask.co/browse/NH-BAD3-1
We decided to go with weighted scores for matching request URI parts to destination path parts. The matches are weighted in descending order from left to right. For every match, exact matches have higher weight than group matches, and group matches have higher weight than wildcard matches.

LGTM

Build is green, merging.


make ipet-parse script multithreaded and add a tqdm progress bar

what the description says. I parallelized this script for two reasons:

It is much faster (n times faster than before)
It consumes less memory than the previous variant, because an experiment and the contained TestRun object is freed immediately after it was parsed and stored to disk, instead of continuing to parse until all files were processed.

Drawback: Files such as meta data, error files etc. are simply discarded if passed to the script. However, I believe this is the same behavior as before. In order to associate a test run with an OUT file and completely differently named ERR/META/whatever file, one would need to setup a custom TestRun object writing custom Python code.
Check out the progress bar.

What happened to travis?

I dont know. Two weeks ago, everything was fine.


Crash during Zoom screen share of Microsoft Remote Desktop

ℹ Computer information

Windows build number: 19041.423
PowerToys version: v0.20.0
PowerToy module:

📝 Provide detailed reproduction steps (if any)
I'm not sure at what point PowerToys crashed, but it was while I was on a Zoom call screen sharing a remote desktop (connected through Remote Desktop Connection).
Exception message:
Version: 1.0.0
OS Version: Microsoft Windows NT 10.0.19041.0
IntPtr Length: 8
x64: True
Date: 08/03/2020 13:50:18
Exception:
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'Timer'.
at System.Timers.Timer.set_Enabled(Boolean value)
at System.Timers.Timer.Start()
at PowerLauncher.MainWindow.OnVisibilityChanged(Object sender, DependencyPropertyChangedEventArgs e)
at System.Windows.UIElement.RaiseDependencyPropertyChanged(EventPrivateKey key, DependencyPropertyChangedEventArgs args)
at System.Windows.UIElement.OnIsVisibleChanged(DependencyObject d, DependencyPropertyChangedEventArgs e)
at System.Windows.DependencyObject.OnPropertyChanged(DependencyPropertyChangedEventArgs e)
at System.Windows.FrameworkElement.OnPropertyChanged(DependencyPropertyChangedEventArgs e)
at System.Windows.DependencyObject.NotifyPropertyChange(DependencyPropertyChangedEventArgs args)
at System.Windows.UIElement.UpdateIsVisibleCache()
at System.Windows.PresentationSource.RootChanged(Visual oldRoot, Visual newRoot)
at System.Windows.Interop.HwndSource.set_RootVisualInternal(Visual value)
at System.Windows.Interop.HwndSource.set_RootVisual(Visual value)
at System.Windows.Window.SetRootVisual()
at System.Windows.Window.SetRootVisualAndUpdateSTC()
at System.Windows.Window.SetupInitialState(Double requestedTop, Double requestedLeft, Double requestedWidth, Double requestedHeight)
at System.Windows.Window.CreateSourceWindow(Boolean duringShow)
at System.Windows.Window.CreateSourceWindowDuringShow()
at System.Windows.Window.SafeCreateWindowDuringShow()
at System.Windows.Window.ShowHelper(Object booleanBox)
at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
Log file:
2020-08-03.txt

I got the same crash, but there was no remote desktop session involved.
2020-08-03.txt

tracked in #5400


Plans to release code for training of application specific discriminators?

Awesome work and example notebook. Are there any plans to release the training code?

Hi. Thanks for your interest in our work. Currently the paper is under review process. We do plan to release the training code as well.
Best

Any idea when the training code will be released?


No callback if command not found

When running a command that doesn't exist, no error callback is generated and the task exits abruptly.
For example,
gulp.task('runme', function(cb) {
    new $.run.Command('nonexistant').exec(functioen (err) {
      console.log('This is where I'm supposed to tell you that I do not exist', err);
      cb(err);
    });
});

Running gulp runme just outputs:
[14:38:48] Using gulpfile ~/myproject/gulpfile.js
[14:38:48] Starting 'runme'...
nfantone@ubuntu-laptop ~/myproject $ echo $?
0

And ends silently with status 0.
It would be useful to inform the user that the command/binary that is to be run is not installed in the system and raise a non-zero error status (like POSIX sh standard, 127).

Up for a PR?


[FEA] Standardize a Workflow's representation of its output columns

In a robust training pipeline, online preprocessing Workflows will need to know the columns output by the preprocessing Workflow, and the model will need to know the columns output by the online Workflow (or the preprocessing Workflow if there is no online preprocessing) in order to build the appropriate input tensors. Moreover, they will need to know which columns are list-like (or sparse or multi-hot or whatever you want to call it) in order to know what types of inputs to build. Creating a standard and intuitive representation of these columns that can be accessed from, saved by, and loaded to a Workflow will be invaluable in making pipelines simpler and more robust.
I understand that right now there is a method Workflow.create_final_cols that builds a representation like this. It would be good to better document what that representation is and possibly even build an object for it so that its attributes can be better codified and e.g. tab completed. I also wonder, depending on the amount of work it takes to generate this final representation, if this might be better done by a property like Workflow.output_columns that just computes this "final" representation based on whatever Ops are currently in the Workflow.
Once we have something like this, I can even envision a TensorFlow framework_util which can map from this representation to a set of input tensors (and even take care of constructing the SparseTensor multi-hot embedding inputs from values and nnz input tensors). Such a pipeline might look like:
input_columns = load_column_schema("path/to/columns/exported/by/preprocessing/workflow/columns.extension")
online_proc = Workflow(input_columns)
online_proc.add_cat_preprocess(HashBucket(...))

inputs = framework_utils.tensorflow.build_inputs(online_proc.output_columns) # probably return a dict of {name: tensor}
outputs = build_whatever_model(**model_params)(inputs)
model = tf.keras.Model(inputs=list(inputs.values()), outputs=outputs)

We're part of the way there with the API overhaul - we now have a complete representation of the output column names at least, but we still need to be able to get the dtypes reliably.

We're part of the way there with the API overhaul - we now have a complete representation of the output column names at least, but we still need to be able to get the dtypes reliably.

tracked in https://github.com/NVIDIA/NVTabular/issues/943 now


(MODULES-3683) Add Supported DSC Resources to build system

This adds the supported DSC Resources to the type building system by
moving both the xDscResources and the dscresources folder to a new
staging folder called thedescresources. This is a hacky fix, but avoids
having to modify the MOF parsing and building classes to support reading
more than one folder.
This also changes the custom DSC resource detection we did previously.
It relied on the directory name being xDscResources, since we have moved
things this now looks for thedscresources instead.

#247 should be finished / merged prior to this.


Semantic segmentation plugin

Added support for multiple models of semantic segmentation including
deeplabv3plus which should work better than the existing deeplabv3
Mainly I was just trying here to see how hard it was to bring a new model in, but this does work a little bit better than the plain deeplabv3 plugin.

Hi! This is excellent and exactly the kind of flexibility with model choices I had in mind when I changed the GIMP-ML architecture to be based on the Torch Hub format.
I will gladly merge this PR, but I'm not too eager about the implication of creating a separate fork of GIMP-ML just yet. I would prefer these changes to be part of the main repo, but this is up to @kritiksoman. I guess I will have to fork GIMP-ML for good at some point if he remains unresponsive.
The code for the plugin and the added hubconf.py look good. There are some minor issues and cosmetic adjustments I would prefer to be included before this PR can be merged, though:

The plugin script must have the executable bit set for GIMP to recognize it as a plugin. I assume you are developing on Windows and this just worked semi-accidentally? Anyway, please apply git update-index --chmod=+x plugins/semanticsegmentation.py to make sure it works on Unix platforms also.
Please set progress=True for load_state_dict_from_url() in the hubconf.py file. This is required for having the model download progress displayed in GIMP. Better yet if you make this a parameter of the model callables in hubconf.py.
I would name the plugin file semantic_segmentation.py and the model file DeepLabv3.py. You can remove the previous DeepLabv3 plugin and model.
Please follow PEP 8 for the code style. I say this mostly regarding the variable name style, where you should prefer snake_case instead of camelCase for Python code unless there is a good reason not to.
For consistent formatting applying an auto-formatter would be nice. Good options are PyCharm's built-in auto-formatting and black.

Oops, I've been programming in 3 different languages recently so I'm not hot on case styles and the like! Will fix these things in a couple of days once I've got time after I've finished marking student work.
Next thoughts for things I want in gimp is this deblur: it's surprisingly simple and the results look great. I'll try to add that at some point soon.
https://github.com/csdwren/SelfDeblur/blob/master/selfdeblur_nonblind.py

Next thoughts for things I want in gimp is this deblur: it's surprisingly simple and the results look great. I'll try to add that at some point soon.

Looking forward to your PR. :+1:
You can apply the same logic I used for adding MiDaS to the monocular depth plugin to provide a choice between the existing DeblurGANv2 model and SelfDeblur: https://github.com/valgur/GIMP-ML/blob/master/plugins/monocular_depth.py

Think I've fixed the code style etc. on this now, and added the progress callbacks

Code and permissions fixed now, think it is good to merge, but I can't test on unix here?

Looks good to me now. Thank you!


Uncaught InvalidStateError: Failed to execute 'createMediaElementSource'

Each time that I press Play button, the console triggers this error:
"Uncaught InvalidStateError: Failed to execute 'createMediaElementSource' on 'AudioContext': HTMLMediaElement already connected previously to a different MediaElementSourceNode"

Works fine on Firefox, what browser do you use ?

Google Chrome 43.0.2357.132 in Windows 10 RTM x64.

Too, works fine here, but console triggers this error.

Google Chrome Version 45.0.2454.101 in Ubuntu also have same issue and I am not able to play song,Is there any solution for this

@nt1m This is very focused in Mozilla Firefox, but need tests on Google Chrome.


 font-size larger than expected when .form-group-sm stacks

I am trying to create a responsive form.  The default form layout is horizontal.  When the user hits the small browser size it will put the form back into the standard form layout with label on top of the input field.  I have my default form group class set to "form-group form-group-sm".  This seems to display just fine in the horizontal view but when it changes to the standard form layout the font styles appear to be reset to the medium.
I have attached a working sample via JSFiddle.  In the example, if you slide the middle divider to simulate window size in the result window you will see the issue in the first text box field.  The second textbox is meant to simulate the medium form group size.  I don't know if this is by design but it causes inconsistent form layout between devices.
https://jsfiddle.net/ddelella/02vaLa9s/8/

Hi @ddelella!
You appear to have posted a live example (https://fiddle.jshell.net/ddelella/02vaLa9s/8/show/light/), which is always a good first step. However, according to the HTML5 validator, your example has some validation errors, which might potentially be causing your issue:

line 42, column 3 thru line 42, column 42: Bad value   for attribute action on element form: Must be non-empty.

You'll need to fix these errors and post a revised example before we can proceed further.
Thanks!
(Please note that this is a fully automated comment.)

I'm experiencing the same issue. I've attempted to fix the validation in the fiddle:
https://jsfiddle.net/02vaLa9s/9/
The issue is that the font size rule for form-group-sm control labels is wrapped in the following media query:
@media (min-width: 768px) {
  .form-horizontal .form-group-sm .control-label {
    padding-top: 6px;
    font-size: 12px;
  }
}

My workaround is to add the innards of the query to my site styles:
.form-horizontal .form-group-sm .control-label {
  padding-top: 6px;
  font-size: 12px;
}

I'm experiencing the same issue. I've attempted to fix the validation in the fiddle:
https://jsfiddle.net/02vaLa9s/9/
The issue is that the font size rule for form-group-sm control labels is wrapped in the following media query:
@media (min-width: 768px) {
.form-horizontal .form-group-sm .control-label {
padding-top: 6px;
font-size: 12px;
}
}
My workaround is to add the innards of the query to my site styles:
.form-horizontal .form-group-sm .control-label {
padding-top: 6px;
font-size: 12px;
}

Punting as a won't fix given how late into v3's lifecycle we are. <3


[backport] Support default dtype in L.BinaryHierarchicalSoftmax

Backport of #5648

[automatic post] Jenkins, test this please.

Jenkins CI test (for commit 0632b1240d1f360dfb79e601bfe6a2fb635986e4, target branch v5) failed with status FAILURE.
(For contributors, please wait until the reviewer confirms the details of the error.)

Jenkins, test this please.

Jenkins CI test (for commit 0632b1240d1f360dfb79e601bfe6a2fb635986e4, target branch v5) succeeded!


[Help] Implement arrow_serialize/deserialize for newtype over num_complex

I'd like to create a newtype struct that contains num_complex::Complex32 and can convert to/from arrow.
struct Phasor(num_complex::Complex32);

I'm having trouble understanding the documentation for implementing ArrowSerialize and ArrowDeserialize with types other than privatives (as shown in the complex_example.rs) can anyone help point me in the right direction? Here is what I have so far, though I know I'm not approaching this right...
impl arrow2_convert::field::ArrowField for Phasor {
    type Type = Self;

    fn data_type() -> arrow2::datatypes::DataType {
        arrow2::datatypes::DataType::Extension(
            "phasor".to_string(),
            Box::new(arrow2::datatypes::DataType::Struct(vec![
                Field::new("re", arrow2::datatypes::DataType::Float32, false),
                Field::new("im", arrow2::datatypes::DataType::Float32, false),
            ])),
            None,
        )
    }
}

impl arrow2_convert::serialize::ArrowSerialize for Phasor {
    type MutableArrayType = arrow2::array::MutableStructArray;

    fn new_array() -> Self::MutableArrayType {
        Self::MutableArrayType::new(
            <Self as arrow2_convert::field::ArrowField>::data_type(),
            vec![],
        )
    }

    fn arrow_serialize(v: &Self, array: &mut Self::MutableArrayType) -> arrow2::error::Result<()> {
        let real: &mut MutablePrimitiveArray<PhasorType> = array.value(0).unwrap();
        real.try_push(Some(v.re()));
        let imag: &mut MutablePrimitiveArray<PhasorType> = array.value(1).unwrap();
        imag.try_push(Some(v.im()));

        array.push(true);
        Ok(())
    }
}

impl arrow2_convert::deserialize::ArrowDeserialize for Phasor {
    type ArrayType = arrow2::array::StructArray;

    fn arrow_deserialize(v: Option<???>) -> Option<Self> {
        v.map(|t| Phasor::new(t.get(0).unwrap(), t.get(1).unwrap()))
    }
}

arrow2_convert::arrow_enable_vec_for_type!(Phasor);

I've seen issue #79 reference adding support for remote types, but until then and to better understand arrow2 I'd like to understand how to do this manually. Thanks in advance!

@graybc thanks for raising this. You're right. Customizing non-primitive types need more work. As a long-term solution, in addition to #79 some of the reworking of the ArrowSerialize and ArrowDeserialize traits in the next release should help support this use-case better.
For the short-term, the code below is a possible workaround. It manually implements part of what we would do for remote type. However, the way data types are specified, it might be a bit tricky to specify the data type exactly the way you have it in your example. It should be possible by manually implementing the ArrowField trait for the LocalComplex32 struct below, but I haven't tried it.
#[test]
fn test_complex() {
    use num_complex::Complex32;

    #[derive(Debug, PartialEq)]
    struct Phasor(num_complex::Complex32);

    #[derive(arrow2_convert::ArrowField, arrow2_convert::ArrowSerialize, arrow2_convert::ArrowDeserialize)]
    struct LocalComplex32 {
        re: f32,
        im: f32,
    }

    impl From<&num_complex::Complex32> for LocalComplex32 {
        fn from(c: &num_complex::Complex32) -> Self {
            Self { re: c.re, im: c.im }
        }
    }

    impl From<LocalComplex32> for num_complex::Complex32 {
        fn from(c: LocalComplex32) -> Self {
            Self::new(c.re, c.im)
        }
    }

    impl arrow2_convert::field::ArrowField for Phasor {
        type Type = Self;

        fn data_type() -> arrow2::datatypes::DataType {
            LocalComplex32::data_type()
        }
    }

    impl arrow2_convert::serialize::ArrowSerialize for Phasor {
        type MutableArrayType = <LocalComplex32 as arrow2_convert::serialize::ArrowSerialize>::MutableArrayType;

        fn new_array() -> Self::MutableArrayType {
            LocalComplex32::new_array()
        }

        fn arrow_serialize(v: &Self, array: &mut Self::MutableArrayType) -> arrow2::error::Result<()> {
            let local: LocalComplex32 = (&v.0).into();
            LocalComplex32::arrow_serialize(&local, array)
        }
    }

    impl arrow2_convert::deserialize::ArrowDeserialize for Phasor {
        type ArrayType = <LocalComplex32 as arrow2_convert::deserialize::ArrowDeserialize>::ArrayType;

        fn arrow_deserialize(v: <&Self::ArrayType as IntoIterator>::Item) -> Option<Self> {
            v.map(|t| Phasor(t.into()))
        }
    }

    arrow2_convert::arrow_enable_vec_for_type!(Phasor);

    let original = vec![Phasor(Complex32::new(10., 10.)), Phasor(Complex32::new(20., 20.)), Phasor(Complex32::new(30., 30.))];
    let array: Box<dyn Array> = original.try_into_arrow().unwrap();
    let round_trip: Vec<Phasor> = array.try_into_collection().unwrap();
    assert_eq!(round_trip, original);
}

Hi @ncpenke,
Thanks so much, that makes a lot of sense! I'll keep an eye out for the next release as well but this is a very cool project and has really helped me get started with Apache Arrow and arrow2.


Fix CI

see https://github.com/open-mmlab/mmocr/pull/472

Anything wrong with ubuntu-18.04?

I don't know, something in latest os break the installation


Ref script fee fixes

Check everywhere that could use a ref script (not sure if it's allowed in all these spots)

Duplicate size for different ref inputs with the same ref script as per Haskell code

look in spent inputs too (as this is possible to ref a script from)

Possibly addresses #354

I'm going to wait at least a little bit until we can see if #354 is fixed by this before merging. If the guy who made the issue responds with what the actual inputs were (he only posted final cbor so any utxo/orig script info isn't preserved) then we can make test vectors out of that.


metering cluster registration fix

I don't believe the additional logic is necessary. Just needed to fix the owner to point to metering-controller-manager. Updated in commit:
https://github.com/redhat-marketplace/redhat-marketplace-operator/commit/35c07e156a6334b37ca9eb764b11a7067d9550fc
If you encounter another error, let me know

closing this one will retest with Dan commit


Added input filters tab on workflow page and improved dataset page

Checklist before merging

[X] I added an appropriate entry to CHANGELOG.md

Closes #428
Closes #444
Closes #441
Closes #437

This looks great!

There is a tiny unclear detail I spotted in the pagination section.
I start with

Then I click on "11", and the page changes correctly, but "12" is highlighted in gray:

Is this the expected behavior?

Another minor glitch.
I start with

Then I select a given well in the dropdown menu (without clicking apply), and the size of some element changes

This triggers a small layout change, which moves the columns horizontally - and it looks like something is happening in the data.
Would it be doable to keep the size fixed, when selecting e.g. well=A90?

Another tiny glitch in the dropdown menus:
Either the true or false row should be highlighted, but not both at the same time.

When I try to edit the zarr_dirr but receive a 422 from the backend, the spinner keeps going forever

Within the dataset-filters modal:

The attribute values can be strings, numbers or boolean (boolean is currently missing).

The attribute values can be strings, numbers or boolean (boolean is currently missing).

This also applies to the new workflow-task input-filters tab:

Another tiny glitch in the dropdown menus:
Either the true or false row should be highlighted, but not both at the same time.

Changed the hover background color of slim-select to light grey. This applies globally, not only in this page.

When I try to edit the zarr_dirr but receive a 422 from the backend, the spinner keeps going forever

Handled this case:

I can do this only setting a fixed width to the dropdowns. I would set 180px for attributes and 100px for types. This has an implication on the columns that have a long name in the header.

I like this new option 1 much better! Makes for more horizontal compactness. We may have some attributes that are 2 lines long, but probably most won't be :)

I confirm that after https://github.com/fractal-analytics-platform/fractal-web/commit/6d62eb12c2db35191229643fa1e03307bc4ff822 fixes the bug I just spotted is not there any more, thanks.


Adding second API

Using OPEN AI API
Adding generated image to the HTML

The OPEN AI will convert the lyrics of the song into a single image. We can use this functionality to bring more images into the Foto Music app.

I thought that we agreed on 'Foto Music', not 'Photo Music'.
I don't like how the lyrics are showing on the right side, they are long. Probably something I have to play with.
Otherwise, your code seems to work fine and gathers all the needed info.

I thought that we agreed on 'Foto Music', not 'Photo Music'. I don't like how the lyrics are showing on the right side, they are long. Probably something I have to play with. Otherwise, your code seems to work fine and gathers all the needed info.

Oh okay, sorry I didn't realise it was deliberate. We can change it back if you prefer it but I think the app's name should be consistent. Let's say that a user in the UK is looking for our app to interact with it, the name of the app with an F is likely to be misspelled.
The lyrics being in the right hand side is because it's the only place where they can be displayed leaving space for other elements under the artist's photo. They come in this column format from the API. There is a way to display them like a whole single text but it makes it really confusing and difficult to follow. We can talk about this things later!
Thank you for the feedback! :)


Is still actively maintained?

Related to: https://github.com/jondot/sneakers/issues/452

[bump]

That is a real shame as I was just looking at it for a new project and then noticed the commit history.

It depends on what you mean by "dead". Sneakers hasn't massively changed in years. I may have been less responsive as I have dozens of RabbitMQ projects to work on but at some point small  projects reach maturity and do not change much.

Also, this is open source software, so instead of asking others to "actively maintain" something for you, you can contribute and ask for committer permissions. If anyone wants them
and is ready to review PRs, triage issues and answer questions, let me know.

Hey @michaelklishin sorry if my message came across as entitled. I didn’t mean to say I deserve your free time. I just meant it would have been nice if the community had picked up rabbitmq some more over some other back-ends.
Good point on contributing though. Do you have any thoughts on which areas there can be some nice improvements for new contributors?

Right now I want to clear the PR backlog and reduce the number of open issues (including by closing them as too old or after triaging).
Like I said above, I think Sneakers has matured to the point I don't expect it to
accumulate many large features. It has a ton of work around integration tests, metrics,
concurrency and maybe efficiency, so basically operational areas that are never "solved" problems. But it has a fairly fixed scope otherwise IMO.
Sadly I don't have the permissions to add contributors. @jondot I would like to be able to :)
But I can merge PRs, update release notes and push tags, which can acts as a short term way of
producing new releases.


🙅 Inaccurate model coref predictions master thread

Master thread for collecting incorrect and/or problematic coreference predictions with the pretrained models. These can be interesting test cases when training the next version of the model.

For this article, the model struggles with NASA's James Webb Space Telescope.
This is the mentions array:

[Mauna Kea: [Mauna Kea, Mauna Kea, Mauna Kea, Mauna Kea, Mauna Kea, Mauna Kea, Mauna Kea, Mauna Kea, Mauna Kea, Mauna Kea, Mauna Kea, Mauna Kea, it, Mauna Kea, Mauna Kea], Hawaii: [Hawaii, Hawaii, Hawaii, Hawaii, Hawaii, its, Hawaii, Hawaii, Hawaii, Hawaii], Spain: [Spain, Spain, Spain, Spain, Spain, Spain], Earth: [Earth, Earth, Earth, Earth], astronomers: [astronomers, their], the world's largest telescope in Hawaii: [the world's largest telescope in Hawaii, the telescope, the telescope, it, the Webb telescope, the telescope, it, Thirty Meter Telescope, Its, the telescope on La Palma, the telescope in Spain, this telescope], the islands': [the islands', their], Meter Telescope officials: [Meter Telescope officials, their], their backup site atop a peak on the Spanish Canary island of La Palma: [their backup site atop a peak on the Spanish Canary island of La Palma, it, it, the site], La Palma: [La Palma, La Palma, La Palma, La Palma, La Palma, La Palma, La Palma, La Palma, La Palma, La Palma], Mauna Kea's: [Mauna Kea's, Mauna Kea's], Bolte, who has used existing Mauna Kea telescopes: [Bolte, who has used existing Mauna Kea telescopes, he], Bolte: [Bolte, Bolte, Bolte, The telescope group's Bolte], Webb: [Webb, Webb], Mather: [Mather, He, Mather, he], bright stars: [bright stars, them], Loeb: [astrophysicist Avi Loeb, who chairs Harvard University's astronomy department, Loeb, Loeb, he], The Native Hawaiian opponents: [The Native Hawaiian opponents, themselves, their, They], the telescope group: [the telescope group, The telescope group], protest leader Kealoha Pisciotta: [protest leader Kealoha Pisciotta, Pisciotta], Thirty Meter Telescope officials: [Thirty Meter Telescope officials, they], the Canary Islands: [the Canary Islands, the Canary Islands, the Canary Islands], Others: [Others, their, their], Jos Manuel Vilchez, an astronomer with Spain's Higher Council of Scientific Research and a former member of the scientific committee of the Astrophysics Institute of the Canary Islands: [Jos Manuel Vilchez, an astronomer with Spain's Higher Council of Scientific Research and a former member of the scientific committee of the Astrophysics Institute of the Canary Islands, We, We], Vilchez: [Vilchez, Vilchez, Vilchez, Vilchez], Native Hawaiians: [Native Hawaiians, their, they, Native Hawaiians]]

Webb is broken out as if it is a last name when it is the part of the telescope's name.

Hey @svlandeg ,
I have observed that the live demo on https://huggingface.co/coref/ is surprisingly correct at multiple cases, when My local implementation of the same model fails.
To ensure this furthermore, I have tried to resolve co-reference with both of the models, latest one - neuralcorefv4.0 and neuralcoref-lg-3.0.0. And, our results are way poorer than your live demo. I am attaching a screenshot for your better understanding of the situation. Have a look, and please respond.
Does the live demo implements a different model than both of the above mentioned ones. if So; How can we get them to implement in our project.?????

Fig: our implementation results ( neuralcoref v3 + neuralcored v4)

Fig:  your live demo result

I wasn't involved with this project when the demo environment was created. However, note that it's not just the version that was trained that makes a difference, but also the specific hyperparameters used for making the predictions. So that is definitely something you can "play" with too.

I wasn't involved with this project when the demo environment was created. However, note that it's not just the version that was trained that makes a difference, but also the specific hyperparameters used for making the predictions. So that is definitely something you can "play" with too.

Thanks,
I actually tried different options for these Hyper-parameters, But, none of the model release' uploaded here could match the demo one.

Same here. I tried all kinds of options for hyperparameters, and I am unable to reproduce the result. In the sample page, the score is usually like 3~15, where on my environment the result is always like -2~0.7. I am wondering how exactly to reproduce the result on the sample page.
Thank you!
@svlandeg @thomwolf

Same here.
With exactly the same sentence provided from the sample site, I tried all kinds of options for hyperparameters, but I am still unable to reproduce the result. I installed Spacy and Neuralcoref from source on a brand new Docker, so it should not be a problem of environment.
In the sample page, the score is usually like 3 to 15, where on my environment the result is always like -2 to 2.
I am wondering how exactly to reproduce the result on the sample page.
Thank you very much!
@svlandeg @thomwolf
resolved by compiling spacy 2.1.0 and neuralcoref from source code.

hello,
you confirm that the demo results can be achieved by compiling spacy 2.1.0 and neuralcoref from source code?

Same here.
With exactly the same sentence provided from the sample site, I tried all kinds of options for hyperparameters, but I am still unable to reproduce the result. I installed Spacy and Neuralcoref from source on a brand new Docker, so it should not be a problem of environment.
In the sample page, the score is usually like 3 to 15, where on my environment the result is always like -2 to 2.
I am wondering how exactly to reproduce the result on the sample page.
Thank you very much!
@svlandeg @thomwolf
resolved by compiling spacy 2.1.0 and neuralcoref from source code.

hello,
you confirm that the demo results can be achieved by compiling spacy 2.1.0 and neuralcoref from source code?

Not exactly. I am just saying that this would increase the accuracy on my side, from usable to usable.

Surprised to see the following.
On the example sentence in the README, neuralcoref predicts accurately:

But on a slight modification, where we switch sister to brother, and swap the pronouns, we get an incorrect prediction on the second sentence:

It would be very helpful if someone could shed some light on what model and combination of package versions are used in the demo environment. Like others here I'm not able to reproduce what I see on the demo in my local setup, even when rolling spacy back to 2.1.3 and building neuralcoref from source. It feels like the model served in the demo environment is a different model or it was trained with different word embeddings. Are pretrained neuralcoref models tied to any specific spacy language model tag? I've played with the parameters like others here with some improvement but I'm still seeing systematic differences with the live demo.

It would be very helpful if someone could shed some light on what model and combination of package versions are used in the demo environment. Like others here I'm not able to reproduce what I see on the demo in my local setup, even when rolling spacy back to 2.1.3 and building neuralcoref from source. It feels like the model served in the demo environment is a different model or it was trained with different word embeddings. Are pretrained neuralcoref models tied to any specific spacy language model tag? I've played with the parameters like others here with some improvement but I'm still seeing systematic differences with the live demo.

Totally agree!!! Online demo is an ideal starting point, but only if it is reproducible :(

I'm not able to share much in the way of text for confidentiality reasons, but I'm noticing that the pre-trained model seems to be gravitating toward resolving "us" to "We." It might be useful to be able to blacklist certain words (e.g. "We") as never been satisfactory coreferents.
\< It is not up to us to rectify things
\---
\> It is not up to We to rectify things

\< It is absolutely an issue, but not only to us
\---
\> It is absolutely an issue, but not only to We

neuralcoref 4.0
spacy 2.3.2

I agree- to have a more customizable blacklist (including they, it, these,
who) would be wonderful. this is great tech as it is but just a suggestion
On Mon, Jul 27, 2020 at 2:57 PM Keating950<EMAIL_ADDRESS>wrote:

I'm not able to share much in the way of text for confidentiality reasons,
but I'm noticing that the pre-trained model seems to be gravitating toward
resolving "us" to "We." It might be useful to be able to blacklist certain
words (e.g. "We") as never been satisfactory coreferents.
< It is not up to us to rectify things
---
> It is not up to We to rectify things
< It is absolutely an issue, but not only to us
---
> It is absolutely an issue, but not only to We

neuralcoref 4.0
spacy 2.3.2

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/huggingface/neuralcoref/issues/215#issuecomment-664577814,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ALGMOQXWSF373JA2LBJJMJDR5XEXZANCNFSM4JBIG4EQ
.

I agree- to have a more customizable blacklist (including they, it, these, who) would be wonderful. this is great tech as it is but just a suggestion

If you're interested in this feature, I've added it in my fork of this project. I'm still making sure it works, so I'm all ears to any feedback and review.

Thanks alot Keating950!
I see that in your fork NO_COREF_LIST = ["i", "me", "my", "you", "your"] no
longer exists in train/document.py, nor in neuralcoref.pyx. So to me it
seems we do not place our custom blacklist directly in the source code.
Does that mean each time neuralcoref is instantiated I just pass the custom
blacklist? for example:
coref = neuralcoref.NeuralCoref(nlp.vocab, greedyness=0.75 ,
blacklist = ["i", "me", "my", "you", "your", "they", "their", "it"] )
Just verify I implement your fork as intended. Thanks
On Sun, Aug 9, 2020 at 3:21 PM Keating950<EMAIL_ADDRESS>wrote:

I agree- to have a more customizable blacklist (including they, it, these,
who) would be wonderful. this is great tech as it is but just a suggestion
If you're interested in this feature, I've added it in my fork
https://github.com/Keating950/neuralcoref of this project. I'm still
making sure it works, so I'm all ears to any feedback and review.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/huggingface/neuralcoref/issues/215#issuecomment-671090891,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ALGMOQQCPSE3FI3ZWR7F6U3R73ZL5ANCNFSM4JBIG4EQ
.

Yup, that's exactly right. I've updated the README. Feel free to open an issue on that repo if you have any other questions.

Thanks alot man.
On Sun, Aug 9, 2020 at 10:31 PM Keating950<EMAIL_ADDRESS>wrote:

Yup, that's exactly right. I've updated the README. Feel free to open an
issue on that repo if you have any other questions.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/huggingface/neuralcoref/issues/215#issuecomment-671138129,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ALGMOQTPEWADAXYJ75QCNWLR75LX5ANCNFSM4JBIG4EQ
.

Thanks for your work @Keating950! Very helpful!

@lauwauw Thanks! I've merged in the latest changes from this repo in light of the renewed interest.


Revert cherry-pick of 61e11c6808e3fd87c574f4277b3d9ff745dc85f0

This commit caused failures on bleeding-edge, and was reverted.
This commit was cherry-picked by issue #29222
@johnmccutchan

Merged in 1.23.0-dev.11.3


Adding Catalan (4 accents)

Adding the files and code related to Catalan with 4 accents: Central, North-Western, Balearic and Valencian.

Thank you!

Thank you for your great work =)


Default certificate subject is too long

Currently TbsCert uses a default subject that is too long. It uses the full public key, encoded and represented as hex. We would do better to use the Subject Key Identifier instead in these cases - this is still deterministic and unique, but considerably shorter.
Thanks @benmaddison for finding this and pointing this out to us.

This was merged into main via PR #165


Added About Us page and edited navbar to have more space

still need to add things to about us page

fixed in new PR


Did you try to implementate deepsmiles to chemical_vae coded by Rafa Gómez-Bombarelli etc.?

Thank you for your great library.
It is such a great idea that we can insert this converter before encoding and after decoding.  Do you actually implement it into any vae model, such as chemical_vae built by Rafa Gómez-Bombarelli etc? Many thanks

No. I have never used any form of deep learning.


Added Italian Translation

Added italian translation for the app

Hi, thanks for the translation! Could you please reformat the file so that the spaces match the other translation files?

Removed the extra lines, I can't figure out where they were generated


Removing field "user_id" when saving instance of Class via meteorMethods

Here's my application: https://github.com/furagi/poller
In the Topics I defined meteorMethod "answer". In this method I creating instance of Comment, like
if(!Meteor.isServer) {
        userId = Meteor.userId();
}
let comment = new Comment({
        text,
        topic_id: this._id,
        user_id: userId
      });

When I call Topic::answer from the server, everything is fine.
But when I call the Topic::answer from client-side , I get validation error. It says, that user_id is undefined. I checked in debugger, and comment.user_id is defined when I call comment.save().
I also checked in debugger, at Comment's event beforeInsert(e) , e.currentTarget.user_id is still defined.
I found temporary solution: for some reasons, when I set user_id in Comment's beforeInsert event, it's works.
Here is my Comment's beforeInsert code:
beforeInsert: function(e) {
      const comment = e.currentTarget;
      let slug = comment._id;
      let fullSlug = `${formatDate(comment.created_at)}:${slug}`;
      if(typeof comment.parent_id == "string") {
        const parent = Comment.findOne(comment.parent_id);
        fullSlug = `${parent.full_slug}/${fullSlug}`;
        slug = `${parent.slug}/${slug}`;
      }
      e.currentTarget.user_id = Meteor.userId();
      e.currentTarget.slug = slug;
      e.currentTarget.full_slug = fullSlug;
    }

You're not providing user_id on topic.answer and your beforeInsert event is only setting the user_id field on the client. Validation is performed on both client and server, so you should also fill this field on the server. Validation error is not coming from the client but probably from the server.


不支持标准jsonschema的验证， 是否考虑引入ajv来支持

Help us to manage our issues by answering the following:

How would you tag this issue?

Enhancement

Describe your issue:

Which versions of UForm?
any version
.

目前您对标准的json-schema验证的需求主要是哪些属性？

maximum, minimum、maxLength, minLength, pattern, format、maxItems, minItems, uniqueItems 等等


Effect size

Dear Dorian
We were wondering if Lesymap provides also effect sizes for the SCCAN solutions? Or there a possibility to calculate them?
Thank you very much for your reply!
Kind regards
Wiebke

No, SCCAN does not produce any effect sizes, and I am not sure how they could be calculated. The effect size is probably useful because p values do not provide info on the strength of relationship. Maybe the out of sample cross validated correlation can provide a similar understanding of that strength.
Pinging @stnava if he has any inputs on SCCAN effect size.

Thank you for your reply!


Cannot use "Auto Format" enum in the AutoFormatType field property

Compiler complains when I try to use the newly(2019 wave 2) implemented "Auto Format" type in a field Property
I created an enum extension to "Auto Format" to add "CustomEnum" value. I then went to a table to add that enum value to the "AutoFormatType" and the compiler complains.
To Reproduce
The below code will throw a compiler error of 'Integer Literal Expected'.
AL Code to reproduce the issue
field(50000; "CustomFieldA"; Decimal)
        {
            AutoFormatType = "Auto Format"::CustomEnum;
        }

Expected behavior
I should be able to define an enum as a value of the newly implemented AutoFormatType, so that I can properly use the event for 'OnAfterResolveAutoFormat'
5. Versions:

AL Language: 4.0.182565

Business Central: In the new client there is no system information in the search

Hi,
the AutoFormatType property does not support adding the enum value. You'll have to add the enum ordinal value (i.e. the integer value) of the option.
br/Gert

If I have to hard define 50000 on the AutoFormatType field and then later have to change the enum value id to 50010, then they are no longer connected. Hard defining an integer for an enum seems counterproductive to the point of what Enum's are supposed to be.
Is there plans to allow for the enum expressions in the AutoFormatType? If not this is a suggestion for a future enhancement.

Hi,
yup, this is a bit where legacy meets new. Unfortunately the old properties don't know about enums. It is on the backlog to improve this but I do not know when we'll be able to do that.
br/Gert


Kneser-Ney Smoothing on Expected Counts: Alignment and Joint n-gram models

The topic of LM training came up again recently.
The aligner produces weighted alignment lattices.  There is some evidence that augmenting the Maximization step in the EM alignment process with the sort of expected-count KN smoothing described in this paper may/should improve the overall quality of the G2P aligner:

https://www3.nd.edu/~dchiang/papers/zhang-acl14.pdf

The same approach may be used to directly train the target joint n-gram model from the resulting alignment lattices.  I previously tried the latter using the WB fractional counts implementation in OpenGrm NgramLibrary, but it seemed to have little impact.  The Zhang paper notes a similar outcome and that EC-KN appears to be much more performant, even compared to the fractional KN implementation employed in Sequitur.
If I'm going to include some form of LM training after all, maybe this represents the most appropriate choice.  There is also reference implementation as a Ghiza add-on:

https://github.com/hznlp/giza-kn

@AdolfVonKleist thanks a lot for such wonderful library 👍 I am having following questions, please share your thoughts.

Can we pass Kenlm based ARPA file in phonetisaurus-train step ? I am not sure the current mit-lm is trained on what data and how is being used and same for RNNLM.

[Does usage same as defined in Ref 1 HMM where emission probabilities comes from alignment module and transition prob comes from LM trained on phonetic sequences passed in training as word pair <Grapheme /t Phoneme> ? but looks like you are trying to improve alignment module itself using KN smoothing]
Note : Got the overview of this work from these references :

https://www.aclweb.org/anthology/N07-1047.pdf (M2M EM -> HMM)
Improving WFST-based G2P Conversion with Alignment Constraints and
RNNLM N-best Rescoring

Thanks a lot !!

You should be able to use kenlm to perform the ARPA training directly.  Just use the command line utilities instead of the python wrappers.
$ phonetisaurus-align --input=cmudict.formatted.dict \
  --ofile=cmudict.formatted.corpus --seq1_del=false
# Train an n-gram model (5s-10s):
$ estimate-ngram -o 8 -t cmudict.formatted.corpus \
  -wl cmudict.o8.arpa
# Convert to OpenFst format (10s-20s):
$ phonetisaurus-arpa2wfst --lm=cmudict.o8.arpa --ofile=cmudict.o8.fst

just replace the estimate-ngram call with an equivalent kenlm command.  You'll need to output it to ARPA text format though, so that you can still transform it into a WFST for inference.
The mitlm call is trained on the output of the alignment - it just treats the aligned and segmented joint token sequences a 'normal' text corpus.

@AdolfVonKleist  Thanks for quick response 👍
Does RNNLM work in current master code. Is this used in same way as mit-lm ? or Is there any goodness achieved with RNNLM over mit-lm ?
And just to clarify, this is what you are saying about last line. [Steps from paper referenced above]

Thanks :)

Hi,
Yes it should work, however the rnnlm code has not been updated since that
earliest release, and is effectively the same as the original Mikolov code
from that time.  The only novel contribution there is the joint token
implementation of the decoder.
I did not find it to yield any significant improvement over mitlm as a pure
alternative, and the training time, as well as decoding time were
significantly slower.
The only place where it yielded a modest boost was when used in ensemble
with mitlm as described in the paper [but again there is a time penalty].
Whether or not that was/is sufficient reason to use the combined system in
a real-world or production setting, as opposed to just the normal joint
ngram models, would probably depend on how heavily you prioritize speed
versus absolute accuracy.
Best,
Joe
2020年4月6日(月) 0:43 smilenrhyme<EMAIL_ADDRESS>
@AdolfVonKleist https://github.com/AdolfVonKleist Thanks for quick
response 👍
Does RNNLM work in current master code. Is this used in same way as mit-lm
? or Is there any goodness achieved with RNNLM over mit-lm ?
And just to clarify, this is what you are saying about last line. [Steps
from paper referenced above]
[image: image]
https://user-images.githubusercontent.com/45142420/78534235-b5e4db00-7807-11ea-9463-60374d6df83e.png
Thanks :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/AdolfVonKleist/Phonetisaurus/issues/24#issuecomment-609629902,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABVUA5U5WV2D6P2Y7WTESLRLGBZVANCNFSM4DW3MMRA
.

Thanks a lot for detailed perspective 👍


list index out of range for multi-faces

It can work.
when It detect multi-faces, I sometime encounter a error "IndexError: list index out of range" as below
[log]
Corrupt JPEG data: 2 extraneous bytes before marker 0xd1
Detected_FaceNum: 1
face is inner of range!
Corrupt JPEG data: 3 extraneous bytes before marker 0xd6
Detected_FaceNum: 2
face is inner of range!
Traceback (most recent call last):
File "realtime_facenet_git.py", line 107, in
cropped[i] = facenet.flip(cropped[i], False)
IndexError: list index out of range

@rouyunpan I think this is an error in Opencv itself. Do these problems occur frequently?

if one of two face is closed to the boundary for a while,
"face0,  bb[0] = 0" --> skip
and then, when I touch Face1, I will encounter this issue.
[log]
...
Detected_FaceNum: 2
i=0, bb[1],[3],[0],[2]: 0 152 192 0 27
face is inner of range!
i=1, bb[1],[3],[0],[2]: 95 209 186 274
Traceback (most recent call last):
File "realtime_facenet_git.py", line 108, in
cropped[i] = facenet.flip(cropped[i], False)

hi
I found this bug.
if Face0 is out of range, you will skip face0 and doesn't be appended in the list "cropped" .
when skipping face0 and appending face1 , the index should be 0 in the list "cropped".
Then, you call facenet.flip() and access the list with index 1, you will encounter "IndexError: list index out of range"
inner exception
if bb[i][0] <= 0 or bb[i][1] <= 0 or bb[i][2] >= len(frame[0]) or bb[i][3] >= len(frame):
print('face is inner of range!')
continue
cropped.append(frame[bb[i][1]:bb[i][3], bb[i][0]:bb[i][2], :])
cropped[i] = facenet.flip(cropped[i], False)
BR

@rouyunpan
First, I acknowledge my mistake.
When debugging the point you pointed out in my code, I found that I had accessed the wrong idx.
I have accepted your suggestion and committed the code again, and I am very grateful for your contribution.

i add
 try :
 except IndexError :
                            print("Oops! IndexError : list index out of range for multi-faces")

in for loop
so it's work correctly.


Enable fetching the largest version instead of just the latest version

Sometimes - we have to go back to a previous major version and patch it. This becomes the latest 'tag'.  When executed in CI/CD this causes an issue since the calculated new version will likely exist in tags - making it incorrect and likely to fail the tagging process.
.e.g tags in date order are:
v6.8.0
v3.2.8  # historical version with a patch fix gets inserted at the end

then
git commit -m "[minor] stuff"

will attempt to tag v2.9.0 - which already exists in our tags.
Suggestion:
git-semver -sort-on version -target minor

properly calculates the tag
v6.9.0

This enables recent patches to continue as planned without affecting the ongoing tagging process. Since the tagging code is encapsulted in this module - there is no reasonable amount of complexity we can add to get around it.

Thanks for pointing this out. Saved a day, will search for alternative


Add the missing namespace_packages

+1

Coverage remained the same at 100.0% when pulling 4dec06d2be859be4e3c3d6cc13f1c0c690e9d7d3 on namespace_packages into 4a3d6051b56d88aa3b652047a6e780e32b9e60d8 on master.

Coverage remained the same at 100.0% when pulling 4dec06d2be859be4e3c3d6cc13f1c0c690e9d7d3 on namespace_packages into 4a3d6051b56d88aa3b652047a6e780e32b9e60d8 on master.


Cheers!

:beers:


Nuspec file change for NetCore Platform Service

@jayaranigarg,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

LGTM.


Upgrade Jetty to 9.4.0.v20161208

Hopefully, it's just a version bump.
Resolves #1874.

@arteam It's failing on Travis CI. That's why I didn't create a PR from https://github.com/dropwizard/dropwizard/tree/jetty-9.4.0. 😉

Sorry, didn't see that branch. The build in fact fails on Travis CI. Will try then to run it several times there to figure out a reason of failing. Unfortunately, the build passes locally, so we can't debug it in a simple way.

Heads up, I ran this branch on appveyor (windows ci) and it passed
By the way, Dropwizard has an appveyor.yml in the repo if one of the owners want to allow permissions, so it'll build all commits 😋

I've set up an integration with AppVeyour here, so now it should build all pull requests.

I've set up the build on another CI sever CircleCI. Let see whether the build fails there or not.

Looks like the build fails on Circle CI at the same test as on Travis CI.
The test suite just hangs up (I guess because the relevant test doesn't
have timeouts).

Would closing these - https://github.com/dropwizard/dropwizard/blob/master/dropwizard-testing/src/test/java/io/dropwizard/testing/DropwizardTestSupportTest.java#L154 help at all?

This might be dumb, but what if we changed all the localhost's to <IP_ADDRESS>? That would remove any reliance on the DNS resolver being set up properly (as localhost is really just an /etc/resolv.conf) thing.

It looks like Jetty's session handling was a major change in 9.4 - http://www.eclipse.org/jetty/documentation/9.4.0.v20161208/upgrading-jetty.html#_upgrading_from_jetty_9_3_x_to_jetty_9_4_0
Would that impact anything?

I believe the issue lies somewhere in handling administrative tasks, but still can figure it out. For instances, many tests in dropwizard-testing took ~30 seconds, because the thread pool for admin tasks doesn't shut down gracefully. Another possibility is that Jetty 9.4.* somehow creates too many TCP sockets that are not closed correctly, so we hit limit on Linux CI boxes.

Looks like I boiled down the problem to DefaultServerFactory and admin thread pools. Will try to poke around there.

@arteam this may be unrelated, but have you ever seen the DefaultServerFactory register the handlers twice:
INFO  [2017-01-16 18:18:33,623] org.eclipse.jetty.util.log: Logging initialized @1206ms
INFO  [2017-01-16 18:18:33,682] io.dropwizard.server.DefaultServerFactory: Registering jersey handler with root path prefix: /
INFO  [2017-01-16 18:18:33,682] io.dropwizard.server.DefaultServerFactory: Registering admin handler with root path prefix: /
INFO  [2017-01-16 18:18:34,371] com.smoketurner.uploader.handler.UploadInitializer: SSL Provider: OpenSSL
INFO  [2017-01-16 18:18:34,388] com.smoketurner.uploader.handler.UploadInitializer: SSL ENABLED (certificate: 'scripts/server.crt', key: 'scripts/server_pkcs8.key', trust store: 'scripts/ca.crt')
INFO  [2017-01-16 18:18:34,392] com.smoketurner.uploader.handler.AccessControlListFilter: Loaded 1 IP filter(s) (1 accept, 0 reject)
INFO  [2017-01-16 18:18:34,394] com.smoketurner.uploader.config.NettyConfiguration: Event Loop: NIO
INFO  [2017-01-16 18:18:39,487] io.dropwizard.server.DefaultServerFactory: Registering jersey handler with root path prefix: /
INFO  [2017-01-16 18:18:39,487] io.dropwizard.server.DefaultServerFactory: Registering admin handler with root path prefix: /
INFO  [2017-01-16 18:18:39,494] io.dropwizard.server.ServerFactory: Starting uploader

That's from a service I've been messing around with which includes a managed-Netty instance, but I don't know why the "Registering XXX handler with root path prefix" messages are showing up twice.

Hm, I have not. Thanks for the pointer, though.

Coverage decreased (-0.005%) to 81.856% when pulling e9a65db6b72ed1a1913b6f61a5bfbc4ddaacbbd6 on upgrade_jetty_94 into c4de62f8add7a752d773427c49a8de76166fd54d on master.

Looks like I found the culprit. I believe, the issue is reported in #523. The OS on CI Linux machines returns astonishingly big amount of CPUs (128). Jetty changed their algorithm for calculating the maximum amount of selector and acceptor threads, see https://github.com/eclipse/jetty.project/commit/2d52280179a7d9f8b2449c49bcb0847e02115082. But, in Dropwizard we still set the amount of acceptors as #CPUs/2 and selectors #CPU. Looks like that's too much for Jetty, and it can't handle such a big amount of threads. I propose not to guess the right amount of threads and delegate it to Jetty.

Coverage increased (+0.005%) to 81.866% when pulling 534819c570705264f691ca19d5ae9ea1cbfcc3b7 on upgrade_jetty_94 into c4de62f8add7a752d773427c49a8de76166fd54d on master.

Thanks for all your hard work @arteam 👍 I think this is good to merge now.

With the optional change, users can still provide overrides in a yaml file and they'll be picked up correct?  I've internally stored things in an Optional, but I haven't used the setters with @JsonProperty before to know whether that works or not?

Yes, they can. The HttpConnectorFactoryTest#testParseFullConfiguration test loads a custom YAML config and verifies that options are applied correctly. I verified it on a sample application and properties were picked up correctly. I think even the @Min annotation worked.

Yay! @jplock, thank you for merging these changes and your suggestions fox fixing the issue.


Localisation file for template linking and transclusion check.

Please add the localisation file for template linking and transclusion check:
Tool: https://tools.wmflabs.org/templatetransclusioncheck/
Source: https://github.com/ChameleonWiki/templateCheck

The translation pipeline lock has been lifted. This can be processed now. Please update this pull request to use the new JSON format.


Spinner was not put right place

What problem does this feature solve?
Spinner is lie on the right of the screen on IE 11
What does the proposed changes look like?
change css in .spinner look like:
.spinner {
/margin: 100px auto;/
width: 50px;
height: 40px;
text-align: center;
font-size: 10px;
margin: auto auto;
position: absolute;
}

This feature request is available on Nuxt community (#c7523)

@TranChiNam Why don't you submit a PR for that? :relaxed:


API-Management Private Link

Hello,
As far as I know, API-Management private link is not yet GA - it's in Public Preview and works only on front-side. Additionally, the link next to API-Management entry leads to event grid page, not APIMs.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 97849a4b-e79a-840e-8cfc-d43b0ea1dec3
Version Independent ID: 6d224a2c-5e53-832a-ecd4-ae30fd472b76
Content: Azure Private Link availability
Content Source: articles/private-link/availability.md
Service: private-link
GitHub Login: @asudbring
Microsoft Alias: allensu

@miqm , thank you for your feedback. We'll review this and get back to you shortly!

@miqm , thank you for bringing this to our notice. We've updated the API-Management private link availability to Preview and the changes will go live soon.
We are closing this issue for now. If there are further questions regarding this matter, please reply and we will gladly continue the discussion.


Fix for Incorrect content size after keyboard closed - issue #4193

It doesn't reset scroll view when tapping on a checkbox/radio after focusing a text field, for example.
There are two problems:
1 - It shouldn't clear the timeout if the target element is not a keyboard element (line 307)
2 - As 'keyboardActiveElement' is being set to null (line 314), resetScrollView event will never be fired (line 552). It doesn't need to be set to null at this point as it will be set to null after keyboardHide executes (line 576)

this fixes the bug for me, i need this NOW :)
how can i monkey patch this into my current app? any chance?

Tracking for 1.2. Thanks

Seeing the same issue and this PR does indeed fix it. +1.


Generate functions in external editors

Godot version:

3.0.4.stable.custom_build

OS/device including version:

Linux desktop-arch 4.17.2-1-ARCH #1 SMP PREEMPT Sat Jun 16 11:08:59 UTC 2018 x86_64 GNU/Linux
Feature Request:
It would be nice if the "Create Function" option from the signal connection dialog worked with external editors.
I use neovim, and when I wire a new connect with "Create Function" enabled, it opens the script with neovim but there is no new function definition.
One flexible solution I could think of is adding a {generated_text} external editor param.
If there are concerns about passing multiline text as a cli flag, then {generated_text} could be the path to a temp file containing the generated function, or the text could be passed on STDIN.
Another is to append and save the generated text to the file before opening the editor. This could be optional as it is not the current behavior.

Duplicate of https://github.com/godotengine/godot/issues/41283.


VESTA plots

We have discussed the possibility of interfacing with VESTA to plot realistic atomic structures.
Here we can brainstorm about this possibility and what plots could be produced.

What does it mean, realistic atomic structures?

Probably realistic atomic structures is not the correct definition, but I was thinking of something like this:

Not sure if this is feasible and useful, though.

We sort of already have that. We can store the structure with the density. The density can basically be anything (electron density, free energy, probability, etc).
https://gemdat.readthedocs.io/en/latest/api/gemdat_volume/#gemdat.volume.Volume.to_vasp_volume
Did you have something else in mind?

Do you still want to pick this up @SCiarella ?

I think we can close this issue and use the plots already implemented in GEMDAT.


A factor is not changed.

| Camera Model                       | { D400 } |
| Firmware Version                   | <IP_ADDRESS> |
| Operating System & Version |   Win 10  |
| Platform                                 | PC |
| SDK Version                            |  2.24, 2.25  |
| Language                            |  C++  |
| Segment			|  others  |
Hello, I saw this white paper.
(https://dev.intelrealsense.com/docs/white-paper-subpixel-linearity-improvement-for-intel-realsense-depth-cameras)
I tried to change A factor using C++ language or .json file or Realsense View.
But It's not changed. How can I change this A factor value?

@Mustardsauce , A-factor functionality requires FW 5.11.9+.
I believe a compatible version will be released soon though I don't have a concrete info. @RealSense-Customer-Engineering , can you comment ?

@Mustardsauce  The new Development Firmware <IP_ADDRESS> is available for download from the firmware releases page.  Among its updates are improved depth linearity.  Details of the additions in this firmware can be found at the link below.
https://support.intelrealsense.com/hc/en-us/community/posts/360034784453

Thank you :)

Hope it works well now. Please feel free to open a new issue if it does not


Add branches to protect against destroyed instance

Destroyed Scrollwatch instances result in a undefined error being thrown. This adds a guard to protect against that.

Thanks for reporting this issue.
Closing this issue for a few reasons.

Code standards and formatting do not match existing code base. In lieu of a formal style guide, please maintain the existing coding standards.
Would prefer a cleaner commit history (git rebase)
Not sure why the checks in commit 117274e are necessary. If possible, could you elaborate on how you could get into a scenario where a check at this point is needed? Also, why would you need multiple checks? Wouldn't one at the top of the method take care of any issues?

Note: I'm currently in the process of releasing v2.0 and have rolled the fix from commit 70997ee into the codebase with credit going to you.


giraffe born cool giraffe

click left mouse to born new giraffe

Merging this into the main branch is not in the plan.
We are going to consider where your contribution (usage example?) should go.

closing for now


hasImages parameter acts odd due to Python having upper-case Booleans`

I was making some request calls to the 'search' endpoint to get some information about paintings, and noticed some very odd behavior regarding the hasImages field. When encoding data via the requests library, the upper case 'T' in 'True' causes issues with the case sensitive nature of the hasImages field. You can see here that there is quite a large difference between manually encoding data and letting the library encode that data for you
Manual Encoding (12111 results):
object_ids_large = session.get(BASE_URL+"/search?hasImages=true&medium=Paintings&q=%22%22").json()

Requests Library Encoding (228 results):
object_ids_small = session.get(BASE_URL+"/search", params={ "hasImages" : True, "medium" : "Paintings","q":""}).json()

I had tried using a string in my params (passing "true"), but that returned 0 results, so however this boolean is interpreted on the backend query behaves strange across languages.

@HStep20 It is because in the first query the param q is set to %22%22 as a string while in the second the q is set to an empty string and thus will be encoded to `` (nothing). The resultant urls are

https://collectionapi.metmuseum.org/public/collection/v1/search?hasImages=true&medium=Paintings&q=""
https://collectionapi.metmuseum.org/public/collection/v1/search?hasImages=True&medium=Paintings&q=


Fix initial database connection issue

If the issuer kit database (mongo) is unavailable for a period of time when the issuer kit api attempts to connect (https://github.com/bcgov/issuer-kit/blob/main/api/src/app.ts#L38) a connection error can be thrown by the MongoClient driver (https://github.com/bcgov/issuer-kit/blob/main/api/src/mongodb.ts), resulting it the following log message:
error: Unhandled Rejection at: Promise  {"name":"MongoServerSelectionError","reason":{"type":"Single","setName":null,"maxSetVersion":null,"maxElectionId":null,"servers":{},"stale":false,"compatible":true,"compatibilityError":null,"logicalSessionTimeoutMinutes":null,"heartbeatFrequencyMS":10000,"localThresholdMS":15,"commonWireVersion":null}}

At this stage the driver has failed to make and initial connection to the database and will never attempt to connect with the database again.  This renders the api completely inoperable.  This manifests as an inability to issue credentials to a wallet, the initial connections are made but the credential issuing flow appears to hang on the wallet and issuer-web side due to this error.  In contrast when the api is able to make the initial connection the database can become unavailable and available again and the driver is able to reconnect.  It is only the initial connection that appears critical.
This issue occurs in our OpenShift environments during rollouts and evacuations due to the fact the API startup process is much faster than the database's startup process.  Therefore in the majority of cases the API has started while the database is unavailable.
There are a couple ways we can deal with this:

Have the api shutdown when it is unable to connect with the database (logging the reason for shutdown, the fact it could not connect).  In addition the serverSelectionTimeoutMS parameter (which defaults to 30 seconds) could be made configurable so it can be adjusted to give the connection attempt a bit more time.  In OpenShift this will cause a new pod to be started which will try to connect to the database.
Come up with a retry process.  Where the api would attempt the initial connection a number of times before finally failing.  This looks like it becomes complicated quickly and may have other side effects.

Therefore it looks like the most appropriate way to handle this issue is using the first option.

@wadeking98, The changes in PR https://github.com/bcgov/issuer-kit/pull/351 don't seem to be working as expected.  The initial server selection timeout does not appear to be configurable.  I can set the SERVER_SELECTION_TIMEOUT environment variable to 300 seconds (5 minutes), and the initial database connection will still timeout after 30 seconds when the database is not available.
Error message is:
Database connection failed MongoServerSelectionError: connect ECONNREFUSED <IP_ADDRESS>:27017
    at Timeout._onTimeout (/opt/app-root/src/node_modules/mongodb/lib/core/sdam/topology.js:438:30)
    at listOnTimeout (internal/timers.js:554:17)
    at processTimers (internal/timers.js:497:7) {
  reason: TopologyDescription {
    type: 'Single',
    setName: null,
    maxSetVersion: null,
    maxElectionId: null,
    servers: Map { 'issuer-kit-db:27017' => [ServerDescription] },
    stale: false,
    compatible: true,
    compatibilityError: null,
    logicalSessionTimeoutMinutes: null,
    heartbeatFrequencyMS: 10000,
    localThresholdMS: 15,
    commonWireVersion: null
  }
}
npm info lifecycle<EMAIL_ADDRESS>Failed to exec start script
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>start: `node lib/`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm timing npm Completed in 34098ms

npm ERR! A complete log of this run can be found in:
npm ERR!     /opt/app-root/src/.npm/_logs/2022-03-15T14_29_04_436Z-debug.log

@wadeking98, @esune suggestion is a good one, it would resolve other issues, like the pod being made available before the database connection is actually established, and would provide k8s more control over the app in general.  The current solution helps, but the pod will still appear to be running before a database connection is available.
On startup the /health endpoint would return a 503 error until the initial database connection is establish.  Once established the /health endpoint would return a 200.
This signaling would replace the app shutdown on initial database connection failure.
An easy feature for you to add?

@WadeBarnes did you still want this feature now that we've fixed the timeout issue?

Yes please


ValueError: Source image data is not complete, please check if one of the image folder is not exist

when i was run  "python tools/make_tusimple_tfrecords.py ",I meet ValueError: Source image data is not complete, please check if one of the image folder is not exist
np_resource = np.dtype([("resource", np.ubyte, 1)])
Traceback (most recent call last):
File "tools/make_tusimple_tfrecords.py", line 32, in
generate_tfrecords()
File "tools/make_tusimple_tfrecords.py", line 22, in generate_tfrecords
producer = lanenet_data_feed_pipline.LaneNetDataProducer()
File "/data-input/lanenet-lane-detection/data_provider/lanenet_data_feed_pipline.py", line 48, in init
raise ValueError('Source image data is not complete, '
ValueError: Source image data is not complete, please check if one of the image folder is not exist

@xukaiyan 按照https://github.com/MaybeShewill-CV/lanenet-lane-detection/tree/master/data/training_data_example 文件结构重新组织你的训练集数据.

解决了，谢谢

请问一下是怎么解决的，我刚刚下载项目，现在不知道怎么操作


sched/taskfiles: skip unnecessary file open/close operations to improve performance

Summary
sched/taskfiles: skip unnecessary file open/close operations to improve performance
The task files should consult the "spawn action" and "O_CLOEXEC flags"
to determine further whether the file should be duplicated.
This PR will further optimize file list duplicating to avoid the performance
regression caused by additional file operations.
Signed-off-by: chao an<EMAIL_ADDRESS>Impact
N/A
Testing
ci-check

Ready to review now

This should be fine from the kernel mode point of view now. I'm just wondering will we lose the benefit from your optimizations because of the copying ?

This should be fine from the kernel mode point of view now. I'm just wondering will we lose the benefit from your optimizations because of the copying ?

This may depend on the application implement, In the previous implementation, all file descriptors would be dup in files_duplist and closed after nxtask_init in files_close_onexec.
exec_module
|
|->nxtask_init
|  |
|   ->group_setuptaskfiles
|     |
|      ->files_duplist    /* all files will be duplicate from parent */
|
|->spawn_file_actions /* do file actions */
 ->files_close_onexec /* close exec files */

In the current implementation,

if the file actions has ACTION_CLOSE/OPEN operation, no needs to dup, and also no needs to close;
if the file descriptor has the O_CLOEXEC flag, it will not perform additional dup/close operations.

exec_module
|
 ->nxtask_init
   |
    ->group_setuptaskfiles
      |
       ->files_duplist    /* pass the file actions to avoid  */
         |
         |->spawn_file_is_duplicateable /* 1. SPAWN_FILE_ACTION_CLOSE: will be close, do not dup  (*)
         |                               * 2. SPAWN_FILE_ACTION_OPEN:  will be open, no needs to dup (*)
         |                               * 3. SPAWN_FILE_ACTION_DUP2:  could be dup
         |                               */
          ->(filep->f_oflags & O_CLOEXEC) /* skip dup */

Of course, this improvement is obvious in flat build, but if it is in kernel mode, it will depend on the ratio of action/cloexec in the file list.
exec_module
|
|->binfmt_copyactions
|
|->nxtask_init
|  |
|   ->group_setuptaskfiles
|     |
|      ->files_duplist    /* pass the file actions to avoid  */
|        |
|        |->spawn_file_is_duplicateable /* 1. SPAWN_FILE_ACTION_CLOSE: will be close, do not dup  (*)
|        |                               * 2. SPAWN_FILE_ACTION_OPEN:  will be open, no needs to dup (*)
|        |                               * 3. SPAWN_FILE_ACTION_DUP2:  could be dup
|        |                               */
|         ->(filep->f_oflags & O_CLOEXEC) /* skip dup */
|
 ->binfmt_freeactions

In addition, not re-dup the file in nxtask_init could also avoid potential problems caused by some files/drivers that do not support multiple instances. If the driver sets the O_CLOEXEC flag after open, but the driver will be dup multiple times after exec, Then it is also unfriendly to the driver developer.

This should be fine from the kernel mode point of view now. I'm just wondering will we lose the benefit from your optimizations because of the copying ?

This may depend on the application implement, In the previous implementation, all file descriptors would be dup in files_duplist and closed after nxtask_init in files_close_onexec.
exec_module
|
|->nxtask_init
|  |
|   ->group_setuptaskfiles
|     |
|      ->files_duplist    /* all files will be duplicate from parent */
|
|->spawn_file_actions /* do file actions */
 ->files_close_onexec /* close exec files */

In the current implementation,
1. if the file actions has ACTION_CLOSE/OPEN operation, no needs to dup, and also no needs to close;

2. if the file descriptor has the O_CLOEXEC flag, it will not perform additional dup/close operations.

exec_module
|
 ->nxtask_init
   |
    ->group_setuptaskfiles
      |
       ->files_duplist    /* pass the file actions to avoid  */
         |
         |->spawn_file_is_duplicateable /* 1. SPAWN_FILE_ACTION_CLOSE: will be close, do not dup  (*)
         |                               * 2. SPAWN_FILE_ACTION_OPEN:  will be open, no needs to dup (*)
         |                               * 3. SPAWN_FILE_ACTION_DUP2:  could be dup
         |                               */
          ->(filep->f_oflags & O_CLOEXEC) /* skip dup */

Of course, this improvement is obvious in flat build, but if it is in kernel mode, it will depend on the ratio of action/cloexec in the file list.
exec_module
|
|->binfmt_copyactions
|
|->nxtask_init
|  |
|   ->group_setuptaskfiles
|     |
|      ->files_duplist    /* pass the file actions to avoid  */
|        |
|        |->spawn_file_is_duplicateable /* 1. SPAWN_FILE_ACTION_CLOSE: will be close, do not dup  (*)
|        |                               * 2. SPAWN_FILE_ACTION_OPEN:  will be open, no needs to dup (*)
|        |                               * 3. SPAWN_FILE_ACTION_DUP2:  could be dup
|        |                               */
|         ->(filep->f_oflags & O_CLOEXEC) /* skip dup */
|
 ->binfmt_freeactions

In addition, not re-dup the file in nxtask_init could also avoid potential problems caused by some files/drivers that do not support multiple instances. If the driver sets the O_CLOEXEC flag after open, but the driver will be dup multiple times after exec, Then it is also unfriendly to the driver developer.

Thanks for this very thorough explanation!


Incorrect handling of EntityRef/Mut Queries + Resources resulting in false conflicts

Bevy version
0.13
What you did
Have a system with EntityRef & ResMut<T>.
What went wrong
System should not have any access conflicts but it does.
Additional information
Minimal example to reproduce:
use bevy::prelude::*;

#[derive(Resource)]
struct Foo;

fn sys(_: Query<EntityRef>, _: ResMut<Foo>) {}

fn main() {
    App::new()
        .insert_resource(Foo)
        .add_systems(Update, sys)
        .run();
}

thread 'main' panicked at /home/yogii/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bevy_ecs-0.13.2/src/system/system_param.rs:542:13:
error[B0002]: ResMut<bevy_bug::Foo> in system bevy_bug::sys conflicts with a previous Res<bevy_bug::Foo> access. Consider removing the duplicate access.
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Encountered a panic in system `bevy_app::main_schedule::Main::run_main`!

Yeah, I don't think our access model is quite right. We have reads_all and writes_all, but that's inadequately granular: components and resources are distinct.

Probably obvious, but I'd like to point out that the opposite EntityMut and Res will also return the same error.
use bevy::prelude::*;

#[derive(Resource)]
struct Foo;

fn sys(_: Query<EntityMut>, _: Res<Foo>) {}

fn main() {
    App::new()
        .insert_resource(Foo)
        .add_systems(Update, sys)
        .run();
}

I've encountered this too, is there any progress here?

World::resource_scope is a convenient work-around until the accesses for EntityRef/EntityMut are fixed. Though it does feel quite boilerplate-y:
fn sys(world: &mut World, state: &mut SystemState<Query<EntityRef>>) {
    world.resource_scope(|world, mut foo: Mut<Foo>| {
        let _query = state.get(world);
    });
}

Just hit this as well, I have to switch to Res and queue my resource update with Commands


Use dnsmasq with connman

This change is

@petrosagg @agherzan @telphan

SGTM

Approved - merge when jenkins says :+1:


Recent bookmarks - auto update UI and update feedback

Expected Behavior
Please describe the behavior you are expecting

When I click on the bookmark icon the "recent bookmarks" should auto add or remove the bookmark without having to refresh the page.

When adding or removing a bookmarked article if the action is unsuccessful there should be a toast notification with simple actions to resolve.

Current Behavior
What is the current behavior?
When I click on the bookmark icon in an article, the icon will change colour (provides feedback),
but I need to manually refresh the page to see the updated bookmarks,

Is this issue up-for-grab. Could you assign this to me @John-Paul-Larkin @Shane-Donlon

It certainly is @revanthnetha.
I assigned you.

@revanthnetha
I just looked at the SideBarSavedPosts function which displays the recent bookmarks.
As part of this PR can you handle the included

Could you check the PR @John-Paul-Larkin . Happy to do Changes if needed :)


Fix LetExpr node not formatting correctly

Purpose
Fix issue in ballerina formatter.
Fixes #21904
Check List

[x] Read the Contributing Guide
[ ] Updated Change Log
[ ] Checked Tooling Support (#)
[ ] Added necessary tests

[ ] Unit Tests
[ ] Spec Conformance Tests
[ ] Integration Tests
[ ] Ballerina By Example Tests

[ ] Increased Test Coverage
[ ] Added necessary documentation

[ ] API documentation
[ ] Module documentation in Module.md files
[ ] Ballerina By Examples

Codecov Report

Merging #22466 into ballerina-1.2.x will not change coverage by %.
The diff coverage is n/a.

@@               Coverage Diff                @@
##           ballerina-1.2.x   #22466   +/-   ##
================================================
  Coverage            14.59%   14.59%
================================================
  Files                   51       51
  Lines                 1411     1411
  Branches               219      219
================================================
  Hits                   206      206
  Misses                1189     1189
  Partials                16       16

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1b7a1d0...64255d4. Read the comment docs.


Newsletters: add 228 (2022-11-30)

[x] Add lede, update releases/RCs, add topic links @harding
[x] LND 7122 @Xekyo (sorry, no Bitcoin Core PRs this week, but at least this is PSBT related)

Hey, I think I may have added a news item to the topic page for the first time. Perhaps glance at that slightly more thoroughly, whether I did that right.

Made edits, including dropping Bitcoin Core 24.0 (thanks for feedback @glozow, @fanquake, @vostrnad, @LarryRuane , @Xekyo, and @jirijakes!), reviewed @Xekyo's contribution (including topic entry, thanks!), checked for updated RCs, and added topic links.  Thanks everyone!

Thank you @harding and @Xekyo for authoring this week and to @LarryRuane @fanquake @glozow @jirijakes @vostrnad for the 🔍 👀


Incorrect results when loading OpenBLAS with OpenMP prior to pytorch on POWER9

For context: We have a POWER9 cluster with big GPUs in our HPC system and are compiling everything from source via EasyBuild. One of such applications is PyTorch which used to show faulty behavior until a recent change that I tracked down to not importing numpy prior to torch anymore. But it reappears when doing that manually
In short:

Loading openblas prior to import torch leads to miscalculations. Load can either happen via import numpy which depends on OpenBLAS or via ctypes.CDLL, which basically dlopens the library
building OpenBLAS without OpenMP support (as done by e.g. conda) makes the problem disappear, which leads me to thinking that this is a bug in OpenBLAS rather than pytorch
The same setup (same OpenBLAS version, compilation settings, ...) works on x86

The test case I'm using is from e.g. https://github.com/xianyi/OpenBLAS/issues/1191#issuecomment-460336212 / https://github.com/pytorch/pytorch/issues/3716#issuecomment-387776917 hence ccing @hartb
#!/usr/bin/env python

import sys
import os
import ctypes
import subprocess

if os.getenv('IMPORT_NUMPY'):
      import numpy as _np
else:
  ctypes.CDLL('/sw/installed/OpenBLAS/0.3.7-GCC-8.3.0/lib/libopenblas.so.0', mode=ctypes.RTLD_LOCAL)

import torch

isOK = True

for sz in [ 127, 128, 129, 130 ]:
        diffs = []
        print("Size = {}".format(sz))
        for i in range(10):
                a = torch.randn(sz, sz)
                q, r = torch.qr(a)
                a_qr = torch.mm(q, r)
                m = float(max(max(x) for x in a - a_qr))
                diffs.append("max diff = {0:.6f}{1}".format(m, " FAIL!" if m > 0.001 else ""))
                if m > 0.001:
                  isOK = False
        print(", ".join(diffs))

if not isOK:
  sys.exit(1)

I see failures already at the 127 sizes consistently. As mentioned compiling OpenBLAS without OpenMP makes this pass, not importing OpenBLAS (either via the ctypes.CDLL or via numpy) prior to torch also works and setting OMP_NUM_THREADS=1 works too, reducing to e.g. 2 or 4 makes the failures less frequent. In that cases they appear more frequently at higher sizes.

Which OpenBLAS version are you using (the path in the CDLL line suggests 0.3.7, current is 0.3.10) ?

I tried using 0.3.10 by compiling it as usual and adapting the LD_LIBRARY_PATH and double checked it gets loaded but the issue persists. However I could confirm that with 0.3.10 DYNAMIC_ARCH=1 was working again (regular tests passed) and there were less failures in the extended lapack-test (but there still some)

There being some lapack-test failures still is largely a function of the compiler used, and I hope the situation is already better with current develop. The original issue looks like the OpenMP build is erroneously relying on OpenMP to prevent a race condition where the plain threads code employs locks. (Which is a bit weird as the "official" stance used to be that compiling with OpenMP is absolutely mandatory for OpenBLAS on POWER)

If I understand https://github.com/xianyi/OpenBLAS/blob/4d36711547e71dbebaded85f9ddd92f9dfbe887a/Makefile.power#L2-L10 correctly then there is only singlethread or OpenMP on POWER. So no "plain thread code"?
I tried with develop, still failing e.g. Nonsymmetric-Eigenvalue-Problem-EIG/xeigtsts, RFP-linear-equation-routines-LIN/xlintstrfs
For reference I also tried using the numpy functions instead of torch and they exhibit the same faulty behavior, so I assume both numpy and pytorch map straight to OpenBLAS.
I'm really puzzled about this only happening when OpenBLAS is loaded before torch

Reviewing the pytorch issue ticket this appears to be a very old bug, pity that it never surfaced here apparently (not that i need it now though...)
A reproducer without torch would be helpful I think, not sure who packages pytorch for ppc64le. Or does the bug require both OpenBLAS and torch to be present ? Multithreaded OpenBLAS allocates some buffer space
for communicating partial results between threads, some of that gets put on the stack but I do not think we are
actively overwriting anybody there.

I failed to reproduce without torch. As mentioned the mystery to me is that loading OpenBLAS before torch triggers the bug but loading it afterwards (no-op as it is loaded by torch) does not trigger the bug. Easiest to test by importing numpy before or not. And even for torch: I manually loaded all libraries that torch loads until only torch libs remained (checked using strace), then loading OpenBLAS, then torch and it still is torch, not any other lib.
Currently got another lead where some minor difference in the make command leads to 0.3.10 passing the issue, trying to isolate and reproduce

I'm afraid I've moved on to other responsibilities since updating that PyTorch issue, but some recollections from when I was looking at it....

We were looking mostly at OpenBLAS 0.2.20, and things seemed improved by the 0.3.3 release.

We didn't see failures with input sizes <= 128x128. I used ftrace to try to see what was happening, and could see that (at least) one of the OpenBLAS routines switched to a different strategy when the input size grew to 129x129. It would use a single-threaded solution for the smaller sizes, and decompose into a parallel solution for the larger sizes. Perhaps the switch-point was based on the vector size provided by the platform. I didn't find any OpenBLAS QR unit tests with input sizes approaching anything like the switch-over point size (maybe the largest dimension of test input sizes was 4-5, or maybe around 20? I can't remember, but much smaller than 129x129 from what I could find).

I tried recreating outside pytorch but wasn't able. I'll attach a test program (qrt.c.gz) I was trying with. A few caveats:

I'm not really a linear algebra or BLAS guy, so it's kind of cobbled together
It's based on the old PyTorch qr code  (https://github.com/pytorch/pytorch/blob/v1.1.0/aten/src/TH/generic/THTensorLapack.cpp). But it's limited to square inputs because: a) torch would also fail with square inputs, and b) it avoids the resizing and narrowing
The torch qr implementation changed quite a bit in PyTorch 1.2.0: https://github.com/pytorch/pytorch/pull/20689

A couple other thoughts,  probably unrelated, but for completeness:

We ran into hangs due to OpenMP non-thread-safety when mixing: OpenMP-enabled PyTorch, PyTorch multiprocessing, and OpenMP-enabled OpenBLAS. See: https://stackoverflow.com/questions/49049388/understanding-openmp-shortcomings-regarding-fork This is at least in part why Anaconda builds their OpenBLAS explicitly without OpenMP support on Linux (https://github.com/AnacondaRecipes/openblas-feedstock/blob/master/recipe/build.sh#L31)

I see that Anaconda also disables TLS in their build (https://github.com/AnacondaRecipes/openblas-feedstock/blob/master/recipe/0003-turn-TLS-off.patch). Looks like that might just be so that they can run  against an older GLIBC.

@hartb Thanks for the notes, I'll add my findings:

I'm seeing this issue with 0.3.7 and 0.3.10
failures occur at 127x127 although the bigger the size, the more often it fails. So that might be a slightly different issue
I'm using pytorch 1.6 however as mentioned even numpy breaks when it is loaded before pytorch (i.e. loading numpy+pytorch -> break, loading pytorch+numpy works). I also could not recreate it without loading pytorch
the fork might actually be an issue. Using gdb I found PyTorch executing a fork through subprocess but that only runs uname, so maybe not the issue. Maybe loading openBLAS library already runs some OpenMP stuff that will then violate the fork conditions while pytorch forks before loading openblas? Just a wild guess, not sure how to verify. Disabling OpenMP on the cluster-wide installed OpenBLAS would likely be an issue as it means slowing down numpy and other "safe" usages
That TLS patch still exists but is not used. That was a bug in OpenBLAS which enabled TLS by default which is no longer the case: https://github.com/AnacondaRecipes/openblas-feedstock/blob/4841a03388955fccb667588f5bdd132822b0b2f5/recipe/meta.yaml#L18 So we can exclude that as the cause

I experimented a bit and found that setting NUM_THREADS to the number of CPU cores (in my case 176) leads to the breakage while setting it to any lower value (e.g. 175) works. Maybe some off-by-one error?
Oh, my... Just tested the fork theory and it really is able to trigger the issue:
#!/usr/bin/env python

import sys
import os
import ctypes
import subprocess

import numpy as np
import platform
if os.getenv('RUN_FORK') and platform.system():
  pass

isOK = True

for sz in [ 127, 128, 129, 180 ]:
        diffs = []
        print("Size = {}".format(sz))
        for i in range(10):
            a = np.random.randn(sz, sz)
            q, r = np.linalg.qr(a)
            a_qr = np.dot(q, r)
            m = float(max(max(x) for x in a - a_qr))
            diffs.append("max diff = {0:.6f}{1}".format(m, " FAIL!" if m > 0.001 else ""))
            if m > 0.001:
              isOK = False
        print(", ".join(diffs))

if not isOK:
  sys.exit(1)

When running this with RUN_FORK set it fails O.o

IIRC (and probably unhelpful) the fork problems are a feature of the GNU libgomp, not shared by the llvm libomp(?)
OTOH I do not get the connection to the number of cpus, so perhaps there are two bugs ?

FWIW I tried to build 0.3.10 with clang and it failed due to some power flags not known to clang...
Yeah, no idea about the NUM_THREADS issue or why it solves this. Maybe some buffer gets corrupted that only shows when all spaces are used?

Ouch... It seems the issue is fixed by https://github.com/xianyi/OpenBLAS/pull/2765

Ah; good find!

Sorry, my brain is mush from too many competing issues (not all OpenBLAS-related) right now, I knew (felt rather?) there was something possibly relevant in recent fixes but had not gotten round to refreshing my memory.

I can confirm, that this is indeed the issue but am not sure why it depends on forks and NUM_THREADS.
I build OpenBLAS with -DDEBUG=1 and used this Python code to check for sync problems leading to buffer double-use:
import re

inUse = [False]*300

with open('/home/alex/np3.log') as f:
    for line in f:
        m = re.search(r'Position -> (\d+)', line)
        if m:
            idx = int(m.group(1))
            if inUse[idx]:
                print('Double use of %s' % idx)
            inUse[idx] = True
        m = re.search(r'Position : (\d+)', line)
        if m:
            idx = int(m.group(1))
            if not inUse[idx]:
                print('Double free of %s' % idx)
            inUse[idx] = False

On the system with 176 cores/threads and at 0.3.10 without the patch I get many messages, after applying the patch I get none. Without the patch and NUM_THREADS=174 I also get none and with NUM_THREADS=175 I get a single Double use of 0 which is strange as it is missing the double free

I found the issue: A fork clears the per-thread memory buffers that are created on application init when using OpenMP. So on every BLAS invocation all threads race to alloc memory which in turn shows the missing sync barrier and degrades performance. Fixed in #2876

Given that there are now 2 quite serious issues (1 specific to POWER and 1 performance issue for all OpenMP builds) I'd suggest to release 0.3.11 earlier so package maintainers can pick those fixes up


Mockups: Collections ui

In progress.
Current plan for first roll-out:

Happy to go with selecting whole collection only (no cherry picking) for first rollout
Our favorite version of the ui involves changing the current variable picker tree so that only clicking on the arrow, not the whole category/entity name, expands/collapses the tree. We'll check with the rest of the eda team to see if they'd be happy with the change.

Duplicate of #757 . Closing


Fix metric web ui can't resolve large number datapoints

What changes are proposed in this pull request?
Fix metric web ui will block when data points number too large to process
Why are the changes needed?
This is a bug. metric web ui can't show pages, when there are too many data points.
Does this PR introduce any user facing changes?
No
Please list the user-facing changes introduced by your change, including
webui

Codecov Report

Merging #14376 (c6ce639) into master (7f30116) will decrease coverage by 25.07%.
The diff coverage is 50.00%.

@@              Coverage Diff              @@
##             master   #14376       +/-   ##
=============================================
- Coverage     41.96%   16.89%   -25.08%
+ Complexity     9341     2698     -6643
=============================================
  Files          1492     1493        +1
  Lines         87257    87362      +105
  Branches      10417    10420        +3
=============================================
- Hits          36621    14761    -21860
- Misses        47667    71362    +23695
+ Partials       2969     1239     -1730

Impacted Files
Coverage Δ

...src/main/java/alluxio/wire/MasterWebUIMetrics.java
0.00% <0.00%> (ø)

...mmon/src/main/java/alluxio/metrics/TimeSeries.java
71.42% <100.00%> (ø)

...mon/src/main/java/alluxio/shell/CommandReturn.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...mon/src/main/java/alluxio/util/ExceptionUtils.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...n/src/main/java/alluxio/wire/AlluxioProxyInfo.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../src/main/java/alluxio/wire/AlluxioMasterInfo.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...src/main/java/alluxio/job/meta/JobIdGenerator.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...n/src/main/java/alluxio/stress/BaseParameters.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...src/main/java/alluxio/client/UnderStorageType.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...main/java/alluxio/underfs/options/ListOptions.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 688 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7f30116...c6ce639. Read the comment docs.

@tieujason330 PTAL!

alluxio-bot, merge this please