bzip2 and zip dependencies_if_needed

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[x] Have you written new tests for your changes? Here's an example.

[x] Have you successfully run brew tests with your changes locally?

Continuing to merge changes from Linuxbrew to Homebrew. Therefore, mentioning @sjackman @imichka
Brining zip and bzip2 related changes from Linux. The PR addresses the issue that not all  distros are created equal and lack bzip2 and zip.
I have written simple tests based on p7zip example.

I'd suggest modelling bzip2_dep_if_needed and unzip_dep_if_needed after xz_dep_if_needed. The generic implementation is…
  def bzip2_dep_if_needed(tags)
    Dependency.new("bzip2", tags)
  end

The macOS implementation is…
def bzip2_dep_if_needed(*); end

and no Linux-specific implementation.

and no Linux-specific implementation.

I agree with this.

Brining zip and bzip2 related changes from Linux. The PR addresses the issue that not all distros are created equal and lack bzip2 and zip.

That's not what you're handling here, though, you're instead installing Homebrew's bzip2 and zip unconditionally regardless of their presence on the system. I don't think that's a good idea and it's not in keeping with the Homebrew ethos.

Some more questions here:

What Linux distros have gzip installed but not bzip2?
Again: why are you unconditionally making everyone using a Linuxbrew bzip2 formula even when the vast majority will have bzip2 present on their system already?
Where does this end in terms of assuming things are/aren't present in Linuxbrew?
Will this approach not cause issues with bottles that are reused between systems that assume the presence of some system libraries and others don't?

I think really what Linuxbrew should start doing to address those problems is:

Make bottles conditional on the distro version and the default (or minimal) distro install
Introduce portable- or similar packages to match the first Linuxbrew run with what's provided by the minimal macOS or OS X system

As a more general point: try to think more about designing Linuxbrew as a system. A bunch of these changes (https://github.com/Homebrew/brew/pull/3809 included) are essentially just hacks to try and get Linuxbrew working on more Linux systems. This way lies madness. Linuxbrew should define (both in terms of documentation and code) what they are willing to support. If you'd rather take the approach of "we're going to try and support literally every case of running Linuxbrew on any Linux distro" then I think the upstreaming efforts may need to part ways (and perhaps be partially reverted).

What Linux distros have gzip installed but not bzip2?

Bare-bones distros, most commonly used in Docker and CI systems.

Again: why are you unconditionally making everyone using a Linuxbrew bzip2 formula even when the vast majority will have bzip2 present on their system already?

This could be conditional on which("bzip2")

Where does this end in terms of assuming things are/aren't present in Linuxbrew?
Will this approach not cause issues with bottles that are reused between systems that assume the presence of some system libraries and others don't?

Linuxbrew uses host executables when possible. Linuxbrew uses no host libraries, except for glibc and libstdc++. Linuxbrew uses brewed versions of glibc and libstdc++ when the host versions are too old.

Make bottles conditional on the distro version and the default (or minimal) distro install

There's nearly unlimited variation in what packages may or may not be installed for a given distribution.

Make bottles conditional on the distro version and the default (or minimal) distro install

Linuxbrew uses a single bottle that's portable to all Linux systems by provided brewed versions of all the necessary libraries.

As a more general point: try to think more about designing Linuxbrew as a system. A bunch of these changes (#3809 included) are essentially just hacks to try and get Linuxbrew working on more Linux systems. This way lies madness. Linuxbrew should define (both in terms of documentation and code) what they are willing to support. If you'd rather take the approach of "we're going to try and support literally every case of running Linuxbrew on any Linux distro" then I think the upstreaming efforts may need to part ways (and perhaps be partially reverted).

We actively support the major distributions of Linux, such as Debian, Ubuntu, Fedora, and CentOS.
I believe Homebrew also had issues with the system version of curl being too old on older distributions of macOS. The solution that worked or Homebrew on macOS should also work for Linuxbrew.

the upstreaming efforts may need to part ways (and perhaps be partially reverted).

Although I appreciate your effort to help upstream Linux patches, Maxim—Thank you!—you don't have the context of when and why patches were introduced, and whether those patches are still needed upstream, or could instead be reverted in Linuxbrew. I suggest we have more internal discussion in Linuxbrew before opening PRs to upstream Linuxbrew patches.

why are you unconditionally making everyone using a Linuxbrew bzip2 formula even when the vast majority will have bzip2 present on their system already?

I could not agree more: everything should be conditional and if the system provides required components - Homebrew should use these. In fact, I was disappointed to see :python and the like go, but, unfortunatly, I don't have enough time to read all the reasoning behind that decision. I think that all tools that Homebrew uses should be checked not only for existence but also that they function properly, have required version, etc. and not rely on MacOS.version. MacOS.version is convenient as it converts multiple comparisons to only a handful, but implies you have to make changes every single time macOS is updated and that's not a super sustainable model. Though, of course, it will survive given the interest that project has in the community.
This PR is merely an attempt to bring changes that currently exist in Linuxbrew to Homebrew.

Although I appreciate your effort to help upstream Linux patches, Maxim—Thank you!—you may not have the context of when and why patches were introduced, and whether those patches are still needed upstream, or could instead be reverted in Linuxbrew. I suggest we have more discussion within the Linuxbrew team before opening PRs to upstream Linuxbrew patches.

:hocho: First of all, thank you for editing the comment thus making it a bit easier to pull out from my back :smile: The context I usually use consists of (1) git log messages (2) comments within the code (not the strongest side of Linuxbrew/Homebrew) (3) Slack (when needed) (4) related GitHub PRs and Issues if changes are hard to make sense of otherwise. Please, let me know of any other context I'm missing. Changes introduced in this as well as #3809 PRs seem straightforward so I only mentioned you here.

I suggest we have more discussion within the Linuxbrew team before opening PRs to upstream Linuxbrew patches.

sounds good

Bare-bones distros, most commonly used in Docker and CI systems.

Which distros specifically, though? Can you name a few?

We actively support the major distributions of Linux, such as Debian, Ubuntu, Fedora, and CentOS.

My point was more: it doesn't seem there's a definition of what you don't support and won't try to e.g. the bare-bones distros you mention above. It's also worth asking what Linuxbrew wants to do differently to tools like Gentoo Prefix which are much more established. With a smaller team and fewer resources spreading yourselves too thin means a tool that's less useful to anyone.

Linuxbrew uses host executables when possible.

That's not the case in https://github.com/Homebrew/brew/pull/3809 with curl or here with bzip2.

There's nearly unlimited variation in what packages may or may not be installed for a given distribution.

That's true but the standard or minimal installation for a distribution provides a minimal subset.

Linuxbrew uses a single bottle that's portable to all Linux systems by provided brewed versions of all the necessary libraries.

I believe Homebrew also had issues with the system version of curl being too old on older distributions of macOS. The solution that worked or Homebrew on macOS should also work for Linuxbrew.

This is not the same solution, though. In this case you're saying "if there's a Homebrew curl: always use it". In the macOS case the relevant formulae have audit checks so that a modern curl can be built and then used. If there was some sort of feature check before this was enabled with curl that would be a smarter move e.g. some test to see what sorts of crypto are supported by the system curl.

In fact, I was disappointed to see :python and the like go, but, unfortunatly, I don't have enough time to read all the reasoning behind that decision.

This was as the vast majority of users are now using bottles exclusively with Homebrew which means that these requirements were only using system versions when building from source and untested by CI. That's not the case here where the vast majority of Linuxbrew users will have bzip2 installed but this requires everyone to have it installed.

I think that all tools that Homebrew uses should be checked not only for existence but also that they function properly, have required version, etc. and not rely on MacOS.version. MacOS.version is convenient as it converts multiple comparisons to only a handful, but implies you have to make changes every single time macOS is updated and that's not a super sustainable model.

Our model is completely sustainable because we rely on a macOS base system that only changes versions once a year. With respect, MacOS.version makes life much easier for Homebrew but just not for Linuxbrew. This is why I've been saying for years Linuxbrew should create their own formulae that are not designed to be used on macOS (i.e. don't use if OS.mac? or MacOS.version).

Reverting changes to Linuxbrew/brew that are no longer needed is an important component of merging Linuxbrew/brew into Homebrew/brew.

👍 . I also think it's worth a discussion at some point between Linuxbrew and Homebrew maintainers as to what the end goal is, a (very loosely) estimated timescale and the similarities and differences in ethos between the projects. This could result in documentation that would make PRs and issues between the projects less confusing.
My starter for ten on this:

Linuxbrew/brew should eventually go away as something that's used by end-users in favour of Linux support being baked sufficiently into Homebrew/brew.
Linuxbrew/brew will remain as a fork (and could be used for those currently on master rather than stable branches) purely to upstream work to Homebrew.
Completing this migration should be aimed to be done by the end of 2018
Homebrew is primarily a non-optioned binary package manager installed into /usr/local and run on macOS. The longer tail we support are older macOS versions, custom prefixes and custom options in formulae.
Linuxbrew/homebrew-core changes never go upstream to Homebrew/homebrew-core unless there's a bug reproducible on both platforms.
Making a change in Homebrew/homebrew-core to add consistency or ease merge conflicts between Homebrew and Linuxbrew is not acceptable.

which distros specifically, though? Can you name a few?

Our ubuntu docker image we use for the circle.ci build does not have bzip2 nor zip installed.
Another (better) example would be Alpine Linux: docker pull alpine && docker run -it alpine, and then run apk -vv info|sort. You will see that the list of packages is quite small :)

Our model is completely sustainable because we rely on a macOS base system that only changes versions once a year. With respect, MacOS.version makes life much easier for Homebrew but just not for Linuxbrew. This is why I've been saying for years Linuxbrew should create their own formulae that are not designed to be used on macOS (i.e. don't use if OS.mac? or MacOS.version).

I agree that it is easier on mac. As I have been doing the core merges for the last year, I see that on Linux it is a little bit more of work; but at the end it is just "some" dependencies missing (libxml2, zip, ncurses, ....). I think we got that already pretty well covered now, as we have now more than 1600 bottles. And I do not get many requests to fix other packages, so I guess the other 3000 or so formulae are less used (or just work out of the box).
When you say we should create our own formulae, how should that work? As patch files that we would apply on top of the homebrew-core repo? I think whatever way we choose, the amount of work is probably equivalent.
I must say that the merges from homebrew-core to linuxbrew-core are working quite well; it is a lot o work during the week, but I think it is because I am doing this mostly alone. You have the same bottleneck in homebrew where most of the updates are done by one or two dedicated persons :) If we had 5 maintainers more everything would be fine :)
I am open for changes though, if it can reduce the workload: but most of the time is spent fixing dependencies and rebuilding bottles, so I am not sure how we can improve this without more maintainers.

Homebrew is primarily a non-optioned binary package manager installed into /usr/local and run on macOS. The longer tail we support are older macOS versions, custom prefixes and custom options in formulae.

One small note: Linuxbrew tries to follow that philosophy too. Though we have some users that can not create a /home/linuxbrew/ folder, and these are stuck with building stuff from source. And these are the cases where we hit the most bugs ...

Linuxbrew/homebrew-core changes never go upstream to Homebrew/homebrew-core unless there's a bug reproducible on both platforms.

Right. I think there was never such a case, at least not last year. We just have the regular sha256 mismatches (at least a few per month), which then get fixed by me or reported if I have no time to fix them myself.

Making a change in Homebrew/homebrew-core to add consistency or ease merge conflicts between Homebrew and Linuxbrew is not acceptable.

+1 on that. I think I never backported anything to core. Maybe once I wanted some linker flags in a different order for a test, but that's all. Merge conflicts are not a problem, I fix some of them each day, and it is working quite well. In fact, there are not so many merge conflicts at the end, and they are all trivial to solve.

BTW, I am not sure this PR is the best place to discuss this :)

Bare-bones distros, most commonly used in Docker and CI systems.

Which distros specifically, though? Can you name a few?

Debian and Ubuntu for example
❯❯❯ docker run debian:9 which gzip bzip2 xz
/bin/gzip
❯❯❯ docker run ubuntu:xenial which gzip bzip2 xz
/bin/gzip

We actively support the major distributions of Linux, such as Debian, Ubuntu, Fedora, and CentOS.

My point was more: it doesn't seem there's a definition of what you don't support and won't try to e.g. the bare-bones distros you mention above.

We don't actively support 32-bit Intel, or distributions of Linux that use a libc other than glibc. We can add more systems to the unsupported list as they come up, but they don't come up too terribly often. Most issues are reported from users running the typical distributions. Docker images help a lot in confirming whether an issue is reproducible on their distribution of Linux, or somehow unique to their system configuration (not reproducible in Docker).

It's also worth asking what Linuxbrew wants to do differently to tools like Gentoo Prefix which are much more established. With a smaller team and fewer resources spreading yourselves too thin means a tool that's less useful to anyone.

I develop data analysis applications on macOS and then deploy them to Linux servers using the same package manager (Brew) on both macOS and Linux. I teach workshops where users install packages using a single set of instructions to install software on their laptops running macOS using Homebrew, on Linux using Linuxbrew, and on Windows using Linuxbrew thanks to Windows Subsystem for Linux. Being able to use the same package manager on multiple operating systems is quite powerful.

Linuxbrew uses host executables when possible.

That's not the case in #3809 with curl or here with bzip2.

@maxim-belkin is fixing that now.

There's nearly unlimited variation in what packages may or may not be installed for a given distribution.

That's true but the standard or minimal installation for a distribution provides a minimal subset.

Here's the list of essential (required) Debian packages. It's pretty safe to assume that these packages will be available on most Linux systems.
See https://www.reddit.com/r/debian/comments/6lnakq/what_are_the_essential_packages/
base-files              libattr1                libuuid1
base-passwd             libblkid1               login
bash                    libc-bin                lsb-base
bsdutils                libc6                   mawk
coreutils               libcomerr2              mount
dash                    libfdisk1               multiarch-support
debconf                 libgcc1                 ncurses-base
debianutils             liblzma5                ncurses-bin
diffutils               libmount1               passwd
dpkg                    libpam-modules          perl-base
e2fslibs                libpam-modules-bin      sed
e2fsprogs               libpam-runtime          sensible-utils
findutils               libpam0g                sysvinit-utils
gcc-6-base              libpcre3                tar
grep                    libselinux1             tzdata
gzip                    libsepol1               util-linux
hostname                libsmartcols1           zlib1g
init-system-helpers     libss2
libacl1                 libtinfo5

>> Reverting changes to Linuxbrew/brew that are no longer needed is an important component of merging Linuxbrew/brew into Homebrew/brew.

> 👍 . I also think it's worth a discussion at some point between Linuxbrew and Homebrew maintainers as to what the end goal is, a (very loosely) estimated timescale and the similarities and differences in ethos between the projects. This could result in documentation that would make PRs and issues between the projects less confusing.

> My starter for ten on this:

> - Linuxbrew/brew should eventually go away as something that's used by end-users in favour of Linux support being baked sufficiently into Homebrew/brew.
> - Linuxbrew/brew will remain as a fork (and could be used for those currently on master rather than stable branches) purely to upstream work to Homebrew.
> - Completing this migration should be aimed to be done by the end of 2018
> - Homebrew is primarily a non-optioned binary package manager installed into /usr/local and run on macOS. The longer tail we support are older macOS versions, custom prefixes and custom options in formulae.
> - Linuxbrew/homebrew-core changes never go upstream to Homebrew/homebrew-core unless there's a bug reproducible on both platforms.
> - Making a change in Homebrew/homebrew-core to add consistency or ease merge conflicts between Homebrew and Linuxbrew is not acceptable.

I'm on board with all of these goals. End of 2018 is a good target date.

Thanks for all the information above.

I'm on board with all of these goals. End of 2018 is a good target date.

Is it worth having a Linuxbrew/brew tracking issue to discuss this? I'm happy to create it.

I've created a tracking issue at https://github.com/Linuxbrew/brew/issues/612

bzip2 and zip have been conditional since https://github.com/Homebrew/brew/pull/3813/commits/b084a2581f32fcbc6fbdf081ffce9a6c53f66ff6
The latest commit (https://github.com/Homebrew/brew/pull/3813/commits/14d7a7a08c9aab7db14ea03f59ec85899e5e3ba6) does the following:

makes xz_dep_if_needed conditional on Linux systems (changes in extend/os/dependency_collector.rb and extend/os/linux/dependency_collector.rb)
removes test/os/dependency_collector_spec.rb
adds Linux-specific tests in test/os/linux/dependency_collector_spec.rb that account for conditional nature of xz, zip, and bzip2
adds macOS-specific tests in test/os/mac/dependency_collector_spec.rb for zip and bzip2 (making sure they are not added as dependencies because they're provided by the host system).

What else could be done:

default implementation of xz_dep_if_needed could be modified to include unless which("xz")

default implementation of xz_dep_if_needed could be modified to include unless which("xz")

That makes sense to me.

if default implementation of xz_dep_if_needed is augmented with unless ... then there will be no need in redefining these on Linux so some of changes will go away.

Thanks @maxim-belkin! A long, painful journey but we got there in the end!

Thanks for this PR, Maxim! And thanks for merging, Mike!


Report Configuation

As a Super User/Creator
I want to update the Draft Report Configuration
so that the selected staff members will get reports notification in the correct schedule
Acceptance Criteria
Selected staff members should get email notification in the configured interval
Scenario 1:
GIVEN ... (you or your condition)
WHEN ... (what you do)
THEN ... (what you see)
Scenario 2:
xxxxxx
Validation Rules?
(If yes, list here)
Screenshot
(Link above will always be the most up-to-date version.)

Tasks

[ ] Create Draft Report Configuration form
[ ] Create BPM Model for the form
[ ] Create report-api endpoints for Draft Report Configuration


Paper elements are not loaded correctly in Chrome

Hello,
When I'm cloning my DOM from an template element, the paper elements are not correctly loaded in Chrome. They work just fine in Firefox though...
I don't get any errors from Polymer in the browser console, so I don't know what exactly is the problem.
I have an example here: http://titannano.github.io/VocabularyTrainer/
It works well in Firefox, but not with Chrome. If anyone wants to test this you will have to use Chrome Canary since the code requires Proxies.
I hope this maybe can get fixed.

You have a ton of imports that are never loaded/registered:

I'm not sure what rel="import-async" is but you should change them to rel="import". If you want the import be async, add the async attribute.

the import-async is there so polymer won't load them. I did this because the paper spinner wouldn't render before all the elements are loaded. This way I wait until the ready event fires
if (window.Polymer && window.Polymer.ImportStatus._ready) {
    Polymer.ImportStatus.whenReady(bootstrap);
} else {
    window.addEventListener('WebComponentsReady', bootstrap);
}

and then replace the other elements
    [].slice.apply(document.querySelectorAll('link[rel="import-async"]')).forEach(element => {
        let node = document.createElement('link');

        node.setAttribute('rel', 'import');
        node.setAttribute('href', element.href);

        queue.push(new Promise(success => node.addEventListener('load', success)));

        element.parentNode.replaceChild(node, element);
    });

Love the ES6 :) You could also put the rest of the imports in a rest.html and load that separately from the spinner.
FWIW, I ran https://gist.github.com/ebidel/cea24a0c4fdcda8f8af2 and still same unregistered elements. This was after the page was fully loaded and the spinner was gone.

I don't get the unregistered errors at all. Do I need to enable any options in chrome to see them?

I disable the spinner as soon as all the elements are loaded. In previous test everything worked fine, but when I put the HTML into the template tags it stopped working in chrome.
        let node = template.content.cloneNode(true);

for some reason this doesn't work in chrome but in Firefox with just the web-components shim.

when I put the HTML into the template tags

Can you provide an example? Sounds like a difference between native imports and the polyfill?

        <template id="word-editor" replace>
            <paper-header-panel flex mode="waterfall" bind-class="{ style-xs-hide: hideEditor() }">

                    <paper-icon-button class="style-xs-show" bind-events="{click:closeEditor}" icon="icons:chevron-left"></paper-icon-button>
                    Word Editor

                        <paper-input type="text" label="{{profile.targetLang}}" bind-value="{{item.text}}"></paper-input>
                        <paper-input type="text" label="{{profile.userLang}}" bind-value="{{item.translation}}"></paper-input>
                        <paper-input type="text" label="Pronounciation" bind-value="{{item.pronounciation}}"></paper-input>
                        <paper-input type="text" label="Priority" bind-value="{{item.priority}}"></paper-input>

                            Accuracy:
                            <paper-progress bind-value="item.getAccuracy()" flex></paper-progress>

                            {{getRemainingTests()}} correct tests remaining for full accuracy

                            <paper-button class="style-warn" bind-enabled="checkNew()" bind-events="{click:removeItem}">Remove</paper-button>

        </template>

this is the main view in a template tag.
as soon as all the elements are loaded I register the template element in my data binding engine
    /** @type {ScopePrototype} */
    let { scope } = DataBinding.makeTemplate('#main-view', {
        profile : null,

        $methods : {
            manageWords : function(){
                UiPage.activate('word-manager');
            },

            trainWords : function(){
                UiPage.activate('word-trainer');
            },

            reset : function(){
                this.profile.progress = 0;

                this.profile.wordList.forEach(word => {
                    word.tests = 0;
                    word.right = 0;
                    word.priority = 0;
                });

                Profile.save(this.profile);
            }
        }
    });

which then clones the template content, processes all my bindings and replaces the template tag with the final DOM sub-tree
let node = template.content.cloneNode(true);
[...]
template.parentNode.replaceChild(node, template);

Try using document.importNode(t.content, true); instead of cloneNode. The former is needed because  custom elements in templates are not upgraded. importNode will bring in the definitions.
Example: http://jsbin.com/kuhucoropi/edit?html,output

thanks a lot! now it works 😄:+1:
How is that with the rest.html meant?
I put it like this:
    <link rel="import" href="components/paper-spinner/paper-spinner.html">
    <link rel="import" href="rest.html">

and the rest.html
<link rel="import" href="components/paper-header-panel/paper-header-panel.html">
<link rel="import" href="components/paper-progress/paper-progress.html">
<link rel="import" href="components/paper-button/paper-button.html">
<link rel="import" href="components/paper-icon-button/paper-icon-button.html">
<link rel="import" href="components/paper-input/all-imports.html">
<link rel="import" href="components/paper-listbox/paper-listbox.html">
<link rel="import" href="components/paper-item/all-imports.html">
<link rel="import" href="components/iron-icons/iron-icons.html">
<link rel="import" href="components/paper-tooltip/paper-tooltip.html">

but this doesn't change that all imports need to be done before any custom element is rendered and so the spinner doesn't show up until everything is loaded.


bert model could not applied in TimeDistributed wrapper.

Describe the Bug
bert model could not applied in TimeDistributed wrapper.
Version Info
*tensorflow 1.12
*keras 2.2.4
Minimal Codes To Reproduce
from keras_bert import load_trained_model_from_checkpoint, Tokenizer
from keras.layers import *
from keras.models import Model

config_path = './chinese_L-12_H-768_A-12/bert_config.json'
checkpoint_path = './chinese_L-12_H-768_A-12/bert_model.ckpt'
dict_path = './chinese_L-12_H-768_A-12/vocab.txt'

bert_model = load_trained_model_from_checkpoint(config_path,
                                                checkpoint_path,
                                                seq_len=None)

for l in bert_model.layers:
    l.trainable = True

MAX_SENTENCE_LENGTH = 128
MAX_SENTENCE_COUNT = 64
l2_reg = regularizers.l2(1e-8)
n_classes = 2

x1 = Input(shape=(MAX_SENTENCE_LENGTH, ), dtype='int32')
x2 = Input(shape=(MAX_SENTENCE_LENGTH, ), dtype='int32')
sentence = bert_model([x1, x2])
sentence = Lambda(lambda x: x[:, 0])(sentence)
texts_in = Input(shape=(MAX_SENTENCE_COUNT, MAX_SENTENCE_LENGTH, 2),
                 dtype='int32')
attention_weighted_sentences = TimeDistributed(sentence)(texts_in)
model = Model(texts_in, attention_weighted_sentences)
model.summary()

2019-09-27 10:06:25.897602: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-09-27 10:06:25.906646: I tensorflow/core/common_runtime/process_util.cc:69] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
Traceback (most recent call last):
  File "/home/phoenixkiller/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/ptvsd_launcher.py", line 43, in <module>
    main(ptvsdArgs)
  File "/home/phoenixkiller/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/lib/python/ptvsd/__main__.py", line 432, in main
    run()
  File "/home/phoenixkiller/.vscode/extensions/ms-python.python-2019.9.34911/pythonFiles/lib/python/ptvsd/__main__.py", line 316, in run_file
    runpy.run_path(target, run_name='__main__')
  File "/home/phoenixkiller/anaconda3/envs/tf1.12/lib/python3.6/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/home/phoenixkiller/anaconda3/envs/tf1.12/lib/python3.6/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/home/phoenixkiller/anaconda3/envs/tf1.12/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/phoenixkiller/source/hnatt_bert_keras/error.py", line 27, in <module>
    attention_weighted_sentences = TimeDistributed(sentence)(texts_in)
  File "/home/phoenixkiller/anaconda3/envs/tf1.12/lib/python3.6/site-packages/keras/engine/base_layer.py", line 431, in __call__
    self.build(unpack_singleton(input_shapes))
  File "/home/phoenixkiller/anaconda3/envs/tf1.12/lib/python3.6/site-packages/keras/layers/wrappers.py", line 198, in build
    if not self.layer.built:
AttributeError: 'Tensor' object has no attribute 'built'

TimeDistributed takes a layer as its input, not the output (tensor) of a layer.


Artifactory private nuget package source authentication

Is there an existing issue for this?

[X] I have searched the existing issues

Package ecosystem
nuget
Package manager version
No response
Language version
No response
Manifest location and content before the Dependabot update
No response
dependabot.yml content
registries:
  nuget-public:
    type: nuget-feed
    url: https://api.nuget.org/v3/index.json

  my-artifactory:
    type: nuget-feed
    url: https://xyz.jfrog.io/artifactory/api/nuget/v3/nuget-local
    username: ${{secrets.ART_USER}}
    password: ${{secrets.ART_PASS}}

Updated dependency
No response
What you expected to see, versus what you actually saw
artifactory private nuget package source authentication is not working. I think part of the reason could be that artifactory requires username/password, but it looks like nuget_config_credential_helpers.rb uses hardcoded user user and token instead of password:

is there a way to support username/password, so that artifactory authentication starts working? Otherwise none of the dependencies get updated, event the ones coming from public nuget feed.
By the way public package updates were working fine until something changed in the nuget implementation. Now none of the packages get updates in projects that also use private packages.
Native package manager behavior
No response
Images of the diff or a link to the PR, issue, or logs
No response
Smallest manifest that reproduces the issue
No response

Thanks @vadimi and sorry about the regression. The crew is working on fixes in this area and should hopefully have some updates

@vadimi this shipped; does it fix the issue?

unfortunately it does not, the error log I'm getting looks like this:
 proxy | 2024/01/25 21:15:05 [124] GET https://xyz.jfrog.io:443/artifactory/api/nuget/nuget-local/Download/MyPackage/1.2.0
  proxy | 2024/01/25 21:15:05 [124] 403 https://xyz.jfrog.io:443/artifactory/api/nuget/nuget-local/Download/MyPackage/1.2.0
  proxy | 2024/01/25 21:15:05 [124] Remote response: {
  proxy |   "errors" : [ {
  proxy |     "status" : 403,
  proxy |     "message" : "Bad credentials"
  proxy |   } ]
  proxy | }

it doesn't look like this PR changes anything around authentication

Tagging @brettfo and @edgarrs ; I know there was some work done on auth improvements this week; is this scenario supported?

This specific scenario isn't supported, there are earlier steps that combine the username and password into a token field like so: user:password.  I'll have to explore a bit to see the best way to support this as well as the other scenarios.

@vadimi I'm still looking into a proper fix for this, but in the meantime you might be able to make a change to your depedabot.yml file.  I don't yet have an Artifactory feed to test this with, but based on how we give the authentication to the NuGet feed, you might be able to pre-combine the username and password into a single token field like this:
 registries:
   nuget-public:
     type: nuget-feed
     url: https://api.nuget.org/v3/index.json

   my-artifactory:
     type: nuget-feed
     url: https://xyz.jfrog.io/artifactory/api/nuget/v3/nuget-local
-    username: ${{secrets.ART_USER}}
-    password: ${{secrets.ART_PASS}}
+    token: "${{secrets.ART_USER}}:${{secrets.ART_PASS}}"
     # combine both username and password with a colon character ':'

thanks @brettfo just tried that, but still received:
  proxy | 2024/01/26 19:41:01 [287] Remote response: {
  proxy |   "errors" : [ {
  proxy |     "status" : 403,
  proxy |     "message" : "Bad credentials"
  proxy |   } ]
  proxy | }

I think part of the reason is that nuget_config_credential_helpers.rb hardcodes Username field in ~/.nuget/NuGet/NuGet.Config to user. Artifactory requires an actual user.

@vadimi I'm surprised the change to token: user:pass didn't work and that makes me wonder what else is going on.  You included part of the  log that reported a 403, can you give the few lines of the log above that, specifically where it lists the URL it was trying to hit?  There have been reported instances of Artifactory NuGet feeds not reporting all of their supported endpoints which means we might not be able to auto-inject the authentication.
On a related note, I'd love to be able to see the raw responses from your Artifactory feed.  If you have some time I'd like to schedule a bit of a back-and-forth where we explore the responses so I can see what's going on.  I can do a video call or we could even do some back-and-forth via email, and any feed or package information I gather would be kept private.  My email address is<EMAIL_ADDRESS>

For anybody following along at home, I've received some logs via email and will continue my investigation of the feed.

Thanks again to @vadimi for providing me with some API responses.  I've distilled the information into this comment so I don't lose track of it.
The Problem
NuGet feeds provided by Artifactory don't support the PackageBaseAddress API endpoint which we use to generate a URL to download a .nupkg file.  The end result was that we tried to create this URL from the base feed address, but that wasn't correct and Artifactory would just return a 404.
The Solution
Artifactory does support the SearchQueryService API endpoint which we can use to list package data by name.  The results of that query look like this:
GET https://<SearchQueryService>?q=Some.Package&prerelease=true
{
  "totalHits": 1,
  "data": [
    {
      // these first few fields correspond to the latest package version available
      "registration": "// same value that we can calculate from the RegistrationsBaseUrl endpoint",
      "version": "1.2.3", // the latest package version available
      "description": "// not interesting",
      "authors": [
        "// not interesting"
      ],
      "totalDownloads": 123, // not interesting
      "verified": true, // not interesting
      "versions": [
        {
          "version": "1.0.0",
          "downloads": 123,
          "@id": "<SOME-URL>/some.package/1.0.0.json" // registration leaf
        },
        {
          "version": "1.2.3",
          "downloads": 123,
          "@id": "<SOME-URL>/some.package/1.2.3.json" // registration leaf
        }
      ],
      "@id": "// seems to be identical to the `registration` field above",
      "id": "Some.Package", // the package name.  This will need to be matched to the original query to protect against partial matches
    }
  ]
}

Once we've matched response.data.id to the original package name, we can then pull the registration leaf from the @id field of each element of the versions array.  That data looks like the following:
GET <SOME-URL>/some.package/1.2.3.json
{
  "listed": true, // n.b., if this value is missing it is considered to be `true`
  "packageContent": "<SOME-URL>/some.package.1.2.3.nupkg", // this is what we really want, the .nupkg URL
  "registration": "// same value that we can calculate from the RegistrationsBaseUrl endpoint",
  "@id": "// URL directly to a JSON file describing the package, we won't need it for this scenario"
}

From this, we can pull down the response.packageContent URL and extract the .nuspec from the .nupkg file as we currently do.
Action

Modify repository_finder.rb to allow for a missing/nil PackageBaseAddress endpoint.
Modify nupkg_fetcher.rb to use the SearchQueryService method described above IFF the PackageBaseAddress endpoint is missing/nil.
Modify the internal proxy to inject authentication for the packageContent field in the response above.

N.b., the original Artifactory response also lists LegacyGallery and PackagePublish endpoints that happen to match the prefix of the packageContent URL, so we might be able to go that route, too.

@vadimi I've created #9117 that fixes Artifactory feeds.

thanks @brettfo , I see the PR got merged, is it ok to get it tested?

Yes, @vadimi ; can you test it again and please let us know the outcomes?

tried it on a few repos - everything worked perfectly fine. thanks @brettfo and @abdulapopoola, appreciate the effort!


Failed to load class for data source: com.databricks.spark.xml

Hello, I am using spark 1.4.0 with scala 2.10.4, when I try to use spark-xml this way,
bin/spark-shell --packages HyukjinKwon:spark-xml:0.1.1-s_2.10
scala> import org.apache.spark.sql.SQLContext
scala> val sqlContext = new SQLContext(sc)
scala> val df = sqlContext.read
.format("com.databricks.spark.xml")
.option("rootTag", "book")
.load("books.xml")

Then I got this error,
java.lang.RuntimeException: Failed to load class for data source: com.databricks.spark.xml

I read from https://issues.apache.org/jira/browse/SPARK-8368,  so do I need to move to spark 1.4.1 or 1.5, or could you please offer me some way to fix it?

When it was in my private repository, I uploaded the package with the package name org.apache.spark.* as described here.
And the problem is, any release is not published and not uploaded yet since this package has moved to here Databricks. Namely, not you cannot use by --packages but have to manually build.
I will publish a release as soon as the pull requests I made are merged.
To cut it short, If you set the package name as described here, that should import properly.
I apologise that I made such a confusion. I think I should change the README.md at my private repository.

@dougneedham
Would you like to try this one please?
$SPARK_HOME/bin/spark-shell --packages com.databricks:spark-xml_2.11:0.2.0

Thanks! This works well for me!


Merge prTesting_201903141023 branch to master branch

OPS Build status updates of commit 3d154d6:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details
Dependent Files

split/updateTestRef.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


remove dangling commas from function params to support node v6.14

fixes #24 by removing two dangling commas on functional arguments

Strange, I would have expected travis to fail as I'm also running tests on node 6 there. I guess I have to check the test & travis setup again..
Thank you very much for pointing out this issue and for fixing it @eleith! 🎉


Support Mlflow as LLM-Provider

is it planned to support MlFlow as LLM-Provider ?

MLFlow is for training models as I understood. Do they provide a normalized API for running inferencing as well?

Yes, MLFlow is used or training/ evaluationg as well as serving Models.
It has unified APIs to return Models with their End-Points https://mlflow.org/docs/latest/llms/deployments/api.html#/
I can work on an extension and provide with pull request.

are there plans to add mlflow as an llm provider now


Error on debugging my website

Hallo,
when i debug my Website and set a breakpoint i can't watch my variables.
I got the error:
error CS0103: The name 'model' does not exist in the current context
See Picture attached.
How can i fix this.
thx
Jens

Is the code running as Release / optimized? Local variables like this may be removed by optimizations. Try running under Debug instead.

I think this is a bug in VS - see: http://stackoverflow.com/questions/36284093/visual-studio-2015-update-2-rtm-and-rc-debugging-context-not-available

This issue is being closed because it has not been updated in 3 months.
We apologize if this causes any inconvenience. We ask that if you are still encountering this issue, please log a new issue with updated information and we will investigate.


Support Number jsons

humanReadableSerializerModule should allow to parase things like:
"price": {
    "amount": 12.34,
    "currency": "€"
}

But right now it forces to use "12.34". We shouldn't lose any precision by using a number of json. We shouldn't lose any precision by using 12.34.
From backend I'm receiving that and with Moshi I hadn't any problem. On moshi I read 12.34 as a String (even it is a number in the json) and I parse it to BigDecimal.

Hi @BraisGabin as far as I am aware it's not possible to encode it that way without precision loss, and the limitation is on the KotlinX serialization side. You can see the issue regarding this here https://github.com/Kotlin/kotlinx.serialization/issues/1051
Specifically this comment https://github.com/Kotlin/kotlinx.serialization/issues/1051#issuecomment-812130020
I'm closing this issue as there is nothing to on our side at the moment.

Good to know, thanks!

Hi, just a quick heads up that I'm working on supporting BigDecimal/BigInteger in this PR https://github.com/Kotlin/kotlinx.serialization/pull/2041.
I'm a contributor to KxS, not a maintainer, so I can't give any timescale.
I'd like to be able to help implement support in in this library when it's available! As a preview, you can see the test I wrote for BigDecimal - I expect a serializer for BigNum will be similar https://github.com/Kotlin/kotlinx.serialization/pull/2041/files#diff-46d155dec31ebfaed1e644398258f05f18332f9af8454854f863c0100019eb41


[PULL] Speedup Slice::compare

Please pull https://code.google.com/r/savetherbtz-leveldb/

commit 0b63bc80fb080f40a96d562b58a5d2f554b59609
Author: Alexey Ivanov<EMAIL_ADDRESS>Date:   Fri Mar 1 15:26:58 2013 +0530

    Speedup Slice::compare

    Micro optimization simplifies code, reduces number of branches and
    overall code footprint(both C and machine code).

See
https://code.google.com/r/savetherbtz-leveldb/source/detail?r=0b63bc80fb080f40a9
6d562b58a5d2f554b59609

Original issue reported on code.google.com by<EMAIL_ADDRESS>on 1 Mar 2013 at 10:51

can you pl provide some numbers on how much improvement you get and which
compiler you are using.

Original comment by<EMAIL_ADDRESS>on 1 Mar 2013 at 6:02

Doesn't this also change the behaviour? It will order by length rather than
lexicographically, no?

Original comment by<EMAIL_ADDRESS>on 1 Mar 2013 at 7:26

If one want to preserve lexicographical order you may pull following patch
instead - this removes one branch from resulting code and makes code smaller.
But I don't suppose that this one will be visible on any benchmarks.

Original comment by<EMAIL_ADDRESS>on 2 Mar 2013 at 7:01
Attachments:

micro.diff

Yes, grouping keys by sizes and sorting lexicographically within each size gave
a big performance boost for our workflow. But I guess some workflows could be
hurt by that.

Original comment by<EMAIL_ADDRESS>on 2 Mar 2013 at 6:39


Module has no default export

I'm using the default snippet in typescript
import sslChecker from "ssl-checker";

const getSslDetails = async hostname => await sslChecker("google.com");
console.log(getSslDetails);

And what if you imported the module like this?

import * as sslChecker from "ssl-checker";

yea, tried all the variations
This expression is not callable.
  Type 'typeof import("/Users/amrsobhy/dldl/functions/node_modules/ssl-checker/lib/index")' has no call signatures.ts(2349)

Seeing the same issue on my end as well

Same for me... any progress? 👀

Sorry for the late response, Should be fixed by now


feat(cmake): add custom target functions

This adds new cmake rules to build our libraries and unit tests. This should help with several things:

cleaner naming schemes
proper use of INTERFACE/PUBLIC/PRIVATE specifiers
way more reasonable handling of (transitive) dependencies allowing for hassle free link time substitution, which is essential for unit testing

Documentation is included in the .cmake script.

@LeoBuron thanks for the review


Base Upgrade Content

We have the base upgrade faq content reviewed by Sri, Pranay, Rohan. We got CS feedback as well.
Harness Developer Pull Request
Thanks for helping us make the Developer Hub better. The PR will be looked at
by the maintainers.
What Type of PR is This?

[ ] Issue
[ ] Feature
[ ] Maintenance/Chore

If tied to an Issue, list the Issue(s) here:

Issue(s)

House Keeping
Some items to keep track of. Screen shots of changes are optional but would help the maintainers review quicker.

[ ] Tested Locally
[ ] Optional Screen Shoot.

Preview environment: https://ci.pr.harness.io/pr-416

closing as this content was merged in another PR:
https://github.com/harness/developer-hub/pull/417


How to configure the action endpoint from within python script

Rasa Core version:
rasa-addons==0.4.3
rasa-core==0.10.4
rasa-core-sdk==0.11.5
rasa-nlu==0.13.7

Python version: 3.6.6
Operating system (windows, osx, ...): GNU/Linux (Arch Linux) 4.18.14
Issue: Unclear on how to set action endpoint in my python script in which the dialogue model is trained.
Content of domain file (if used & relevant):
intents:
  # add your intents
  - get_started
  - greet

entities:
  - forename

slots:
  forename:
    type: text

templates:
  # GENERAL PURPOSE UTTERANCES
  utter_greet:
    - text: "Hallo! Wie kann ich dir weiterhelfen?"

actions:
  # all the utter actions from the templates, plus any custom actions
  - utter_greet
  - action_store_forename

Hey there,
I'm trying to call a custom Action through my action endpoint python -m rasa_core_sdk.endpoint --actions actions after training the NLU through python3.6 -m rasa_nlu.train -c nlu_config.yml --data data/nlu_data.md -o models --fixed_model_name nlu --project current --verbose and my python script for training the dialogue model looks as follows:
# -*- coding: utf-8 -*-

import os
import warnings

from rasa_addons.webchat import WebChatInput, SocketInputChannel
from rasa_core.agent import Agent
from rasa_core.featurizers import MaxHistoryTrackerFeaturizer, BinarySingleStateFeaturizer
from rasa_core.interpreter import RasaNLUInterpreter
from rasa_core.policies import MemoizationPolicy
from rasa_core.policies.fallback import FallbackPolicy
from rasa_core.policies.keras_policy import KerasPolicy
from rasa_core.utils import read_endpoint_config
from ruamel import yaml

def train_dialogue(domain_file, model_path, training_folder):
    policies = [
        MemoizationPolicy(max_history=3),
        KerasPolicy(MaxHistoryTrackerFeaturizer(BinarySingleStateFeaturizer(), max_history=3)),
        FallbackPolicy(fallback_action_name='utter_default', nlu_threshold=0.8, core_threshold=0.4)
    ]

    endpoint_cfg = read_endpoint_config('endpoints.yml', 'action_endpoint')

    agent = Agent(
        domain=domain_file,
        interpreter=RasaNLUInterpreter('models/current/nlu/'),
        policies=policies,
        action_endpoint='http://<IP_ADDRESS>:5055/',  # <--- where to put this endpoint ?
    )

    training_data = agent.load_data(training_folder)

    agent.train(training_data, epochs=300)
    agent.persist(model_path)
    return agent

if __name__ == '__main__':
    warnings.simplefilter('ignore', yaml.error.UnsafeLoaderWarning)

    agent = train_dialogue('domain.yml', 'models/current/dialogue', 'data/core/')

    # create dialogue graph for debugging
    agent.visualize('data/core/stories.md', output_file='debug/stories.png', max_history=15)

    static_asset_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'templates')
    input_channel = WebChatInput(static_assets_path=static_asset_path)
    agent.handle_channel(SocketInputChannel(5000, '/', input_channel))

and my actions.py:
from rasa_core_sdk import Action
from rasa_core_sdk.events import SlotSet

class ActionStoreForename(Action):
    """Stores the users name in a slot"""

    def name(self):
        return "action_store_forename"

    def run(self, dispatcher, tracker, domain):
        forename = next(tracker.get_latest_entity_values('forename'), None)

        # if no name was extracted, use the whole user utterance
        # in future this will be stored in a `name_unconfirmed` slot and the
        # user will be asked to confirm their name
        if not forename:
            forename = tracker.latest_message.get('text')

        return [SlotSet('forename', forename)]

I also have a endpoints.yml file in the same folder with the content:
action_endpoint:
  url: "http://localhost:5055/"

My question: How can I actually manage to add the action_endpoint to my python script?
PS.: Moreover, when running the action endpoint server and curling from it, I run into further issues:
INFO:__main__:Starting action endpoint server...
INFO:rasa_core_sdk.executor:Registered function for 'action_store_forename'.
INFO:__main__:Action endpoint is up and running. on ('<IP_ADDRESS>', 5055)
<IP_ADDRESS> - - [2018-10-24 10:07:33] "GET /version HTTP/1.1" 404 342 0.023045

when using both, curl http://<IP_ADDRESS>:5055/version or curl http://localhost:5055/version.

You don't need an action endpoint for training your dialogue model. And the reason the /version endpoint doesn't work is because it doesn't exist.
If you want to pass an action endpoint to running the bot, please take a look at the run script (https://github.com/RasaHQ/rasa_core/blob/master/rasa_core/run.py#L224), or use the run script to start your bot.
I'm going to close this as it's a usage question. If you would like to know more, please post in our forum

hi i have created once webservice using python wanted to call that service to call from Rasa Nlu could you please update any information/process i am very new to this advance thanks


chore(workflow-dev): Update telegraf env vars

Allows usage of NSQ for pulling router metrics
Updates env vars for turning on prometheus endpoints
Removes stdout-metrics

I tested this on my workflow-dev cluster, and it gets grafana working again. 👍

I didn't manually test this, but the code makes sense and LGTM


Added more info to stats command

Proposed Semver Increment Bump: [MAJOR/MINOR/PATCH]
MINOR
Changes Proposed in this Pull Request (List new items in CHANGELOG.MD)

Add more info to stats command

Adding node version because it's useful, both for devs and for helping
us help devs troubleshoot

(If Applicable) What Issue does it fix?
Fixes...
nothing

Um…I think the commit history might be messed up.

all g we squash anyway


Fixed grammar mistake

changed an "an JSX tree" to "a JSX tree"
changed "transpiled to React.createElement() function tree" to "transpiled to a React.createElement() function tree"

Thank you for MR :)


Ancient8

Chain Name
Ancient8
Chain ID
888888888
RPC URL
https://rpc.ancient8.gg
Block Explorer URL
https://scan.ancient8.gg/
Deployment Funds
No

As you can see in the README update I just pushed, the Multicall3 deployer key has been compromised, and someone used it to deploy a different contract on Ancient8: https://scan.ancient8.gg/address/0xcA11bde05977b3631167028862bE2a173976CA11
As a result, the regular Multicall3 contract cannot be deployed at this address. You may consider asking the chain operator to place Multicall3 as a predeploy on this chain to overwrite the existing code. Ancient8 is an OP Stack chain, and new OP Stack chains do include Multicall3 as a preinstall by default, so they may be willing to overwrite the code in a hard fork.


OperationalPlan cleanup

There are a few issues that can be improved on OperationalPlan.

There is some duplicate / unnecessary code (e.g. returning typed and SI values).
getOperationalPlanSegmentList() returns a mutable List.
Do we need speed (acceleration=0) and wait (speed=0) segments/plans? We can use an acceleration plan with 0 values. This will really reduce the amount of code.
Rather than supplying Segments, perhaps we can supply acceleration and duration lists. This can highly simplify underlying calculations on distance/speed/acceleration at given times, and reduce the class structure.
It is now impossible to cleanly translate an OperationalPlan in to information to sent to a visualization, e.g. Unity. For example, the acceleration has to be determined from the plan as the segments shield these values (and would need to be checked to be an AccelerationSegment with instanceof). This requires cumulative addition of time over the segments to retrieve acceleration at that time from the plan. This may result in rounding errors.

OperationalPlan has been reduced from 1053 to 491 lines of code. The segments are now a Segments object, which contains a list of Segment objects (now a separate class). Each Segment has a constant acceleration, where both the acceleration and speed may be 0. Class Segments is initialized with a start speed for the first segment. Consecutively added segments receive a start speed equal to the end speed of the previous segment.
The constructor of OperationalPlan will set a total duration and length to comply with the minimum of either the path, or the Segments. The special case of a stand-still plan is taken care of.
OperationalPlanBuilder was removed, this is now all easily achieved with Segments. OperationalPlanDemo (test code) has been removed. OperationalPlanTest was updated.


Fixed build.zig for zig version 0.13.0-dev.230+50a141945

Just updated the build script to work properly when working with zig version  0.13.0-dev.230+50a141945

Thanks @tayoldotdev!
I'll tag a new version.


Location of Penetration % total?

I can't find it anywhere in Calcs. I have like 20% fire pen from The Wise Oak flask, 8% from tree, 10% from Chieftain, can't see where's total.
Main hand skill Cyclone, but using Avatar of Fire.

It's in the breakdown for the Effective DPS Modifier:

By the way, the penetration modifier from The Wise Oak isn't supported yet.


Configuration option to change the preferredStatusBarStyle

The following PR adds a config option to determine the status bar style that appears when the user present the YPImagePicker NavigationController. Please see example of the config option below:
        var config = YPImagePickerConfiguration()
        config.preferredStatusBarStyle = .lightContent

OR
        var config = YPImagePickerConfiguration()
        config.preferredStatusBarStyle = .default

@miscampbell This is very cool!
Thanks a ton for taking the time to improve the api for everybody 👏 💪


ui-level

Description
Is there any plan to add a ui-level flow into the Dashboard V2?
Thanks,
Have you provided an initial effort estimate for this issue?
I have provided an initial effort estimate

Whatever the considerations, do not to try to replicate.
Steal ideas from grafana and maybe from here (https://discourse.nodered.org/t/dashboard-2-0-is-now-generally-available/84913/43?u=hotnipi)
I don't have plans to migrate ui-level to DB2 but happy to consult in terms of user feedback, features, improvements, technical proposals ...

Thanks a lot.  That's exactly what I was looking for.


Google cloud storage support?

Now only aws s3 is supported. Are there any plans to add support for Google cloud storage?

Should be able to, I'll look into what would need to be done.

Will be closing this for now, will need it to be supported in different repos before it gets added here.


Handler Template needed.

Hi,
I have been trying to create a release note for one of our classic build pipeline. The details I am trying to write to release note are Build Number and its Associated work item details, but not working. Could you please provide one template with such information?

There are sample templates here.
A basic build one in the sample is also shown below, this lists the build details and WI and CS
# Notes for build
**Build Number**: {{buildDetails.id}}
**Build Trigger PR Number**: {{lookup buildDetails.triggerInfo 'pr.number'}}

# Associated Pull Requests ({{pullRequests.length}})
{{#forEach pullRequests}}
{{#if isFirst}}### Associated Pull Requests (only shown if  PR) {{/if}}
*  **{{this.pullRequestId}}** {{this.title}}
{{/forEach}}

# Global list of WI ({{workItems.length}})
{{#forEach workItems}}
{{#if isFirst}}## Associated Work Items (only shown if  WI) {{/if}}
*  **{{this.id}}**  {{lookup this.fields 'System.Title'}}
  - **WIT** {{lookup this.fields 'System.WorkItemType'}}
  - **Tags** {{lookup this.fields 'System.Tags'}}
  - **Assigned** {{#with (lookup this.fields 'System.AssignedTo')}} {{displayName}} {{/with}}
{{/forEach}}

# Global list of CS ({{commits.length}})
{{#forEach commits}}
{{#if isFirst}}### Associated commits  (only shown if CS) {{/if}}
* ** ID{{this.id}}**
  -  **Message:** {{this.message}}
  -  **Commited by:** {{this.author.displayName}}
{{/forEach}}

Hey, thanks for the quick response.
I have a custom build number format(see screenshot), but it is not reflecting in the release note.

instead, the default build number is taken in the release note

The {{buildDetails.id}} is the unique numeric ID of the build.
I think you need the {{buildDetails.buildNumber}} property
If you want to see all the field available to you use the JSON dump function {{json buildDetails}} and you can pick out what you are after, you can use this for any object

Thanks, @rfennell, Build number property worked.
I have been trying to add some more details to the template like the Description and the Parent UserStory of the associated work item. But not working. below is my template, could you please correct me on what went wrong there?

By default the task does not look for parent or child workitems, only directly associated work items. You have to tell it to populate the related WI array. Until you do this the template you have above, which looks OK, will not show the parents
The details of all the parameters are in the documentation, but the two of interest are

(Advanced) Get Direct Parent and Children for associated work items, defaults to false
(Advanced) Get All Parents for associated work items, recursing back to workitems with no parents e.g. up to Epics, defaults to false

or in YAML
# Generate Release Notes (Crossplatform)
# Description - Generates a release notes file in a format of your choice from the build or release history
- task: XplatGenerateReleaseNotes@3
  inputs:
     ... other arguments
     getParentsAndChildren: True
     getAllParents: True

You should enable one or both of these depending upon the results you want.
Once you enable can see the task building the list via the logs if you set a build variable system.debug=true. You should be able to see the adding of the parent and child WIs

Great!! that works.
Parent is now available in the note. One more help, Is there any way to format the WI description as same as the source? With this, I can finalize the release note.
Orginal:

In the release note:

Yes there is, again it is the documentation

Note: if a field contains escaped HTML encode data this can be returned its original format with triple brackets format {{{lookup this.fields 'System.Description'}}}

Fantastic, thank you @rfennell Now the note looks cool.
It would have been great if I could map the WI and Parent ID to its URL in the release note.

Do you mean you want Hyperlinks in your release notes, that can be done have a look #810 but the core is a template
# Global list of WI ({{workItems.length}})
{{#forEach workItems}}
{{#if isFirst}}## Associated Work Items (only shown if  WI) {{/if}}
*  **{{this.id}}**  {{lookup this.fields 'System.Title'}}
  - **WIT** {{lookup this.fields 'System.WorkItemType'}}
  - **Link** [{{this.id]]({{lookup this.fields 'System.url'}}_

{{/forEach}}

This generates

13 My WI

Bug
Link 13

seems like wrong sysntax here in the line "- Link [{{this.id]]({{lookup this.fields 'System.url'}}_".

Sorry try this
{{#forEach workitems}}
    - [{{this.id}}]({{replace this.url "_apis/wit/workItems" "_workitems/edit"}}) - {{lookup this.fields 'System.Title'}}
{{/forEach}}

The key fact to know is that you can dump all properties of an object like this
{{#forEach workitems}}
    - {{json this}}
{{/forEach}}

You can then see all the property values available and use them to build the strings you need. Remember you have all these Handlebars extensions to help


Add wabt compatibility test

As mentioned in https://github.com/wasdk/wasmparser/issues/33 and https://bugs.chromium.org/p/chromium/issues/detail?id=1066386, we want to generate WAT that is wabt-compatible.
Some existing ones are not wabt-compatible so we exclude those.

@bmeurer @ktran PTAL :)

Thank you for the patch!

:tada: This PR is included in version 1.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Issue with Npgsql.LegacyPostgis

Hi,
I'm trying to use PostgisGeometry type to read spatial data from database table. It's throwing
"The entity type 'PostgisGeometry' requires a primary key to be defined". Is there any additional configuration steps required to do before using it.
Thanks,
Sri

@srinivashavanur Could you post some version information (i.e. provider, framework) along with a minimal reproduction of what you're seeing?

@austindrenski : I'm using Npgsql.EntityFrameworkCore.PostgreSQL 2.1.0 version and I just installed the latest version of legacypostgis. Earlier I was using PostgisGeometry type which belong to NpgSQLTypes but after updating to 2.1 NpgSQLTypes no longer supports PostgisGeometry. I tried using NetTopologySuite but it doesn't support spatial data type PostgisGeometry. Any help would be appreciated.
Thanks in advance,
Sri

Npgsql.LegacyPostgis supports the legacy PostGIS types at the Npgsql ADO.NET level, but the plugin to add support for EF Core doesn't exist yet. #464 tracks adding that plugin, hopefully soon.
Consider checking out the new spatial support based on NetTopologySuite, which is much better than LegacyPostgis.


SSH backend

What does this PR change?
It adds a new "dummy" backend that will not create any infrastructure, rather, it will configure some via SSH.

At the end of the day it was not particularly difficult to extract this PR from #652. Code-wise it is not different, but it's easier to review small blocks.

I am not sure if you like this name!

I am not sure if you like this name!

Can you elaborate a bit more about the reason to choose this name? We didn't use ssh in this backend.
What if we decide to add another backend that connects to existing machines using ssh, set some data (like grains), copy salt files and apply the high state? The name for a backend with this logic would probably be called ssh.

What if we decide to add another backend that connects to existing machines using ssh, set some data (like grains), copy salt files and apply the high state? The name for a backend with this logic would probably be called ssh.

Hmm, yeah, there must be something rather big I've missed here. I thought the intention was to use this module to do exactly that - I thought this was in the making, and we would add provisioner blocks to the null_resources as an additional step.
But possibly I am missing something - what would be the tradeoff in keeping both such an "ssh backend" and a really empty "null backend"?

What if we decide to add another backend that connects to existing machines using ssh, set some data (like grains), copy salt files and apply the high state? The name for a backend with this logic would probably be called ssh.

Hmm, yeah, there must be something rather big I've missed here. I thought the intention was to use this module to do exactly that - I thought this was in the making, and we would add provisioner blocks to the null_resources as an additional step.
But possibly I am missing something - what would be the tradeoff in keeping both such an "ssh backend" and a really empty "null backend"?

We can do it, but I was thinking in the module as a really empty "null backend", to allow us to verify the configurations we have in our root module before connecting to machines.
If you think this is pointless I can update this module with the needed provisioner and connection blocks and also add the provisioner specific properties, like ip_address and port.
What do you think?

As per our discussion, we will want something separate from the SSH backend, which will also come (just later).
This could be used for Travis validations, for example.
As far as the name goes, I could suggest "dummy", which is hopefully clear to users and developers alike. I would not document it in main user READMEs.
Another possible name would be "stub", but that's probably clear only to programmers.
I will leave the final choice to you in any case.


[Delta Lake Direct] Dataset in Databricks configured with Liquid Clustering returning no results when querying with no limit

Describe the bug
Reported in Discord:
I am trying to query data in Databricks via Spice, and got it to return data when using a limit. However, when I use a where clause I am getting errors. I am also running my machine out of RAM when I attempt to setup acceliration. The dataset I am testing on is a sync of data from Maxmind, where every IP address has a record. The query and error are:
sql> select * from table_name_here where ip = '<IP_ADDRESS>';
Error An internal error occurred while processing the query. Show technical details with '.error'
No data returned for query
sql> .error
Status { code: Internal, message: "Arrow error: External error: Parquet error: ParquetObjectReader::get_byte_ranges error: Generic S3 error: request or response body error: operation timed out", source: None }
Using a 200 file, 45.6 GiB dataset on Databricks with liquid clustering enabled.
Using acceleration in-memory leads to OOM error.  File based does sqlite or duckdb also doesn't succeed.

Also, the query above would return a single row based on the data in the table.

@ablyler did you consistenly getting the S3 generic error with where clause?
Underneath we are using delta-rs, so this issue might be related: https://github.com/delta-io/delta-rs/issues/2301.
I am going to test against our own databricks with similar data volume to see if it can reproduced.

@y-f-u yes, it happens consistently. Exposing the timeout is a great idea!

@ablyler Thanks again for reporting this issue!
I’ve investigated and couldn’t reproduce it using the latest version (v1.0.0-rc.1). Could you help confirm if it’s still occurring on your end? If it’s resolved, we’d like to close this issue. Please let us know if you’re still encountering any problems, thanks!

I am not sure, I don't have this environment set up at the moment. I say go ahead and close it and we can reopen if it happens again.

@ablyler Sounds great! I’ll go ahead and close this issue for now. Please let us know if the same issue happens in the future. Thanks again!


Use relative path for depfile

Close #6291.
This has been tested to work as expected on numpy, scipy and scikit-learn.
Suggestions more than welcome on how to add a test for this!

cc @da-woods maybe to approve the workflow to run and see whether that breaks any tests?

@da-woods could you trigger the workflow again, I think they are now passing locally on my machine.

cc @ev-br as the original author of https://github.com/cython/cython/pull/4563, in case you want to have a look at the diff.

I don't have anything of value to add, other than saying thank you for implementing this.

Thanks.


Crashlytics no longer bucketing correctly for builds to device

Description
I'm running into an issue where it looks like both fatal and non-fatals are not being bucketed correctly, but only when i build to a device. I have an app that references another dynamic framework of mine, where I'm calling fatalError("some message"). I also tried with non-fatals with the same effect. In the simulator, these crashes/nonfatals are logged correctly (see images below). However, with the same code, and same build settings, building to an actual device logs the issue as part of the main app as opposed to the dylib. This is causing everything to be bucketed together across my various modules, which is bad because I can't identify individual crashes easily from the dash.
I've heard there are issues with the Enable Debug dylib support setting but that's off for me.

Simulator
Device

Reproducing the issue
Create a vanilla SwiftUI app, and a dynamic framework. Have the app depend on the framework and call fatalError somewhere within the framework. On a simulator, crash stacktrace looks good and buckets correctly. On a device it buckets incorrectly and is all bucketed within App.main and greyed out stack traces to the actual crash
Firebase SDK Version
11.6.0
Xcode Version
16.1
Installation Method
Swift Package Manager
Firebase Product(s)
Crashlytics
Targeted Platforms
iOS
Relevant Log Output
No response
If using Swift Package Manager, the project's Package.resolved

Expand Package.resolved snippet

Replace this line with the contents of your Package.resolved.

If using CocoaPods, the project's Podfile.lock

Expand Podfile.lock snippet

Replace this line with the contents of your Podfile.lock!

Hello @lilidotshi could you provide a complete sample in a separate repository so I can reproduce this? Thanks

@Lyokone here you go: https://github.com/lilidotshi/CrashlyticsDemo
Copy your own google-service.plist file into ConfigurationFiles/development/, and when you run it, run the DemoApp-Debug scheme on an actual device. When the app launches, tap Show Menu and then Crash. This is what's coming up in firebase for me. You can see that where the crash actually happened is grayed out.

If I disable Enable Debug Dylib support, i still get this:

@Lyokone do you need any more info from me?


[v6] Template android app stuck in splash screen

Framework7 version: 6.0.14
Platform and Target: Android API Level 22 (same issue with 23)

Describe the bug
Output cordova android app stuck on splash screen due to javascript error in app.js:

Uncaught SyntaxError: Unexpected token )

To Reproduce
Steps to reproduce the behavior:
Prepare and run android emulator (or install and run apk)

Run framework7 create --ui
In the Create App page, select simple web app and cordova app under app type then click the Create App button at the bottom of the page.
Open a terminal to the app directory
Run npm run cordova-android (app should start in the emulator)

Expected behavior
App splash screen goes away after a few seconds and app continues to the home page.
Actual Behavior
App stuck in splash screen.
Screenshots

Additional context
App works on API Level 24 and above. Cordova v9 supports Android API 22-29, which is what this Framework7 template is configured to use.
I used the template above to show it is not an issue with the code I am working on.

Closing as not related to Framework7 itself. Not sure what Android version you test on, but F7 compatible with Android v7+

config.xml


added iPython support

Based on commit from pgcli to support Jupyter Notebooks.

FYI: https://www.pgcli.com/embedding-pgcli-in-ipython.html
The ipython support is not for Jupyter notebooks but for the ipython console repl. The blog post linked above talks about how to use it. I don't know how we can embed a CLI repl into a Jupyter notebook.

Closing this out given our current priorities.


Trying to refresh a UI Widget inside a Dashboard Widget

Hello Andy,
I'm trying to update the data bindings to a treegrid UI widget which is placed on a dashboard widget; however, I cannot get this treegrid widget to refresh ! In other words, the initial rendering of my treegrid is fine with my hard coded data; however it does NOT refresh after updating its data bindings from the server.
Here are the steps:

Launch the dashboard initially. The treegrid's hard coded data bindings render the grid sucessfully.
Launch the modal window from the dashboard widget. This brings up several configuration tabs which the user can choose from. Once saved, the parameters are submitted to the server, the return data is parsed in my controller. Finally I update the treegrid's datasource.
Now I attempt to update $scope.result.dataModel.treeGridOptions on the $modalInstance.close event; however I cannot figure out how to refresh the widget content (which should re-render my treegrid widget).

I'm wondering if this problem is more related to the widget ui library I'm using (i.e. I use both Kendo UI and jqWidgets), or if I'm totally missing something within your framework.
Ex/ Here's the my initial grid:

Ex/ Here's my modal window where i configure the grid widget, and update the grid's data binding on SAVE event

My widget def is defined here with the modal options template and controller :
    {
            name: 'chart_treegrid',
            title: 'Report Grid',
            templateUrl: 'app/shared/gadget-templates/tree-grid.html',
            dataModelType: TreeGridDataModel,
            settingsModalOptions: {   // for grid gadget config popup
                templateUrl: 'app/shared/include/grid-config/widget-config-grid.html',
                controller: 'GridSettingsCtrl as settings'
            }
        }

The GridSettingsCtrl controller code looks something like :
     (function () {
'use strict';
angular.module('rage')
   .controller('GridSettingsCtrl', ['$rootScope', '$scope', '$modalInstance', 'datacontext', 'gridHierarchyService', 'reportsContext', 'gadgetHelperFactory', 'widget', gridSettings]);

function gridSettings($rootScope, $scope, $modalInstance, datacontext, gridHierarchyService, reportsContext, gadgetHelperFactory, widget) {

    var settings = this;

   // add widget to scope
    $scope.widget = widget;

    // set up result object
    $scope.result = jQuery.extend(true, {}, widget);

    $scope.ok = function () {       // SAVE USERS SETTINGS !
        parseGadgetSettingsData();
        //$modalInstance.close($scope.result);  // moved below at end of grid processing.
    };

    $scope.cancel = function () {
        $modalInstance.dismiss('cancel');
    };

    function parseGadgetSettingsData() {
         // some code omitted for brevity

         var dataAdapter = new $.jqx.dataAdapter(myModel);  // WIRE UP NEW DATA ADAPTER !

        $scope.result.dataModel.treeGridOptions.source = dataAdapter;  // UPDATE RESULT SCOPE
        $scope.result.dataModel.treeGridOptions.columns = colDefs

        $modalInstance.close($scope.result);    // FIRE THE MODAL CLOSE EVENT

        // PRAY MY TREEGRID WIDGET REFRESHES IN MAIN WIDGET CONTAINER ! :-)
    }
  };   // end of gridSettings()
 })();

In this screen shot you can see my debug window in Chrome. I show $scope.result.dataModel.treeGridOptions , and I can indeed confirm that my data records are refreshed from the server API call.

Bottom line: why doesn't my treegrid widget get refreshed ?

it appears to be an issue related to the actual UI Gadget's redraw event.
I've managed to add the k-rebind attribute/directive to the Kendo UI div - and it refreshed just fine !
I'm now working on the jqWidgets redraw event.
This ticket might be a non-issue for Malhar.

I think i follow along, and I'm glad that it is outside the scope of our component. Next time I would do <pre>{{ dataModel.treeGridOptions | json }}</pre> to first check that the data is even getting there. Shall we close this one?

Yes close
thx
From:"Andy Perlitch"<EMAIL_ADDRESS>Date:Wed, Feb 11, 2015 at 11:56 AM
Subject:Re: [malhar-angular-dashboard] Trying to refresh a UI Widget inside a Dashboard Widget (#116)
I think i follow along, and I'm glad that it is outside the scope of our component. Next time I would do {{ dataModel.treeGridOptions | json }} to first check that the data is even getting there. Shall we close this one?
—
Reply to this email directly or view it on GitHub.￼


orchestra/run: add a method to write file

The method writes given data to a temporary file and then uses
put_file() to move the file to required remote machine on the given
path.

In Ceph PR#28692, I moved sudo_write_file() from test_volume_client.py to vstart_runner.py since it was generic enough and could be used by other testsuites too. However, a change like that can't pass through teuthology testing since a method with same name and functionality would be required in teuthology codebase too.

This method already exists:
https://github.com/ceph/teuthology/blob/52d8af779b377ace2b4f4d7d7353259a13ec194c/teuthology/misc.py#L524
It's also used in several places in qa/ already.

@batrick That's brilliant :-D But I thought @rishabh-d-dave wanted 'remote' object context method.
Btw, if someone still wants it, the misc method could just reuse the code of 'remote'.

This method already exists:
https://github.com/ceph/teuthology/blob/52d8af779b377ace2b4f4d7d7353259a13ec194c/teuthology/misc.py#L524
It's also used in several places in qa/ already.

Closing this PR since this method works for me.


Use snake case for attribute.

We convert all keys to snake case prior to working with them.

I have already tested it and it's working. Since it is just variable naming changes, let's merge this one.


WebGL seems unsupported on Raspberry Pi running 64-bit Raspberry Pi OS

While carbonyl claims to have webGL support, it seems to be disabled on hardware that otherwise supports it. I'm not sure if the VideoCore GPU is blacklisted since I can't access the flags page.

Looks like a regression, maybe a missing library, going to take a look at it. It currently works on macOS for what it's worth.

The fix just landed on the main branch! You can give it a try by downloading the latest binaries using CI builds, or using the next Docker image:
$ docker pull fathyb/carbonyl:next
$ docker run -ti fathyb/carbonyl:next


Resolves ssh config issue resolving host

SSH Config issues
Background
When configuring a ssh host in the .ssh/config with a entry to resolve github.com e.g.)
Host MY-GITHUB-ALIAS
   Hostname github.com
   IdentityFile my_authentication_identity

The extension fails to resolve the host name and displays the following error in the log
[Info] GitHubServer> No response from host https://MY-GITHUB-ALIAS/xxxxx/repo: getaddrinfo ENOTFOUND MY-GITHUB-ALIAS

Root cause
The ssh config file was ignoring the Hostname property in favour of HostName in protocol.ts. The man pages for ssh config define the entry a Hostname and the result was the wrong value was being assigned to host in parseSshProtocol().
Resolution
Align with the ssh config specification as defined by

SSH_CONFIG(5)

To avoid configuration brittleness Hostname or HostName will now resolve correctly.
Related Issues

Fixes #1541

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: Rusty-1 sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

Thank you for the review Rachel!
On Tue, 26 May 2020, 21:31 Rachel Macfarlane<EMAIL_ADDRESS>wrote:

@RMacfarlane approved this pull request.
Thank you!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/microsoft/vscode-pull-request-github/pull/1830#pullrequestreview-418614509,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABMUCCZDNSDBKC5YLUL5QWTRTQKIJANCNFSM4NIYZRNQ
.


1.2.16 isn't published to hex.pm

Hi @bitwalker
When 1.2.16 will be released to Hex.pm? There is a lovely feature for adding types to Structs definitions.
Also, if this is not possible, can I add type definition for proto struct definition?

I'll make a release today, sorry for the delay!

Released!


fixed double logging issue

Coverage decreased (-0.01%) to 95.157% when pulling 631b37dbc6e19b4e4ace5524727b8cc7fa991bfc on fix_double_logging into b4cf43c1fa1fe8350720c55ba2c33a8032c0476e on master.


How to use Windows ConPTY API from a process whose output has been redirected?

I am working on an application which creates a Windows pseudoconsole (using the ConPTY API, according to the official manual which is really good).
It starts a PowerShell session (for test), and shows everything ConPTY sends from that session via the pseudoconsole, and everything works really well. Until I redirect the parent program output to anywhere.
Here's my full code listing for reference.
#include <string>
#include <stdexcept>
#include <iostream>
#include <vector>

#include <Windows.h>

using namespace std::string_literals;

void throwLastError(int number)
{
	throw std::runtime_error("Error "s + std::to_string(number) + ": "s + std::to_string(GetLastError()));
}

STARTUPINFOEXW prepareStartupInformation(HPCON hPCon)
{
	STARTUPINFOEXW startupInfo{sizeof(STARTUPINFOEXW)};
	SIZE_T bytesRequired = 0;
	if (InitializeProcThreadAttributeList(nullptr, 1, 0, &bytesRequired))
		throw std::runtime_error("InitializeProcThreadAttributeList wasn't expected to succeed at that time.");

	const auto threadAttributeList = static_cast<LPPROC_THREAD_ATTRIBUTE_LIST>(calloc(bytesRequired, 1));
	startupInfo.lpAttributeList = threadAttributeList;

	if (!InitializeProcThreadAttributeList(threadAttributeList, 1, 0, &bytesRequired))
		throwLastError(4);

	if (!UpdateProcThreadAttribute(
			threadAttributeList,
			0,
			PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE,
			hPCon,
			sizeof(HPCON),
			nullptr,
			nullptr)
	)
		throwLastError(5);

	return startupInfo;
}

PROCESS_INFORMATION startProcess(STARTUPINFOEXW startupInfo, const std::wstring &commandLine)
{
	std::vector<wchar_t> cmdLineBuffer;
	for (auto x : commandLine)
		cmdLineBuffer.push_back(x);
	cmdLineBuffer.push_back(L'\0');

	PROCESS_INFORMATION processInfo{nullptr};
	if (!CreateProcessW(
		nullptr,
		cmdLineBuffer.data(),
		nullptr,
		nullptr,
		FALSE,
		EXTENDED_STARTUPINFO_PRESENT,
		nullptr,
		nullptr,
		&startupInfo.StartupInfo,
		&processInfo)
	)
		throwLastError(6);

	return processInfo;
}

int main()
{
	auto h = GetStdHandle(STD_OUTPUT_HANDLE);
	auto type = GetFileType(h);
	std::cout << "STD_OUTPUT_HANDLE " << h << " " << type << std::endl;

	try
	{
		HANDLE inPipeRead = nullptr, inPipeWrite = nullptr;
		if (!CreatePipe(&inPipeRead, &inPipeWrite, nullptr, 0))
			throwLastError(1);

		HANDLE outPipeRead = nullptr, outPipeWrite = nullptr;
		if (!CreatePipe(&outPipeRead, &outPipeWrite, nullptr, 0))
			throwLastError(2);

		HPCON hPCon = nullptr;
		if (CreatePseudoConsole(COORD { 80, 25 }, inPipeRead, outPipeWrite, 0, &hPCon) != S_OK)
			throwLastError(3);

		CloseHandle(inPipeRead);
		CloseHandle(outPipeWrite);

		auto startupInfo = prepareStartupInformation(hPCon);

		auto processInfo = startProcess(startupInfo, L"powershell.exe");

		std::cout << "PID: " << processInfo.dwProcessId << "\nTID: " << processInfo.dwThreadId << "\n";

		char buffer[1024];
		DWORD readBytes = 0;
		while (ReadFile(
			outPipeRead,
			buffer,
			sizeof buffer - 1,
			&readBytes,
			nullptr
		))
		{
			buffer[readBytes] = 0;
			std::cout << "Read bytes: " << readBytes << "\n";
			std::cout << buffer << "\n";
		}
	}
	catch (const std::runtime_error &x)
	{
		std::cerr << x.what() << "\n";
	}
}

So, if I start this program from terminal or from an IDE, it works (and tells how much data it has read from the PTY pipe). But if I start it with output redirected (e.g. myprogram.exe > C:\Temp\somefile.txt, or even myprogram.exe | out-host in pwsh), then it stops working: PowerShell then somehow inherits the stdin and stdout, doesn't write anything to the PTY pipe and uses the stdout/stdin instead.
For diagnostics, I've added output of GetFileType(GetStdHandle(STD_OUTPUT_HANDLE)), which helps to detect if the output has been redirected (since it's not always obvious in practice). For pristine stdout, it should write 2, and other numbers for other output modes.
How can I overcome this issue? Is it possible for my program to work even if its own stdout/stderr were redirected somewhere? I thought one of the points of PTY was to create a separate isolated environment, which won't interfere with the parent one.

I've been able to reproduce the same issue even on the test C# GUI application. Steps to reproduce:

Open GUIConsole.WPF.csproj, and replace <OutputType>WinExe</OutputType> with <OutputType>Exe</OutputType>, to make it console application with redirectable output:
https://github.com/microsoft/terminal/blob/0a7310dee4c36f1c423c58b958ffbdbb164ce140/samples/ConPTY/GUIConsole/GUIConsole.WPF/GUIConsole.WPF.csproj#L8
Compile it and start GUIConsole.WPF.exe, observe that it works well.
Start GUIConsole.WPF.exe > C:\Temp\file.txt, observe how it breaks (no console output, the window is empty).

Thanks for expressing appreciation for that document! You know, this is tickling a memory of handle inheritance causing us some significant trouble in the past.
While I go look for the change we made on the OS intended to make this better, could you try setting startupInfo.StartupInfo.dwFlags |= STARTF_USESTDHANDLES?
Enabling the use of hStdIn... et al but leaving them set to null should force the new process to not inherit the redirected default handles.

A-ha! We did do some work to improve handle inheritance, but only if the inherited handles were console handles (so: the spawned application would connect to its parent's stdin/stdout even if it was supposed to be using a pseudoconsole)
    //
    // If we are establishing a Pseudoterminal Connection, close
    // any in/out/err handles that may have been copied from the parent.
    // We will re-make new connections to the given PTY instead of
    // using the automatically inherited handles.
    //

    if ((ProcessParameters->ConsoleFlags & CONSOLE_USING_PTY_REFERENCE) != 0)
    {
        ConsoleCloseIfConsoleHandle(&ProcessParameters->StandardInput);
        ConsoleCloseIfConsoleHandle(&ProcessParameters->StandardOutput);
        ConsoleCloseIfConsoleHandle(&ProcessParameters->StandardError);
    }

We didn't do the same for all other inherited handles, so... I'm going to file a bug on the docs to make sure we document the need for STARTF_USESTDHANDLES.

We didn't do the same for all other inherited handles, so... I'm going to file a bug on the docs to make sure we document the need for STARTF_USESTDHANDLES.

It's not handle inheritance. CreateProcess is called with bInheritHandles as FALSE. What happens is that the standard handles get duplicated to the child, not inherited (i.e. the handle values are not the same). It's an old hack to make spawning console applications magically work when the parent's console is inherited but handle inheritance is disabled, and the standard handles aren't explicitly set. Specifically, this occurs in the following case:

the target executable is flagged as a console application
inheritance is disabled
the startup info doesn't include the flag STARTF_USESTDHANDLES
the creation flags do not specify CREATE_NEW_CONSOLE, CREATE_NO_WINDOW, or DETACHED_PROCESS

Maybe I'm missing something, but it seems to me that the pseudoconsole case is like the latter three scenarios (i.e. a new console or no console). If so, then the implementation of NtCreateUserProcess in the kernel should be updated to skip duplicating the parent's standard handles to the child, and to set the standard handle values in the child's ProcessParameters to NULL.

@DHowett, thanks for your answer!
Yes, your suggestion seems to work. I'll perform some additional experiments this week to see if it covers my use case, and let you know if there are any additional questions.
For now, I guess we could close the issue? Or would you like for it to stay open until the documentation changes are made?

Or would you like for it to stay open until the documentation changes are made?

That's a disappointing resolution. If a child process isn't getting spawned with the same console as the parent, the system simply shouldn't duplicate the parent's standard I/O handles to the child. It should work the same way as CREATE_NEW_CONSOLE, CREATE_NO_WINDOW, and DETACHED_PROCESS.


The 2D view is black in desktop browser when presenting

After going to the present mode with Oculus Rift, I will see the 2D view in browser is black. The image in HMD is normal. I have checked some demo as below, both of them have this problem after clicking the VR button.
https://googlevr.github.io/webvr-ui/examples/basic.html
https://googlevr.github.io/webvr-polyfill/examples/basic/
I also checked the result in https://threejs.org/examples/webvr_cubes.html from three.js r84 and r87. They are good. I suppose webvr-polyfill has some issues when using VREffect.js.

it can be resolved by
if  (isDesktop) {   renderer = new THREE.WebGLRenderer({antialias: true}); }
It seems like a specific issue from the desktop browser.


Update mcumgr library to fix wrong callback state

Describe the bug
mcumgr's library has callbacks which report the status of a firmware update with 3 states - start, ongoing and complete. However, complete is not used.
This is fixed in this PR to the upstream mynewt-mcumgr project code: https://github.com/apache/mynewt-mcumgr/pull/137
Therefore this is a request to pull the change into zephyr's version of mcumgr so that it will be included for Zephyr 2.7

Included here: https://github.com/zephyrproject-rtos/mcumgr/pull/43

@cfriedt This is low priority bug, so it will not go into the 2.7. Should we re-target it? Also the bug is in the mcumgr rather than in Zephyr, in common code, so is it needed to keep this as a bug here? It has already been fixed in mcumgr but we are not able to change west.yml to include the fix at this point.

@de-nordic - we should be able to pull into main. What is preventing the west.yml update?

@cfriedt The mcumgr updates come in snapshot updates. To bring this fix in we would need to bring in features/enhancements that ere not "blockers".
Alternative is to create mcumgr-zephyr-2.7-lts branch and start bringing the fixes there, and use this branch in west.yml for lts.
Do we want to start the mcumgr-zephyr-2.7-lts branch already? It may happen anyway.

Is it possible to get this included in a 2.7 backport, just the bug fix not the addition of features? We currently manually have this patched in a modified 2.6 branch and the idea was we would scrap it when moving to 2.7 because the fix would be included, but if the idea is this won't be fixed in 2.7 then that really doesn't sit well for our development

@cfriedt So this could go in: https://github.com/zephyrproject-rtos/zephyr/pull/39101 ?

@cfriedt So this could go in: #39101 ?

I think the issue there is that it would be tied to a PR rather than the main branch of mcumgr.
Would it make sense to merge zephyrproject-rtos/mcumgr#43 into the master branch of mcumgr and then update #39101 to use the resulting master commit?
Or if these are changes specific to v2.7 that have been cherry-picked, they will need to wait until after v2.7.0 unless they have "Blocker" status.
I'll bring this up in the release meeting in any case though.

@cfriedt So this could go in: #39101 ?

I think the issue there is that it would be tied to a PR rather than the main branch of mcumgr.
Would it make sense to merge zephyrproject-rtos/mcumgr#43 into the master branch of mcumgr and then update #39101 to use the resulting master commit?

@cfriedt  Yes I will merge it, need reviews from people, also can you join in. What I am asking is whether such set of changes, as done with the snapshot in that PR, is acceptable, and whether sha in west can be updated to point to it, when merged.
Review here https://github.com/zephyrproject-rtos/mcumgr/pull/43.
No cherry picking. this is normal snapshot update.

Fixed by #39658


Use cases for different storage methods?

I have a SvelteKit app that I need to persist what I see as two different kinds of user state...

Auth State: I am currently using an http only cookie to persist an authentication token and that seems to be working, though Im not sure about security issues of having that token visible in the browser cookies inspector.

User App State: I need to persist some user settings such as the current "space" or "page" they are using in the app, as they can have multiple workspaces and pages. I am not sure whether this would best be stored in one of the various local storage options, one of the various cookie type options, or something else etc.

Any suggestions?
Thanks!

Auth State: I am currently using an http only cookie to persist an authentication token and that seems to be working, though Im not sure about security issues of having that token visible in the browser cookies inspector.

For the authentication token, you have several options:

A cookie, that will be transparent for you, as they can easily be send and receive with HTTP request. (But keep in mind that this lib don't have (yet?) built-in storage for server cookie, meaning that you have to implement your own way to read cookie at server side)
An item in localStorage/sessionStorage: you will need to get the token from the authentication, store it. Then when you need to call you API/pages you will have to read the storage and pass the data to the request (in most of case through a custom header)

Concerning the security issue, it depend of the kind of token, a JWT token can have a signature that ensure the token is not forged, and signature can be user specific (instead of application specific).
But like any application, the cookie should not store clear personal and critical information, and like for any application if cookies are stolen, impersonation can be done.
If security concern is an import part of you application, you should set the TTL of the token short, and store it (at browser side) in temporary/short live storage, like sessionStorage or session cookies

2. User App State: I need to persist some user settings such as the current "space" or "page" they are using in the app, as they can have multiple workspaces and pages. I am not sure whether this would best be stored in one of the various local storage options, one of the various cookie type options, or something else etc.

Depending of the size of the data to store, cookie or browser storage can be used:

Cookie are not meant to store lots of data (as there are sent to every request's headers)
Browser storage only exist at browser side and the server don't know them (except if you explicitly setup your app to send them)

For the kind of storage, you have 2 kind:

temporary (browsing session)
"long term"

The temporary storage are:

sessionStorage (only in browser): when the browser is closed, everything in sessionStorage is deleted
Session cookie (available in browser* and server): when the browser is closed, all session cookies are removed

The "long term" storage are:

localStorage (only in browser): depending of the browser policies, this storage can have a maximum capacity per website/domain/etc. and maybe some purge strategy if the capacity is exceed.
cookie (available in browser* and server): The cookies have a limited size, there is not defined limit, but 4kB seem to by the most common limit
IndexedDB (only in browser): depending of the browser policies, this storage can have a maximum capacity per website/domain/etc. and maybe some purge strategy if the capacity is exceed.

*About cookies in browser: For cookie to be accessible in browser, the httponly flag need to be disabled. But it will expose the cookie to any JavaScript code and then potentially expose personal/critical data.

And for the mix use of storage type, you can do it. Further more you should do it if you have different kind of scope.
For example storing the user theme should be in localStorage (as it need to be keep between browsing session, but it's not the end of the world if the data is loose), but storing user last action/visited page, sessionStorage is more than enough as the information don't hold lots of meaning in future browsing.
There are lots of different use case, and storage depend on them. But you should not limit yourself to use only one storage as it can restrict features/functionality of one use case.

I hope that answers your questions, if not, don't hesitate to ask more :wink: