[SPARK-22767][SQL] use ctx.addReferenceObj in InSet and ScalaUDF

What changes were proposed in this pull request?
We should not operate on references directly in Expression.doGenCode, instead we should use the high-level API addReferenceObj.
How was this patch tested?
existing tests

cc @mgaido91 @viirya @kiszk

Test build #84826 has started for PR 19962 at commit b8c0689.

Test build #84826 has finished for PR 19962 at commit b8c0689.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84826/
Test PASSed.

Test build #84837 has started for PR 19962 at commit 3922ff4.

LGTM, pending Jenkins.

Test build #84837 has finished for PR 19962 at commit 3922ff4.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84837/
Test FAILed.

Merged build finished. Test FAILed.

retest this please

Test build #84843 has started for PR 19962 at commit 3922ff4.

LGTM, left only one minor comment

Test build #84843 has finished for PR 19962 at commit 3922ff4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84843/
Test PASSed.

Test build #84852 has started for PR 19962 at commit 2caef80.

LGTM, thanks

LGTM

Test build #84852 has finished for PR 19962 at commit 2caef80.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84852/
Test PASSed.

thanks, merging to master!


PolyBlock clarification

If polyblock a is connected to polyblock b, is polyblock b necessarily connected to polyblock a (or is that something we need to figure out ourselves)?

Yes, both blocks should have knowledge of the blocks they are connected to.


EVP_MD_meth_dup() on global EVP_MDs leaks memory in OpenSSL 3.0

When EVP_MD_meth_dup() is called on a global EVP_MD in OpenSSL 3.0, the origin of the newly allocated EVP_MD is initially set to EVP_ORIG_METH in EVP_MD_meth_new(), but is later overwritten with the origin of the source EVP_MD in memcpy(), which turns EVP_MD_meth_free() into a no-op.
The following excerpt works with OpenSSL 1.1.1, but leaks memory with OpenSSL 3.0:
#include <openssl/evp.h>
#include <stdlib.h>

int main(void) {
        EVP_MD *md = EVP_MD_meth_dup(EVP_sha256());
        if (md) EVP_MD_meth_free(md);
}

A potential fix might be to set origin back to EVP_ORIG_METH in EVP_MD_meth_dup() after the call to memcpy():
diff --git a/crypto/evp/evp_lib.c b/crypto/evp/evp_lib.c
index f78df52a..42331b0b 100644
--- a/crypto/evp/evp_lib.c
+++ b/crypto/evp/evp_lib.c
@@ -822,6 +822,7 @@ EVP_MD *EVP_MD_meth_dup(const EVP_MD *md)
         CRYPTO_RWLOCK *lock = to->lock;

         memcpy(to, md, sizeof(*to));
+        to->origin = EVP_ORIG_METH;
         to->lock = lock;
     }
     return to;

Fix in #16159.


Add API docs

Following our discussion in #1220, this PR adds API docs to the documentation. The API docs are auto-generated with mkdocstrings. We use the Google-style in docstrings to document parameters, return values, exceptions raised, etc..
Important note: (and I realize it's a bit weird to talk about this only now, sorry) the proposed mkdocstrings configuration uses options that are only available to Insiders. Since you maintain Python-Markdown, I will gladly give you access for free. The Insiders version is not published on PyPI, it has to be installed from GitHub, or built and stored in a private index. We provide a solution to do exactly this, PyPI Insiders, but there is a bit of setup to use it locally (see docs), so I understand if you do not want to do that. The alternative is to rely on the community version of mkdocstrings-python, and override it in CI with the Insiders one (if CI does indeed build and publish docs). Or we can do nothing of that and just use the community version. In that case, the options that won't be available are:

cross-refs in separate signatures (we don't use this one anyway since we render signatures in headings)
symbol types in headings and in the ToC (module, function, attribute, etc.)
auto-summaries

Apologies for not mentioning this earlier. It can feel like I had an agenda behind this, but really I was just focused on the technical side of things (making the docs look good) and I totally forgot to speak about Insiders :bow: Happy to hear your thoughts on this matter.

There are still some inconsistencies in docstrings (leading space in front of summary, trailing blank line at the end of the docstring). Imports are probably not ordered.
I've just based my PR on the branch as is, but I can of course rework commits to cleanly separate type changes (in function signatures) from docs changes.

Auto-summaries are the bullet point lists of submodules, classes, functions and attributes added automatically at the end of each docstring. It's not a big loss since these links also appear in the ToC.

I'm inclined to just go with the community version. However, if we leave the settings in place for the now unsupported features will that work, or will we get an error?

I do my best to always make insiders features compatible with the community version. Having these features enabled in the community version will not cause any error, the features will just do nothing.

I'm not sure what to think about this. I am annoyed this wasn't mentioned earlier. If it had been, I would have said no thanks.

Just wanted to note that there was no Insiders program in place when I opened the issue in February 2022 :slightly_smiling_face: I've only started it 5-6 months ago. Now working full-time on my own projects, and this is what allowed me to iterate quickly based on your feedback :slightly_smiling_face:

Ah, I missed your last comment when posting mine. Alright, I'll get to making tests pass in CI :slightly_smiling_face:

I've updated the templates in mkdocstrings-python to wrap parameter names (any object name really) in code spans, this way the spell checkers do not consider them. There were just a few words in docstrings that had to be corrected (indent, ints), and I replaced every occurrence of double backticks with single backticks (``thing`` -> `thing`). The last remaining warning, issued by mkdocs-section-index, will be fixed in its next release.

Moving out of draft since we're already reviewing the PR.

This is looking good. The only CI test failing is the check for a changelog entry. Adding an entry in docs/changelog.md will result in all tests passing. I know that we don't generally include changelog entries for changes which don't effect the behavior of the code, but this touches a lot of code (by adding type annotations) and a clear explanation should exist in the timeline of changes.
However, before I merge, I want to do a read-through of the generated documentation. I may be making a few changes to the various docstrings. Unless I see a repeating pattern, I expect I'll make the changes myself as I find them. Of course, my time is limited to I don't know how long this will take.

I want to do a read-through of the generated documentation

That is great, thanks. Take your time of course, there's no rush merging this :slightly_smiling_face: Definitely feel free to push directly on this branch again.

I've been looking at the generated site locally and it bugs me that reference/markdown/core/ and reference/markdown/ both contain docs for the exact same classes/methods/etc. In fact, in the code everything in markdown/core.py is imported into markdown/__init__.py and listed in __all__ in that module. I don't think there is any reason for anyone to ever import from markdown.core directly.
I wonder if we should just exclude core from the API docs. Any thoughts or observations about how others have handled this?

If everything from markdown.core is importable directly from markdown, then yes we could probably not render the markdown.core module in the API docs.
I'm not sure how others handle this. In some of my projects, some objects are documented twice because I expose some of them in the top-level module. It bugs me as well but I don't really mind it, as I allow users to import from both locations.

As I've been working through the various modules, something I have noticed is that the objects get ordered alphabetically. However, in the code, they are ordered logically. For example, in each pf the processor modules, the classes follow the same pattern. First the base class, then each of the classes in their order of precedence. Sometimes some of these classes even subclass other subclasses. With the logical order, it all fits together really well.
However, in the documentation, it feels like a jumbled mess. The base class is seemingly randomly in the middle and subclasses often are listed before their base class. Related objects are not near each other, being separated by unrelated objects. In the end, the relationships between objects is not as clear from reading the docs.
I realize that one way to address this is in how objects are named. But we have a legacy code case which many existing extensions already rely on. We can't be mixing everything up by renaming all our objects so that they get ordered correctly in the docs.
Therefore, I would much prefer if the order in the code was maintained in the docs. Is that possible?
Frustratingly, there does not seem to be any single location in mkdocstrings' documentation were all of the config options are listed. I'm hoping this is an option and I just haven't found where it is documented.

Since mkdocstrings is language agnostic, and Python is just one of the supported languages, each language handler is responsible for documenting its options. The docs for the Python handler options are here: https://mkdocstrings.github.io/python/usage/configuration/general/.
In particular, you're looking at members_order: https://mkdocstrings.github.io/python/usage/configuration/members/#members_order :)

These are actually labels, such as property, cached, writable, instance-attribute, etc.
If you don't want to show any label, it's probably possible to override the labels template and put nothing in it. It should be something like this:
docs/
    templates/
        python/
            nature/
                labels.html  # empty

# mkdocs.yml
plugins:
- mkdocstrings:
    custom_templates: docs/templates

So it appears that some things do not get carried over to the subclass from a parent class, while other things do. I'm not sure if this was intentional or an oversight. For example, fully documented methods defined in a parent class show up in every subclass. However, __init__ methods do not. Consider this simple example:
class Foo:
    """ Foo class """
    def __init__(self, foo: str):
        """
        Create a Foo instance

        Arguments:
            foo: A foo.
        """
        self.foo = foo

    def foo(self) -> str:
        """ Return foo """
        return self.foo

class Bar(Foo):
    """ Bar class """
    def bar(self) -> str:
        """ Return bar """
        return 'bar'

On the Foo class we get the type annotations and the list of arguments (circled in blue below). However, those are missing from the Bar class (as noted in red). However, the foo method is fully documented on both classes.

It's an oversight :slightly_smiling_face: The renderer currently doesn't look into inherited members for __init__ methods when merging them in the class docstring. Will fix!

I have gotten through all of the files. The inlinepatterns are a little incomplete, mostly because of the issue mentioned in the previous comment. So if that issue is fixed, that will help a lot.
I didn't spend a lot of time on the built-in extensions. Still missing a lot of type annotations. I'm not really convinced that they (extension docs) add much value. Maybe we do not include them???
I am curious why you decided to go with generated pages rather than injecting documentation into Markdown pages. I see that some config options could be unique per injected object. I'm thinking that might be desirable in some instances.

I just pushed a changelog entry. All tests should pass now. This is mostly ready to go. Ideally, we would see a fix for mkdocstrings/python#106 and mkdocstrings/python#110 before we publish the docs.
I also need to decide what we are going to do with the extension docs. Specifically, the API docs for each builtin extension. For the most part, I don't see much value in them. A couple have some potentially helpful functions which could be utilized by other extensions, but for the most part, looking at the source code is going to be more helpful to potential extension devs.  And then there is the weirdness of the Extension subclass for every extension repeating all of the methods of the base class. While I generally prefer that behavior and want it to remain elsewhere in the API docs, in the case of the Extension class, I would prefer the base class methods to not be repeated. It makes it less clear (at a glance) what specifically has been defined by the extension (a label which identified a method/attribute/etc as being inherited might help here). If we could define a different set of options for the extensions, that might be helpful. Otherwise, I'm inclined to remove them from the documentation.
Finally, It would be nice if there was a label for deprecated objects. Sure, we can include a note in the docstring (even wrapping it in an admonition), but a mkdocstring label seems like an appropriate thing as well. The issue, I suppose, is that there is no standard way of marking an object as deprecated in Python. In our case, we provide our own decorator. Ideally, any object which is wrapped with that decorator would get a deprecated label (see an example here).

I'm not really convinced that they (extension docs) add much value

Having worked on mkdocstrins and markdown-exec and their ability to report headings from nested Markdown instances to upper layers in the conversion stack, I'd say that any API doc for extensions like toc are super, super welcome. Yes, we can read the code, and with type annotations it will be even more helpful, but I really believe having everything in the same place (API docs) provides additional value.

I am curious why you decided to go with generated pages rather than injecting documentation into Markdown pages.

I went with generated pages because I absolutely do not have your knowledge regarding Python-Markdown's code base, so wouldn't have been able to do what you did here (added/improved docstrings), but yes we can definitely switch to manual injection if you prefer that.

I would prefer the base class methods to not be repeated.

With manual injection (or additional logic in the pages generation script), it's possible to exclude members from the rendered docs.

inherited labels

I'll think about it. This would be a nice addition indeed.

deprecated labels

Definitely possible with a Griffe extension! I'll check the sources and come up with a working example :)

I'd say that any API doc for extensions like toc are super, super welcome.

I find it interesting that you called out toc specifically, as that is the most obvious one among 2 or 3. For the rest, though, I think my point stands. Although, it doesn't make sense to only document some and not others.

With manual injection (or additional logic in the pages generation script), it's possible to exclude members from the rendered docs.

However it is accomplished, I would like to see inherited members excluded from the Extension subclasses. Possibly, we could exclude inherited members from everything under markdown/extensions/.

yes we can definitely switch to manual injection if you prefer that.

Back in the beginning I would have said absolutely yes.  Now that we have everything working, it seems like a lot of work for no immediate gain. The one thing that I keep coming back to though, is that future edits (or even reorganization) would require less understanding of the innerworkings of mkdocstrings if we were using manual injection. As it stands now, we have a custom script which uses an API I have not seen any documentation for.

For the rest, though, I think my point stands.

I don't have a strong opinion on this, you know best!

custom script

This script is a modified version of the one we can find in mkdocstrings' docs: https://mkdocstrings.github.io/recipes/#automatic-code-reference-pages.
It uses mkdocs-gen-files, mkdocs-literate-nav and file paths manipulation to achieve auto-generation of pages and navigation.
It sure makes things less straight-forward and a bit harder to understand.

Griffe 0.36.4 is live. You'll now see two deprecation warnings in the UnescapePostprocessor docs, one coming from the Griffe extension, and one coming from the docstring itself. Feel free to remove the dynamically inserted one by removing the self._insert_message(cls, message) lines in the extension code.

Awesome. Those changes helped a lot. With the Extension class not showing all of the inherited methods, it is much cleaner and easier to read.
Which leads me to a new issue. Various extensions have default config options defined. It would seem that the API docs should include those so I added a docstring for each. As a reminder, configs is a class attribute. so I added a docstring and got this:

That is not easy to read. I realize we can also define class attributes with an Attributes section in the class docstring. But that doesn't show the values assigned. I'm not sure of how to address this.

I think perhaps the last issue I raised would be addressed if attributes got the behavior of separate_signature: true.  Maybe that option should not apply to attributes and they should always be separate??

Although, that isn't perfect as we don't get a code block with preserved newlines.

If you install Black in the environment, mkdocstrings will use it to format the attribute value.
It's not currently possible to render separate signatures for attributes only. I'll think about it. There might be a way to hide the value, but I'm not sure you'd want that.

If you install Black in the environment, mkdocstrings will use it to format the attribute value.

But it is already formatted the way I want it. mkdocstrings is failing to wrap the code in a pre and so the formatting is lost when rendered in a browser. I don't see how Black can fix that.
To b clear, I understand that we don't want a pre tag nested in a h1-6 tag. However, when the signature is separate from the header, then the signature should be a normal code block. Specifically, it should be wrapped in a pre tag so that whitespace is preserved. I haven't checked, but is there a separate template we could override for separate signatures?

It's not currently possible to render separate signatures for attributes only. I'll think about it.

That is very unfortunate, They look so very different from other objects that I don't see how anyone would consider it okay to render them the same.
I'm thinking maybe we do need to switch to injected documentation. Then, rather than rendering a whole module, we can render individual objects and define custom settings for each.

Turns out it was easy to make attributes always have a separate signature. I just overrid the relevant template blocks and removed the if config.separate_signature check (and the else block`) from each (see 13e6254). Much better IMO.
The problem remains that the signature is all on one line and the formatting in the source code is not preserved. I have taken a closer look and see that the newlines are not preserved in the rendered HTML so the issue is not a missing pre tag (actually the pre tag is present). I also tried installing Black as I see the template filter actually calls Black if it is installed. However, that made no difference. The signature for each attribute is still a single line.

As you noticed, there's always a <pre> tag. mkdocstrings doesn't use the source directly, it uses a custom AST built from the source to be able to render cross-references within code blocks (function signatures, attribute values), as well as allowing configuration options (whether to show type annotations, etc.).
Glad you found a solution with overridden templates, however note that by displaying the source directly, you lose the cross-references functionality (which will become free when we reach the $500/month goal).

however note that by displaying the source directly, you lose the cross-references functionality

If there is a way to retain formatting and have cross-references, then that would be great. However, if I only get to have one, I'll take formatting. The cross references do little to no good when the code is unreadable.

I know I broke things. Ran out of time. I will fix it when I have more time.

I think this is almost ready to go. I need to take another pass at inlinepatterns.py (which has a lot of stuff). Other than that, I am happy with it now.
@facelessuser could you take a look at the docstring for EmStringItem namedtuple. It's not clear to me how to document that better.
https://github.com/Python-Markdown/markdown/blob/5a2fee340845ee5a70f6570c35b53fd4a37ade1c/markdown/inlinepatterns.py#L184-L185
This is what it currently looks like in the documentation.

Or maybe we need to document its use on the processors which make use of it. Not sure how to proceed here.

@facelessuser could you take a look at the docstring for EmStringItem namedtuple. It's not clear to me how to document that better.

I'll try to take a look sometime this week.

Something that we need to document for extension devs is the collections of processors and the specific name and precedence given to each so that an extension can properly insert itself in the proper location. Thus far, extension devs have had to read the source. But all of the relevant info is in the various build_* functions:

markdown.preprocessors.build_preprocessors
markdown.blockprocessors.build_block_parser
markdown.treeprocessors.build_treeprocessors
markdown.inlinepatterns.build_inlinepatterns
markdown.postprocessors.build_postprocessors

There are a few ways we could document this.

Use the show_source feature of mkdocstrings. Each of the functions only contains the relevant info so there is no info overload. But I'd rather not have show_source enabled for everything; only these specific functions. Not sure how to do that. Maybe there is a way to define custom options for specific objects??
Insert a table in the docstring for each of the functions. However, a manually edited table could become out-of-sync with the code if we are not careful. Therefore, I would prefer an automatically generated table. I'm assuming some sort of plugin could do this. But I'm not sure where to start.

I'm thinking I would prefer the second option, but am open to the first if its an easy fix.

I'm not sure exactly what you would like the table to look like, but I suppose it would be possible to build it dynamically, at build time, using markdown-exec or similar extensions/plugins:
```python exec="on"
# import all processors and sort them by priority
# then output a markdown table
print("Priority | Processor")
print("--- | ---")
for priority, processor in processors:
    print(f"{priority} | {processor}")
```

I believe this is sufficiently complete to merge. We can always tweak various things in future PRs.
Thanks to @pawamoy for getting this going and following through over many months. Without all your work (and the mkdocstrings plugin to make it possible) this never would have happened.

Very nice last commit, adding link to the source on GitHub :+1:
You're welcome, thanks to you @waylan for your valuable feedback, we improved and fixed many things in mkdocstrings thanks to it :slightly_smiling_face: I enjoyed working on this PR with you!

Ha, too bad the docs deployment job failed!

Ignoring the auth key problem with the automated deploy, there is still an issue.
We deploy using MkDocs' organization setup, which means deploying from a directory other than the root directory of the project (see linked docs for details). However, trying to build results in a failure:
λ mkdocs build --config-file ../md/mkdocs.yml
INFO    -  Cleaning site directory
INFO    -  Building documentation to directory: C:\code\md\site
ERROR   -  Error reading page 'reference/markdown/index.md': Extension module
           'scripts/griffe_extensions.py' could not be found
Traceback (most recent call last):
  File "c:\code\md\venv\lib\site-packages\griffe\extensions\base.py", line 415, in _load_extension
    ext_object = dynamic_import(import_path)
  File "c:\code\md\venv\lib\site-packages\griffe\importer.py", line 61, in dynamic_import
    module = import_module(module_path)
  File "C:\Users\wlimberg\AppData\Local\Programs\Python\Python38\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'scripts/griffe_extensions'

It seems that extensions paths are not relative to the config file like they should be. Presumably this would be a bug in mkdocstrings. It should be altering the path to be absolute based on the location of the config file before passing it on to griffe.
As a reminder, MkDocs validates its own known config options and ensures that paths are properly adjusted. However, as this is a config setting for a plugin, MkDocs doesn't know about it and therefore doesn't validate or adjust it. That needs to be handled by the plugin.
Until this bug is fixed, we can't deploy the documentation. I would file a bug, but I'm not sure if this goes in mkdocsstring, mkdocstring-python or elsewhere.

Ah, I see, thanks for the explanation. I wonder if MkDocs 1.5's new feature would solve this issue:
plugins:
- mkdocstrings:
    handlers:
      python:
        options:
          extensions:
          - !relative $config_dir/scripts/griffe_extensions.py

That might provide a workaround. Although I would encourage you to fix the issue within your plugin and not rely on that feature. That feature needs to exist for Markdown extensions which don't have any knowledge of MkDocs. However, a MkDocs plugin certainly has knowledge of MkDocs and should work out-of-the-box with any supported configuration and/or environment.

I mostly agree. However, now that this feature exists, I wonder if it is worth it to add code in mkdocstrings-python to transform the extensions path as relative to the config file directory. You know, the less code to maintain, the better. We should definitely document this though. I'll think a bit more about it :slightly_smiling_face:

I would consider relying on that feature to be a bug. In fact, I would have objected to the feature being added if it wasn't for the fact that there is no workaround for Markdown extensions. I might have argued for the feature to only be available to Markdown extensions within the config file if I had known it was being considered before release.

That represents a lot of boilerplate code in every plugin using file paths, recording MkDocs config file path in on_config, passing it down the call chain to the objects that need it to build relative paths. Many different implementations, many missing ones, probably buggy ones. Could be alleviated with a guide on how to implement paths so that they are relative to the config file, sure. Instead of this guide we have the !relative feature, and I think that's just as good, if not better. Users have to use it anyway for Markdown extensions, so it doesn't cost them that much (one line changes) to do the same thing for plugins. And they only have to do it if they need it. Most users have no issues relying on the current working directory. Those who want their docs to work with -f can use !relative.

One annoying aspect is that projects like ReadTheDocs have to implement support for such custom tags, since they don't want to use MkDocs' loader to avoid running arbitrary Python code.

Well I strongly disagree. I spent a lot of time working on MkDocs config validation framework and any plugin can (and probably should) use that framework (as documented here) to address this and many other potential issues with configs. In other words, you can reuse the same validator that MkDocs uses for path based config options and get the correct expected behavior for free (see FilesystemObject and its various subclasses). But, anyway, this is getting off-topic.
Back to the issue at hand... I tried using the !relative feature so I could at least get the current version deployed and it doesn't work. I can build my documentation just fine until I add the !relative tag to the config. With that one change, I get the error: Error reading page 'reference/markdown/index.md': cannot pickle '_thread.RLock' object. I'm not sure what exactly is going on here, but the error specifically relates to the first of the mkdocstrings generated pages. It could be a bug with mkdocstrings, a bug with MkDocs, or a bug with one of the extensions. The full traceback follows...
% mkdocs build
INFO    -  Cleaning site directory
INFO    -  Building documentation to directory: /Users/waylan/code/md/site
ERROR   -  Error reading page 'reference/markdown/index.md': cannot pickle
           '_thread.RLock' object
Traceback (most recent call last):
  File "/Users/waylan/code/md/venv/bin/mkdocs", line 8, in <module>
    sys.exit(cli())
  File "/Users/waylan/code/md/venv/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/waylan/code/md/venv/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/waylan/code/md/venv/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/waylan/code/md/venv/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/waylan/code/md/venv/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/waylan/code/md/venv/lib/python3.9/site-packages/mkdocs/__main__.py", line 286, in build_command
    build.build(cfg, dirty=not clean)
  File "/Users/waylan/code/md/venv/lib/python3.9/site-packages/mkdocs/commands/build.py", line 322, in build
    _populate_page(file.page, config, files, dirty)
  File "/Users/waylan/code/md/venv/lib/python3.9/site-packages/mkdocs/commands/build.py", line 175, in _populate_page
    page.render(config, files)
  File "/Users/waylan/code/md/venv/lib/python3.9/site-packages/mkdocs/structure/pages.py", line 271, in render
    self.content = md.convert(self.markdown)
  File "/Users/waylan/code/md/markdown/core.py", line 353, in convert
    root = self.parser.parseDocument(self.lines).getroot()
  File "/Users/waylan/code/md/markdown/blockparser.py", line 116, in parseDocument
    self.parseChunk(self.root, '\n'.join(lines))
  File "/Users/waylan/code/md/markdown/blockparser.py", line 135, in parseChunk
    self.parseBlocks(parent, text.split('\n\n'))
  File "/Users/waylan/code/md/markdown/blockparser.py", line 157, in parseBlocks
    if processor.run(parent, blocks) is not False:
  File "/Users/waylan/code/md/venv/lib/python3.9/site-packages/mkdocstrings/extension.py", line 125, in run
    html, handler, data = self._process_block(identifier, block, heading_level)
  File "/Users/waylan/code/md/venv/lib/python3.9/site-packages/mkdocstrings/extension.py", line 207, in _process_block
    data: CollectorItem = handler.collect(identifier, options)
  File "/Users/waylan/code/md/venv/lib/python3.9/site-packages/mkdocstrings_handlers/python/handler.py", line 264, in collect
    mutable_config = dict(copy.deepcopy(config))
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 205, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 205, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 205, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 237, in _deepcopy_method
    return type(x)(x.__func__, deepcopy(x.__self__, memo))
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/copy.py", line 161, in deepcopy
    rv = reductor(4)
TypeError: cannot pickle '_thread.RLock' object

Thanks for trying, looks like a bug in mkdocstrings. I'll investigate and fix it ASAP.
About the config validation framework: it's indeed great. However in the case of mkdocstrings and its handlers, it's not as straight-forward to use it.
The thing is, each handler can declare its own configuration. Currently they declare it as simple dicts with almost no validation (not great, I know, though it allowed you to add your own source config item). Let say we try to re-declare it using the config validation framework. The issue is that, each handler is optional, and is lazily loaded. It means that, if we want to load a handler's config to graft it onto mkdocstrings' own config, we have to implement a handlers discovery mechanism, and somehow merge everything together at import time. It's not that hard to implement, but it would take time to design and implement properly.
Even then, the Python handler passes down options from docstring_options directly to Griffe. That includes extensions. If we start validating these options, this will create much more maintenance work, as new Griffe options won't immediately be usable by mkdocstrings-python, and will require a new release of the handler each time there's a new/modifed/removed option... unless we have a mix of validation for known items, and no validation (and no errors) for unknown ones. But you can see how all this is not trivial, and will require a lot of work before we get it right. !relative shaves off some of that work (once we fix the bug you discovered), so I believe you can understand how attractive it can be.

I got the docs deployed, but there appears to be a bug in the generator script. Nothing in the extensions subdirectory is included. Presumably another relative path issue.

I got the documentation deployed using the following:
mkdocs build
cd ../Python-Markdown.github.io
ghp-import -p -b master ../md/site/

Notice that I had to use mkdocs build from the root dir of the Markdown repo, which avoids the bugs in mkdocstring. Then I changed to the Pages repo and ran ghp-import directly (the command that mkdocs gh-deploy` wraps), pointing it at the dir that contains the built site.

custom templates: indeed, not relative to config file dir, we have an issue opened already, have to fix
extensions modules not appearing: indeed, the script that generates those relies on the cwd, just opened https://github.com/mkdocstrings/mkdocstrings/issues/625


[pyhanko-certvalidator] Ability to skip nonce validation in OCSP response

Issue
One of the CAs I'm working with currently has an incorrect implementation of the nonce extension for OCSP. They are responding with the nonce value, but it's always the same. I suspect this might be because the response is cached on the CA's side. This leads to OCSPValidationError being raised in certvalidator. There are also some issues related to this in the wbond/certvalidator repo.
Proposed Solution
Add an option in the ValidationContext to disable nonce extension entirely. If this sounds okay, I can make a Pull Request for the same :)

Some additional notes:

AIOHttpOCSPFetcher and RequestsOCSPFetcher already have a flag called request_nonces to disable nonce extension, but this is always True when the fetchers are initialised.
This error currently causes the whole process to fail instead of falling back to CRL fetching.

Yes, that's by design (as it is intended to the default). You can, however, initialise (and resource-manage) your own fetchers and pass those to the validation context. Does that work for you?

Thanks for the idea. I did something simpler:
validation_context = ValidationContext(allow_fetching=True)
validation_context.revinfo_manager._fetchers.ocsp_fetcher.request_nonces = False

I'm modifying an internal attribute, but it seems like a more maintainable solution than resource-managing my own fetchers.

That's definitely a bug. Can you post a sample stack trace here?

Traceback (most recent call last):
  File "certvalidator/pyhanko_certvalidator/fetchers/common_utils.py", line 237, in queue_fetch_task
    wait_event: asyncio.Event = running_jobs[tag]
KeyError: (b'\xef\x9 --- trimmed ---'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "pyHanko/pyhanko/cli/runtime.py", line 50, in pyhanko_exception_manager
    yield
  File "pyHanko/pyhanko/cli/commands/signing/plugin.py", line 132, in _callback
    _callback_logic(plugin, infile, outfile, **kwargs)
  File "pyHanko/pyhanko/cli/commands/signing/plugin.py", line 111, in _callback_logic
    result = PdfSigner(
  File "pyHanko/pyhanko/sign/signers/pdf_signer.py", line 1423, in sign_pdf
    result = asyncio.run(
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "pyHanko/pyhanko/sign/signers/pdf_signer.py", line 1485, in async_sign_pdf
    validation_info = await signing_session.perform_presign_validation(
  File "pyHanko/pyhanko/sign/signers/pdf_signer.py", line 1654, in perform_presign_validation
    signer_path = await self._perform_presign_signer_validation(
  File "pyHanko/pyhanko/sign/signers/pdf_signer.py", line 1753, in _perform_presign_signer_validation
    signer_cert_validation_path = await validator.async_validate_usage(
  File "certvalidator/pyhanko_certvalidator/__init__.py", line 269, in async_validate_usage
    validated_path = await self.async_validate_path()
  File "certvalidator/pyhanko_certvalidator/__init__.py", line 136, in async_validate_path
    self._path = candidate_path = await find_valid_path(
  File "certvalidator/pyhanko_certvalidator/__init__.py", line 36, in find_valid_path
    await async_validate_path(
  File "certvalidator/pyhanko_certvalidator/validate.py", line 145, in async_validate_path
    return await intl_validate_path(
  File "certvalidator/pyhanko_certvalidator/validate.py", line 1111, in intl_validate_path
    await _check_revocation(
  File "certvalidator/pyhanko_certvalidator/validate.py", line 1298, in _check_revocation
    await verify_ocsp_response(
  File "certvalidator/pyhanko_certvalidator/revinfo/validate_ocsp.py", line 547, in verify_ocsp_response
    await validation_context.revinfo_manager.async_retrieve_ocsps(
  File "certvalidator/pyhanko_certvalidator/revinfo/manager.py", line 214, in async_retrieve_ocsps
    ocsp_response_data = await fetchers.ocsp_fetcher.fetch(
  File "certvalidator/pyhanko_certvalidator/fetchers/requests_fetchers/ocsp_client.py", line 44, in fetch
    return await self._perform_fetch(
  File "certvalidator/pyhanko_certvalidator/fetchers/requests_fetchers/util.py", line 36, in _perform_fetch
    return await queue_fetch_task(
  File "certvalidator/pyhanko_certvalidator/fetchers/common_utils.py", line 262, in queue_fetch_task
    return _return_or_raise(result)
  File "certvalidator/pyhanko_certvalidator/fetchers/common_utils.py", line 267, in _return_or_raise
    raise result
  File "certvalidator/pyhanko_certvalidator/fetchers/common_utils.py", line 251, in queue_fetch_task
    result = await async_fun()
  File "certvalidator/pyhanko_certvalidator/fetchers/requests_fetchers/ocsp_client.py", line 89, in _fetch
    ocsp_response = await ocsp_job_get_earliest(
  File "certvalidator/pyhanko_certvalidator/fetchers/common_utils.py", line 303, in ocsp_job_get_earliest
    ocsp_resp = await ocsp_job
  File "certvalidator/pyhanko_certvalidator/fetchers/requests_fetchers/ocsp_client.py", line 57, in _fetch_single
    return process_ocsp_response_data(
  File "certvalidator/pyhanko_certvalidator/fetchers/common_utils.py", line 205, in process_ocsp_response_data
    raise errors.OCSPValidationError(
pyhanko_certvalidator.errors.OCSPValidationError: Unable to verify OCSP response since the request and response nonces do not match
Error: Generic processing error.


behavior is not very correct on web

I have gone through these following points
ALL DONE
Issue Description

Open the https://github.com/GeekyAnts/NativeBase#6-nativebase-for-web from the README.md , use Chrome. and then:
PHONE MODE

WEB MODE

Expected behaviour

Label animates upward when input is selected and animates downward when input is erased. -- from: https://docs.nativebase.io/Components.html#inline-label-headref

Actual behaviour
i cannot input in phone mode, and in web mode the placeholder label can block input

From demo

I dont get why phone mode here

hey @SupriyaKalghatgi , dont you see the WEB MODE?
dont you have a phone? dont you access web site from phone?
why you think i cheat you.ok, well, nothing. just close it.


AttributeError: module 'pandas.compat' has no attribute 'string_types'

today I got this error:
AttributeError: module 'pandas.compat' has no attribute 'string_types'
and the code is here:
https://github.com/python-streamz/streamz/blob/f9b4ce38fc8251c3807387e37e935e157e07d0df/streamz/dataframe/core.py#L248
the problem is , there's no pd.compat.string_types now.
pandas removed it at Mar 29, 2019, see here:
https://github.com/pandas-dev/pandas/pull/25903

Missed this depr, thanks for posting.
Should be changed to
if (isinstance(c, str) and c.isidentifier()))

Would you care to submit this fix as a PR?

(also a test which uses dir() on a dataframe would be useful, as this should have been caught before)


Icons alignment to use NiaIcons

Changes

Changed icons used to use Nia Design System defined icons.
Replaced ArrowBack from Rounded to Filled variant as Rounded was not used, but Filled was.

Hi,
The android tests are passing locally, but failing in CI.
Can anyone please help me understand how to fix it?

Please could you fix the merge conflicts.

@dturner Done.

Thank you for your contribution!


[Event Request] Codeunit 5752 Get Source Doc. Outbound

The following events are required for this object:
    [IntegrationEvent(false, false)]
    local procedure OnGetOutboundDocsBeforeUpdateShipmentHeaderStatus(var WhseShptHeader: Record "Warehouse Shipment Header")
    begin
    end;

    procedure GetOutboundDocs(var WhseShptHeader: Record "Warehouse Shipment Header")
    var
        WhseGetSourceFilterRec: Record "Warehouse Source Filter";
        WhseSourceFilterSelection: Page "Filters to Get Source Docs.";
    begin
        WhseShptHeader.Find;
        WhseSourceFilterSelection.SetOneCreatedShptHeader(WhseShptHeader);
        WhseGetSourceFilterRec.FilterGroup(2);
        WhseGetSourceFilterRec.SetRange(Type, WhseGetSourceFilterRec.Type::Outbound);
        WhseGetSourceFilterRec.FilterGroup(0);
        WhseSourceFilterSelection.SetTableView(WhseGetSourceFilterRec);
        WhseSourceFilterSelection.RunModal();
        OnGetOutboundDocsBeforeUpdateShipmentHeaderStatus(WhseShptHeader); //EVENT REQUEST
        UpdateShipmentHeaderStatus(WhseShptHeader);

        OnAfterGetOutboundDocs(WhseShptHeader);
    end;

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: 12439.


Get Started: Next level

[x] Change PR target to master when #1382 is merged.
It's called "Getting Started" for now because of #1406 but either that gets fixed or I just remove the tutorials/get-started and rename this new one eventually.

You may disregard these recommendations if you used the Edit on GitHub button from dvc.org to improve a doc in place.

❗ Please read the guidelines in the Contributing to the Documentation list if you make any substantial changes to the documentation or JS engine.
🐛 Please make sure to mention Fix #issue (if applicable) in the description of the PR. This causes GitHub to close it automatically when the PR is merged.
Please choose to allow us to edit your branch when creating the PR.
Thank you for the contribution - we'll try to review it as soon as possible. 🙏

WDYT of the new intro so far @shcheklein ?
See https://dvc-landing-get-started-lnjjgt.herokuapp.com/doc/getting-started

It's called "Getting Started" for now because of #1406 but either that gets fixed or I just remove the tutorials/get-started and rename this new one eventually.

Actually this may be tricky because we also have a 301 redirect now from ^/doc/get-started(/.*)? to /doc/tutorials/get-started$1 🙁 so should we keep the "Getting Started" name for the new new version, @shcheklein ?

I don't understand the problem to be honest - we just move it outside and that's it. Change redirects if needed. Let's not spend time on "Next get started" at all for now as agreed.
Until I merge changes to the existing one with example get started migrated to DVC 1.0.

Sure, I'm not working on this. I was doing the new DVC files&dirs guide: I added a redirect for it, and realized we have the old Get Started redirect, so I wanted to leave a comment here for future reference.

I don't understand the problem ... Change redirects if needed

Not easy to change a 301 redirect: browsers that have visited it before cache it for an undefined length of time 😕

Closing as stale for now.


Replace deprecated action product/list call with findProducts

product/list is deprecated replace it's occurrence with product/findProducts

PR has been merged, What is  process to close the task?


add integration tests that use the distribution artifact

We should have some simple integration smoke tests that rely on the actual artifact we expect end users to use.
For systems that publish a single node "test cluster" artifact, this will hopefully be simple. For others, I'm open to brainstorming once we've identified them.

I have been thinking about how to test YCSB with Docker. Currently have a POC here:
https://github.com/risdenk/ycsb-testing
I need to update the README but it will take a release version and binding to run the tests. There needs to be some more parameterization of the run_test script. The tests are all run serially but could be parallelized if we wanted.
Travis is currently running the tests:
https://travis-ci.org/risdenk/ycsb-testing


The ServiceInfo field does not exist in Azure cost details fields.  Perhaps it should be ServiceInfo1?

My best guess is that on 20th May 2022, someone did a search and replace of the number 1 and made it a superscript for the key fields in
https://github.com/MicrosoftDocs/azure-docs/commit/87acd2bd0b2faebe5983d78da1c81386b85704cd
This has resulted in:
ServiceInfo¹
when in fact there is a column "ServiceInfo1" in the cost details.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: d79af6b6-535b-fe98-2bf4-670b0cc1e71f
Version Independent ID: e75818d6-3f51-dd56-04b4-a92a523906d5
Content: Understand usage details fields - Microsoft Cost Management
Content Source: articles/cost-management-billing/automate/understand-usage-details-fields.md
Service: cost-management-billing
Sub-service: cost-management
GitHub Login: @bandersmsft
Microsoft Alias: banders

@drtimf
Thanks for your feedback! We will investigate and update as appropriate.


vagg19_net KeyError normalization at matconvnet.py

i am trying to run this code. and i completed setting the requirement( Deeppy, cuDnn, choose imagenet-vgg-verydeep-19), vgg-verydeep-19 file is located at this project directory.
but i get error like below.
Traceback (most recent call last):
File "neural_artistic_style.py", line 139, in
run()
File "neural_artistic_style.py", line 99, in run
layers, img_mean = vgg19_net(args.vgg19, pool_method=args.pool_method)
File "/home/ubuntu/LeeHD/neural_artistic_style-master/neural_artistic_style-master/matconvnet.py", line 35, in vgg19_net
img_mean = matconvnet['normalization'][0][0][0].astype(dp.float_)
KeyError: 'normalization'
what should i do to fix this error?

MatConvNet recently switched over from beta16 to beta17 and changed their pre-trained models accordingly. Neural-Artistic-Style apparently only works with the beta16 format, which you can download from here

Hey, thank you both for the info. I have now switched to the new file format. Hopefully it won't change anytime soon. :)

Hey, I got the same issue and when I download new imagenet-vgg-verydeep-19.mat from beta16 then I got following problem
Traceback (most recent call last):
  File "neural_artistic_style.py", line 139, in <module>
    run()
  File "neural_artistic_style.py", line 117, in run
    args.smoothness)
  File "/Users/yaroslav/Documents/NaUKMA/NeuronArtist/neural_artistic_style/style_network.py", line 88, in __init__
    self.x._setup(x_shape)
AttributeError: 'Parameter' object has no attribute '_setup' ```

Does someone know how to solve this problem?
Thanks

Hi, I think you need to pull the latest version of DeepPy/Neural artistic style.

Did you mean this one https://github.com/andersbll/deeppy ?

Yes.

Thanks, I've sold this problem, I think the reason was that I wasn't use sudo command when run python setup.py install


⚠ Samsung has degraded performance

In 1761471, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2157 ms

Resolved: Samsung performance has improved in 3efdead.


Mix escript.install downgraded to livebook 0.1.0

Environment

Elixir & Erlang/OTP versions (elixir --version):

Erlang/OTP 23 [erts-11.2.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] [dtrace]

Elixir 1.12.0 (compiled with Erlang/OTP 23)

Operating system: MacOS Big Sur (11.6)
How have you started Livebook (mix phx.server, livebook CLI, Docker, etc): N/A
Livebook version (use git show if running with mix): Livebook 0.2.3, 0.3.1
Browsers that reproduce this bug (the more the merrier): N/A
Include what is logged in the browser console: N/A
Include what is logged to the server console: N/A

Current behavior
When I run mix escript.install hex livebook (in the README) the version of livebook installed is 0.1.0
I started on 0.2.3 and wanted to upgrade, so I ran that command expecting it to install the latest version. It downgraded me to 0.1.0
I ran mix escript.install hex livebook 0.3.1 and it correctly installed Livebook 0.3.1
To see if it would repro, I ran mix escript.install hex livebook again, and it again installed 0.1.0
I can also repro by running mix escript.uninstall livebook and then mix escript.install hex livebook so new users to livebook may also be getting 0.1.0 installed when they follow these instructions
Start of CLI output:
$ mix escript.install hex livebook
Resolving Hex dependencies...
Dependency resolution completed:
New:
  castore 0.1.13
  cowboy 2.9.0
  cowboy_telemetry 0.3.1
  cowlib 2.11.0
  earmark_parser 1.4.17
  jason 1.2.2
  livebook 0.1.0
  mime 1.6.0

Click to show complete output (168 lines)
$ mix escript.install hex livebook

Resolving Hex dependencies...
Dependency resolution completed:
New:
  castore 0.1.13
  cowboy 2.9.0
  cowboy_telemetry 0.3.1
  cowlib 2.11.0
  earmark_parser 1.4.17
  jason 1.2.2
  livebook 0.1.0
  mime 1.6.0
  phoenix 1.5.7
  phoenix_html 2.14.1
  phoenix_live_dashboard 0.4.0
  phoenix_live_view 0.15.0
  phoenix_pubsub 2.0.0
  plug 1.10.0
  plug_cowboy 2.5.2
  plug_crypto 1.2.2
  ranch 1.8.0
  telemetry 0.4.2
  telemetry_metrics 0.4.0
  telemetry_poller 0.4.0
* Getting livebook (Hex package)
* Getting castore (Hex package)
* Getting earmark_parser (Hex package)
* Getting jason (Hex package)
* Getting phoenix (Hex package)
* Getting phoenix_html (Hex package)
* Getting phoenix_live_dashboard (Hex package)
* Getting phoenix_live_view (Hex package)
* Getting plug_cowboy (Hex package)
* Getting telemetry_metrics (Hex package)
* Getting telemetry_poller (Hex package)
* Getting telemetry (Hex package)
* Getting cowboy (Hex package)
* Getting cowboy_telemetry (Hex package)
* Getting plug (Hex package)
* Getting mime (Hex package)
* Getting plug_crypto (Hex package)
* Getting cowlib (Hex package)
* Getting ranch (Hex package)
* Getting phoenix_pubsub (Hex package)
Resolving Hex dependencies...
Dependency resolution completed:
Unchanged:
  castore 0.1.13
  cowboy 2.9.0
  cowboy_telemetry 0.3.1
  cowlib 2.11.0
  earmark_parser 1.4.17
  jason 1.2.2
  mime 1.6.0
  phoenix 1.5.7
  phoenix_html 2.14.1
  phoenix_live_dashboard 0.4.0
  phoenix_live_view 0.15.0
  phoenix_pubsub 2.0.0
  plug 1.10.0
  plug_cowboy 2.5.2
  plug_crypto 1.2.2
  ranch 1.8.0
  telemetry 0.4.2
  telemetry_metrics 0.4.0
  telemetry_poller 0.4.0
All dependencies are up to date
==> earmark_parser
Compiling 1 file (.yrl)
Compiling 2 files (.xrl)
Compiling 3 files (.erl)
Compiling 32 files (.ex)
Generated earmark_parser app
===> Compiling telemetry_poller
===> Compiling ranch
===> Compiling telemetry
==> telemetry_metrics
Compiling 7 files (.ex)
Generated telemetry_metrics app
==> jason
Compiling 8 files (.ex)
Generated jason app
==> castore
Compiling 1 file (.ex)
Generated castore app
==> phoenix_pubsub
Compiling 11 files (.ex)
Generated phoenix_pubsub app
===> Compiling cowlib
===> Compiling cowboy
===> Compiling cowboy_telemetry
==> mime
Compiling 2 files (.ex)
Generated mime app
==> plug_crypto
Compiling 5 files (.ex)
Generated plug_crypto app
==> plug
Compiling 1 file (.erl)
Compiling 40 files (.ex)
warning: System.stacktrace/0 is deprecated. Use __STACKTRACE__ instead
  lib/plug/conn/wrapper_error.ex:23: Plug.Conn.WrapperError.reraise/3

warning: MIME.valid?/1 is deprecated. Use MIME.extensions(type) != [] instead
  lib/plug/mime.ex:32: Plug.MIME.valid?/1

Generated plug app
==> phoenix_html
Compiling 8 files (.ex)
warning: got "@impl true" for function handle_text/2 but no behaviour specifies such callback. The known callbacks are:

  * EEx.Engine.handle_begin/1 (function)
  * EEx.Engine.handle_body/1 (function)
  * EEx.Engine.handle_end/1 (function)
  * EEx.Engine.handle_expr/3 (function)
  * EEx.Engine.handle_text/3 (function)
  * EEx.Engine.init/1 (function)

  lib/phoenix_html/engine.ex:52: Phoenix.HTML.Engine (module)

warning: function handle_text/3 required by behaviour EEx.Engine is not implemented (in module Phoenix.HTML.Engine)
  lib/phoenix_html/engine.ex:1: Phoenix.HTML.Engine (module)

Generated phoenix_html app
==> plug_cowboy
Compiling 5 files (.ex)
Generated plug_cowboy app
==> phoenix
Compiling 66 files (.ex)
Generated phoenix app
==> phoenix_live_view
Compiling 22 files (.ex)
warning: got "@impl true" for function handle_text/2 but no behaviour specifies such callback. The known callbacks are:

  * Phoenix.Template.Engine.compile/2 (function)
  * EEx.Engine.handle_begin/1 (function)
  * EEx.Engine.handle_body/1 (function)
  * EEx.Engine.handle_end/1 (function)
  * EEx.Engine.handle_expr/3 (function)
  * EEx.Engine.handle_text/3 (function)
  * EEx.Engine.init/1 (function)

  lib/phoenix_live_view/engine.ex:325: Phoenix.LiveView.Engine (module)

warning: function handle_text/3 required by behaviour EEx.Engine is not implemented (in module Phoenix.LiveView.Engine)
  lib/phoenix_live_view/engine.ex:148: Phoenix.LiveView.Engine (module)

warning: MIME.valid?/1 is deprecated. Use MIME.extensions(type) != [] instead
  lib/phoenix_live_view/upload_config.ex:388: Phoenix.LiveView.UploadConfig.accept_option!/1

Generated phoenix_live_view app
==> phoenix_live_dashboard
Compiling 36 files (.ex)
Generated phoenix_live_dashboard app
==> livebook
Compiling 83 files (.ex)
Generated livebook app
Generated escript livebook with MIX_ENV=prod
Found existing entry: /Users/callum/.mix/escripts/livebook
Are you sure you want to replace it with "livebook"? [Yn]
* creating /Users/callum/.mix/escripts/livebook
$ livebook --version
Erlang/OTP 23 [erts-11.2.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] [dtrace]

Elixir 1.12.0 (compiled with Erlang/OTP 23)

Livebook 0.1.0

Expected behavior
I'd expect this command to install the latest version

Can you please run mix local.hex to update to latest Hex and try again? :)

That fixed it, thanks! :) Hex upgraded from 0.21.1 to 0.21.3 and now it's installing livebook 0.3.2

Yeah, it was a Hex bug. Glad to hear it is working!


Keyboard appears on non-text entry steps

I am experiencing the keyboard showing on custom non-text entry steps.
 It makes for pretty annoying UX.
Is there any workaround to prevent this from happening?
All of my steps are custom, extending Step, with layouts implementing StepLayout.

Which version/commit of ResearchStack are you using?
Could you please attach a screenshot? And if your repository is public, a link to the layout, Step and StepLayout with the keyboard inappropriately popping up?

I have same issue. I guess the following line is the reason.
imm.toggleSoftInput(InputMethodManager.HIDE_IMPLICIT_ONLY, 0);
// StepSwitcher.java ln:160-163

The bug is occurred on especially Samsung Galaxy S4 phone.

@liujoshua I'm experiencing the same issue. I've been able to reproduce on the nexus 5x emulator running API 21. Repo here.

Thank you all for bringing this to our attention. I've entered it into our internal JIRA as a high priority bug.

I fixed this issue on the develop branch, see the PR https://github.com/ResearchStack/ResearchStack/pull/374
Here is a screenshot of the changes for the lazy

Thanks for this!!

On Apr 8, 2017, at 4:09 PM, Michael DePhillips<EMAIL_ADDRESS>wrote:
Closed #359 https://github.com/ResearchStack/ResearchStack/issues/359.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/ResearchStack/ResearchStack/issues/359#event-1035263528, or mute the thread https://github.com/notifications/unsubscribe-auth/AGaRgSb7kleUcUtK80jCXdfvCx2xveYAks5rt-mAgaJpZM4Mbtka.


Fix assigned type of assignment nested in literals

Fixes #12946
Previously, this code would cause the compiler to recur infinitely:
// @noImplicitAny: true
var target, x, y;
target = [x = 1, y = x];

That's because the assigned type of an assignment that is nested in an array literal includes the type of the array literal itself. In other words, to get the assigned type of x=1, the compiler will union the type of 1 and [x=1, y=x]. This cause infinite recursion.
The union code is intended to support destructuring assignment defaults, so this change makes the recursion only happen when the array literal is on the left-hand side of an assignment.
Note that it also applies to object literals, and to the initializer of for-of loops.

@sandersn  could you add test for object literal? and initializer of for-of loops

@yuit Lines 5 and 6 in assignmentNestedInLiterals test object literals.
I tried to make a test for for-of and couldn't get the same untyped behaviour. However, there is a negative test in conformance/es6/for-ofStatements/for-of45
var k: string, v: boolean;
var map = new Map([["", true]]);
for ([k = "", v = false] of map) {
    k;
    v;
}

If you leave off the for-of check in isDestructuringAssignmentTarget, then v=false ignores the type of map and incorrectly narrows v: false instead of v: boolean.
Here's what I tried:
var target, x, y;
target = [x = 1, y = x];
// compare to
var i, a, b;
for (i of [a = 1, b = a]) {
  foo;
}

Never mind, I got it to fail this time. I'll update the test case.

thanks @sandersn !


Cannot read property 'map' of undefined

I created the following schema:
// Create CacheSearch Schema
const CacheSearch = new dynamoose.Schema({
    updated_at: {
        type: String,
        required: false,
    },
    created_at: {
        type: String,
        required: true,
        rangeKey: true
    },
    cache_id: {
        type: String,
        hashKey: true,
        required: true
    },
    uses: {
        type: Number,
        required: true,
        default: 1
    },
    search: {
        type: Object
    },
    flights: [Object],
    flights_back: [Object]
},{ useDocumentTypes: true, useNativeBooleans: true});

// end Create CacheSearch Schema

I inserted the data from Aws `console

But when I tried to use the .scan ()  i received the following error log in the
CacheSearch.scan().exec()
        .then( (res) => {
            console.log(res);
        }).catch(err => {console.error(err)})

TypeError: Cannot read property 'map' of undefined
at dedynamofy (/var/www/crawler/node_modules/dynamoose/lib/Attribute.js:419:17)
at Attribute.parseDynamo (/var/www/crawler/node_modules/dynamoose/lib/Attribute.js:498:13)
at Schema.parseDynamo (/var/www/crawler/node_modules/dynamoose/lib/Schema.js:182:24)
at toModel (/var/www/crawler/node_modules/dynamoose/lib/Scan.js:58:12)
at Array.map (<anonymous>)
at Response.<anonymous> (/var/www/crawler/node_modules/dynamoose/lib/Scan.js:211:35)
at Request.<anonymous> (/var/www/crawler/node_modules/aws-sdk/lib/request.js:364:18)
at Request.callListeners (/var/www/crawler/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
at Request.emit (/var/www/crawler/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/var/www/crawler/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/var/www/crawler/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/www/crawler/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /var/www/crawler/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/var/www/crawler/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/var/www/crawler/node_modules/aws-sdk/lib/request.js:685:12)
at Request.callListeners (/var/www/crawler/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
at Request.emit (/var/www/crawler/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/var/www/crawler/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/var/www/crawler/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/www/crawler/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /var/www/crawler/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/var/www/crawler/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/var/www/crawler/node_modules/aws-sdk/lib/request.js:685:12)
at Request.callListeners (/var/www/crawler/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
at Request.emit (/var/www/crawler/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/var/www/crawler/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/var/www/crawler/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/www/crawler/node_modules/aws-sdk/lib/state_machine.js:14:12)

@Onnion Can you provide those two items you imported into your table so we can debug better?

@Onnion Also, could you rerun the test with the latest from the master branch.  There have been several changes to Attribute.js to fix some issues.  Thanks

I also got the same error message. I first created a new item by hand in the AWS console, then tried to match up the dynamoose Model with the data. The value of the field that throws an error was set to a List of Map elements in the console when I created the item.
System
node v8.6.0
dynamoose ^0.8.6

Model Definition
const Test = dynamoose.model('ToyApi-Test', {
  id: {
    type: String,
    hashKey: true,
  },
  name: {
    type: String,
  },
  order: {
    type: Number,
    index: {
      global: true,
      throughput: 1,
      name: 'order-index',
      project: true,
    },
  },
  tests: {
    type: [Object], // Mis-matched data type?
  },
})

Trace
/home/app/src/node_modules/aws-sdk/lib/request.js:31
            throw err;
            ^

TypeError: Cannot read property 'map' of undefined
    at dedynamofy (/home/app/src/node_modules/dynamoose/lib/Attribute.js:419:17)
    at Attribute.parseDynamo (/home/app/src/node_modules/dynamoose/lib/Attribute.js:498:13)
    at Schema.parseDynamo (/home/app/src/node_modules/dynamoose/lib/Schema.js:182:24)
    at Response.<anonymous> (/home/app/src/node_modules/dynamoose/lib/Model.js:400:14)
    at Request.<anonymous> (/home/app/src/node_modules/aws-sdk/lib/request.js:364:18)
    at Request.callListeners (/home/app/src/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
    at Request.emit (/home/app/src/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/home/app/src/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/home/app/src/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/home/app/src/node_modules/aws-sdk/lib/state_machine.js:14:12)

Query
Test
  .query('order')
  .eq(1)
  .exec()
  .then((res) => {
    console.log(res)
  })
  .catch(console.error)

Model Update
  tests: {
    type: String, // No error thrown
  },

Result After Datatype Update
[ NewModel {
    order: 1,
    id: '720ae023-ce1a-4532-adaf-20f81113bf52',
    name: 'sum-api',
    tests: '{"L":[{"M":{"request":{"M":{"path":{"S":"/1/2/3"},"header":{"M":{"Content-Type":{"S":"text/html, application/json"}}},"method":{"S":"GET"},"query":{"M":{}}}},"description":{"S":"should respond with sum of path integers"},"response":{"M":{"body":{"S":"6"},"code":{"N":"200"}}}}}]}' },
  lastKey: undefined,
  count: 1,
  scannedCount: 1 ]


Cryptic error message when ROI metadata and parsing inputs disagree

EDIT:
The underlying issue of why the shape of the array did not correspond to the metadata table was actually wrong metadata that was passed initially. I mistakenly parsed this as a 4x5 sites plate and the yokogawa_to_zarr parsing follows this logic of rows & cols specified.
It's actually a 5x5 dataset though. And the metadata said as much. Thus, when it tried to process things by ROIs, that was based on the metadata. But the parsing was based on other parameters. Those two were never checked against each other and thus lead to this very hard to trace bug.
How do we avoid this?

Avoid different tasks relying on different ground truths. In this case, make the parsing also depend on metadata, not on rows & cols parameter. See here for an issue for that specifically: https://github.com/fractal-analytics-platform/fractal/issues/126
What checks would actually make sense to ensure that the metadata corresponds to the actual data on disk?

Original issue:
I'm trying to process some new datasets and perform illumination correction on them. But when running any of the new ROI-based tasks in the tasks branch (happens for both illumination correction and ROI-based labeling), I get an error during the ROI assignment.
Traceback (most recent call last):
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/parsl/dataflow/dflow.py", line 768, in sanitize_and_wrap
    new_inputs.extend([dep.result()])
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/parsl/dataflow/dflow.py", line 286, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/parsl/dataflow/dflow.py", line 471, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/parsl/app/errors.py", line 131, in reraise
    reraise(t, v, tb)
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/six.py", line 718, in reraise
    raise value.with_traceback(tb)
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/parsl/app/errors.py", line 159, in wrapper
    return func(*args, **kwargs)  # type: ignore
  File "../../fractal/fractal_cmd.py", line 600, in app
    return dict_tasks[task](zarrurl, **kwargs_)
  File "/net/nfs4/pelkmanslab-fileserver-jluethi/data/homes/jluethi/mwe_fractal/fractal/tasks/illumination_correction.py", line 246, in illumination_correction
    data_czyx_new[ind_ch, s_z:e_z, s_y:e_y, s_x:e_x] = da.stack(
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/dask/array/core.py", line 1893, in __setitem__
    dsk = setitem_array(out, self, key, value)
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/dask/array/slicing.py", line 1868, in setitem_array
    raise ValueError(
ValueError: shape mismatch: value array of shape (21, 2160, 2560) could not be broadcast to indexing result of shape (21, 1, 2560)

The y indices in data_czyx_new[ind_ch, s_z:e_z, s_y:e_y, s_x:e_x] seem to get calculated wrong leading to a shape of 1 instead of 2160.
This happens with /data/homes/jluethi/mwe_fractal/examples/FMI_Conny_IllumCorr/example_fmi_conny_IllumCorr_argolight.sh, but not with the 2x2 test case. So maybe it's some edge case we haven't covered yet.
I checked the AnnData ROI table, that has been parsed correctly. All the positions are as they should be, all FOVs have the correct x & y sizes.

Will debug a bit further to see if I can find the source. Maybe something when all sites actually start with positive positions (no negative stage coordinates)? Or something else specific to this metadata?

The list_indices calculated here appear to be fine (correct spacing in x & y dimension, nothing that would be reduced to a 1):

The issue seems to be here:
data_czyx_new[ind_ch, s_z:e_z, s_y:e_y, s_x:e_x] = da.stack(tmp_zyx, axis=0)
data_czyx_new[ind_ch, s_z:e_z, s_y:e_y, s_x:e_x] is actually just of that shape:

Which is weird, given that the indices are:
print(ind_ch, s_z,e_z, s_y,e_y, s_x,e_x)
0 0 21 8639 10799 0 2560

Oh, I think figured it out! x & y sizes of the data_czyx_new were switched. Thus, the array was limited in size and sometimes only returned the left-over shapes.
Given an data_czyx_new like this of shape (4, 21, 8640, 12800)

Trying to access this with data_czyx_new[0,0:21, 8639:10799, 0:2560] doesn't throw an error of out of bounds indexing. It just returns an array with y size of 1. Weird behavior of dask tbh...
I wonder if this is coming from a switch up in the parsing! (would explain why the 2x2 test case still work). Will investigate whether the two metadata files differ somewhere in that aspect.

Correction: The underlying issue of why the shape of the array did not correspond to the metadata table was actually wrong metadata that was passed initially. I mistakenly parsed this as a 4x5 sites plate and the yokogawa_to_zarr parsing follows this logic of rows & cols specified.
It's actually a 5x5 dataset though. And the metadata said as much. Thus, when it tried to process things by ROIs, that was based on the metadata. But the parsing was based on other parameters. Those two were never checked against each other and thus lead to this very hard to trace bug.
How do we avoid this?

Avoid different tasks relying on different ground truths. In this case, make the parsing also depend on metadata, not on rows & cols parameter. See here for an issue for that specifically: https://github.com/fractal-analytics-platform/fractal/issues/126
What checks would actually make sense to ensure that the metadata corresponds to the actual data on disk?

After giving it some thought, I think #126 is the only case of Fractal taking duplicate inputs (getting the same info from two places) that I can think of. If someone can think of other issues where this comes up, please open a separate issue for it.
Closing this for now as we'll address the root cause in #126


Standardize paths, fix path traversal

As per RFC-3986 5.2.4 (Remove Dot Segments), we want to remove ..s. We can use Foundation's URL.standardized features for that.
However, I am sleepy, so this may have missed something obvious.
Fixes swhitty/FlyingFox/issues/24

I love the choice of branch name btw hahah. From now on I’m definitely going to call path traversals ‘naughty paths’.

Looks like we found another place where CoreLibsFoundation differs from Darwin Foundation.  I have updated the test to pass here, but maybe we should normalise the behaviour across all platforms
https://github.com/swhitty/FlyingFox/commit/f969347cdf7170f2b4124c2fb1296eebfbdbbaf3

That is certainly very annoying. You would think it would be very easy for the two Foundations to have the same behaviour in that context, weird.


Getting ValueError: Error when checking target: expected dense_1 to have shape in Ubuntu 16,Python 2.7

I have tried to execute the below code in Ubuntu16 64 bit with Python 2.7 in Virtual environment
Unique Labels  Count 1252
Input RawData has lot of Text files(.txt) and their equivalent .lab files.
When I ran the code with lesser number of labels (in 100s), the code ran fine.
from magpie import Magpie

magpie = Magpie()
magpie.train_word2vec('InputCustomization/RawData')
magpie.fit_scaler('InputCustomization/RawData')
magpie.init_word_vectors('InputCustomization/RawData')

uniLabels = []
with open('InputCustomization/UniqueLabels.txt') as fl:
     for aLine in fl:
         aLine = aLine.strip()
         uniLabels.append(aLine)

print len(uniLabels)

magpie.train('InputCustomization/RawData', uniLabels, test_ratio=0.2, epochs=2)
print("Hello PREDICTED OUTPUT will be printed After now  ")
print("********************")
#print(magpie.predict_from_text('Stephen Hawking studies black holes'))
print(magpie.predict_from_file('InputCustomization/RawData/T100075.txt'))

Error I am getting :

$ python hellomagpie.py
/home/rajeshkumar/TrailGitClones/magpie_git/magpie_py2Dupl/local/lib/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.
Fitted to 46611 vectors
Fitted to 7013 vectors
WARNING! Overwriting already trained word2vec model.
WARNING! Overwriting already fitted scaler.
Fitted to 46611 vectors
Fitted to 7013 vectors
1252
WARNING:tensorflow:From /home/rajeshkumar/TrailGitClones/magpie_git/magpie_py2Dupl/local/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py:497: calling conv1d (from tensorflow.python.ops.nn_ops) with data_format=NHWC is deprecated and will be removed in a future version.
Instructions for updating:
`NHWC` for data_format is deprecated, use `NWC` instead
Traceback (most recent call last):
  File "hellomagpie.py", line 51, in <module>
    magpie.train('InputCustomization/RawData', uniLabels, test_ratio=0.2, epochs=2)
  File "/home/rajeshkumar/TrailGitClones/magpie_git/magpie_py2Dupl/local/lib/python2.7/site-packages/magpie-2.0-py2.7.egg/magpie/main.py", line 107, in train
    verbose=verbose,
  File "/home/rajeshkumar/TrailGitClones/magpie_git/magpie_py2Dupl/local/lib/python2.7/site-packages/keras/engine/training.py", line 1630, in fit
    batch_size=batch_size)
  File "/home/rajeshkumar/TrailGitClones/magpie_git/magpie_py2Dupl/local/lib/python2.7/site-packages/keras/engine/training.py", line 1480, in _standardize_user_data
    exception_prefix='target')
  File "/home/rajeshkumar/TrailGitClones/magpie_git/magpie_py2Dupl/local/lib/python2.7/site-packages/keras/engine/training.py", line 123, in _standardize_input_data
    str(data_shape))
**ValueError: Error when checking target: expected dense_1 to have shape (1252,) but got array with shape (1251,)**

Please guide me so that the issue is resolved.

That's a very peculiar error @rajeshkumargp, in Magpie there's no logic around the number of labels so it should work fine regardless of their number. We managed to train those models on >10000 of labels, so it's definitely possible. You say that it works for couple of hundred labels but not 1252 - is there a threshold number when it stops working? It would be great if you could also enable me to reproduce the error on my machine so I can investigate.
One minor thing, magpie.init_word_vectors('InputCustomization/RawData') is equivalent to the two lines above, so you don't need them. It's enough if you run:
magpie = Magpie()
magpie.init_word_vectors('InputCustomization/RawData')

print len(uniLabels)
magpie.train('InputCustomization/RawData', uniLabels, test_ratio=0.2, epochs=2)

I rechecked list of unique labels and found that 'ALabel'  and '   ALabel' , although same . This causes error.
I am closing this issue.


It appears the marketplace rejects certain "reserved" keywords as extension name

I wasn't able to publish under <publisher>.ext or <publisher>.extension. Marketplace Manage as well as the APIs returned (slightly edited):
Done in 22.60s.
 WARNING  LICENSE.md, LICENSE.txt or LICENSE not found
 INFO  Publishing '<publisher>.extension v0.4.0'...
 ERROR  The Extension Id already exist in the Marketplace. Please use the different Id

It'd be great to detect earlier in the process and upgrade docs to list all reserved keywords if there are more. There's also a chance I could be wrong.
Version used:
<EMAIL_ADDRESS>

Unfortunately, the Marketplace started restricting extension names and makign them unique despite the publisher.

ERROR  The Extension Id already exist in the Marketplace. Please use the different Id

A better error message should be sent from the service, clarifying what is meant by Extension Id.
cc @isidorn @sandy081

@holazz I suggest picking a different extension name, since open-in-browser is taken.

We should also communicate this change clearly to extension authors. @prashantvc can you please take the lead here?

Unfortunately, the Marketplace started restricting extension names and makign them unique despite the publisher.

ERROR  The Extension Id already exist in the Marketplace. Please use the different Id

A better error message should be sent from the service, clarifying what is meant by Extension Id.
cc @isidorn @sandy081
@holazz I suggest picking a different extension name, since open-in-browser is taken.

I noticed that both igordvlpr.open-in-browser and techer.open-in-browser exist in the Marketplace. Isn't ${publisher}.${name} as  Extension Id ?

Unfortunately, the Marketplace started restricting extension names and makign them unique despite the publisher.

ERROR  The Extension Id already exist in the Marketplace. Please use the different Id

A better error message should be sent from the service, clarifying what is meant by Extension Id.
cc @isidorn @sandy081
@holazz I suggest picking a different extension name, since open-in-browser is taken.

I noticed that both igordvlpr.open-in-browser and techer.open-in-browser exist in the Marketplace. Isn't ${publisher}.${name} as  Extension Id ? @joaomoreno

@holazz marketplace is moving to the new model. @prashantvc can provide more details what happens with the existing name conflicts.

We are working new extension model where extension names must be unique. As @holazz suggested could you pick different name instead of open-in-browser
We will update the error message to be more descriptive

We have changed the error message for better clarity and call for action to fix the issue

Keeping it opened until the fix goes live :)

@prashantvc can provide more details what happens with the existing name conflicts

@prashantvc Can you explain what will happen to the existing name conflicts? Will they be forced to change IDs - if so, how do you decide which ones stay and which ones will be renamed?

When we migrate to new system extension identifier will just be name from the package.json.
In case there is a conflict, for example ms.python and google.python, we will use the current convention of {publisher}.{name}.

I am seeing this error when publishing a new version of an existing extension. The publisher and ID haven't changed AFAICT and there is only one extension with this ID in the marketplace.

Any chance to address this? I wasted more than hour trying to debug blindly why marketplace rejected the upload with a defective error message.
I had to rename my extension "name" to make it unique to the entire marketplace (yep, all publishers), despite the fact that this requirements was never specified.

@ssbarnea I am extremely sorry about it. We have added a better error message and steps to resolve it.
You should see it soon in the production


Various platform updates

Add support for ppc64le on Debian & Ubuntu
Add support for:

AlmaLinux 8 & 9
Amazon Linux 2023
Debian 12

Remove support for:

Amazon Linux 2 & 2022
Debian 9

Set sane default for Windows System Drive
Update and improved unit and integration tests

Signed-off-by: Lance Albertson<EMAIL_ADDRESS>Description
Describe what this change achieves
Issues Resolved
List any existing issues this PR resolves
Check List

[ ] A summary of changes made is included in the CHANGELOG under ## Unreleased
[ ] New functionality includes testing.
[ ] New functionality has been documented in the README if applicable.

Released as: 1.1.0


Elasticsearch: Request Entity Too Large

Description
I've tried to enable code indexing in our instance using Elasticsearch, but I get the following error for a lot of repositories:
workergroup.go:102:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 20 items, backoff for a few seconds
indexer.go:128:func2() [E] Codes indexer handler: index error for repo 713: elastic: Error 413 (Request Entity Too Large)
workergroup.go:102:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 1 items, backoff for a few seconds

I've changed the setting http.max_content_length in the Elasticsearch config to the maximum possile value<PHONE_NUMBER>b but the error still comes up.
This also comes up for a lot of repositories, not just our biggest ones.
I'm unsure how the indexer works, does it take the whole sourcecode of a branch and pumps it into elastic? Is some kind of batching per x files needed?
Gitea Version
1.21.0
Can you reproduce the bug on the Gitea demo site?
No
Log Gist
No response
Screenshots
No response
Git Version
No response
Operating System
No response
How are you running Gitea?
Running Gitea on Linux amd64 with the official binary and Elasticsearch within Docker
Database
None

The repository sizes (as reported in the gitea web ui) from some repositories that I picked out of the log are:

1.1 MiB
7.8 MiB
2 MiB
598 MiB
154 MiB
2.8 MiB

That makes me wonder if the small repositories are batched together with the big ones which then exceeds the request limit

Me 2 having the same issue.
My repo size in UI is 319MiB not indexing

Unfortunately, after a quick look, I think your guess is right ... maybe Gitea does put everything into one request and send it to elasticsearch, since the first elasticsearch PR: #10273
https://github.com/go-gitea/gitea/blob/688d4a1f719d2df4d2626453f4bc042c1874a375/modules/indexer/code/elasticsearch/elasticsearch.go#L182-L188

Unfortunately, after a quick look, I think your guess is right ... maybe Gitea does put everything into one request and send it to elasticsearch, since the first elasticsearch PR: #10273
https://github.com/go-gitea/gitea/blob/688d4a1f719d2df4d2626453f4bc042c1874a375/modules/indexer/code/elasticsearch/elasticsearch.go#L182-L188

Yeah! you're right Gitea is putting everything in 1 request before sending it to elastic search, not batching it, as shown in my error logs below:
...queue/workergroup.go:102:doWorkerHandle() [E] Queue "code_indexer" failed to handle batch of 1 items, backoff for a few seconds

After finding out I pushed my elasticsearch maximum limit to<PHONE_NUMBER>b then I'm getting the following error 429 (too many requests):
 ...exer/code/indexer.go:128:func2() [E] Codes indexer handler: index error for repo 28: elastic: Error 429 (Too Many Requests): [in_flight_requests] Data too large, data for [<http_request>] would be  [1272753908/1.1gb], which is larger than the limit of [1073741824/1gb] [type=circuit_breaking_exception]

I'm trying to fix this issue in the code by batching the requests

Unfortunately, after a quick look, I think your guess is right ... maybe Gitea does put everything into one request and send it to elasticsearch, since the first elasticsearch PR: #10273
https://github.com/go-gitea/gitea/blob/688d4a1f719d2df4d2626453f4bc042c1874a375/modules/indexer/code/elasticsearch/elasticsearch.go#L182-L188

Fixed by #29075


On import create dashboard from template, also create any included resources

as a user
with admin
when i see a list of resources (dashboard/task/variable for now)
then i would like the ability to export resource as JSON,
and import it back from that JSON, to be instantiated as that resource.
If imported JSON is not valid for that resource, I would like to see an error.
what happens to other resources bundled with that resource? like variables included in dashboard?

closed by: https://github.com/influxdata/influxdb2-js/pull/42


fix(src/maps/OsmRasterTileStyle.ts): change out dated url to recommended format

About this pull request
Haven't seen you for a long time.
This commit changes legacy load balancing subdomain url to new unified osm tiles url according to operation group's announcement.
Currently discovering and submitting PRs to projects on Github that still use legacy URL. See https://github.com/openstreetmap/operations/issues/737
They also announced this on Twitter: https://twitter.com/OSM_Tech/status/1601625779036893190
Related issues
None
What you checked

[ ] Existing feature works correctly
[ ] Added changes works correctly

Related URLs
None
Screenshots
Before
None
After
None

Hi LaoshuBaby-san! Happy New Year!
And I am really, really sorry for the delay in responding to this Pull Request🙏
When I enabled renovate, a lot of automatic Pull Requests were created, making it difficult to notice contributors' Pull Requests...😭


[BUG] RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument weight in method wrapper_CUDA__native_layer_norm)

File "/home/pete/PycharmProjects/Time-Series-Classification-master/model/mmm4tsc.py", line 224, in forward
fused = self.visual_expert(concat)
File "/home/pete/anaconda3/envs/mamba/lib/python3.8/site-packages/zeta/nn/modules/visual_expert.py", line 106, in call
normalized = self.norm(x)
File "/home/pete/anaconda3/envs/mamba/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/pete/anaconda3/envs/mamba/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/pete/anaconda3/envs/mamba/lib/python3.8/site-packages/torch/nn/modules/normalization.py", line 196, in forward
return F.layer_norm(
File "/home/pete/anaconda3/envs/mamba/lib/python3.8/site-packages/torch/nn/functional.py", line 2543, in layer_norm
return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument weight in method wrapper_CUDA__native_layer_norm)

Upvote & Fund

We're using Polar.sh so you can upvote and help fund this issue.
We receive the funding once the issue is completed & confirmed by you.
Thank you in advance for helping prioritize & fund our backlog.

@dddlli hey can you publish the full code for me please?

@dddlli let me know if not I will close this issue ?


Facilitate rapid releasing  by adding auto

I'm going to hold off on this until after there's a final (non-alpha) release, as auto does not produce PEP 440-compatible prerelease version strings.

I'm going to hold off on this until after there's a final (non-alpha) release, as auto does not produce PEP 440-compatible prerelease version strings.

IMHO it is already worth of 0.1.0 release -- we use it daily ;)

Version 0.1.0 released.  I'll set up auto later; the need for this to be a single standalone file prevents us from using versioneer or the like, so I'll need to figure out how to get auto to update the version number in the source file.


[perftests] Clean up after testSupprocessSpawnWorkingDirectory

Spawning processes with a working directory may leave the thread working
directory set.  Clean up by resetting it to the process level working
dir.
rdar://problem/48978131

@swift-ci please smoke test

@swift-ci please smoke test os x

@swift-ci please smoke test os x


Array List or Tuple of integer is not supported

Seems like hashids.encode did not accept Array or Tuple argument. while the site said it could convert array of numbers like [27, 986] into “3kTMd”.
it work if hashids.encode(1,2,3),
but if i pass in array like hashids.encode([1,2,3])
it will return
TypeError: int() argument must be a string, a bytes-like object or a number, not 'tuple' / 'list'
this is not an issue if using hashids.js
it could be simply fix by adding another IF to check the values is list or not in line 226
Thanks!

Hi @pohlee. I am very sorry for letting you wait for over two years.

while the site said it could convert array of numbers like [27, 986] into “3kTMd”.

I am not sure whether that is mentioned anywhere for the python version.
However, you can simply unpack your tuple or list when calling like this:
hashids.encode(*[1,2,3])