import image which no tag will not display by ctr

Description
In the docker, Image has only hash value, example:
$ sudo docker images
REPOSITORY                                                        TAG                 IMAGE ID            CREATED             SIZE
<none>                                                               <none>                  6308a9db5272        5 hours ago         312MB

when use ctr to import the image, there are no any error, but 'ctr i ls` can not find this image

Steps to reproduce the issue:

docker save -o tmp.tar 621ba461ce0f
ctr i import tmp.tar
ctr i ls

Describe the results you received:
ctr i ls display nothing
Describe the results you expected:
should has result, like:
# ctr i ls
REF                                                                     TYPE                                                 DIGEST                                                                  SIZE      PLATFORMS   LABELS
sha256:6308a9db5272754bf7928f2d364e89482653e4aa4a7898e76333b0b12fa2c428 application/vnd.docker.distribution.manifest.v2+json sha256:25d509ac82512d68448c4a6adcdbcf42eb6a3d3be01992af8ea5b556f8db9929 306.7 MB linux/amd64

Output of containerd --version:
v1.3.7

Any other relevant information:

use the latest is ok
[root@centos7 temp]# docker images
REPOSITORY                                 TAG                 IMAGE ID            CREATED             SIZE
alpine                                     3.11.6              f70734b6a266        3 months ago        5.61MB
[root@centos7 temp]# docker save -o tmp.tar alpine:3.11.6
[root@centos7 temp]# ctr i ls
REF TYPE DIGEST SIZE PLATFORMS LABELS
[root@centos7 temp]# ctr i import tmp.tar
unpacking docker.io/library/alpine:3.11.6 (sha256:d7c36d94b31ad7fe08a6505409b303a66dae8bc3c842eb8d244225ebf7133b9a)...done
[root@centos7 temp]# ctr i ls
REF                             TYPE                                       DIGEST                                                                  SIZE    PLATFORMS   LABELS
docker.io/library/alpine:3.11.6 application/vnd.oci.image.manifest.v1+json sha256:d7c36d94b31ad7fe08a6505409b303a66dae8bc3c842eb8d244225ebf7133b9a 5.6 MiB linux/amd64 -

Hi @yylt The archive result from docker save by image ID doesn't contain any repo:tag information. containerd cann't do the assumption to use image ID as name in image service(one important component in containerd) when there is no image name/tag.
If there is requirement for this, I think containerd client API can help.
close it and please feel free to reopen it if necessary.  Thanks


Add TF 2.3.0rc to testing

Can you add a version check on this line https://github.com/tensorflow/addons/blob/master/tensorflow_addons/optimizers/tests/lookahead_test.py#L125?

Looks like there's an error when compiling Windows + 2.3rc0

Can we have more details on the bulid?

Not really since this is appears to be a bash segfault. Trying to replicate it on my local windows box and I as of now I believe this to be the issue:
https://github.com/bazelbuild/bazel/issues/3636
Will dig into how we can avoid this.

Welp... spent way too much time working through that for a relatively straight forward fix. @tensorflow/sig-addons-maintainers ready for review.

BTW, can we also upgradte bazel version in this (or next) PR? We are quite lag behind TF, which uses 3.1.0 in TF 2.3.0rc* and 2.0.0 in TF 2.2.*

BTW, can we also upgradte bazel version in this (or next) PR? We are quite lag behind TF, which uses 3.1.0 in TF 2.3.0rc* and 2.0.0 in TF 2.2.*

Should be no issue with 3.1.0 bazel and RC1-- good catch. Will see how the tests run

Seems good :-) But there are two files staying at version 1.1.0 haha.
https://github.com/tensorflow/addons/blob/master/tools/install_deps/bazel_linux.sh#L6
https://github.com/tensorflow/addons/blob/master/.github/workflows/ci_test.yml#L14


⚠ Bofrost.de has degraded performance

In 7cdb928, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1069 ms

Resolved: Bofrost.de performance has improved in a3d264c after 6 minutes.


Added support for Target session Id setter & getter

Description
Ref: [MOB-16753], [MOB-16754]
Added Target session Id getter and setter APIs in the SDK to support hybrid implementations.

Added new public APIs getSessionId and setSessionId
Added unit, functional and integration tests to verify the functionality.

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] I have signed the Adobe Open Source CLA.
[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

can we also update the objc test app with these new apis?

The objC test app is updated for both sessionId and tntId as part of the other pr https://github.com/adobe/aepsdk-target-ios/pull/116.


Retry topic subscription in a different thread?

Hi! Opening this issue because I am not 100% sure the caveats about thread-safety mentioned in the README apply to my use case.
I have an application that only consumes using a consumer group. In order to avoid strict deploy ordering between producers and this application when new topics are created, I'm thinking about the following:

Instantiate a consumer, and put it on a each_message block.
Pass the consumer and the list of topics to subscribe to a thread.
In that thread, the consumer checks for topic existence and subscribes to topics if/when they exist, with spaced retries for missing topics.
When all topics have been joined, the thread exits, and the main loop will be processing everything.

Would that be thread-safe?

Do you need a consumer for that? I think this will work:
kafka = Kafka.new(...)

# List of required topics.
topics = ["topic-1", ...]

until topics.empty?
  topics.each do |topic|
    topics.delete(topic) if kafka.has_topic?(topic)
  end

  sleep 10 # let's wait a bit
end

# start your app here!

Problem is, I have several services creating topics at different times. When a deploy happens, if service A has started to produce, I want to consume. If service B starts producing 10 minutes later, then I'll consume too, but meanwhile A has been processed.

Do these topics follow a predictable naming scheme? If so, subscribing by regex should allow for dynamically picking up new topics.

You need to pass refresh_topic_interval: to consumer.

There is no pattern. Technically I could OR the topics in a regexp, but I see two problems:

When all topics have been created, I do not need to recheck again.
The point of not calling subscribe is to not subscribe to a non-existing topic. If autocreation is disabled, you can't. If enabled, you do not own the attributes of that topic anyway (retention policy, number of partitions, etc.), the producer is the owner of them.

To be 100% clear, I do not need this to be supported explicitly.
It'd be enough if I can call subscribe from a different thread myself, which the consumer is in the each_message main loop. But the docs say the lib is not thread-safe, so I wonder if this particular use-case would be still OK, or else if support for it could be added.

Hmm, you'd have to test this out because I'm not sure it'll work – but maybe if you converted each topic name to a regex before subscribing you'd get "lazy" subscriptions? That is, if the topic doesn't exist, the regex won't match anything and so won't subscribe, but once the topic is created it'll match and the subscription will be set up.

I'm not sure making subscribe thread safe is worth the effort and added complexity – subscribing out-of-thread seems to be a somewhat esoteric use case. But dynamically picking up topics as they're created doesn't sound crazy, and is in fact something I'd be looking into doing soon. But the regex approach should already work there.

Ah, I've read the source, I see what you mean.
If I am not mistaken, if topic T exists on boot, and topic S is created later, since the scan happens on each iteration of the main loop, and that happens with each poll, event if T is quiet, the consumer would subscribe to S in a timely fashion right? Won't be blocked because T is sending nothing.
Yes, I think that'd work. In this approach, the only thing that I'd miss would be the ability to disable scans, because in the end I have only a bunch of fixed topics to subscribe to. Maybe that leaks there, because re-scanning now an then makes sense for a widlcard pattern, say, but not for this use case, in which we are tunneling fixed strings just to leverage this existing code.
Unfortunately, I am running 0.5.5, though, don't even have this feature available. So, my comment above is feedback for master, but for 0.5.5, if subscribing in a different thread is not thread-safe, I'll see if I can hack something.
I think we can close this one, thank you!

Cool! I remember there being a setting that controlled the scan frequency, but I could be wrong.


failure to fetch data sometimes, or route not loading

I don't know how to debug this exactly or where to start. When ever I traverse through routes in my single page app sometimes going back to the route the server stops responding and doesn't fetch any data to present to the app. I thought it was my routes at first but I'm not sure where to start or even look at. Is it maybe something to do with my API, could too many ajax calls be getting called? Any advice? The strange thing is, on chrome hitting refresh it will say 'Waiting for localhost..' and on firefox it will refresh normally.

@kkotwal94 Hmm. Do you have some code for us to look at and some reproducible steps? I think I get what you're saying but if you have something I can reproduce reliably it can really speed up our debugging process. :D

Yea, I'm not sure if this is it. Here was how I reproduced it, between the route /reportsall and reports/:id i just spammed back and forward buttons (alt+right, alt+left) and I kept doing it until it would just say undefined where the data should be. I can hit refresh but it will state waiting for localhost and take a lot longer than usual to refresh. On Firefox this issue will not ever happen, same with IE. I can start with this, maybe this will give an idea:
Here is the handler: for /reportsall
https://github.com/kkotwal94/IsoReportProcessor/blob/master/app/components/Report.js

Here is the handler for /reports/:id
https://github.com/kkotwal94/IsoReportProcessor/blob/master/app/components/ReportView.js

they are both similar. I console.logged to make sure the data was being obtained and it was @ ReportView however the view presented undefined. The console data showed everything that was supposed to be loaded however.
Here is my route Overview, might be the problem on how I set this up too:
import React from 'react';
import { Route, DefaultRoute } from 'react-router';
import App from 'components/App';
import Login from 'components/Login';
import Signup from 'components/Signup';
import Dashboard from 'components/Dashboard';
import Report from 'components/Report';
import Employees from 'components/Employees';
import AddReports from 'components/AddReports';
import AllEmployees from 'components/AllEmployees';
import MyEmployees from 'components/MyEmployees';
import Profile from 'components/Profile';
import ReportsToComplete from 'components/ReportsToComplete';
import Readme from 'components/Readme';
import Stats from 'components/Stats';
import NewReport from 'components/NewReport';
import TemplateReport from 'components/TemplateReport';
import ReportView from 'components/ReportView';
import EditReport from 'components/EditReport';
import AssignReport from 'components/AssignReport';
import AssignReported from 'components/AssignReported';
import AddSubreport from 'components/AddSubreport';
import NotFound from 'components/NotFound';
import ReportHandler from 'components/ReportHandler'
import Index from 'components/Index';
import { history } from 'react-router';

const routes = (
<Route>
 <Route name ="dashboard" path="/dashboard" handler={App}>
    <Route name ="dash" path="/dashboard" handler={Dashboard}/>
    <Route name ="assignReport" path="/user" handler={AssignReport}/>
    <Route name ="assignReported" path="/user/:id" handler={AssignReported}/>
    <Route name ="employees" path="/employees" handler={Employees}/>
    <Route name ="reports" path="/reports" handler={ReportHandler}>
      <Route name ="report" path="/reportsall" handler={Report}/>
      <Route name ="singlereports" path=":id" handler={ReportView}/>
      <Route name ="addsubreport" path=":id/addsubreport" handler={AddSubreport}/>
      <Route name ="editreports" path=":id/edit" handler={EditReport}/>
    </Route>
      <Route name ="MyEmployees" path="/MEmployees" handler={MyEmployees}/>
      <Route name ="AllEmployees" path="/AEmployees" handler={AllEmployees}/>
      <Route name ="Profile" path="/profile" handler={Profile}/>
      <Route name ="reportstocomplete" path="/reportsc" handler={ReportsToComplete}/>
    <Route name ="addReport" path="/addReport" handler={AddReports}/>
    <Route name ="readme" path="/readme" handler={Readme}/>
    <Route name ="statistics" path="/statistics" handler={Stats}/>
    <Route name ="signup" path="/signup" handler={Signup} />
    <Route name ="newreport" path="/newreport" handler={NewReport}/>
    <Route name ="templatereport" path="/templatereport" handler={TemplateReport}/>
    <Route name ="404" path="/404NotFound" handler={NotFound}/>
    <Route name ="login" path="/" handler={Login}/>
 </Route>
</Route>
	);

export default routes;

I set this up differently a lot of times, and it occurred however I adjusted my routes, I don't think this is the issue though.

I'm starting to think this is a ajax call thing, and how im writing my API, will try some stuff out and reply back later!


Does not work with argocd multiple sources

argo-vault-plugin version 1.13.1 installed as sidecar container
Tried add this manifest
project: myProject
source:
  repoURL: 'https://gitlab.local/k8s/testproject/testapp.git'
  path: ./
  targetRevision: HEAD
destination:
  server: 'https://kubernetes.default.svc'
  namespace: test
sources:
  - repoURL: 'https://gitlab.local/k8s/testproject/testapp.git'
    path: ./
    targetRevision: master
    plugin:
      name: argocd-vault-plugin-helm
      env:
        - name: HELM_ARGS
          value: -f $app_values/global/values.yaml
  - repoURL: 'https://git.local/k8s/testproject/env.git'
    targetRevision: feature/argocd
    ref: app_values

And I get an error:
Unable to save changes: application spec for testapp is invalid: InvalidSpecError: Unable to generate manifests in ./: rpc error: code = Unknown desc = plugin sidecar failed. error generating manifests in cmp: rpc error: code = Unknown desc = error generating manifests: `sh -c helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE ${ARGOCD_ENV_HELM_ARGS} . | argocd-vault-plugin generate - ` failed exit status 126: sh: 2: argocd-vault-plugin: Exec format error Error: open /global/values.yaml: no such file or directory
Maybe I'm doing something wrong)

i have same issue, anyone have solution?

remove plugin part and try again as it's not required with sidecar method anymore

remove plugin part and try again as it's not required with sidecar method anymore

If I understand correctly, the same error
 sources:
    - path: ./
      repoURL: 'https://gitlab.local/k8s/testproject/testapp.git'
      targetRevision: master
      plugin:
        env:
          - name: HELM_ARGS
            value: '-f $app_values/global/values.yaml'
    - repoURL: 'https://git.local/k8s/testproject/env.git'
      targetRevision: feature/argocd
      ref: app_values

Unable to create application: application spec for testapp is invalid: InvalidSpecError: Unable to generate manifests in ./: rpc error: code = Unknown desc = plugin sidecar failed. error generating manifests in cmp: rpc error: code = Unknown desc = error generating manifests: `sh -c helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE ${ARGOCD_ENV_HELM_ARGS} . | argocd-vault-plugin generate - ` failed exit status 126: sh: 2: argocd-vault-plugin: Exec format error Error: open /global/values.yaml: no such file or directory

Seems the issue is with ArgoCD.
https://github.com/argoproj/argo-cd/issues/12476

@ilya251188 was the Argo CD issue linked above able to help your issue?

Does somebody have any updates? Can we hope multiple sources will be supported?
Currently, we have to use single source with helm dependency like as workaround.

Our current workaround is to pull and store the helm chart into the same git repo we store the values in.
repo
|- chart-1.0
  |- Chart.yaml
  |- ...
  |- our-values.yaml

So the app would look something like that
...
source:
  repoURL: 'https://our-git/our-repo.git'
  path: chart-1.0
  targetRevision: HEAD
  plugin:
    name: argocd-vault-plugin-helm
    env:
      - name: HELM_ARGS
        value: '-f our-values.yaml'

Are there any plans to fix this in future? I'm facing the same issue and it's quite disturbing

Same here, currently can't use the multiple sources with a custom plugin :/

Except the workaround proposed by @olzemal, did someone find another alternative?
I am having the same issue and trying to find a solution for use case when Helm charts are published by an external third party such as Bitnami and I just maintain the values.<environment>.yaml in my own GitHub repo.

The dirty solution might be to clone the git repository with values files explicitly during init phase of plugin. I'm attaching pseudocode example of what I have in mind. It's not tested and most likely not working out of the box:
      argocd-vault-plugin-helm:
        init:
          command: [sh, -c]
          args: ["git clone https://github.com/my-values-repo ../my-values-repo"]
        generate:
          command: ["bash", "-o", "pipefail", "-c"]
          args: ['helm template $ARGOCD_APP_NAME --include-crds -n $ARGOCD_APP_NAMESPACE -f ../my-values-repo/values.yaml} . | argocd-vault-plugin generate -']

Git binary must be of course available inside the sidecar (if you use default argocd image it's there). The problem might be credentials so you might have to mount there SSH key from some secret or use some username/password combination. my-values-repo string can be probably replaced with $ARGOCD_APP_VALUES_REPO env varible defined in Application manifest.

Help please :) I have the same issue!

As linked above this is an issue with ArgoCD. https://github.com/argoproj/argo-cd/issues/12476 We have no control over how ArgoCD works AVP is just a custom plugin.

The dirty workaround might be to clone the git repository with values files explicitly during init phase of plugin.

I am trying the similar solution by cloning on-premise (Github Enterprise server) repos in the plugin. However, I agree with you that it is a dirty trick. By the way did you get correct argocd-native solution for deploying Application (helm workload) using multiple sources (values.yaml)? Thanks


Draft: fix CI setup

Hi @serge-sans-paille @emmenlau,
Our Mac dev ran into the same issues as you wrt my changes to gather/scatter and insert on Arm. This MR is a cleanup of yours, which I've used to:

add a macOS universal build workflow
fix the tests on M1
and fix the QEMU runs on Azure Pipelines.

Fixes #694
See #721
Closes #725

Thanks for doing this! We've merged the CI update, I'm relying on you to update the scatter & gather implementation ;-)

The instancing of batch<double> with a 0-sized register on NEON breaks the compile-time iteration I used for the gather/scatter implementations. Should've realised it sooner, but first it hits SFINAE, and only then Clang parses the static_asserts for checking that condition.

@amyspark can you please resolve the conflict? Otherwise looks good!


this example is out of date, please update to latest msal library and android x

Hi team,
this example is out of date, please update to latest msal library and android x #13

@hamiltonha @iambmelt Can you guys take a look at this?

Please update it. Developers come in here from https://docs.microsoft.com/en-us/azure/active-directory-b2c/code-samples

Thanks @marcelmedina, we'll take a look.

@jennyf19 @hamiltonha This is our most up-to-date sample with B2C
https://github.com/Azure-Samples/ms-identity-android-java#b2cmodefragment-class
@rpdome has been the primary contributor to this sample

@iambmelt thanks so much.
@mmacy maybe we need to discuss w/android team about which sample should be used in the docs? @hamiltonha should be able to help with this.

Followed all instructions and build was clean. Running on Pixel w/ SDK 24 as instructed. App loads and then proceeds to crash the emulator.


More Documents like this

It would be great to have a "MoreLikeThis" feature in Tantivy.
An efficient, effective "more-like-this" query generator would be a great contribution.
Elasticsearch and Lucene both support it:
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-mlt-query.html
https://lucene.apache.org/core/7_2_0/queries/org/apache/lucene/queries/mlt/MoreLikeThis.html

Note: this came up when trying to add support for Tantivy in Django-Haystack

If it's helpful, here's how Whoosh (pure search engine implemented in Python) is doing it:
https://github.com/mchaput/whoosh/blob/main/src/whoosh/searching.py#L543-L585

Duplication of #799

Is this issue still open ?

ah no it has been implemented


Refactor and option for mobile

A little cleanup and a new feature. The color/emoji dict for eimg() has been moved to YAML, and the eimg() command itself now has the option for desktop (default) or mobile. When user indicates mobile, a smaller image is generated that is the proper width for the mobile client.

More features and fewer lines? In MY crimsoBOT? It's more likely than you think.


TypeError: Cannot read property 'Symbol(Symbol.iterator)' of undefined  Ln -1 Col -1

I'm getting (with a few of my html files):
WARNING: TypeError: Cannot read property 'Symbol(Symbol.iterator)' of undefined Ln -1 Col -1 /src/analysis/vg-selection.html
And I don't see anything special in the html template itself (e.g of /src/analysis/vg-selection.html) that should error like that:

hmmm, It would be helpful if you could whittle down your html until you can find something that triggers it; then post the related view-model property for it.

OK, so I found the issue.
It's this part from the HTML above:
<a click.delegate="vg.collapsed = !vg.collapsed">
So i agree the code isn't the best - and that this logic should have been placed in a function in the VM, and not inline as it's - but still the linter shouldn't fail on this - as linter wise - it's legit as the code works.
I do understand why it's complicated to validate - so I suggest that instead of validating this type of code (which is complex and not a good approach generally speaking) - maybe a warning should be written - that it's not recommended and that the linter can't verify this type of code...

Generally if something isn't supported I'll at least triage it to stop the warnings. What is the view-model property for vg and what is vg type?

Had the same issue, since the code seem to be failing here: https://github.com/MeirionHughes/aurelia-template-lint/blob/develop/source/rules/binding.ts#L236
..because of args is undefined, it looks like it could be easily patched. I could open a PR but I'm not sure in what way this should behave, would reporting the issue as a warning be good enough?
For now, I've monkey patched it by doing access.args || [] so that it skips the problematic parts of the templates but at least it still proceeds and checks the rest instead of just throwing an error.


attempt adding command palette

adding @rohitpotato's https://github.com/rohitpotato/svelte-command-palette to solve #130

didnt like how all the actions must be declared upfront https://github.com/rohitpotato/svelte-command-palette/issues/19

performance got hit by 24 points when adding this thing

pausing this exploration for now but might reopen later


Add setting to skip confirmation when marking all feeds read

@Minoru  I see that this feature has been added to the latest Newsboat 2.21. But is is possible to set in the configuration file of Newsboat, whether the user wants this feature or not?
For, some users (old) it might be useful to not to confirm as they have been using Newsboat for a long time, and they know important shortcuts. It was more frictionless for them, but now there is an additional step.

Originally posted by @anishmittal2020 in https://github.com/newsboat/newsboat/issues/1006#issuecomment-703259122
We can add a setting like the existing confirm-exit. I think it should default to yes, because we don't have a way to undo this action — better be safe than sorry.

can you plz telll where is this existing  "confirm-exit" setting, and where do we have to add the new one?

Generally you could find answers to those by running git grep confirm-exit, and clicking through to the referenced issue #1006 and looking at the PR that fixed it (#1096).
So in this case, what has to be done is:

add an if in case OP_MARKALLFEEDSREAD in src/feedlistformaction.cpp. See src/view.cpp for how it's done for confirm-exit;
add an entry for the new setting in src/configcontainer.cpp (again, look for "confirm-exit" and do likewise);
document the new setting in doc/configcommands.dsv;
make a commit mentioning this issue and submit a PR :rocket:

thanks for the info. Plz assign this to me , I'd like to work on this

Sure! Looking forward to your PR @tarishij17 ^_^

@Minoru any preferrable name of the new setting?

Plz review the PR and let me know for any updations : https://github.com/newsboat/newsboat/pull/1215

Fixed by #1215.


Some Discord invitation links broken

Hello! These Discord invitation links seem to be expired:

Move Language Discord
Move @ Mysten Labs
Move @ StarCoin

The 0L one works, though.

Starcoin discord link: https://discord.gg/starcoin


atexit fails with runfile

# file.py
import atexit
def f():
    print("Ran f successfully")
atexit.register(f)

I may be misunderstanding how runfile works, but I expect it to function similar to python file.py from terminal. Above prints successfully per latter but not former; tried below to force an "exit" to no avail (though program execution does stop, as evident from failed followup print):
sys.exit(0)
# raise SystemExit  # try one at a time
print("Couldn't exit")  # <-- won't execute

Can we "exit" runfile such that f() is called?

@OverLordGoldDragon, please read the Python docs before reporting issues like this (firsr sentence of the atexit docs):

The atexit module defines functions to register and unregister cleanup functions. Functions thus registered are automatically executed upon normal interpreter termination

The relevant words here are interpreter termination: since our IPython consoles are interactive interpreters (not regular Python processes), they don't terminate after runfile. Well, they don't terminate unless you restart their kernels. In that case, I think atexit should take place.

runfile resembled a pure terminal in some respects, I figured maybe it ran as such and simply redirected output to the IPython console; thanks for clarifying. Tested restarting kernel, atexit doesn't execute (no longer needed regardless):
import atexit
def f():
    with open('ran.txt', 'w') as f:
        f.write('')
atexit.register(f)


HTML tags stripping

It looks like that Huawei's AppGallery doesn't support HTML tags in the release notes and description, like Google Play does.

Please consider adding automated stripping of HTML tags to allow sharing of the same description with Google Play.

@rtsisyk Hi!  But inspired your idea I'm going to add new param text which would hight priorited then filePath. So you can create function in gradle build script wich would fetch release noted and strip html from text on your side.
For example
    releaseNotes = listOf(
        ru.cian.huawei.publish.ReleaseNote(
            lang = "ru-RU",
            text = getReleaseNotes("$projectDir/release-notes-ru.txt")
        ),
        ru.cian.huawei.publish.ReleaseNote(
            lang = "en-US",
            filePath = "$projectDir/release-notes-en.txt"
        )
    )

and
fun getReleaseNotes(filePath: String): String {
	return File(filePath).readAll().replaceAll("\\<[^>]*>","")
}

What do you think about it?

The best option would be to support Triple-T structure out of the box.

@rtsisyk How does Triple-T Release Notes structure help us to solve problem that AppGallery doesn't support html tags?

@rtsisyk Hi! Please try a new 1.4.1-SNAPSHOT version with a new removeHtmlTags parameter. For more information see README/snapshot-1.4.1.
Pay attention I've changed releaseNotes block. Instead of
releaseNotes = listOf(
                ru.cian.rustore.publish.ReleaseNote(
                          lang = "en-US",
                          filePath = "$projectDir/release-notes-en.txt"
                ),
            )

new configuration
          releaseNotes = ru.cian.huawei.publish.ReleaseNotesExtension(
              descriptions = listOf(
                  ru.cian.huawei.publish.ReleaseNote(
                          lang = "en-US",
                          filePath = "$projectDir/release-notes-en.txt"
                  )
              ),
              removeHtmlTags = false
          )

And I remember you are use a trick to support Google Play languages. A new version should be
huaweiPublish {
  instances {
    huaweiRelease {
      credentialsPath = "$rootDir/huawei-appgallery.json"
      buildFormat = 'aab'
      deployType = 'draft' // confirm manually
      releaseDescriptions = []
      def localeOverride = [
          'am' : 'am-ET',
          'gu': 'gu_IN',
          'iw-IL': 'he_IL',
          'kn-IN': 'kn_IN',
          'ml-IN': 'ml_IN',
          'mn-MN': 'mn_MN',
          'mr-IN': 'mr_IN',
          'ta-IN': 'ta_IN',
          'te-IN': 'te_IN',
      ]
      def files = fileTree(dir: "$projectDir/src/google/play/release-notes",
          include: '**/default.txt')
      files.each { File file ->
        def path = file.getPath()
        def locale = file.parentFile.getName()
        locale = localeOverride.get(locale, locale)
        releaseDescriptions.add(new ru.cian.huawei.publish.ReleaseNote(locale, path))
      }
      releaseNotes = new ru.cian.huawei.publish.ReleaseNotesExtension(releaseDescriptions)
    }
  }
}


cdcon 2021: define what timezones we are going to support

2-3 day event (eastern time)

Americas (2 day event) will start @ 7am EST


16 jest test cases rendering components

Added few basic test cases which should be helpful for you guys to understand how the test cases are written
all the basic test cases are here
3 important key notes

understand how the test case is written (format)
see what were the criteria's mentioned for them to pass
understand how to use Moxios and Stubs to generate fake requests

Great job writing these. It gives us a good idea of how to structure test cases going forward with our components. Made great use of utilizing the utils files.


Migration of db fails when upgrading to 2.14.0

Bug description
Error when moving from 2.13.1 docker image to 2.14.0 (latest). Failed migration: while backing up db before migration: read /data/portainer.db: permission denied. Also I do see files in backups/common/ like portainer.db.035.20220628192528 but they are 0 bytes.
I am able to rollback to the image 2.13.1 and access the db fine.
Expected behavior
New version migration completes successfully.
Portainer Logs
level=info msg="Encryption key file portainer not present"
level=info msg="Proceeding without encryption key"
level=info msg="Loading PortainerDB: portainer.db"
level=info msg="Opened existing store" version=35
level=info msg="2022/06/28 19:25:28 [INFO] [database, backup] [message: creating db backup]"
level=info msg="2022/06/28 19:25:28 [INFO] [database, backup] [message: Copying db file from /data/portainer.db to /data/backups/common/portainer.db.035.20220628192528]"
level=info msg="2022/06/28 19:25:28 [ERROR] [database, backup] [message: Failed] [error: read /data/portainer.db: permission denied]"
level=fatal msg="Failed migration: while backing up db before migration: read /data/portainer.db: permission denied"
Steps to reproduce the issue:
Pulled new image (latest) using docker-compose and started container.
Technical details:

Portainer version:
Docker version (managed by Portainer):
Kubernetes version (managed by Portainer):
Platform (windows/linux):
Command used to start Portainer (docker run -p 9443:9443 portainer/portainer):
Browser:
Use Case (delete as appropriate): Using Portainer at Home, Using Portainer in a Commercial setup.
Have you reviewed our technical documentation and knowledge base? Yes/No

Additional context
Add any other context about the problem here.

I ran into this same problem and figured out it had to do with file locks on the portainer.db file. My Portainer data directory is in network mount. To get around this for the upgrade I temporarily moved the data directory to the host machine, the DB migration succeeded and everything came up. Then I moved the data directory back to the network mount and it came up fine, not needing to copy the portainer.db file anymore.
Hope that helps.

My data directory is also a network volume as well, I figured this may have something to do with the issue but had not gone as far as moving it local. Moving it locally is a work around for now but don't think that's a long term solution if this is going to break every database upgrade going forward.

Thanks for the feedback on this - we've opened an internal issue around this to see if we can figure out why this is happening and get a resolution sorted.

@nickwest @TheAceMan
Are you able to give me a bit more detail on the sort of network mounts you're using? We're trying to replicate this behavior internally so we can find a fix,

Here is my docker-compose config
services:
portainer:
container_name: portainer
image: portainer/portainer-ce:latest
ports:
- "$PORTAINER_PORT:9000/tcp"
volumes:
- portainer_data:/data
volumes:
portainer_data:
driver_opts:
o: "username=xyz,password=xyz"
type: "cifs"
device: \<IP_ADDRESS>\portainer\data

@jamescarppe The mounts in my setup are NFS mounts sourcing from a Windows File Server on each Docker Swarm host, then simple file mounts in Docker.

Closing, as there's a fix for this in 2.14.1 which will be released imminently.

Not quite.
https://github.com/portainer/portainer/pull/10721 resolved an issue with the upgrade and --rollback process where the database was being backed up to a name that the rollback was not expecting. This was actually only in BE (also known as EE) Portainer - and not CE.

Thanks, should I open a new bug? I am getting a portainer.db: permission denied when I upgrade to CE 2.19.4. But works fine when I went back to CE 2.19.3.
2.19.4 log:
2023/12/07 12:45AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:369 > encryption key file not present | filename=portainer
2023/12/07 12:45AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:392 > proceeding without encryption key |
2023/12/07 12:45AM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
2023/12/07 12:45AM INF github.com/portainer/portainer/api/datastore/backup.go:16 > Backing up database | from=/data/portainer.db to=/data/backups/portainer.db.bak
2023/12/07 12:45AM WRN github.com/portainer/portainer/api/datastore/backup.go:19 > failed to create backup file | error="read /data/portainer.db: permission denied"
2023/12/07 12:45AM FTL github.com/portainer/portainer/api/cmd/portainer/main.go:142 > failed migration | error="while backing up database: read /data/portainer.db: permission denied" stack_trace=[{"func":"(*Store).MigrateData","line":"45","source":"migrate_data.go"},{"func":"initDataStore","line":"140","source":"main.go"},{"func":"buildServer","line":"395","source":"main.go"},{"func":"main","line":"640","source":"main.go"},{"func":"main","line":"250","source":"proc.go"},{"func":"goexit","line":"1598","source":"asm_amd64.s"}]

Yes please, if you could open a bug, we will look into it.
It could be that the recent change in 2.19.4 around backup and rollback caused something else.

Thanks, bug entered.
https://github.com/portainer/portainer/issues/10780

Thank you.
Note that you may be right that the locking was accidently reintroduced - I don't know.
We'll look into it though.

Thankyou @TheAceMan !


Typo

There is no 'tabBarTitle' in a StackNavigator.

You are correct that there is no tabBarLabel (your comment says tabBarTitle but the PR removes tabBarLabel) in a StackNavigator. However, if the StackNavigator is nested within a TabNavigator, and if the TabNavigator's routes do not have a title specified, the StackNavigator screen that is present will display it's title in the tabBarLabel. Take a look at the following example:
import React from 'react';
import { Button, ScrollView } from 'react-native';
import { StackNavigator, TabNavigator } from 'react-navigation';

const MyNavScreen = ({ navigation }) => (
  <ScrollView>
    <Button
      onPress={() => navigation.navigate('Profile', { name: 'Jordan' })}
      title="Open profile screen"
    />
    <Button
      onPress={() => navigation.navigate('NotifSettings')}
      title="Open notifications screen"
    />
    <Button
      onPress={() => navigation.navigate('SettingsTab')}
      title="Go to settings tab"
    />
    <Button onPress={() => navigation.goBack(null)} title="Go back" />
  </ScrollView>
);

const MyHomeScreen = ({ navigation }) => (
  <MyNavScreen navigation={navigation} />
);

const MyProfileScreen = ({ navigation }) => (
  <MyNavScreen
    navigation={navigation}
  />
);

const MyNotificationsSettingsScreen = ({ navigation }) => (
  <MyNavScreen navigation={navigation} />
);

const MySettingsScreen = ({ navigation }) => (
  <MyNavScreen navigation={navigation} />
);

const MainTab = StackNavigator({
  Home: {
    screen: MyHomeScreen,
    path: '/',
    navigationOptions: {
      title: 'Welcome',
    },
  },
  Profile: {
    screen: MyProfileScreen,
    path: '/people/:name',
    navigationOptions: ({ navigation }) => ({
      title: `${navigation.state.params.name}'s Profile!`,
    }),
  },
});

const SettingsTab = StackNavigator({
  Settings: {
    screen: MySettingsScreen,
    path: '/',
    navigationOptions: () => ({
      title: 'Settings',
    }),
  },
  NotifSettings: {
    screen: MyNotificationsSettingsScreen,
    navigationOptions: {
      title: 'Notifications',
    },
  },
});

const StacksInTabs = TabNavigator(
  {
    MainTab: {
      screen: MainTab,
      path: '/'
    },
    SettingsTab: {
      screen: SettingsTab,
      path: '/settings'
    },
  },
  {
    tabBarPosition: 'bottom',
    animationEnabled: false,
    swipeEnabled: false,
  }
);

export default StacksInTabs;

When I navigate to the Notifications screen, the tabBarLabel updates to display "Notifications". When I go back to the Settings screen, the tabBarLabel updates to "Settings". These titles are set inside the tab's screens, which are StackNavigators.
The documentation covering this is vague. I will close this PR for now since this is not a typo. I encourage you to open another PR with improved documentation to help others understand this feature 😄


Publish outboxer to GitHub Container Registry

To be able to use outboxer in the ODS, I propose to publish a docker image to the new GitHub Container Registry. The main advantage over the old GitHub Packages Docker registry is that public container images can be accessed anonymously (e.g. without authentication). And the GitHub Container Registry will supersede the GitHub Packages Docker registry one day (see here and here).
GitHub Container Registry does require a personal access token (PAT) and does not work with the default GitHub secret GITHUB_TOKEN. So before merging this, a PAT must be created and added to the repository secret with the name GHCR_PAT. The PAT must have the scopes read:packages and write:packages. Currently, there is a bug in the PAT creating page, that always adds the not needed repo scope and does not allow to remove it. But the repo scope can be removed in the edit screen after creating the PAT.
Currently, every commit to the main branch will get published as a new docker image, so we can iterate faster while fixing the last bugs. Once we use the outboxer in the ODS we should, of course, add proper versioning.
Any feedback or concern about this process is highly appreciated.

Added the GHCR_PAT as CI secret

Added the GHCR_PAT as CI secret

@georg-schwarz you must also enable the Github Container Registry for the jvalue org

@georg-schwarz you must also enable the Github Container Registry for the jvalue org

According to the CI the image has been build and published successfully. But I can not see it here, because any new package/image is private by default. So there is one last step: you must change the visibility of the image to public (see here).

According to the CI the image has been build and published successfully. But I can not see it here, because any new package/image is private by default. So there is one last step: you must change the visibility of the image to public (see here).

Done, can you recheck please?

Done, can you recheck please?

Done, can you recheck please?

It works now 👍  🚀

Done, can you recheck please?

It works now 👍  🚀


Enforce non-zero CertModel

Fix #2581.
Signed-off-by: Steven Bellock<EMAIL_ADDRESS>

I'll add the unit tests for CSR in a subsequent push.

I'll add the unit tests for CSR in a subsequent push.

The unit test has been added.

@steven-bellock , have you validated spdm-emu?
I notice spdm-emu failed on set_cert with latest libspdm. Would you please take a look?
libspdm_set_certificate - 80010001
do_certificate_provising_via_spdm - 80010001
do_session_via_spdm - 80010001

have you validated spdm-emu?

I have not. I'll have a look at it.

@jyao1 the spdm-emu issue is due to the Requester using the illegal combination

MULTI_KEY_CONN_RSP is true.
Erase is false.
CertModel is 0.

In particular if MULTI_KEY_CONN_RSP is true then the Integrator needs to use the libspdm_set_certificate_ex function.

As an aside, after re-reading the specification and looking at this pull request, it might be too aggressive with the key_pair_id check. For SET_CERTIFICATE it currently has
        if (spdm_context->connection_info.multi_key_conn_rsp) {
            if (key_pair_id == 0) {
                return LIBSPDM_STATUS_INVALID_PARAMETER;
            }

But maybe this should only be checked if Erase is true? I'll make a follow-up issue and pull request.

Yes, please submit PR to fix. We need this patch for Q2 release.

I have a fix for spdm-emu but #2746 needs to be fixed on the libspdm side first.


new errors, not sure if svui or not?

199x ...ddOns\Blizzard_ArtifactUI\Blizzard_ArtifactPerks.lua:24: Usage: GetAppearanceInfoByID(artifactAppearanceID)
[C]: in function GetAppearanceInfoByID' ...ddOns\Blizzard_ArtifactUI\Blizzard_ArtifactPerks.lua:24: in function RefreshModel'
...ddOns\Blizzard_ArtifactUI\Blizzard_ArtifactPerks.lua:180: in function `TryRefresh'
...ddOns\Blizzard_ArtifactUI\Blizzard_ArtifactPerks.lua:81: in function <...ddOns\Blizzard_ArtifactUI\Blizzard_ArtifactPerks.lua:80>
Locals:
(*temporary) = nil

2x ...ars\libs\LibActionButton-1.0\LibActionButton-1.0-57.lua:1205: attempt to index field 'parent' (a nil value)
...ars\libs\LibActionButton-1.0\LibActionButton-1.0-57.lua:1205: in function <...ars\libs\LibActionButton-1.0\LibActionButton-1.0.lua:1202>
Locals:
self = LAB10ChargeCooldown2 {
0 =
}
(*temporary) = nil
(*temporary) = LAB10ChargeCooldown2 {
0 =
}
(*temporary) = UIParent {
0 =
firstTimeLoaded = 1
variablesLoaded = true
}
(*temporary) = "attempt to index field 'parent' (a nil value)"
tinsert =  defined =[C]:-1
lib =  {
GetAllButtons =  defined @SVUI_ActionBars\libs\LibActionButton-1.0\LibActionButton-1.0.lua:367
nonActionButtons =  {
}
ACTION_HIGHLIGHT_MARKS =  {
}
callbacks =  {
}
numOverlays = 1
activeButtons =  {
}
UnregisterAllCallbacks =  defined @AngryAssignments\libs\CallbackHandler-1.0\CallbackHandler-1.0.lua:200
unusedOverlayGlows =  {
}
actionButtons =  {
}
RegisterCallback =  defined @AngryAssignments\libs\CallbackHandler-1.0\CallbackHandler-1.0.lua:116
UnregisterCallback =  defined @AngryAssignments\libs\CallbackHandler-1.0\CallbackHandler-1.0.lua:179
NumChargeCooldowns = 4
buttonRegistry =  {
}
CreateButton =  defined @SVUI_ActionBars\libs\LibActionButton-1.0\LibActionButton-1.0.lua:143
eventFrame =  {
}
ChargeCooldowns =  {
}
}

The first one has nothing to do with SVUI as far as I can tell as SVUI doesn't touch the GetAppearanceInfoByID or anything to do with Artifact Perks (yet).
The second errorseems to be an incompatability with AngryAssignments... I'll have a look at it and see what I can do.

No further update or sight of this error. I'm going to assume it's now closed by updating to latest LibActionButton


Fixed a typo

imparative -> imperative

Coverage remained the same at 100.0% when pulling 98e09783dae3c9742db8b4f21417fadfa7bf184a on premithk:master into 0a975ca9c32bfefa046220015491c6fbd3e9720d on evilsoft:master.

Coverage remained the same at 100.0% when pulling 98e09783dae3c9742db8b4f21417fadfa7bf184a on premithk:master into 0a975ca9c32bfefa046220015491c6fbd3e9720d on evilsoft:master.

👍 have a squirrel.

@premithk do you mind if I add you to the contributors list for documentation?

Sure, ill be honored. Thank you, really interested in this lib, though am just a starter in functional programming
-----Original Message-----
From: "Ian Hofmann-Hicks"<EMAIL_ADDRESS>Sent: 26/02/2018 21:29
To: "evilsoft/crocks"<EMAIL_ADDRESS>Cc: "Premith"<EMAIL_ADDRESS>"Mention"<EMAIL_ADDRESS>Subject: Re: [evilsoft/crocks] Fixed a typo (#235)
@premithk do you mind if I add you to the contributors list for documentation?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

@premithk awesome!
Welcome to FP, it is super fun!
We have a gitter channel here if you wanna ask any questions or show off your code or anything really!


Fix Incorrect Height and Width for Image Perturbations

Description
[TODO]
Please include a summary of the change, motivation and which issue is fixed. Any dependencies changes should also be included.
Fixes #2045
Type of change
Please check all relevant options.

[ ] Improvement (non-breaking)
[x] Bug fix (non-breaking)
[ ] New feature (non-breaking)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Testing
Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.

[ ] Test A
[ ] Test B

Test Configuration:

OS
Python version
ART version or commit number
TensorFlow / Keras / PyTorch / MXNet version

Checklist

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes

Codecov Report

Merging #2046 (21bf5b7) into dev_1.14.0 (e73c1e9) will decrease coverage by 13.21%.
The diff coverage is 83.33%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@               Coverage Diff               @@
##           dev_1.14.0    #2046       +/-   ##
===============================================
- Coverage       80.05%   66.85%   -13.21%
===============================================
  Files             291      291
  Lines           25564    25564
  Branches         4621     4621
===============================================
- Hits            20466    17090     -3376
- Misses           4003     7559     +3556
+ Partials         1095      915      -180

Impacted Files
Coverage Δ

...cks/poisoning/perturbations/image_perturbations.py
53.84% <83.33%> (-35.90%)
:arrow_down:

art/attacks/evasion/carlini.py
7.55% <0.00%> (-86.74%)
:arrow_down:

art/attacks/evasion/pe_malware_attack.py
14.74% <0.00%> (-84.34%)
:arrow_down:

art/attacks/poisoning/gradient_matching_attack.py
8.61% <0.00%> (-80.53%)
:arrow_down:

art/attacks/evasion/graphite/utils.py
10.37% <0.00%> (-79.25%)
:arrow_down:

art/optimizers.py
21.42% <0.00%> (-78.58%)
:arrow_down:

...s/extraction/functionally_equivalent_extraction.py
14.19% <0.00%> (-78.40%)
:arrow_down:

..._air_flickering/over_the_air_flickering_pytorch.py
17.91% <0.00%> (-76.87%)
:arrow_down:

art/attacks/evasion/decision_tree_attack.py
23.28% <0.00%> (-76.72%)
:arrow_down:

art/attacks/evasion/graphite/graphite_blackbox.py
9.12% <0.00%> (-76.50%)
:arrow_down:

... and 99 more


pip install fails with gcc-c++-9.2.1

Minor issue, on fedora 31 with gcc-c++-9.2.1 the pip install fails due to syntax warning/errors.
Was able to work around it by adding to setup.py cmake_args:
'-DCMAKE_CXX_FLAGS=-Wno-error=parentheses'
Hopefully syntax can be corrected or the above added to avoid compilation errors.
Thank you :)
In file included from /tmp/pip-install-w3131icp/cppkin/build/temp.linux-x86_64-3.7/Third_Party/include/boost/mpl/aux_/na_assert.hpp:23,
                 from /tmp/pip-install-w3131icp/cppkin/build/temp.linux-x86_64-3.7/Third_Party/include/boost/mpl/arg.hpp:25,
                 from /tmp/pip-install-w3131icp/cppkin/build/temp.linux-x86_64-3.7/Third_Party/include/boost/variant/variant_fwd.hpp:19,
                 from /tmp/pip-install-w3131icp/cppkin/build/temp.linux-x86_64-3.7/Third_Party/include/boost/variant/variant.hpp:26,
                 from /tmp/pip-install-w3131icp/cppkin/build/temp.linux-x86_64-3.7/Third_Party/include/boost/variant.hpp:17,
                 from /tmp/pip-install-w3131icp/cppkin/src/BinaryAnnotation.h:5,
                 from /tmp/pip-install-w3131icp/cppkin/src/span_impl.h:10,
                 from /tmp/pip-install-w3131icp/cppkin/src/Span.h:5,
                 from /tmp/pip-install-w3131icp/cppkin/src/SpanContainer.h:7,
                 from /tmp/pip-install-w3131icp/cppkin/src/SpanContainer.cpp:2:
/tmp/pip-install-w3131icp/cppkin/build/temp.linux-x86_64-3.7/Third_Party/include/boost/mpl/assert.hpp:188:21: error: unnecessary parentheses in declaration of ‘assert_arg’ [-Werror=parentheses]
  188 | failed ************ (Pred::************
      |                     ^
/tmp/pip-install-w3131icp/cppkin/build/temp.linux-x86_64-3.7/Third_Party/include/boost/mpl/assert.hpp:193:21: error: unnecessary parentheses in declaration of ‘assert_not_arg’ [-Werror=parentheses]
  193 | failed ************ (boost::mpl::not_<Pred>::************
      |                     ^
cc1plus: all warnings being treated as errors

Thanks, i will review it :). feel free to provide a solution if you want and i will merge it :).


MWPW-141290 | Icon stack text content to contain link and plain text

Icon in brick icon stack can contain picture, text and links. Adding the support to display icon and text in correct order and support link text and plain text combination.
Resolves: MWPW-141290
Test URLs:

Before: https://main--milo--adobecom.hlx.page/drafts/mathuria/masonry/fragments/brick-cards?martech=off
After: https://mwpw-141290--milo--aishwaryamathuria.hlx.page/drafts/mathuria/masonry/fragments/brick-cards?martech=off

Validated the change in the below url :
https://mwpw-141290--milo--aishwaryamathuria.hlx.page/drafts/mathuria/masonry/fragments/brick-cards?martech=off


Ruby Warnings

This patch fixes three kinds of Ruby warnings that we see when requiring this gem.
$  ruby -rlicense_finder -wep                                                                                ~/s/g/p/LicenseFinder (master|✔)
<internal:.../rubygems/core_ext/kernel_require.rb>:136: warning: <internal:.../rubygems/core_ext/kernel_require.rb>:136: warning: loading in progress, circular require considered harmful - .../license_finder-7.1.0/lib/license_finder/packages/go_package.rb
	from <internal:.../rubygems/core_ext/kernel_require.rb>:135:in  'require'
	from <internal:.../rubygems/core_ext/kernel_require.rb>:141:in  'rescue in require'
	from <internal:.../rubygems/core_ext/kernel_require.rb>:141:in  'require'
	from .../license_finder-7.1.0/lib/license_finder.rb:13:in  '<top (required)>'
	from <internal:.../rubygems/core_ext/kernel_require.rb>:136:in  'require'
	from <internal:.../rubygems/core_ext/kernel_require.rb>:136:in  'require'
	from .../license_finder-7.1.0/lib/license_finder/core.rb:9:in  '<top (required)>'
	from <internal:.../rubygems/core_ext/kernel_require.rb>:136:in  'require'
	from <internal:.../rubygems/core_ext/kernel_require.rb>:136:in  'require'
	from .../license_finder-7.1.0/lib/license_finder/package_manager.rb:157:in  '<top (required)>'
	from <internal:.../rubygems/core_ext/kernel_require.rb>:136:in  'require'
	from <internal:.../rubygems/core_ext/kernel_require.rb>:136:in  'require'
	from .../license_finder-7.1.0/lib/license_finder/package_managers/go_modules.rb:3:in  '<top (required)>'
	from <internal:.../rubygems/core_ext/kernel_require.rb>:136:in  'require'
	from <internal:.../rubygems/core_ext/kernel_require.rb>:136:in  'require'
	from .../license_finder-7.1.0/lib/license_finder/packages/go_package.rb:3:in  '<top (required)>'
	from <internal:.../rubygems/core_ext/kernel_require.rb>:136:in  'require'
	from <internal:.../rubygems/core_ext/kernel_require.rb>:136:in  'require'
	from .../license_finder-7.1.0/lib/license_finder/package.rb:174:in  '<top (required)>'
	from <internal:.../rubygems/core_ext/kernel_require.rb>:136:in  'require'
	from <internal:.../rubygems/core_ext/kernel_require.rb>:136:in  'require'

.../license_finder-7.1.0/lib/license_finder/packages/npm_package.rb:97: warning: ambiguity between regexp and two divisions: wrap regexp in parentheses or add a space after '/' operator
.../license_finder-7.1.0/lib/license_finder/package_managers/yarn.rb:99: warning: character class has duplicated range
.../license_finder-7.1.0/lib/license_finder/package_managers/yarn.rb: warning: character class has duplicated range
.../license_finder-7.1.0/lib/license_finder/package_managers/yarn.rb:100: warning: character class has duplicated range
.../license_finder-7.1.0/lib/license_finder/package_managers/yarn.rb: warning: character class has duplicated range
.../license_finder-7.1.0/lib/license_finder/package_managers/yarn.rb:178: warning: character class has duplicated range
.../license_finder-7.1.0/lib/license_finder/package_managers/yarn.rb: warning: character class has duplicated range
.../license_finder-7.1.0/lib/license_finder/package_managers/yarn.rb: warning: character class has duplicated range
.../license_finder-7.1.0/lib/license_finder/package_managers/yarn.rb: warning: character class has duplicated range
.../license_finder-7.1.0/lib/license_finder/package_managers/rebar.rb:44: warning: character class has duplicated range: /(?<name>\w+)─(?<version>[\S.]+)\s*/
.../license_finder-7.1.0/lib/license_finder/package_managers/rebar.rb: warning: character class has duplicated range: /(?<name>\w+)─(?<version>[\S.]+)\s*/

The first commit is a simple fix for / ambiguity.

The next commit is a fix for an obvious circluar require between package.rb and packages/go_package.rb introduced via 860de9ea6f25b91c8dea1558a687e3d76a523d35.

The third commit fixes a redundant Regexp literal.

You may notice some more warnings in package_managers/yarn.rb, but this seems to be already fixed via f57745be34bb7774577ea577671ea7b5c3246576 in master branch.

@amatsuda Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.

@amatsuda This Pull Request contains an obvious fix. Signing the Contributor License Agreement is not necessary.


Bump to 0.3.0

Description

Additional Changes:

Update conda-env and pre-commit dependencies
Update CONTRIBUTING.rst
Add team.rst and team.yml

Related issues

https://github.com/sphinx-contrib/napoleon/issues/38
https://github.com/pangeo-data/xESMF/issues/170

Checklist

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] My changes generate no new warnings
[x] Any dependent changes have been merged and published in downstream modules

If applicable:

[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass with my changes (locally and CI/CD build)
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[ ] I have noted that this is a breaking change for a major release (fix or feature that would cause existing functionality to not work as expected)

Redoing #266 because I forgot to update the changelog.


Allow adding hostIPC to inferenceservice pod spec

/kind feature
Add hostIPC to pytorch pod spec to be able to use VGPUS
I tried to add hostIPC to InferenceService spec but its not propagated to the actual pod so the MPS server cannot communicate with the inferenceservice pod.

I have tried to add to InferenceService:
cat<<EOF | kubectl apply -f -
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: model1
spec:
  predictor:
    tolerations:
    - key: "k8s.amazonaws.com/vgpu"
      operator: "Exists"
      effect: "NoSchedule"
    hostIPC: true

But it does not get propagated to inference service Pod.
I its listed here as supported. Is there a way to use it?
** Without this option is not possible to use VGPUs with kserve **
Thanks in advance.


RTDE protocol v1 is not supported in setup outputs

When trying to setup outputs in the rtde client using protocol v1 the rtde client will crash and the output when it is crashing can be seen below.
./build/examples/rtde_client_example
WARN /src/Universal_Robots_Client_Library/include/ur_client_library/comm/pipeline.h 406: No realtime capabilities found. Consider using a realtime system for better performance
terminate called after throwing an instance of 'urcl::UrException'
  what():  Could not parse received package. This can occur if the driver is started while the robot is booting - please restart the driver once the robot has finished booting. If the problem persists after the robot has booted, please contact the package maintainer.
Aborted (core dumped)
****

This happens because the control_package_setup_outputs class tries to parse a buffer that is longer than the received one, because the answer from the robot is different whether you use protocol v1 or v2. The error occurs in the setupOutputs function, when trying to parse the package received from the robot. I will update the client library to support protocol v1 as well, by updating the control_package_setup_outputs class to handle parsing protocol v1.

Should be resolved by #114 -> closing


Error loading java agent with jdk-16

Hello,
I'm trying to run minecraft 1.16.5, but i keep getting "error loading java agent" error.
I can run any minecraft version >1.16.5 using jdk 17+ just fine, but versions <1.17 with jdk <17 won't start.
I'm running gradle 8.10.2
Build scan

run task doesn't specify any java agent, if you have questions about running various Paper versions on various JDK versions you should ask in the Paper Discord.


Unable to download pre-trained models

unable to download one of the two pre-trained models in the website, did you provide?

Hi,
Which model are you not able to download? Is there any error from dropbox?
Marek

this is pre-trained models, its name here

Hi,
I don't really understand what you mean, try using this address:
https://www.dropbox.com/sh/v754z1egib0hamh/AADGX1SE9GCj4h3eDazsc0bXa?dl=0
It is the same address as in the readme.
Marek

unable to download one of the two pre-trained models in the website, can you check it ?

Hi, I just tried it and it works. What message are you getting? The link should lead to dropbox.
Marek

Hi, the link indeed lead to dropbox ,but when download it ,hints error.

Hi, I need more details, what kind of an error?
Are those direct links working?
https://www.dropbox.com/s/kfblc61u2jjfg1k/DAN.npz?dl=0
https://www.dropbox.com/s/czv8fhzyg1ggmx5/DAN-Menpo.npz?dl=0

@MarekKowalski  Hi, our country cannot access dropbox, could you please upload models to google driver?

Hi,
Thanks for letting me know, I have added links to Google Drive in the readme.
Marek


Fix empty name in email address

A "name" property must exist in the model in order for the Mail class to correctly display the name in addresses.

@gnovaro you can check if the same error occurs in other models such as customers, leads, companies, etc. initially there should be no further problem.


SliderJoint, Angular Limit on wrong Axis?

Godot version:
3.1 beta3
OS/device including version:
Windows 10
Issue description:
I would call this not a bug, but a design flaw because this class is not working as documented. (IMHO)
The docs say about the function: "Piston kind of slider between two bodies in 3D."
But actually, the slider slides along the X-Axis ....and rotates around the X-Axis. This would allow twisting the connected piston rod but not the typical "up/down" movement. So the expected behaviour of a piston slider would be to allow the rotation either on the z- (preferred) or y-axis when sliding on the x-axis.
Here's a screen shot of a non-working SliderJoint configuration:

In the foreground you see a slider joint with a linear limit (two cyan rectangles) and an angular limit (90° segment to the left) set. The indicated limits are exactly performed the way they are shown when the game runs. But as I said, the angular limit makes IMHO no sense for a piston like movement.
Steps to reproduce:
Add slider joint to a project, edit the linear and angular limits.
Minimal reproduction project:
I stepped upon this problem while creating a 3D Joint sampler/demo. I'll add this here when it is ready. But I'll add a workaround (use Generic6DOFJoint instead) to that project to get it working.

Generic6DOFJoint can be used for use cases like this. Documentation needs to be updated so it only mentions slider and not piston, which is not an accurate description of this joint.


Optional parameter to make Buffer() always use current window

Add and optional boolean parameter to Buffer function to control going to other windows.
Default value is true to maintain current behavior
This allows to always reuse current window when switching buffers, similar to default :buffer command
(I personally use Window() to jump to an already "visible" buffer 😉 )
If you prefer, this also could be done with an option variable like in https://github.com/girishji/scope.vim/commit/f4bb40bb781b6878bc7f426a1a2ed085eba02c28 instead of changing the function API

Thanks!


COSIM: Virtual Mode ISA Tests now failing after Chipyard bump to v1.7.0

Seems like they have been failing for a while... as there seems to be a failure with CEP v4.0 as well (prior to chipyard bump)
So, for some reason they fail on Ubuntu w/Questasim, but not RHEL7 w/XCellium. Further investigation required.

They current work on XCellium, but not Questasim. (v4.5_development_bchetwynd branch)

Current log excerpt from virtual mode failures:``
INFO:       15 cep_tb.cpuId[2].driver Entering shIpc_EnableMode Loop simTime = 1
INFO:       15 cep_tb.cpuId[3].driver Entering shIpc_EnableMode Loop simTime = 1
INFO:       15 cep_tb.system_driver Entering shIpc_EnableMode Loop simTime = 1
       0 cep_tb.system_driver C_LOG: loadMemory: Loading file ./riscv_wrapper.img to SCRATCHPAD RAM with maxByteCnt of 655360B , fileOffset = 0B, destOffset = 0B
       0 cep_tb.system_driver C_LOG: loadMemory: flushing cache line
       0 cep_tb.system_driver C_LOG: loadMemory: Setting program loaded flag AND releasing all tiles from reset...
INFO:    31985 cep_tb.system_driver Program is now loaded
INFO:    31995 cep_tb.system_driver Releasing all tiles from reset...
INFO:    35025 cep_tb.cpuId[1].driver SingleCoreOnly = 1
INFO:    35025 cep_tb.cpuId[2].driver SingleCoreOnly = 1
INFO:    35025 cep_tb.cpuId[3].driver SingleCoreOnly = 1
INFO:    35025 cep_tb.cpuId[2].driver C2 Pass/Fail Detected!!! 1/0... Put it to sleep
INFO:    35025 cep_tb.cpuId[3].driver C3 Pass/Fail Detected!!! 1/0... Put it to sleep
       0 cep_tb.cpuId[0].driver C_LOG: access::RunClk mSlotId=0 mLocalId=0 numClk=1000
       0 cep_tb.cpuId[1].driver C_LOG: access::RunClk mSlotId=0 mLocalId=1 numClk=1000
       0 cep_tb.cpuId[2].driver C_LOG: access::RunClk mSlotId=0 mLocalId=2 numClk=1000
       0 cep_tb.cpuId[3].driver C_LOG: access::RunClk mSlotId=0 mLocalId=3 numClk=1000
INFO:    35065 cep_tb.cpuId[1].driver C1 Pass/Fail Detected!!! 1/0...


[Feature Request] Improved UI feedback on mistyped character

It would be great to have the option to only highlight mistyped characters, instead of whole words with mistyped characters in them.
This would mean only coloring the incorrect character, not the word.
Furthermore adding the option to show the mistyped character, instead of the intended-to-be-typed character, in the Prompt-Text-Area would allow the user to recognize what was typed incorrectly, not only where.
With this option enabled it could also be argued that the Input-Text-Field could be disabled since all feedback could exist inside the Prompt-Text-Area.
What do you think:

Is this even wanted? Or is relying on CTRL+BACKSPACE, without further feedback the intended use.
If there should be more UI feedback, how many options should be there?

I'm willing to start implementing these features but would like to check with you what you think about it first.

Here's one version that I've been using https://github.com/bolphen/ttyper/commit/c4a19803690de1a7fb85c6c4387af207e4b90401
A byproduct of marking individual characters is that it also fixes some issues mentioned in https://github.com/max-niederman/ttyper/issues/77 (for example, accented letters like ä or é being counted as two characters).

I wonder if this could be upstreamed, though I don't see a PR for that.
Is there interest for such a feature?

@Sntx626 Just made a PR. Though there's already another one #90 doing a similar thing.


Sets disk_size to base_size+data_size

This pull requests sets a constant BASE_DISK_SIZE to 60 and adds that to what is requested by the use. It also limits the user to disks between 0 and 100GB (not counting the 60 that we add).
When creating the machines from API data we also subtract that constant so as to not make it confusing for the user, i.e, requesting a machine with disk_size_gb=0 and then seeing listed a machine with disk_size_gb=60. However, this should be handled by the backend. Opened a task here: https://github.com/inductiva/tasks/issues/97
handles: https://github.com/inductiva/tasks/issues/87

QA
@IvanPombo @Pbarbosa92
Launch machines of several types. For example:
import inductiva

mg = inductiva.resources.MPICluster(
    machine_type="c2-standard-16",
    num_machines=3,
    data_disk_gb=20,
)
mg.start()

mg = inductiva.resources.MachineGroup(
    machine_type="c2-standard-4",
    num_machines=1,
    data_disk_gb=0,
)
mg.start()

See if they are launched successfully. Then test the registered disk sizes, for example inductiva resources list. You can also test using inductiva.resources.machine_groups.list()

QA

1000GB drive

-1 GB drive

Works as expected.


Enable CDN for thumbnails

For users who are not based in the United states or within a couple thousand miles of it, thumbnails will load slowly because they need to come from our server half a world away. We should make sure that thumbnails are cached through Cloudflare.

We had Cloudflare's CDN enabled for the cc.engineering domain but not for its subdomains. This has been fixed.
It may take a while before common queries are cached.

Hi @aldenstpage. The thumbs endpoint is showing internal server error. Is it because that maybe queries are not cached yet?

Could you include some search terms and links? When I do a search, all of the thumbnails load.

https://api.creativecommons.engineering/v1/images?q=cat Every thumbnail I tried on this search result gives the error.
GIF for reference (opening url and thumbnail pairs)

Thanks for raising that! I hadn't noticed a configuration issue because the frontend falls back to full resolution images if the thumbnails don't load properly. It's fixed now.


[3.6 ] Bug fix/fix one shard traversal instantiation

Scope & Purpose
Instantiation of AQL queries with graph traversals in One-Shard would previously fail. This is already fixed in devel.

[X] Bug-Fix for 3.6
[X] The behavior in this PR can be (and was) manually tested

Related Information

[X] There is a JIRA Ticket number: ES-544

Testing & Verification
This change is trivial.
In devel there are a bunch of additional tests that cover this in conjunction with satellite graphs, which I will maybe backport separately with another bugfix.
Documentation

[X] Added a Changelog Entry

http://jenkins.arangodb.biz:8080/view/PR/job/arangodb-matrix-pr/9279/


nix::fcntl add FcntlArg::F_READAHEAD for freebsd.

Set/clear the amount of read ahead for the file descriptor.

Thanks


add AMax F32/F16 feature

add AbsMax feature to hipBlasLt with type F32 and F16

Add to sample or use bench instead.

remove sample first.
will develop ExtOp samples together.

Add to sample or use bench instead.

add AMax sample

@TonyYHsieh  per our dicussion, the AMax's output scalar should support FP32.

@TonyYHsieh per our dicussion, the AMax's output scalar should support FP32.

add support FP32 output scale for FP16/FP32 input


The splited context representation

Hi, Thanks very much for sharing your code for this project, it has been quite helpful.
I have some problem concerning your splitting strategy. The resulting context representation could be regarded as padded from the left(and also right, because the input data has already been padded on the right). Do you think it is plausible to left pad the context? Have you tried to explicitly split the query and context and then right pad them to fixed-length. I find it really hard to explicitly split them for the length of query is not fixed which makes left zeros on the context side hard to handle in a quick fashion.

I haven't tried explicit pad alignment because BERT by itself injects positional embeddings (refer paper for details). It would be a good thing to try using the masks which tell you the length of each query and context.
Thanks,
Ankit


[stable/grafana] syntax error in deployment.yaml in grafana 3.5.6

Describe the bug
Today, a new helm chart for grafana was release.  This was in it:
https://github.com/helm/charts/blob/9e0d2f23ced8ee73f51a09420b45b097bc89ab04/stable/grafana/templates/deployment.yaml#L78
It caused the YAML parser much chagrin.  Those quotes around the key name are no bueno.  The next line "args": also has the same issue.  Seems like it may have been a bug in converting from JSON to YAML?
Version of Helm and Kubernetes:
v2.14.1 and EKS v1.12.6
Which chart:  grafana
What happened:  YAML parser fail, doesn't install, rolled back to 3.5.4 I believe from 3.5.6
What you expected to happen: Things installed
How to reproduce it (as minimally and precisely as possible):  Just try to do a helm install grafana/stable -n grafana --namespace grafana and watch it go.  Make sure do do the repo update and the delete --purge first.
Anything else we need to know:  Not really.

Sorry it's due to my last PR I don't understand why some "" have been added
I'll fix that quikly


Disable plumage stylesheet

PR here to generate the staging pages. Personally I prefer the cleaner look without backgrounds/shadows, but that is up to taste.
No background also makes it a bit easier to seamlessly integrate tables, images, and maps, into the existing pages.

Nice idea. I am happy with that change.
One thing I have spotted that goes wrong: in the bar with the tabs at the top, we get now an extra entry at the right (i.e. to the left of "Home", which reads "OSCOVIDA: Open Science COVID Analysis". Maybe the now de-activated style file suppressed display of that site title? It points to the / of the page, where as "Home" next to it points to /index.html - so only "Home" should remain (that's as it was before).
Can you fix this?
Other than that it looks great to me.

Added a minimalist-style.css file which hides the navbar-brand element which fixes that problem, does it look alright now?


Feature requests for OGC GDC

I put here some possible improvements that could be useful to better support OGC GDC API:
For GET /coverage

[ ] Support subset: currently the bbox + datetime selection is reflected in the bbox + datetime parameters. Allow to choose if the user wants to use the subset parameter instead, with the same content. datetime=loTime/hiTime&bbox=loLon,loLat,hiLon,hiLat becomes subset=Lat(loLat:hiLat),Lon(loLon:hiLon),time(["loTime":"hiTime"]) maybe a switch to select if using subset or bbox + datetime?
[x] Support f parameter to select the requested file format.
[ ] Optionally support other possibly useful parameters like scale, crs etc. https://m-mohr.github.io/geodatacube-api/#tag/OGC-API-Coverages/operation/describe-coverage

Support subset: currently the bbox + datetime selection is reflected in the bbox + datetime parameters. Allow to choose if the user wants to use the subset parameter instead, with the same content. datetime=loTime/hiTime&bbox=loLon,loLat,hiLon,hiLat becomes subset=Lat(loLat:hiLat),Lon(loLon:hiLon),time(["loTime":"hiTime"]) maybe a switch to select if using subset or bbox + datetime?

What's the advantage we gain? Looks like the same to me.

Support f parameter to select the requested file format.

Is already supported. You need to specify mutliple coverage links with different media types to enable it. It then allows to select from the media types and sends them to f. Unfortunately, the Coverages spec is really weak in this regard and underspecifies how things are connected. There's no other good way to figure out what values for f are supported, excpect for parsing the OpenAPI file (which EURAC doesn't provide afaik).

* [ ]  Optionally support other possibly useful parameters like `scale`, `crs` etc. https://m-mohr.github.io/geodatacube-api/#tag/OGC-API-Coverages/operation/describe-coverage

Good idea, but here as well. Some of them are not discoverable, so from a UI perspective it's not clear where to get the allowed values from... crs can probably be read from the crs property (do you expose that?) but otherwise, scale may also work with the corresponding metadata, but properties I'm not sure how that could be made available in a user-friendly way.

CRS is available:


Feat: Adding select atom

What's the purpose of this pull request?
Adds the Select atom to store-ui

Hey the preview seems not be working https://deploy-preview-876--storeui.netlify.app/?path=/docs/atoms-select--default-select


KCE-65 Update kpsr-tutorial to use json examples instead of YAML examples

KCE-65

Please squash the commits -- it will require reapproval but I'll do that immediately.


Apply API changes and fix test fails

Removed CBLDatabaseConfiguration.flags and used kC4DB_Create | kC4DB_VersionVectors by default.
Renamed create object functions that use New to Create.
Renamed CBLDocument_ToMutable to CBLDocument_MutableCopy.
Renamed CBLDocument_ToJSON to CBLDocument_CreateJSON.
Fixed cbl++/Document’s adopt() to return null when document is not found instead of throwing an exception.
Fixed Save Empty Document to check RevID in non-version-vector format.
Fixed C++ Batch With Exception Test.
NOTE: This PR will be merged to dev branch.

Linux build error:
[2021-03-10T01:37:33.731Z] /var/lib/jenkins/mobile/workspace/_pipeline_couchbase-lite-C_PR95/test/QueryTest.cc:210:10: error: unknown type name '__unused'
[2021-03-10T01:37:33.731Z]     for (__unused auto &result : results)
[2021-03-10T01:37:33.731Z]          ^

Windows build failure:
[2021-03-10T01:35:45.362Z] CMake Error at test/CMakeLists.txt:39 (add_executable):
[2021-03-10T01:35:45.363Z]
[2021-03-10T01:35:45.363Z]   Cannot find source file:
[2021-03-10T01:35:45.363Z]
[2021-03-10T01:35:45.363Z]     C:/Jenkins/workspace/_pipeline_couchbase-lite-C_PR-95/vendor/couchbase-lite-core/vendor/fleece/MSVC/memmem.cc

I'm fixing those failures on a separate PR.


Fixes OS error arising from too many files open

Previously, DataStore  did not judiciously close files, resulting in opening a large number of files that
could result in an OSError related to too many files being open.  This merge provides a solution for the netCDF, scipy, and h5netcdf backends.

The intent of this PR is to address (or at least partially address) the following issues:

https://github.com/pydata/xarray/issues/463
https://github.com/pydata/xarray/issues/798
https://github.com/CCI-Tools/cate-core/issues/102
https://github.com/MPAS-Dev/MPAS-Analysis/issues/49

@shoyer, all the checks "pass" but there are still errors in the "allowed" list.  If you get a change could you please provide me some perspective on whether these are errors on my end or not?  I'm not exactly sure how to interpret them.
Once I know I have correctness in this code I plan to fix the inlines you graciously highlighted above.  I think we are getting close here, assuming that I have enough testing to demonstrate we have accurately fixed the too many open file issue.  Any additional ideas you have for tests would be really helpful too.

@pwolfram the allowed failures are pre-existing, not related to this change.

Thanks @shoyer, Does that mean if the checks pass the code is at least minimally correct in terms of not breaking previous design choices?  E.g., does this imply that we are ok except for cleanup / implementation details on this PR?

Does that mean if the checks pass the code is at least minimally correct in terms of not breaking previous design choices? E.g., does this imply that we are ok except for cleanup / implementation details on this PR?

If the checks pass, it means that it doesn't directly break anything that we have tests for. Which should cover most functionality. However, we'll still need to be careful not to introduce performance regressions -- we don't have any automated performance tests yet.

@shoyer, I just realized this might conflict with #1087.  Do you foresee this causing problems and what order do you plan to merge this PR and #1087 (which obviously predates this one...)?  We are running into the snag with #463 in our analysis and my personal preference would be to get some type of solution into place sooner than later.  Thanks for considering this request.
Also, I'm not sure exactly the best way to test performance either.  Could we potentially use something like the "toy" test cases for this purpose?  Ideally we would have a test case with O(100)  files to gain a clearer picture of the performance cost of this PR.
Please let me know what you want me to do with this PR-- should I clean it up in anticipation of a merge or just wait for now to see if there are extra things that need fixed via additional testing?  Note I have the full scipy, h5netcdf and pynio implementations that can also be reviewed because they weren't available when you did your review yesterday.

Don't worry about #1087 -- I can rebase it.
On Wed, Jan 11, 2017 at 8:54 PM Phillip Wolfram<EMAIL_ADDRESS>wrote:

@shoyer https://github.com/shoyer, I just realized this might conflict
with #1087 https://github.com/pydata/xarray/pull/1087. Do you foresee
this causing problems and what order do you plan to merge this PR and
#1087 https://github.com/pydata/xarray/pull/1087 (which obviously
predates this one...)? We are running into the snag with #463
https://github.com/pydata/xarray/issues/463 in our analysis and my
personal preference would be to get some type of solution into place sooner
than later. Thanks for considering this request.
Also, I'm not sure exactly the best way to test performance either. Could
we potentially use something like the "toy" test cases for this purpose?
Ideally we would have a test case with O(100) files to gain a clearer
picture of the performance cost of this PR.
Please let me know what you want me to do with this PR-- should I clean it
up in anticipation of a merge or just wait for now to see if there are
extra things that need fixed via additional testing? Note I have the full
scipy, h5netcdf and pynio implementations that can also be reviewed because
they weren't available when you did your review yesterday.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/pull/1198#issuecomment-272075886, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABKS1o6yrNYYulbAMkLTHaVKLQA0u3Kjks5rRbIEgaJpZM4LfwBQ
.

This should be totally fine without performance or compatibility concerns as long as we set autoclose=False by the default.
In the long term, it would be nice to handle autoclosing automatically (invoking it when the number of open files exceeds some limit), but we should probably be a little more clever for that.

Thanks @shoyer.  This makes sense.  I think the path forward on the next round of edits should include making sure existing tests using open_mfdataset use both options to autoclose.  If we do this we could future-proof ourselves against loss due to accidental breaking of this new functionality and avoid potentially contaminating existing workflows via performance concerns.
Documentation is also obviously required.
FYI as a heads up, I probably won't be able to get to this mid-week at the earliest but it appears we are close to a viable solution.

I appreciate your work on this too-many-files-open error - I think your fixes will add a lot of value to the NetCDF multi-file functionality.  In this notebook using K-Means clustering on multi-file NetCDF data sets I have repeatedly experienced the too-many-open files error, even with attempts to adjust via ulimit.  I can test out the notebook again as this PR is finalized.

@PeterDSteinberg, did this PR fix the issue for you?  I obviously need to update it but just wanted to confirm that the current branch resolved the too-many-open files error issue.  Also, do you have any idea of the performance impact of these changes I'm proposing?

@shoyer and @PeterDSteinberg I've updated this PR to reflect requested changes.

There are still a few more issues that need ironed out.  I'll let you know when I've resolved them.

I'm just chiming in to signify my interest in seeing this issue solved. I have just hit "OSError: Too many open files". The data itself is not even huge, but it's scattered across many files and it's a PITA to revert to manual concatenation -- I've grown used to dask doing the work for me ;-)

@pwolfram this looks pretty close to me now -- let me know when it's ready for review.

@shoyer, the pushed code represents my progress.  The initial PR had a bug-- essentially a calculation couldn't be performed following the load.  This fixes that bug and provides a test to ensure that this doesn't happen.  However, I'm having trouble with h5netcdf, which I'm not very familiar with compared to netcdf.  This represents my current progress, I just need some more time (or even inspiration from you) to sort out this last key issue...

E   RuntimeError: dictionary changed size during iteration

h5py/_objects.pyx:119: RuntimeError
============================================================================================ 1 failed, 1415 passed, 95 skipped in 116.54 seconds =============================================================================================
Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x10f16e598>
Traceback (most recent call last):
  File "/Users/pwolfram/anaconda/envs/test_env_xarray35/lib/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable

I'll take a look tomorrow. Getting all these backends to behave correctly
and consistently is a constant battle.
On Sat, Feb 4, 2017 at 9:20 PM Phillip Wolfram<EMAIL_ADDRESS>wrote:

@shoyer https://github.com/shoyer, the pushed code represents my
progress. The initial PR had a bug-- essentially a calculation couldn't be
performed following the load. This fixes that bug and provides a test to
ensure that this doesn't happen. However, I'm having trouble with h5netcdf,
which I'm not very familiar with compared to netcdf. This represents my
current progress, I just need some more time (or even inspiration from you)
to sort out this last key issue...
I'm getting the following error:
================================================================================================================== FAILURES ==================================================================================================================
___________________________________________________________________________________________ OpenMFDatasetTest.test_4_open_large_num_files_h5netcdf ___________________________________________________________________________________________
self = <xarray.tests.test_backends.OpenMFDatasetTest testMethod=test_4_open_large_num_files_h5netcdf>
@requires_dask
@requires_h5netcdf
def test_4_open_large_num_files_h5netcdf(self):>       self.validate_open_mfdataset_large_num_files(engine=['h5netcdf'])

xarray/tests/test_backends.py:1040:

xarray/tests/test_backends.py:1018: in validate_open_mfdataset_large_num_files
self.assertClose(ds.foo.sum().values, np.sum(randdata))
xarray/core/dataarray.py:400: in values
return self.variable.values
xarray/core/variable.py:306: in values
return _as_array_or_item(self._data)
xarray/core/variable.py:182: in _as_array_or_item
data = np.asarray(data)
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/numpy/core/numeric.py:482: in asarray
return array(a, dtype, copy=False, order=order)
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/array/core.py:1025: in array
x = self.compute()
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/base.py:79: in compute
return compute(self, **kwargs)[0]
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/base.py:179: in compute
results = get(dsk, keys, **kwargs)
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:537: in get_sync
raise_on_exception=True, **kwargs)
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:500: in get_async
fire_task()
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:476: in fire_task
callback=queue.put)
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:525: in apply_sync
res = func(*args, **kwds)
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:268: in execute_task
result = _execute_task(task, data)
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:248: in _execute_task
args2 = [_execute_task(a, cache) for a in args]
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:248: in
args2 = [_execute_task(a, cache) for a in args]
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:245: in _execute_task
return [_execute_task(a, cache) for a in arg]
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:245: in
return [_execute_task(a, cache) for a in arg]
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/async.py:249: in execute_task
return func(*args2)
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/dask/array/core.py:52: in getarray
c = a[b]
xarray/core/indexing.py:401: in getitem
return type(self)(self.array[key])
xarray/core/indexing.py:376: in getitem
return type(self)(self.array, self.updated_key(key))
xarray/core/indexing.py:354: in updated_key
for size, k in zip(self.array.shape, self.key):
xarray/core/indexing.py:364: in shape
for size, k in zip(self.array.shape, self.key):
xarray/core/utils.py:414: in shape
return self.array.shape
xarray/backends/netCDF4.py:37: in getattr
return getattr(self.datastore.ds.variables[self.var], attr)
../../anaconda/envs/test_env_xarray35/lib/python3.5/contextlib.py:66: in exit
next(self.gen)
xarray/backends/h5netcdf.py:105: in ensure_open
self.close()
xarray/backends/h5netcdf.py:190: in close
close_ds(self.ds)
xarray/backends/h5netcdf.py:70: in _close_ds
find_root(ds).close()
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/h5netcdf/core.py:458: in close
self._h5file.close()
../../anaconda/envs/test_env_xarray35/lib/python3.5/site-packages/h5py/_hl/files.py:302: in close
self.id.close()
h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper (/Users/travis/miniconda3/conda-bld/work/h5py-2.6.0/h5py/_objects.c:2840)
???
h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper (/Users/travis/miniconda3/conda-bld/work/h5py-2.6.0/h5py/_objects.c:2798)
???
h5py/h5f.pyx:282: in h5py.h5f.FileID.close (/Users/travis/miniconda3/conda-bld/work/h5py-2.6.0/h5py/h5f.c:3905)
???
h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper (/Users/travis/miniconda3/conda-bld/work/h5py-2.6.0/h5py/_objects.c:2840)
???
h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper (/Users/travis/miniconda3/conda-bld/work/h5py-2.6.0/h5py/_objects.c:2798)
???

???
E   RuntimeError: dictionary changed size during iteration

h5py/_objects.pyx:119: RuntimeError
============================================================================================ 1 failed, 1415 passed, 95 skipped in 116.54 seconds =============================================================================================
Exception ignored in: <function WeakValueDictionary.init..remove at 0x10f16e598>
Traceback (most recent call last):
File "/Users/pwolfram/anaconda/envs/test_env_xarray35/lib/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/pull/1198#issuecomment-277498382, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABKS1he81D4jBPR8uq8EpiC1dHMBlTbnks5rZVwXgaJpZM4LfwBQ
.

I can reproduce issues with h5netcdf on my machine. Of course, I get slightly different error messages -- either an error about closing a closed file from h5py or a segfault.
So far, I've verified that it isn't a multi-threading issue -- I get the same errors when dask is run in single-threaded mode (dask.set_options(get=dask.async.get_sync). That's not encouraging, suggesting that this may be a legitimate h5netcdf or (perhaps more likely) h5py bug.

@shoyer, it is quite possible the error is on my end because the error message implies that modification of the data prior to closing it may be the issue.  However, I'm not getting that type of an error following a similar philosophy for the other backends, which is confusing.
Do you have advice on hunting this bug further?  You obviously have more experience with h5py and h5netcdf than I do.

I'm still working on it. A possible culprit is some sort of dangling reference that stops the file from being actually closed by h5py. If we can't figure it out in a few days, we can just disable autoclose for h5netcdf.

@shoyer, I'll leave the ball in your court for now but am happy to take another look before we go the disabling autoclose route for h5netcdf.  Please let me know when/if you'd like me to take another look.

@shoyer, I'd like to see this code integrated if possible.  Should we just disable autoclose for h5netcdf and merge?

@shoyer, this should essentially be ready to go now with the exception that I didn't come to a clean resolution on testing the too many open files issue via resource.  Thoughts on this issue?

I obviously need to squash prior to the merge, however...

Note, if we include 3961621 for resource-limited testing we will need to do something special for windows on appveyor.  We could skip this in the short-term to make this merge and deal with these types of issues in #1309, which is what probably makes the most sense to help keep the scope of this PR more limited.

@shoyer, I've collapsed all the commits to a single commit.  At this point I would say this meets the scope of this PR and it should be ready to merge once CI finishes its check and it passes.  Please feel free to take another pass on the code for changes that you would like to see made.

@PeterDSteinberg and @vnoel, do you mind stress-testing this PR to make sure all works as expected?  You can install easily via pip with
pip -v install git+ssh://git@github.com/pwolfram/xarray@fix_too_many_open_files

@rabernat, I suspect you have some datasets that could be used to stress test this PR too... thanks for the help and advice yesterday.

Hey @pwolfram, I installed your branch and tried to open_mfdataset on 1474 hdf5 files.
I got the following:
  File "/users/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/backends/api.py", line 524, in open_mfdataset
  File "/users/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/backends/api.py", line 524, in <listcomp>
  File "/users/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/backends/api.py", line 299, in open_dataset
  File "/users/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 203, in __init__
  File "/users/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 178, in _open_netcdf4_group
  File "netCDF4/_netCDF4.pyx", line 1848, in netCDF4._netCDF4.Dataset.__init__ (netCDF4/_netCDF4.c:13992)
OSError: Too many open files

@vnoel, did you use autoclose=True?  Can you please share the code snippet you used to instantiate this?

Note, the force push was just to get travis-ci to "reboot".  There was no real change to the code.

Is something wrong with travis-ci right now?  It doesn't look like it is running the tests...

with that keyword the open_mfdataset goes through !
my code crashes later because the data is too big to fit in memory, but that's another problem ;-)

Awesome, thanks @vnoel for testing this for us all on your dataset!

@shoyer, do you mind restarting the Travis-CI?

I restarted all the timed out tests on Travis, hopefully they pass now!

They did- thank you!  I'll take a quick look at your comments now so we can hopefully get this merged soon.

Note: tests pass before submitting changes.

@shoyer- can you please have another look following edits as you requested?

@shoyer, all tests are passing now following your recommended edits.

The tests passed so I will squash the commits...

@shoyer, we can roll back the squash if you want because for whatever reason my interactive rebase removed your name from tagged to the commit too, which is not what I expected.  This is obviously suboptimal and the choice of having two commits versus one is up to you.  I think the commit prior to the squash was 9c274eb.

@pwolfram no worries, I am not concerned at all about credit for my commit :). It's possible to do gymnastics with git rebase to preserve history in cases like this but it's questionable if it's worth the trouble.

Thanks @shoyer.  Ok, so we pass checks following merge of #1311.  Is there anything else that we need to do on this PR prior to merging?

Just seeing this now.  I'm very glad to see progress here.  Thanks @pwolfram

Subclasses also work in place of fixtures in many cases (we use them in
much of the existing code).
On Wed, Mar 22, 2017 at 12:30 PM Phillip Wolfram<EMAIL_ADDRESS>wrote:

@pwolfram commented on this pull request.
In xarray/tests/test_backends.py
https://github.com/pydata/xarray/pull/1198#discussion_r107510011:

     with self.assertRaisesRegexp(IOError, 'no files to open'):

       open_mfdataset('foo-bar-baz-*.nc')

       for close in [True, False]:

I think I understand how to use fixtures for both of these arguments now...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/pull/1198#discussion_r107510011, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABKS1tp6OBPnEY9sseevAKirVhHD0XR1ks5roXbrgaJpZM4LfwBQ
.

@shoyer, if we generally cover test_backends for autoclose=True, then we should get the pickle testing for free:
xarray/tests/test_backends.py:181:    def test_pickle(self):
xarray/tests/test_backends.py:191:    def test_pickle_dataarray(self):
xarray/tests/test_backends.py:792:    def test_bytesio_pickle(self):

or was there some other test that is needed?

@shoyer, if we generally cover test_backends for autoclose=True, then we should get the pickle testing for free

Agreed, that should do it.

@shoyer, that subclass-based approach you outlined worked (fixture parameters really don't work with classes as far as I could tell).  We now have more comprehensive, named testing.  Note, there was one minor point that required more explicitly specification that arose from the more rigorous testing:
The scipy backend can handle objects like BytesIO that really aren't file handles and there doesn't appear to be a clean way to close these types of objects.  So, at present I'm explicitly setting _autoclose=False if they are encountered in the datastore.  If this needs to be changed, particularly since it doesn't affect existing behavior, I'd prefer this be resolved in a separate issue / PR if possible.

@shoyer, I had a minor bug that is now removed.  The last caveat no longer applicable:

The scipy backend can handle objects like BytesIO that really aren't file handles and there doesn't appear to be a clean way to close these types of objects. So, at present I'm explicitly setting _autoclose=False if they are encountered in the datastore. If this needs to be changed, particularly since it doesn't affect existing behavior, I'd prefer this be resolved in a separate issue / PR if possible.

I'll let you know when tests pass and this is ready for your final review.

@shoyer, this is ready for the final review now.  Coveralls appears to have hung but other tests pass.

@shoyer, all tests (including coveralls) passed.  Please let me know if you have additional concerns and if we could merge fairly soon, e.g., because of https://github.com/MPAS-Dev/MPAS-Analysis/issues/151 I would really appreciate it.

OK, in it goes!

Thanks a bunch @shoyer!