Add timestamp transform

Other than comments I left, LGTM. I don't think any of those are blocking but several worth considering to make it a little better :) Thanks @chilland!

Dig the last updates! Thanks @chilland!


fix(jsii-pacmak): computeIfAbsent throws ConcurrentModificationException

Java generated modules use cache to store the result of resolveClass which is called when ever jsiiStaticGet/jsiiGet is called. The cache is implemented as a  HashMap, to fill the cache we use computeIfAbsent:
this.cache.computeIfAbsent(MODULE_TYPES.get(fqn), this::findClass);');

code

When either of the get methods is called on a type which in itself includes a type, the first get (the root type get) will call get on the inner type. When computeIfAbsent is called from the root get, the compute method will trigger another call to computeIfAbsent to resolve the inner type, since all types share the same cache it means that the inner computeIfAbsent call will modify the cache while the root computeIfAbsent is not yet completed, resulting it a ConcurrentModificationException.
This effects all Java >= 9 applications as a change to the behavior of computeIfAbsent was introduced in Java 9.  From the Javadocs:
Throws:
ConcurrentModificationException - if it is detected that the mapping function modified this map

This PR replaces computeIfAbsent with explicitly checking if the value exists in the cache and if not adding it (essentially old style computeIfAbsent).
Tests
Steps to verify fix:

Build and package a sample module (see repro) using a local version of jsii and jsii-pacmak.
Add the generated jar to a local mvn repo.
Added the package as a dependency to a Java11 application.
Verify the exception is no longer thrown.

Repro
A repro can be found in this repo
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

copy @iliapolo

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-Blkkw9bQFn8A
Commit ID: d82887b745a49c89e2e3a24403aeb7add3b9ab91
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

🤩 wow

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-Blkkw9bQFn8A
Commit ID: 8fd896cae924b4e4355b1c644cafc2ac89b118d3
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-Blkkw9bQFn8A
Commit ID: 63d5d5d68af966cfdedb73d86f341e4a359394ac
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@NetaNir What did we change that suddenly caused this? Can you please include a reference to the PR that introduced this break?

@NetaNir What did we change that suddenly caused this? Can you please include a reference to the PR that introduced this break?

@eladb updated description to include the PR which introduced the use of computeIfAbsent

@NetaNir What did we change that suddenly caused this? Can you please include a reference to the PR that introduced this break?

@eladb updated description to include the PR which introduced the use of computeIfAbsent

Thanks!

When computeIfAbsent is called from the root get, the compute method will trigger another call to computeIfAbsent to resolve the inner type,

We kinda knew that this would have to be the issue, but do we know HOW/WHY?

(I also originally expected $Module.resolveClass() to be in the stack trace twice, but of course that's not true. The ultimate complete stack trace would look like this:
(stack goes ⬇ that way)
(...some stack frames...)
$Module.resolveClass
HashMap.computeIfAbsent      <----- 2⃣ Error occurs here *AFTER* the frames below have unwound
$Module.findClass
(... what goes here...?)
$Module.resolveClass
HashMap.computeIfAbsent       <---- 1⃣ This completes successfully first

@rix0rrr to answer your (... what goes here...) section:

Class.forName will possibly load up a newly discovered class (let's call it Foo)

Can be a base class/interface, some property's type, ...

Class initialization of Foo might involve jsii calls

Getting values of const fields, for example

The result of those calls may be something that $Module.resolveClass is used for (e.g: class/interface instance, struct, enum)

At this point, we'll get back to computeIfAbsent

I'm trying to come up with a test that blows up before the fix is introduced.

Getting values of const fields, for example

Got it. So that should be easy enough to write a test for then. So it's basically a matter of:
class SomeClass {
  public static const OTHER: OtherClass = new OtherClass();
}

?

Although of course that won't break if OtherClass is referenced in any other capacity...

Getting values of const fields, for example

Got it. So that should be easy enough to write a test for then. So it's basically a matter of:

Yeah that's what I thought but it appears the patch to trigger this situation depends on a bunch of other factors I'm trying to identify (possibly the properties being typed "any" or something.

Oh I missed a critical element of the repro: it only happens with Java >= 9. It seems related to a change in class loading behavior between 8 and 9+.

@RomainMuller Its not a change in class loading, its a change in the implementation of computeIfAbsent, which is Java >= 9 throws the ConcurrentModificationException when it detects a dirty mutation.
https://stackoverflow.com/questions/54824656/since-java-9-hashmap-computeifabsent-throws-concurrentmodificationexception-on
Regarding the repro, did you look at @NetaNir's repo?
It wasn't that straight-forward to reproduce, but I do think the repro can probably be simplified.
Also, since the recursion stems from Class.forName, it can only happen when a class has static initializers that eventually call jsiiGet - right? Since Class.forName only loads the class, and doesn't instantiate if.

So the flow is:

Resolving some class involves loading another
Loading the other class involves resolving yet another

An easy way to trigger is to have a const property causing loading of an enum (it'll then try to resolve the enum when loading up it's values)
Added a test to this effect. Interestingly, it works in Java 8, fails in Java 9. I suspect Java 9 resolves classes more eagerly than 8.

@RomainMuller

Interestingly, it works in Java 8, fails in Java 9. I suspect Java 9 resolves classes more eagerly than 8.

https://stackoverflow.com/questions/54824656/since-java-9-hashmap-computeifabsent-throws-concurrentmodificationexception-on

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-Blkkw9bQFn8A
Commit ID: ac7b8c7eed5e65d497f7a21ea28d63446382768a
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@RomainMuller

Interestingly, it works in Java 8, fails in Java 9. I suspect Java 9 resolves classes more eagerly than 8.

https://stackoverflow.com/questions/54824656/since-java-9-hashmap-computeifabsent-throws-concurrentmodificationexception-on

Ohhhhh

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-Blkkw9bQFn8A
Commit ID: 38cde9960c48d7ebcde715be09aa1b2217ce832f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@MergifyIO refresh


Typo fix in kubeadm/implementation-details.md

“goes thought proxy” → “goes through proxy”
This is my first PR to this project; if I've missed a detail, please let me know.

Thanks.
/lgtm
/approve


url_sig: fix memory leak with urlParse and pristine flag

This fixes a memory leak in url_sig when the pristine flag is set in the section of code that calls urlParse.  This PR further cleans up error messages to help with identifying malformed client requests.

[approve ci]

[approve ci clang-analyzer]

[approve ci clang]

[approve ci]

Is this a 9.0.x candidate?

This one would have to include PR https://github.com/apache/trafficserver/pull/5219 as well.

Cherry-picked to v9.0.x branch.

This has merge conflicts, and we'll need a new 8.1.x PR. Also, feel free to recommend / suggest other PRs that this depends on, such that it makes the back porting easier.


Lint/UselessAccessModifier reports wrong modifier

This is probably related to https://github.com/bbatsov/rubocop/issues/3422. It looks like UselessAccessModifier is reporting the wrong line number for the following code:
class HostsControllerTest < ActionController::TestCase
  context 'this is a context' do
    test 'that 1 is 1' do
      assert 1, return_one
    end

    private

    def return_one
      1
    end
  end

  test 'that 2 is 2' do
    assert 1, return_two
  end

  private

  def return_two
    2
  end
end

Here's the warning:
test.rb:18:3: W: Lint/UselessAccessModifier: Useless private access modifier.
  private
  ^^^^^^^

Notice it's reporting line 18 instead of 7.

Lint/UselessAccessModifier:
  ContextCreatingMethods:
    - context

Does this resolve your problem? With this config, the cop shouldn't find any offenses

@maxjacobson so I want the cop to find the first private as useless since it is in fact useless. I think the person who wrote the code in this example was trying to ensure that return_one could not be used outside of the context block which is already the case (without having to call private).

Hm interesting. Do you have a heuristic for how rubocop should know that the first one is useless?

What about if the access modifier is in a code block (as opposed to a class or module)?

Hm. It sort of conflicts with common patterns like rails's concerning and class_methods, which let you define blocks where private can be meaningful. The ContextCreatingMethods config exists to support that use-case. I could imagine another config which takes a separate list of methods which shouldn't have any access modifiers in them maybe?

I guess my question would then be: why does this code pass then when I don't have ContextCreatingMethods set?
class Test
  context "this is a context" do
    private
    def test
       puts 'ok'
    end
  end
end

Another thought: is it maybe possible to have the cop prefer access modifiers in the class/module body over ones inside of blocks?

Also, this cop does correctly give me an error (it just points to the one I wouldn't expect). So it's quite trivial. I'm happy to close this out if you think it'd be too much work for not enough gain.

I think the crux of the issue is that rubocop doesn't know anything about the behavior of methods defined by users or gems (like context in that last example) and it's not obvious how it should lint for them, because it can't predict what they're doing with the code defined in the blocks passed to them.
The ContextCreatingMethods thing is one way to give it a hint. Without a hint, I kind of wonder if this cop should just ignore them? That might be another way of stating your suggestion to "prefer access modifiers in the class/module body over ones inside of blocks".

I'm happy to close this out if you think it'd be too much work for not enough gain.

If we can identify a useful change that would improve rubocop, I think it's worth keeping this issue open. For what it's worth, I'm just a random contributor (I added the ContextCreatingMethods config when I hit a similar issue).

I took a quick look at the RSpec code and found that context and similar methods create classes dynamically and evaluates the give blocks in the scope of those classes. This is just too specialized behavior for RuboCop to be aware of. A new cop in rubocop-rspec would make sense, I think.
I'm closing the issue. Let us know if you disagree.


Document correct scoped imports for each component

It is possible to import most (all?) components with a scoped import e.g. import { Slider } from 'rmwc/Slider', but the correct paths to import from aren't mentioned in each component's docs.

Good call. I’ll add it to the docs.


Add coverage

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@62cf50e). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master      #49   +/-   ##
=========================================
  Coverage          ?   54.92%
=========================================
  Files             ?        9
  Lines             ?      315
  Branches          ?        0
=========================================
  Hits              ?      173
  Misses            ?      142
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 62cf50e...659a777. Read the comment docs.


Add check for nonexistant if when adding IP

LGTM once the tests pass (I assume this is because the libcalico change is not yet in).

Required libcalico fix: https://github.com/projectcalico/libcalico/pull/45


⚠ Bofrost.be has degraded performance

In 65c68e3, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 805 ms

Resolved: Bofrost.be performance has improved in 821f599.


Feat prms tests

[ ] Closes #xxxx
[ ] Tests added
[ ] Performance benchmarks added
[ ] Performance regression benchmarks run
[ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
[ ] New functions/methods are listed in api.rst or it's sub rsts?

Codecov Report

Merging #241 (0b8c01d) into develop (fb9a0f3) will decrease coverage by 0.28%.
The diff coverage is 57.14%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
@@             Coverage Diff             @@
##           develop     #241      +/-   ##
===========================================
- Coverage    79.19%   78.91%   -0.28%
===========================================
  Files           51       51
  Lines         6536     6531       -5
===========================================
- Hits          5176     5154      -22
- Misses        1360     1377      +17

Files
Coverage Δ

pywatershed/hydrology/prms_runoff.py
82.59% <100.00%> (ø)

pywatershed/hydrology/prms_snow.py
90.76% <100.00%> (+0.15%)
:arrow_up:

pywatershed/hydrology/prms_soilzone.py
88.50% <100.00%> (+0.27%)
:arrow_up:

pywatershed/analysis/process_plot.py
16.93% <0.00%> (ø)

pywatershed/utils/prms5util.py
31.77% <33.33%> (+0.02%)
:arrow_up:

... and 6 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


[automation] fix root owned _out directory issue

Signed-off-by: Igor Bezukh<EMAIL_ADDRESS>What this PR does / why we need it:
There is an issue with _out directory being root-owned
when building the Kubevirt project for the first time on
a clean environment.
The root cause is that we don't explicitly create the
_out directory, leaving it to be done by rsync during the
sync out procedure. Therefore rsync creates the _out directory
with incorrect ownership, thus _out become root-owned.
Proposed solution is just to explicitly create the OUT_DIR
in case SYNC_OUT is turned on.
Release note:
NONE

/hold
checking what went wrong

@enp0s3 I am a little bit surprised by this. The whole purpose of using rsync was that it should at the end have the outside user. Is this related to changes in rsync?

@enp0s3 I am a little bit surprised by this. The whole purpose of using rsync was that it should at the end have the outside user. Is this related to changes in rsync?

@rmohr Perhaps something indeed changed in there. However I see that it happens only _out doesn't exist on the host, do rsync had to create the directory before syncing, it creates it as root owned. Anyway I see that my solution here doesn't work. What I can do is just create the _out directory is if SYNC_OUT is on and that's it. I won't count on rsync ownership decisions for non existing directories (although it is reasonable for rsync to create a non existing directory with same ownership that of the user who called rsync, maybe it a problem to do so, I don't know)

/unhold

/retest

@dhiller @rmohr Hi. Can you please take another look?

/lgtm
/approve
/retest


ESM transition TODO

In tinyhttp 2 ( https://github.com/talentlessguy/tinyhttp/milestone/2) there will be no CommonJS builds. Most modules are hand-written so it can't all be transitioned at once. This issue is to keep track of the progress.
3rd-party modules

[ ] es-mime-types (PR)
[ ] es-fresh (PR)
[ ] es-vary (PR)

more coming soon

WIP on https://github.com/talentlessguy/tinyhttp/tree/esm


How can I read the complete documents inside a collection?

Is there an option to read inside a Collection e.g. students the complete documents?
My aim is that I can see the complete documents inside a dataframe, e.g. import pandas as pd; df=pd.json_normalize(result['...']['...']).
from arango import ArangoClient

# Initialize the ArangoDB client.
client = ArangoClient()

# Connect to database as  user.
db = client.db(<db>, username=<username>, password=<password>)

print(db.collections())
students = db.collection('students')
#studens.getAllDocuments()
# for ...
#print(students)

Hi @rosenberg12,
You can use students.all() or students.export(...)

@joowani thank you very much! That worked.


Allow absolute uris in the index thumbnail field

At this point, the thumbnail path in the index can only contain a relative path, which is appended to the duraspace domain/path.
Do we need to update this, so users can specify a thumbnail image from an external location?

Collection thumbnails have been set to use absolute uris from the index. There is no way to use thumbnails in the repository. Will have to re address this if that requirement is introduced in the future


possible docker build issue

In your Dockerfile you have RUN apt update and RUN apt install ... in separate lines. This is not recommended, since docker's caching can cause old versions of your packages to get installed.
From the docs:

Always combine RUN apt-get update with apt-get install in the same RUN statement. For example:
   RUN apt-get update && apt-get install -y \
       package-bar \
       package-baz \
       package-foo

Using apt-get update alone in a RUN statement causes caching issues and subsequent apt-get install instructions fail.

Ooh I didn’t realize we are using apt instead of apt get to begin with. That’s fascinating about the caching issues, do you have any idea as to why? Would be happy to accept a PR

i think the basic idea is that the result of each line is cached, so if you have a cached apt update from last week, and then add some new packages to the apt install line,  you might get the versions from an older apt repository. probably not a crucial thing here, but i'll put up a PR. also, as you mentioned, apt-get might be better for scripts -- you get a warning about it when you run docker build

well, the build blew up! the docker build works fine locally. maybe it's because there are 0 code changes, so most of the tests aren't running...


Support Swift-only install

Right now the code assumes that there is a 'compute' group. This might
not actaully be the case.

retest


Update: qBittorrent.qBittorrent version 4.3.7

[X] Have you signed the Contributor License Agreement?
[X] Have you checked that there aren't other open pull requests for the same manifest update/change?

[X] Does your manifest conform to the 1.0 schema?

@KaranKad would you look at that, called it 😁

Microsoft Reviewers: Open in CodeFlow

Cannot pass WD PUA scan...

8/6/2021 12:09:27 PM
Status: Completed
##[error] Installer https://sourceforge.net/projects/qbittorrent/files/qbittorrent-win32/qbittorrent-4.3.7/qbittorrent_4.3.7_setup.exe/download Scan information: 'Error information: BlockingDetectionFound'
##[error] Result: Failed

8/6/2021 12:09:34 PM
Status: Completed
##[error] Installer https://sourceforge.net/projects/qbittorrent/files/qbittorrent-win32/qbittorrent-4.3.7/qbittorrent_4.3.7_x64_setup.exe/download Scan information: 'Error information: BlockingDetectionFound'
##[error] Result: Failed

Trying again since the SmartScreen warning is gone and its not flagged as PUP from Microsoft on Virustotal anymore

boo

/azp run

.


remove transition test which was only necessary for ie8, which is no …

…longer supported
Given that readme indicates that ie8 is no longer supported, i got rid of the transition test. It's not needed for ie versions > 8 , right?

According to https://caniuse.com/#feat=css-transitions IE9 doesnt support it either. Then again, should we really still support that old stuff? Maybe get rid of IE below 11 for a v1 ? what are your thoughts @paxperscientiam  @GenaBitu  ?

Ok, let's talk browser support:
I would really like to see some automated browser support testing as part of the lints. I am familiar with amilajack/eslint-plugin-compat (for eslint) and ismay/stylelint-no-unsupported-browser-features (for stylelint) - there should be something similar to that for csslint as well.
Will try to add a PR for this if we reach some conclusion.
I tend to support even IE8, I generally use the lints as a guideline and use cover 95%, not < 0.1% as the preferred coverage setting.
I think this discussion is largely subjective (i. e. you see a lot of arguments about this...). I agree that IE8 should die, but the fact is that some people still use it and I want my sites to work for as many people as possible. Also, bear in mind that in some countries, the browser usage chart looks vastly different.
To sum it up (as if this comment wasn't long enough...): 1) We should talk browser coverage, not individual browsers. 2) We should have automatic lints. 3) I support the aforementioned coverage "query".

These days, projects include pre-built files under "dist" that are meant to be ready for consumption. Since this repo includes source files, any developer is free to choose to ingest them into their own build process.
In short, if your preferred compatibility profile for your dist files is insufficient for some dev, they are free to use the source files.
To put it another way, the "dist" files are a mere convenience and not guaranteed to meet every developer's needs.
Does that help?

Hi,
this should be fixed as of #870, let me know if it isn't :) Thanks!


Issue with ios

Hi everyone.
I have implemented the package for android and it works fine.
But I have received an issue with ios.
Can you please explain how to fix it.
Failed to build iOS app
Could not build the precompiled application for the device.
Error (Xcode): Cycle inside Runner; building could produce unreliable results.
Cycle details:
→ That command depends on command in Target 'Runner': script phase “[CP] Copy Pods Resources”
○ Target 'Runner'
○ That command depends on command in Target 'Runner': script phase “[CP] Embed Pods Frameworks”
○ Target 'Runner'
○ That command depends on command in Target 'Runner': script phase “Thin Binary”
○ Target 'Runner' has process command with output '/Users/dev/Documents/Flutter/Emery/emery-mobile-app/build/ios/Debug-iphoneos/Runner.app/Info.plist'
○ Target 'Runner' has copy command from '/Users/dev/Documents/Flutter/Emery/emery-mobile-app/build/ios/Debug-iphoneos/NotificationServiceExtension.appex' to '/Users/dev/Documents/Flutter/Emery/emery-mobile-app/build/ios/Debug-iphoneos/Runner.app/PlugIns/NotificationServiceExtension.appex'
○ Target 'Runner' has copy command from '/Users/dev/Documents/Flutter/Emery/emery-mobile-app/ios/Flutter/AppFrameworkInfo.plist' to '/Users/dev/Documents/Flutter/Emery/emery-mobile-app/build/ios/Debug-iphoneos/Runner.app/AppFrameworkInfo.plist'

Hi @Korefey
Sorry to hear that you're experiencing this.
The “Cycle inside Runner” error could be possibly due to cyclic dependencies among your build phases or due to derived data or conflicts in pod dependencies. By fixing these, you should be able to resolve the issue.
Although this issue appears to be related to configuration or implementation rather than our SDK, I am happy to assist you. Here are some suggestions that might help you troubleshoot the problem:

Clean and rebuild the project : In Xcode, go to Product > Clean Build Folder
Remove Derived Data : In terminal, run rm -rf ~/Library/Developer/Xcode/DerivedData
Clean the project : flutter clean
Check Podfile : Ensure that there are no conflicts or redundant entries
Update dependencies & Re-install Pods : Run pod install --repo-update in ios directory
Rebuild flutter project : Run flutter pub get and flutter build ios

I also found this SOF question that looks relevant to the issue, in case it helps.
Please let us know if you are still experiencing the issue. It would be helpful if you could provide additional details such as:

Steps to reproduce
Xcode, flutter and customerio-flutter version in use
Podfile
AppDelegate
Logs, if any
Minimal reproducible app

Hi @ami-aman
I created new project, full filled all recommendations from manual and also received some issues.
Please check it on github https://github.com/Korefey/customer_ios

Hi @Korefey
Sorry to hear that despite following the recommendations, you are still experiencing issues. I appreciate you sharing a reproducible app. I will review your GitHub repository and get back to you as soon as possible with feedback or solutions.
In the meantime, please feel free to add any additional details here if you have them.

Hey @Korefey
Thank you for sharing the demo app. I was able to reproduce the issue and resolve it by making adjustments to the build phases of your project. Below is a screenshot of the correct order of the build phases that helped resolve the issue. I hope this helps.

You may also refer our flutter sample app for more reference.
Since the issue is related to project configuration and not our SDK, I will close this ticket. Please feel free to reopen it or open a new issue if you have any other queries related to our SDK.


api/streams: added persistency

This code updates  streams config file when a  stream is created/deleted with api/streams.

This code totally break my existing YAML config. I think streams should be added to go2rtc.json file, like HomeKit cameras. And go2rtc shouldn't change YAML config from code. Because there is no safe way to do it.

ESPHome also does something similar, they take YAML files as input and generate JSON files to have them persisted.

This feature would also come in handy for things like:

https://github.com/AlexxIT/go2rtc/issues/183

So that streams that gets disabled from API/UI are remembered if go2rtc restarts.

I think it's possible to store dynamic part inside go2rtc.yaml, but it should be special single line in text format. It will be easy to find and replace. For example:
db: '{...}'


List of Environment Variables

Hi, there's a way to find a list of all supported environment variables and what each var does?

@vasartori as far as I know, we don't have a central place to list out all the env vars.
@ywk253100 Right?

If you search on code for os.Getenv we found a lot of env vars but we don't know the behaviour of this vars. Some are intuitives, others not so much...

@vasartori  We'll have a discussion about adding ENV vars in the document soon.

@vasartori Env variables should be transparent to users, all configurations can be set by configuration file harbor.cfg, and some of them can be set on web UI either.

@ywk253100 IMHO it not so good. If I want change a small thing, like the number job of workers or if I need put a new ldap filter, I need make a new deploy. In some cases It is not fast. It could be only a restart.
I like to know what an option do, just to understand what are happen in background.

@vasartori The new deploy is not needed. About the detail of reconfiguring Harbor, refer to the "Managing Harbor's lifecycle" part of installation guide here. And some common configuration items, such as LDAP filter, can be modified on the web UI directly.
You want to try the "hard way" to understand very well what happened, that's good. But for most of the users, they don't need it, so the guide may not be provided. If any detail you want to know, just open issues.

@ywk253100 OK, I understand!


feat: replace get_test_client(controller, napp) with starlette test client and include httpx as a dep

For more information, check out https://github.com/kytos-ng/kytos/discussions/337

Landed on https://github.com/kytos-ng/kytos/pull/375


consistent pandas_read_csv_kwargs for file and CDS sources

See #180 for full details and background.
CDS and file sources now have consistent pandas_read_csv_sources such that a resource from  theCDS has the same behaviour whether opened automatically or saved for a later date. Several other small refactorisations also included.
I have not yet modified the ecmwf_api default kwargs as that has more bespoke requirements. But this is due to them being TAB sperated values, not COMMA seperated values.

Latest commit is following an offline discussion to improve the visibility and explicitness of the kwarg handling in the read_csv method

Can this be merged?


More Badgers Needed

This issue is created to request more badgers. It is left open for further discussion.

This is a placeholder comment to ensure the comment body is not blank.


sds: additional support for symlink-based key rotation.

There are a few limitations in our existing support for symlink-based
key rotation:

We don't atomically resolve symlinks, so a single snapshot might have
inconsistent symlink resolutions for different watched files.
Watches are on parent directories, e.g. for /foo/bar/baz on /foo/bar,
which doesn't support common key rotation schemes were /foo/new/baz
is rotated via a mv -Tf /foo/new /foo/bar.

The solution is to provide a structured WatchedDirectory for Secrets to
opt into when monitoring DataSources. SDS will used WatchedDirectory
to setup the inotify watch instead of the DataSource path. On update, it will
read key/cert twice, verifying file content hash consistency.
Risk level: Low (opt-in feature)
Testing: Unit and integration tests added.
Fixes #13663
Fixes #10979
Fixes #13370
Signed-off-by: Harvey Tuch<EMAIL_ADDRESS>

@mattklein123 @lizan @tsaarni can you take a look? This will address #13663, #10979 and #13370, I'd like to obtain consensus on the API before adding implementation and tests.

I've updated the API to avoid modifying DataSource, introducing a new concept of an opt-in WatchedDirectory. I've also added a draft implementation and a very basis integration test. I'd like to achieve agreement on API and design, at which point I can clean up the implementation and fill out the rest of the tests. @lizan @mattklein123 @tsaarni PTAL.

This now uses hash-based comparison of files after reading twice and validated consistency. This is similar to what gRPC is doing.

PTAL, if this makes sense now in terms of API and hash-based comparison implementation then I can fill in the missing tests/docs and move this out of draft.

@mattklein123 @lizan this is now ready for review.

@mattklein123 do deal with failure scenarios I had to add some stats, which then forced me to fix SDS stats more generally, since they were broken before (they all ended up anonymously in the root namespace). PTAL and LMK if there's anything else needed around stats here.

Sure, will merge as some folks are depending on this, but I'll take an AI to do the docs followups next week. Thanks.


ExtSlice Support

I am playing around a bit with tangent after your helpful reply in reddit ;)
So I was trying to reshape an array. First I had:
  File "<stdin>", line 2
    r2 = np.sum(wrapped ** 2, axis=1)[:, np.newaxis]
                                   ^
TangentParseError: Extended Slices are not supported

Then I tried with reshape
r2 = np.sum(wrapped ** 2, axis=1).reshape(-1, 1)
and it gave: AttributeError: 'NoneType' object has no attribute 'reshape'
Then I tried
r2 = np.sum(wrapped ** 2, axis=1)
r2 = r2.reshape(-1, 1)

which gave AttributeError: module 'builtins' has no attribute 'r2'
and there I finally got the hint and did:
r2 = np.sum(wrapped ** 2, axis=1)
r2 = np.reshape(r2, (-1, 1))

Which worked. I guess it would be nice for new users to add a line to the docs not to use the array methods and instead use directly numpy functions.

Good point. We don't currently have the machinery to detect that r2 is an object with methods. I'll add a line to the docs soon making this clear.
We should support ExtSlice, though, that should have worked off the bat.

Should be fixed in 0ef0225f707c9eaa7510ee9099d29361c1208d51.


Publish last changes to bower and npm

It would be great to have the changes of PR #73 published to NPM and Bower!

Just published the latest version. Let us know if there is a problem.

I think the project has not been built, so es_ES is unavailable in dist/ files...


Remove id type naming restrictions

This pull request modifies the mkId template function to use the type, constructor, and destructor names of the provided type.
For example, you can now do the following:
newtype MyType = MT { extractMt :: Int } deriving (Show, Typeable)
$(mkId ''MyType)

Looks good, thank you!

No problem. Thanks for integrating it so quickly.


HDDS-4023. Delete closed container after all blocks have been deleted.

https://issues.apache.org/jira/browse/HDDS-4023

Thanks for working on this @ChenSammi.
I wonder if it would be simpler to remove empty containers as part of Container Report processing? In AbstractContainerReportHandler#updateContainerState, we could check the size and number of keys of the reported containers in the CLOSED branch of the switch statement, and then take action to delete an empty container there? I have a feeling it might be simpler, but I am not sure. The disadvantage of doing it in the Container Report Processing, is that we are dealing with only a single replica at that stage. However if the container is CLOSED in SCM, and a report says it is empty then we should be good to simply remove the container from SCM and issue the delete container command when processing the container report.
In looking at how this all works, I noticed there is a bug in the existing method AbstractContainerReportHandler#updateContainerStats, which might stop your change working:
  private void updateContainerStats(final ContainerID containerId,
                                    final ContainerReplicaProto replicaProto)
      throws ContainerNotFoundException {
    if (isHealthy(replicaProto::getState)) {
      final ContainerInfo containerInfo = containerManager
          .getContainer(containerId);

      if (containerInfo.getSequenceId() <
          replicaProto.getBlockCommitSequenceId()) {
        containerInfo.updateSequenceId(
            replicaProto.getBlockCommitSequenceId());
      }
      if (containerInfo.getUsedBytes() < replicaProto.getUsed()) {
        containerInfo.setUsedBytes(replicaProto.getUsed());
      }
      if (containerInfo.getNumberOfKeys() < replicaProto.getKeyCount()) {
        containerInfo.setNumberOfKeys(replicaProto.getKeyCount());
      }
    }
  }

This logic assumes the bytes used and keyCount is always increasing on the container. However in this case, where blocks are deleted the keyCount and bytesUsed can decrease. Therefore I don't think the stats will ever get updated in SCM via the container reports when blocks are removed.
I then noticed you recently made a change in ReplicationManager to update these counts here:
https://github.com/apache/hadoop-ozone/pull/1295/files#diff-218132dfe72e6e39b9eaaf59737adcf2R780
I think the ReplicationManager part of that change should be reverted and it handled via the ContainerReports by fixing the bug I mentioned above. The only reason we need the Replication Manager change is to work around the bug above. If it was fixed, we don't need that logic in RM.
If we decide to keep your changes inside replicationManager, then we would need a few more tests added to TestReplicationManager to cover the new scenarios.
I also have a question - when the replicas are all deleted, and we call:
containerManager.updateContainerState(container.containerID(),
          HddsProtos.LifeCycleEvent.CLEANUP);

Does this somehow cause the container to be removed from SCM memory and the persistent store?

I raised HDDS-4131 / https://github.com/apache/hadoop-ozone/pull/1339 discuss and address the bytesUsed and KeyCount metric problem I mentioned above.

Hi @sodonnel thanks for review the code and raise the discussion.

I wonder if it would be simpler to remove empty containers as part of Container Report processing? In AbstractContainerReportHandler#updateContainerState, we could check the size and number of keys of the reported containers in the CLOSED branch of the switch statement, and then take action to delete an empty container there? I have a feeling it might be simpler, but I am not sure. The disadvantage of doing it in the Container Report Processing, is that we are dealing with only a single replica at that stage. However if the container is CLOSED in SCM, and a report says it is empty then we should be good to simply remove the container from SCM and issue the delete container command when processing the container report.
In looking at how this all works, I noticed there is a bug in the existing method AbstractContainerReportHandler#updateContainerStats, which might stop your change working:
  private void updateContainerStats(final ContainerID containerId,
                                    final ContainerReplicaProto replicaProto)
      throws ContainerNotFoundException {
    if (isHealthy(replicaProto::getState)) {
      final ContainerInfo containerInfo = containerManager
          .getContainer(containerId);

      if (containerInfo.getSequenceId() <
          replicaProto.getBlockCommitSequenceId()) {
        containerInfo.updateSequenceId(
            replicaProto.getBlockCommitSequenceId());
      }
      if (containerInfo.getUsedBytes() < replicaProto.getUsed()) {
        containerInfo.setUsedBytes(replicaProto.getUsed());
      }
      if (containerInfo.getNumberOfKeys() < replicaProto.getKeyCount()) {
        containerInfo.setNumberOfKeys(replicaProto.getKeyCount());
      }
    }
  }

This logic assumes the bytes used and keyCount is always increasing on the container. However in this case, where blocks are deleted the keyCount and bytesUsed can decrease. Therefore I don't think the stats will ever get updated in SCM via the container reports when blocks are removed.

This logic works for OPEN container, but not for CLOSED container.

I then noticed you recently made a change in ReplicationManager to update these counts here:
https://github.com/apache/hadoop-ozone/pull/1295/files#diff-218132dfe72e6e39b9eaaf59737adcf2R780
I think the ReplicationManager part of that change should be reverted and it handled via the ContainerReports by fixing the bug I mentioned above. The only reason we need the Replication Manager change is to work around the bug above. If it was fixed, we don't need that logic in RM.
You can find my comments in HDDS-4131.

If we decide to keep your changes inside replicationManager (for this change), then we would need a few more tests added to TestReplicationManager to cover the new scenarios.

TestReplicationManager uses Mockito which is not a real end-to-end test. I add a end-to-end test in TestBlockDeletion.

I also have a question - when the replicas are all deleted, and we call:
containerManager.updateContainerState(container.containerID(),
          HddsProtos.LifeCycleEvent.CLEANUP);

Does this somehow cause the container to be removed from SCM memory and the persistent store?
Container of DELETED state  currently is still in SCM memory and persistent store.  Of course we can just delete the Container from memory and DB when it's state changes to DELETED. But I feel like current way is a more safe way,  like a Container trash bin. We can add a new purge DELETED Container action next so user can manually delete these Container when they sure these information are no longer needed.   Overall, I'm open to this question, fine with either way.

The failed UT is irrelevant, fixed by https://github.com/apache/hadoop-ozone/pull/1379.

Hi, @sodonnel  do you have time for review?

As I said before, I have a feeling that it might be better to have a separate service for deleting empty containers, and it would perhaps naturally be part of a "merge small containers" service, which we will need to create eventually. However, the changes to the replication manager here are not too big, and you could argue the current solution fits here OK. Therefore I am OK with keeping this logic in the replication manager. However when it comes to merging small containers in the future, I believe we should create a separate service and not clutter RM with that new feature too.
The patch looks mostly good to me, there are just a couple of areas we need to consider.

What do we do with the containers in DELETED state, and with no replicas? I don't think we should keep them in SCM forever, but we are also worried about removing them. However that said, if the replicas have all been deleted, what use is there in keeping the container in SCM?

Consider if a datanode was dead for some time, and is then restarted. It will start reporting a replica for the DELETED container and with the current logic I don't think we will clean it up as isContainerEmpty(...) will always return false due to it checking for only CLOSED containers. I feel that this should be handled in the ContainerReportHandler code. If a replica is reported for a container which is DELETED in SCM, then we should instruct the datanode to delete the replica. What do you think? The parameter hdds.scm.unknown-container.action currently controls what to do if a DN reports a containers which SCM does not know about. The default is to log a warning, but for a DELETED container, I think we should just delete the replica.

There is another corner case, where the dead node may have a container which is out of sync with the others, as it was dead when the blocks should have been removed. Therefore we had 3 empty container replicas, plus one non-empty dead one. The 3 empty replicas are removed, and the container moved to "DELETED". Then the dead node starts reporting a non-empty replica for the DELETED container. This is actually a wider problem, in that I believe there are scenarios where container replicas can get out of sync with one another and there is nothing to reconcile them and fix it.

In ContainerSafeModeRule, I think we need to consider DELETING and DELETED. If there are a lot of DELETED containers in the cluster, then they will never get replicas and the safemode rule will never get met. I think this could be as simple as excluding DELETING and DELETED when loading the containerMap (right now OPEN and CLOSING are excluded) so we don't wait for replicas on these containers.

In ReplicationManager.isContainerUnderReplicated(...), should we immediately return true if the containerState is DELETING or DELETED?

Are there any other places outside of ReplicationManager and ContainerSafeModeRule where we monitor under / over replicated containers I may have missed?

Thinking some more about what to do with containers in the DELETED state in SCM. If we are not confident to delete them immediately and automatically, maybe we should provide an command "purge empty containers" which can be run by an admin, or via Recon. We should show in Recon how many empty containers are present in the cluster and then provide an option to remove them.
Alternatively we could have some threshold in SCM, so that when the DELETED containers crosses some threshold, they are all removed.
We could do either of these in a separate jira if this idea makes sense.

Thanks @sodonnel for very thoughtful suggestions.

2. Consider if a datanode was dead for some time, and is then restarted. It will start reporting a replica for the DELETED container and with the current logic I don't think we will clean it up as `isContainerEmpty(...)` will always return false due to it checking for only CLOSED containers. I feel that this should be handled in the ContainerReportHandler code. If a replica is reported for a container which is DELETED in SCM, then we should instruct the datanode to delete the replica. What do you think? The parameter hdds.scm.unknown-container.action currently controls what to do if a DN reports a containers which SCM does not know about. The default is to log a warning, but for a DELETED container, I think we should just delete the replica.

Agree. we should just delete the replica in the case.

3. There is another corner case, where the dead node may have a container which is out of sync with the others, as it was dead when the blocks should have been removed. Therefore we had 3 empty container replicas, plus one non-empty dead one. The 3 empty replicas are removed, and the container moved to "DELETED". Then the dead node starts reporting a non-empty replica for the DELETED container. This is actually a wider problem, in that I believe there are scenarios where container replicas can get out of sync with one another and there is nothing to reconcile them and fix it.

The extra stale replica is a real problem, not just in empty container deletion case, but also for non-empty CLOSED container case.  After a delete block command is confirmed by 3 datanodes, this delete block command is finished and removed from block deletion log. So if after that, a fourth replica shows up,  SCM will not send an new delete block command to this replica.
But this fourth replica will have all remaining delete block commands executed if it's datanode is still on.   Maybe we can rely on over replicated container replica deletion logic to remove this extra fourth replica.  But it's hard to choose which one to delete if there deleteTransactionId and blockCommitSequenceId are different. Currently handleOverReplicatedContainer doesn't check these two fields of each replica.

4. In ContainerSafeModeRule, I think we need to consider DELETING and DELETED. If there are a lot of DELETED containers in the cluster, then they will never get replicas and the safemode rule will never get met. I think this could be as simple as excluding DELETING and DELETED when loading the containerMap (right now OPEN and CLOSING are excluded) so we don't wait for replicas on these containers.

Got catch!

5. In `ReplicationManager.isContainerUnderReplicated(...)`, should we immediately return true if the containerState is DELETING or DELETED?

Right.
I also agree we should purge these deleted containers after a while.   We can have a timer task to periodically remove the DELETED containers from SCM DB.  What do you think?

After a second thought,  deleting the container record in SCM DB immediately while keep it in memory maybe a better and clean choice.  So if there is stale container replica, it can be deleted based on in memory information.  And next time when SCM start, SCM doesn't need to handle DELETED containers anymore.

@sodonnel  @elek  could you help to take another look?

Sorry for the slow reply on this. I have been caught up on some other things.

After a second thought, deleting the container record in SCM DB immediately while keep it in memory maybe a better and clean choice. So if there is stale container replica, it can be deleted based on in memory information.

I think this is a good enough idea for now. If SCM is up for a very long time, perhaps in the future we will want to add a thread to clear all the in memory DELETED containers. One small concern is that if a container goes DELETED and then SCM is restarted soon after. Then a DN is restarted and reports a stale replica, it will just be seen as an unknown container. The default position there, is to log a warning. The config hdds.scm.unknown-container.action controls this. This is all an edge case - most of the time, all DNs should be up anyway.
I left just one comment on a suggested refactor in the container report handler, when dealing with replicas from a DELETED container.
Could you also add a test in TestContainerReportHander to check the logic around deleting a replica from a DELETED container?

Sorry for the slow reply on this. I have been caught up on some other things.

After a second thought, deleting the container record in SCM DB immediately while keep it in memory maybe a better and clean choice. So if there is stale container replica, it can be deleted based on in memory information.

I think this is a good enough idea for now. If SCM is up for a very long time, perhaps in the future we will want to add a thread to clear all the in memory DELETED containers. One small concern is that if a container goes DELETED and then SCM is restarted soon after. Then a DN is restarted and reports a stale replica, it will just be seen as an unknown container. The default position there, is to log a warning. The config hdds.scm.unknown-container.action controls this. This is all an edge case - most of the time, all DNs should be up anyway.
I left just one comment on a suggested refactor in the container report handler, when dealing with replicas from a DELETED container.
Could you also add a test in TestContainerReportHander to check the logic around deleting a replica from a DELETED container?

Thanks @sodonnel ,  a new commit to address the concerns.

This change looks almost good now. I wonder about two final things:

In updateContainerStats(...) do you think we should return if the container is DELETING or DELETED without making any updates? If this is a stale replica, they it may not be empty and hence could update the container stats incorrectly. If the Container is already in DELETING or DELETED state, then we can ignore any changes to it, as we know the stale replica will get removed anyway.

I am wondering if we could receive a stale replica when the container is DELETING. Then a replica would get added in updateContainerReplica(...). Then later the container will go to DELETED and the state replica will get reported again - at that point we will send a delete command, but the replica will never get removed from memory now I think. Would it make sense to send the delete command for any replicas received when the container is DELETING or DELETED?

This change looks almost good now. I wonder about two final things:
1. In `updateContainerStats(...)` do you think we should return if the container is DELETING or DELETED without making any updates? If this is a stale replica, they it may not be empty and hence could update the container stats incorrectly. If the Container is already in DELETING or DELETED state, then we can ignore any changes to it, as we know the stale replica will get removed anyway.

2. I am wondering if we could receive a stale replica when the container is DELETING. Then a replica would get added in `updateContainerReplica(...)`. Then later the container will go to DELETED and the state replica will get reported again - at that point we will send a delete command, but the replica will never get removed from memory now I think. Would it make sense to send the delete command for any replicas received when the container is DELETING or DELETED?

This change looks almost good now. I wonder about two final things:
1. In `updateContainerStats(...)` do you think we should return if the container is DELETING or DELETED without making any updates? If this is a stale replica, they it may not be empty and hence could update the container stats incorrectly. If the Container is already in DELETING or DELETED state, then we can ignore any changes to it, as we know the stale replica will get removed anyway.

2. I am wondering if we could receive a stale replica when the container is DELETING. Then a replica would get added in `updateContainerReplica(...)`. Then later the container will go to DELETED and the state replica will get reported again - at that point we will send a delete command, but the replica will never get removed from memory now I think. Would it make sense to send the delete command for any replicas received when the container is DELETING or DELETED?

There is following logic in ReplicatioManager, which will handle the replicas reported during container state is DELETING.
So we only need to send delete replica command for DELETED container during container report proceess, and let the ReplicationManager handle the DELETING container replica, because when a container in in DELETING state, It's sure that scm send out some replica deletion commands,  but we cannot tell replica stale or not in container report.
if (state == LifeCycleState.DELETING) {
handleContainerUnderDelete(container, replicas);
return;
}
@sodonnel

There is following logic in ReplicatioManager, which will handle the replicas reported during container state is DELETING.

Sorry I missed that. You are correct. I am +1 on this change as it is now, so feel free to commit it.
@linyiqun I do agree that I think this could be handled more cleanly and efficiently in the container report handler. However its probably not much of an overhead for replication manager. I am happy for us to commit the change as it is, and we can see how it performs in practice. Worst case we have to refactor the change out of RM into the report handler. What do you think?

@linyiqun I do agree that I think this could be handled more cleanly and efficiently in the container report handler. However its probably not much of an overhead for replication manager. I am happy for us to commit the change as it is, and we can see how it performs in practice. Worst case we have to refactor the change out of RM into the report handler. What do you think?

+1 for this, @sodonnel .
@ChenSammi , can you add a TODO comment while committing like below? That will be helpful for us to revisit this in the future.
// TODO: container report handling the empty containers.
if (isContainerEmpty(container, replicas)) {
deleteContainerReplicas(container, replicas);
}
+1 from me.

Thanks @sodonnel  and @linyiqun for the review.
Basically I think report handler is not a good place to handle all the empty container deletion process.  It can tell which one is empty , but it lacks of the facilities in ReplicationManager, such as inflightDeletion, such as handle send command to extra replica for DELETING state container, or resend command.  In future,  when container compation is considered, we can move this container deletion logic to be together.


Web rendered card actions UI states differences

Could be issue with online editor only, but I've noticed different rendering of web version for action buttons. See sample record below:

edited:
this could be cards explorer web editor issue only. The visualizer renders UI states
Thanks!

This should be fixed now, please let me know if not


Fixed Incorrect Link for Windows/.NET Core CLI

Title
Fixed Incorrect Link for Windows/.NET Core CLI
Summary
The Windows .Net Core CLI Getting Started link was pointing to the VSCode/Ionide markdown file. Updated it to the point the correct .NET Core CLI Getting Started markdown file.
Details
No further details.

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot


Extracted CSS - no way to copy/save

I am on your website now and pasted my HTML into the left "Type or Paste Your HTML" pane.
I selected my Preferences from the top "Extract" and "Indent CSS With" links and clicked the green "Extract!" button.
The right "Extracted CSS" pane populated correctly with the code.
However, I cannot find a way to SELECT ALL > COPY the code for use in my CSS file.   There seems to be no controls or buttons or links  to do so.
Please advise, as I find this tool EXTREMELY useful and would like to use it.

@TygerTech - I’ve noticed this, too. I should have submitted an issue. Thank you for doing so.
I did find a workaround. You can click in the “Extracted CSS” pane, hit command + a (mac) or control + a (windows), and then hit command + c (mac) or control + c (windows) to copy it to your clipboard. It will look like only a portion was selected, but in reality, everything will be in your clipboard.

@TygerTech - I’ve noticed this, too. I should have submitted an issue. Thank you for doing so.
I did find a workaround. You can click in the “Extracted CSS” pane, hit command + a (mac) or control + a (windows), and then hit command + c (mac) or control + c (windows) to copy it to your clipboard. It will look like only a portion was selected, but in reality, everything will be in your clipboard.

THANK YOU!!!  I appreciate it!


replace with Math.min method call

Make sure set the target branch to develop
What is the purpose of the change
XXXXX
Brief changelog
XX
Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

[x] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Coverage increased (+0.07%) to 51.911% when pulling 420218d61036e3316fef82ae8e03d98bc5efd1d5 on drgnchan:master into b4240d5cea8d001c21b9c0d73f5aa700fcd0d568 on apache:master.

Codecov Report

Merging #2796 (420218d) into master (b4240d5) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master    #2796      +/-   ##
============================================
+ Coverage     45.96%   45.98%   +0.01%
- Complexity     4316     4319       +3
============================================
  Files           547      547
  Lines         36236    36235       -1
  Branches       4808     4807       -1
============================================
+ Hits          16657    16663       +6
+ Misses        17491    17484       -7
  Partials       2088     2088

Impacted Files
Coverage Δ
Complexity Δ

...ache/rocketmq/broker/topic/TopicConfigManager.java
59.23% <100.00%> (+0.22%)
16.00 <0.00> (ø)

...org/apache/rocketmq/store/ha/WaitNotifyObject.java
75.00% <0.00%> (-7.70%)
10.00% <0.00%> (-1.00%)

...pache/rocketmq/test/listener/AbstractListener.java
66.66% <0.00%> (-7.41%)
9.00% <0.00%> (-1.00%)

...ketmq/client/impl/consumer/PullMessageService.java
71.11% <0.00%> (-4.45%)
8.00% <0.00%> (-1.00%)

...ketmq/common/protocol/body/ConsumerConnection.java
95.83% <0.00%> (-4.17%)
13.00% <0.00%> (-1.00%)

...a/org/apache/rocketmq/store/StoreStatsService.java
28.85% <0.00%> (-0.66%)
25.00% <0.00%> (-1.00%)

...rocketmq/client/impl/factory/MQClientInstance.java
50.14% <0.00%> (-0.15%)
90.00% <0.00%> (-1.00%)

...main/java/org/apache/rocketmq/store/CommitLog.java
66.42% <0.00%> (-0.11%)
79.00% <0.00%> (ø%)

.../apache/rocketmq/logging/inner/LoggingBuilder.java
63.92% <0.00%> (+0.31%)
3.00% <0.00%> (ø%)

...he/rocketmq/client/impl/consumer/ProcessQueue.java
60.46% <0.00%> (+0.46%)
33.00% <0.00%> (+1.00%)

... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b4240d5...420218d. Read the comment docs.

Plz create an issue before you submit a pull request and change the target branch to develop.

Plz create an issue before you submit a pull request and change the target branch to develop.

sorry,i will make another pr according to the checklist.


Shutting down sandboxes may hang

When sandboxes are terminated along with the other services, they don't shutdown.
It seems the issue is related to the Thrift client not being able to contact the datalayer anymore:
1615983259.380000,1615983259380,"2021-03-17 12:14:19.380","INFO","org.microfunctions.sandbox.frontend","Server is shutting down..."
1615983259.381000,1615983259381,"2021-03-17 12:14:19.381","INFO","org.microfunctions.sandbox.frontend","producer: stopping ..."
1615983259.381000,1615983259381,"2021-03-17 12:14:19.381","INFO","org.microfunctions.sandbox.frontend","producer: stopped"
1615983259.385000,1615983259385286,"INFO","ip-10-10-2-87.eu-central-1.compute.internal","wf-knix-786e01191e5d8564c9e4b93f827b2915-00001-deployment-n7zmn","0l","sand@sand","test_wf","786e01191e5d8564c9e4b93f827b2915","test","2021-03-17 12:14:19.385","[FunctionWorker] Done"
1615983259.381000,1615983259381,"2021-03-17 12:14:19.381","INFO","org.microfunctions.sandbox.frontend","consumer: stopping ..."
1615983269.009000,1615983269009,"2021-03-17 12:14:29.009","INFO","org.microfunctions.sandbox.frontend","consumer: stopped"
1615983269.009000,1615983269009,"2021-03-17 12:14:29.009","INFO","org.microfunctions.sandbox.frontend","datalayer: stopping ..."
1615983269.009000,1615983269009,"2021-03-17 12:14:29.009","INFO","org.microfunctions.sandbox.frontend","datalayer: stopped"
1615983269.009000,1615983269009,"2021-03-17 12:14:29.009","INFO","org.microfunctions.sandbox.frontend","Frontend stopped"
[2021-03-17 12:15:08,989] [INFO] Shutting down the queue service...
[2021-03-17 12:15:08,989] [INFO] Terminating queue service: pid: 10...
[2021-03-17 12:15:08,997] [INFO] Shutting down fluent-bit...
[2021-03-17 12:15:10,999] [INFO] Terminating fluent-bit: pid: 7...
[2021-03-17 12:15:11,000] [INFO] Shutdown complete.
failed to resolve sockaddr for dl-knix-2.datalayer:4998
Traceback (most recent call last):
  File "/opt/mfn/SandboxAgent/sandboxagent.py", line 408, in run
    self._get_and_handle_message()
  File "/opt/mfn/SandboxAgent/sandboxagent.py", line 118, in _get_and_handle_message
    lqm = self._local_queue_client.getMessage(self._instructions_topic, POLL_TIMEOUT)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/LocalQueueClient.py", line 78, in getMessage
    lqm = self.queue.getAndRemoveMessage(topic, timeout)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/local_queue/service/LocalQueueService.py", line 208, in getAndRemoveMessage
    return self.recv_getAndRemoveMessage()
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/local_queue/service/LocalQueueService.py", line 221, in recv_getAndRemoveMessage
    (fname, mtype, rseqid) = iprot.readMessageBegin()
  File "/usr/local/lib/python3.6/dist-packages/thrift/protocol/TCompactProtocol.py", line 334, in readMessageBegin
    proto_id = self.__readUByte()
  File "/usr/local/lib/python3.6/dist-packages/thrift/protocol/TCompactProtocol.py", line 313, in __readUByte
    result, = unpack('!B', self.trans.readAll(1))
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 62, in readAll
    chunk = self.read(sz - have)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 283, in read
    self.readFrame()
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 287, in readFrame
    buff = self.__trans.readAll(4)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 62, in readAll
    chunk = self.read(sz - have)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 126, in read
    buff = self.handle.recv(sz)
  File "/opt/mfn/SandboxAgent/sandboxagent.py", line 223, in sigterm
    raise KeyboardInterrupt
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/DataLayerClient.py", line 210, in putMapEntry
    status = self.datalayer.putEntryToMap(self.keyspace, self.maptablename, mapname, row, loc)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/data_layer/service/DataLayerService.py", line 1907, in putEntryToMap
    self.send_putEntryToMap(keyspace, table, mapName, keyValuePair, locality)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/data_layer/service/DataLayerService.py", line 1920, in send_putEntryToMap
    self._oprot.trans.flush()
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 304, in flush
    self.__trans.write(buf)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 149, in write
    message='Transport not open')
thrift.transport.TTransport.TTransportException: Transport not open

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 99, in open
    addrs = self._resolveAddr()
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 42, in _resolveAddr
    socket.AI_PASSIVE | socket.AI_ADDRCONFIG)
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
failed to resolve sockaddr for dl-knix-2.datalayer:4998
Traceback (most recent call last):
  File "/opt/mfn/SandboxAgent/sandboxagent.py", line 408, in run
    self._get_and_handle_message()
  File "/opt/mfn/SandboxAgent/sandboxagent.py", line 118, in _get_and_handle_message
    lqm = self._local_queue_client.getMessage(self._instructions_topic, POLL_TIMEOUT)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/LocalQueueClient.py", line 78, in getMessage
    lqm = self.queue.getAndRemoveMessage(topic, timeout)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/local_queue/service/LocalQueueService.py", line 208, in getAndRemoveMessage
    return self.recv_getAndRemoveMessage()
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/local_queue/service/LocalQueueService.py", line 221, in recv_getAndRemoveMessage
    (fname, mtype, rseqid) = iprot.readMessageBegin()
  File "/usr/local/lib/python3.6/dist-packages/thrift/protocol/TCompactProtocol.py", line 334, in readMessageBegin
    proto_id = self.__readUByte()
  File "/usr/local/lib/python3.6/dist-packages/thrift/protocol/TCompactProtocol.py", line 313, in __readUByte
    result, = unpack('!B', self.trans.readAll(1))
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 62, in readAll
    chunk = self.read(sz - have)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 283, in read
    self.readFrame()
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 287, in readFrame
    buff = self.__trans.readAll(4)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 62, in readAll
    chunk = self.read(sz - have)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 126, in read
    buff = self.handle.recv(sz)
  File "/opt/mfn/SandboxAgent/sandboxagent.py", line 223, in sigterm
    raise KeyboardInterrupt
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/DataLayerClient.py", line 210, in putMapEntry
    status = self.datalayer.putEntryToMap(self.keyspace, self.maptablename, mapname, row, loc)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/data_layer/service/DataLayerService.py", line 1907, in putEntryToMap
    self.send_putEntryToMap(keyspace, table, mapName, keyValuePair, locality)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/data_layer/service/DataLayerService.py", line 1920, in send_putEntryToMap
    self._oprot.trans.flush()
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 304, in flush
    self.__trans.write(buf)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 149, in write
    message='Transport not open')
thrift.transport.TTransport.TTransportException: Transport not open

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 99, in open
    addrs = self._resolveAddr()
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 42, in _resolveAddr
    socket.AI_PASSIVE | socket.AI_ADDRCONFIG)
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
failed to resolve sockaddr for dl-knix-2.datalayer:4998
Traceback (most recent call last):
  File "/opt/mfn/SandboxAgent/sandboxagent.py", line 408, in run
    self._get_and_handle_message()
  File "/opt/mfn/SandboxAgent/sandboxagent.py", line 118, in _get_and_handle_message
    lqm = self._local_queue_client.getMessage(self._instructions_topic, POLL_TIMEOUT)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/LocalQueueClient.py", line 78, in getMessage
    lqm = self.queue.getAndRemoveMessage(topic, timeout)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/local_queue/service/LocalQueueService.py", line 208, in getAndRemoveMessage
    return self.recv_getAndRemoveMessage()
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/local_queue/service/LocalQueueService.py", line 221, in recv_getAndRemoveMessage
    (fname, mtype, rseqid) = iprot.readMessageBegin()
  File "/usr/local/lib/python3.6/dist-packages/thrift/protocol/TCompactProtocol.py", line 334, in readMessageBegin
    proto_id = self.__readUByte()
  File "/usr/local/lib/python3.6/dist-packages/thrift/protocol/TCompactProtocol.py", line 313, in __readUByte
    result, = unpack('!B', self.trans.readAll(1))
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 62, in readAll
    chunk = self.read(sz - have)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 283, in read
    self.readFrame()
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 287, in readFrame
    buff = self.__trans.readAll(4)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 62, in readAll
    chunk = self.read(sz - have)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 126, in read
    buff = self.handle.recv(sz)
  File "/opt/mfn/SandboxAgent/sandboxagent.py", line 223, in sigterm
    raise KeyboardInterrupt
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/DataLayerClient.py", line 210, in putMapEntry
    status = self.datalayer.putEntryToMap(self.keyspace, self.maptablename, mapname, row, loc)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/data_layer/service/DataLayerService.py", line 1907, in putEntryToMap
    self.send_putEntryToMap(keyspace, table, mapName, keyValuePair, locality)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/data_layer/service/DataLayerService.py", line 1920, in send_putEntryToMap
    self._oprot.trans.flush()
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 304, in flush
    self.__trans.write(buf)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 149, in write
    message='Transport not open')
thrift.transport.TTransport.TTransportException: Transport not open

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 99, in open
    addrs = self._resolveAddr()
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 42, in _resolveAddr
    socket.AI_PASSIVE | socket.AI_ADDRCONFIG)
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
failed to resolve sockaddr for dl-knix-2.datalayer:4998
Traceback (most recent call last):
  File "/opt/mfn/SandboxAgent/sandboxagent.py", line 408, in run
    self._get_and_handle_message()
  File "/opt/mfn/SandboxAgent/sandboxagent.py", line 118, in _get_and_handle_message
    lqm = self._local_queue_client.getMessage(self._instructions_topic, POLL_TIMEOUT)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/LocalQueueClient.py", line 78, in getMessage
    lqm = self.queue.getAndRemoveMessage(topic, timeout)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/local_queue/service/LocalQueueService.py", line 208, in getAndRemoveMessage
    return self.recv_getAndRemoveMessage()
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/local_queue/service/LocalQueueService.py", line 221, in recv_getAndRemoveMessage
    (fname, mtype, rseqid) = iprot.readMessageBegin()
  File "/usr/local/lib/python3.6/dist-packages/thrift/protocol/TCompactProtocol.py", line 334, in readMessageBegin
    proto_id = self.__readUByte()
  File "/usr/local/lib/python3.6/dist-packages/thrift/protocol/TCompactProtocol.py", line 313, in __readUByte
    result, = unpack('!B', self.trans.readAll(1))
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 62, in readAll
    chunk = self.read(sz - have)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 283, in read
    self.readFrame()
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 287, in readFrame
    buff = self.__trans.readAll(4)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 62, in readAll
    chunk = self.read(sz - have)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 126, in read
    buff = self.handle.recv(sz)
  File "/opt/mfn/SandboxAgent/sandboxagent.py", line 223, in sigterm
    raise KeyboardInterrupt
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/DataLayerClient.py", line 210, in putMapEntry
    status = self.datalayer.putEntryToMap(self.keyspace, self.maptablename, mapname, row, loc)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/data_layer/service/DataLayerService.py", line 1907, in putEntryToMap
    self.send_putEntryToMap(keyspace, table, mapName, keyValuePair, locality)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/data_layer/service/DataLayerService.py", line 1920, in send_putEntryToMap
    self._oprot.trans.flush()
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 304, in flush
    self.__trans.write(buf)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 149, in write
    message='Transport not open')
thrift.transport.TTransport.TTransportException: Transport not open

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 99, in open
    addrs = self._resolveAddr()
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 42, in _resolveAddr
    socket.AI_PASSIVE | socket.AI_ADDRCONFIG)
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
failed to resolve sockaddr for dl-knix-2.datalayer:4998
Traceback (most recent call last):
  File "/opt/mfn/SandboxAgent/sandboxagent.py", line 408, in run
    self._get_and_handle_message()
  File "/opt/mfn/SandboxAgent/sandboxagent.py", line 118, in _get_and_handle_message
    lqm = self._local_queue_client.getMessage(self._instructions_topic, POLL_TIMEOUT)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/LocalQueueClient.py", line 78, in getMessage
    lqm = self.queue.getAndRemoveMessage(topic, timeout)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/local_queue/service/LocalQueueService.py", line 208, in getAndRemoveMessage
    return self.recv_getAndRemoveMessage()
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/local_queue/service/LocalQueueService.py", line 221, in recv_getAndRemoveMessage
    (fname, mtype, rseqid) = iprot.readMessageBegin()
  File "/usr/local/lib/python3.6/dist-packages/thrift/protocol/TCompactProtocol.py", line 334, in readMessageBegin
    proto_id = self.__readUByte()
  File "/usr/local/lib/python3.6/dist-packages/thrift/protocol/TCompactProtocol.py", line 313, in __readUByte
    result, = unpack('!B', self.trans.readAll(1))
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 62, in readAll
    chunk = self.read(sz - have)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 283, in read
    self.readFrame()
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 287, in readFrame
    buff = self.__trans.readAll(4)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 62, in readAll
    chunk = self.read(sz - have)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 126, in read
    buff = self.handle.recv(sz)
  File "/opt/mfn/SandboxAgent/sandboxagent.py", line 223, in sigterm
    raise KeyboardInterrupt
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/DataLayerClient.py", line 210, in putMapEntry
    status = self.datalayer.putEntryToMap(self.keyspace, self.maptablename, mapname, row, loc)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/data_layer/service/DataLayerService.py", line 1907, in putEntryToMap
    self.send_putEntryToMap(keyspace, table, mapName, keyValuePair, locality)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/data_layer/service/DataLayerService.py", line 1920, in send_putEntryToMap
    self._oprot.trans.flush()
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 304, in flush
    self.__trans.write(buf)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 149, in write
    message='Transport not open')
thrift.transport.TTransport.TTransportException: Transport not open

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 99, in open
    addrs = self._resolveAddr()
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 42, in _resolveAddr
    socket.AI_PASSIVE | socket.AI_ADDRCONFIG)
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
failed to resolve sockaddr for dl-knix-2.datalayer:4998
Traceback (most recent call last):
  File "/opt/mfn/SandboxAgent/sandboxagent.py", line 408, in run
    self._get_and_handle_message()
  File "/opt/mfn/SandboxAgent/sandboxagent.py", line 118, in _get_and_handle_message
    lqm = self._local_queue_client.getMessage(self._instructions_topic, POLL_TIMEOUT)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/LocalQueueClient.py", line 78, in getMessage
    lqm = self.queue.getAndRemoveMessage(topic, timeout)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/local_queue/service/LocalQueueService.py", line 208, in getAndRemoveMessage
    return self.recv_getAndRemoveMessage()
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/local_queue/service/LocalQueueService.py", line 221, in recv_getAndRemoveMessage
    (fname, mtype, rseqid) = iprot.readMessageBegin()
  File "/usr/local/lib/python3.6/dist-packages/thrift/protocol/TCompactProtocol.py", line 334, in readMessageBegin
    proto_id = self.__readUByte()
  File "/usr/local/lib/python3.6/dist-packages/thrift/protocol/TCompactProtocol.py", line 313, in __readUByte
    result, = unpack('!B', self.trans.readAll(1))
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 62, in readAll
    chunk = self.read(sz - have)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 283, in read
    self.readFrame()
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 287, in readFrame
    buff = self.__trans.readAll(4)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 62, in readAll
    chunk = self.read(sz - have)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 126, in read
    buff = self.handle.recv(sz)
  File "/opt/mfn/SandboxAgent/sandboxagent.py", line 223, in sigterm
    raise KeyboardInterrupt
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/DataLayerClient.py", line 210, in putMapEntry
    status = self.datalayer.putEntryToMap(self.keyspace, self.maptablename, mapname, row, loc)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/data_layer/service/DataLayerService.py", line 1907, in putEntryToMap
    self.send_putEntryToMap(keyspace, table, mapName, keyValuePair, locality)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/data_layer/service/DataLayerService.py", line 1920, in send_putEntryToMap
    self._oprot.trans.flush()
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 304, in flush
    self.__trans.write(buf)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 149, in write
    message='Transport not open')
thrift.transport.TTransport.TTransportException: Transport not open

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 99, in open
    addrs = self._resolveAddr()
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 42, in _resolveAddr
    socket.AI_PASSIVE | socket.AI_ADDRCONFIG)
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
failed to resolve sockaddr for dl-knix-2.datalayer:4998
Traceback (most recent call last):
  File "/opt/mfn/SandboxAgent/sandboxagent.py", line 408, in run
    self._get_and_handle_message()
  File "/opt/mfn/SandboxAgent/sandboxagent.py", line 118, in _get_and_handle_message
    lqm = self._local_queue_client.getMessage(self._instructions_topic, POLL_TIMEOUT)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/LocalQueueClient.py", line 78, in getMessage
    lqm = self.queue.getAndRemoveMessage(topic, timeout)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/local_queue/service/LocalQueueService.py", line 208, in getAndRemoveMessage
    return self.recv_getAndRemoveMessage()
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/local_queue/service/LocalQueueService.py", line 221, in recv_getAndRemoveMessage
    (fname, mtype, rseqid) = iprot.readMessageBegin()
  File "/usr/local/lib/python3.6/dist-packages/thrift/protocol/TCompactProtocol.py", line 334, in readMessageBegin
    proto_id = self.__readUByte()
  File "/usr/local/lib/python3.6/dist-packages/thrift/protocol/TCompactProtocol.py", line 313, in __readUByte
    result, = unpack('!B', self.trans.readAll(1))
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 62, in readAll
    chunk = self.read(sz - have)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 283, in read
    self.readFrame()
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 287, in readFrame
    buff = self.__trans.readAll(4)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 62, in readAll
    chunk = self.read(sz - have)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 126, in read
    buff = self.handle.recv(sz)
  File "/opt/mfn/SandboxAgent/sandboxagent.py", line 223, in sigterm
    raise KeyboardInterrupt
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/DataLayerClient.py", line 210, in putMapEntry
    status = self.datalayer.putEntryToMap(self.keyspace, self.maptablename, mapname, row, loc)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/data_layer/service/DataLayerService.py", line 1907, in putEntryToMap
    self.send_putEntryToMap(keyspace, table, mapName, keyValuePair, locality)
  File "/opt/mfn/SandboxAgent/../FunctionWorker/python/data_layer/service/DataLayerService.py", line 1920, in send_putEntryToMap
    self._oprot.trans.flush()
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TTransport.py", line 304, in flush
    self.__trans.write(buf)
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 149, in write
    message='Transport not open')
thrift.transport.TTransport.TTransportException: Transport not open

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 99, in open
    addrs = self._resolveAddr()
  File "/usr/local/lib/python3.6/dist-packages/thrift/transport/TSocket.py", line 42, in _resolveAddr
    socket.AI_PASSIVE | socket.AI_ADDRCONFIG)
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

It starts with the LocalQueue exception. Basically, the sandbox shutdown shuts down the queue at some point, but that should be the last process to shut down.
Besides that, when the components are shutting down, are the workflows undeployed first? If not, these errors will happen and the sandbox will be eventually shut down. But I guess they will keep gettig restarted, because the underlying K8s has not been told about the shutdown? Is that the problem?

Is this resolved by commit f5f059f3b705d8743bad4627b5da5df940601586?

This commit may be reated to resolving this issue: https://github.com/knix-microfunctions/knix/commit/c277126fb9199ea428a1581aa86721190ac8574f

Another commit that handles shutdowns more properly: https://github.com/knix-microfunctions/knix/commit/c05b4902eeb6fb724aede707864220f4cd706d61

This issue only manifests itself during the uninstallation of the entire platform. The above commits should alleviate this problem. If it persists, please reopen.


Add taxon checks for signaling pathways

These were found from a "PAINT" issue (https://github.com/geneontology/paint/issues/25)
The annotation of RCN1 (aka P36054) to GO:0070884 regulation of calcineurin-NFAT signaling cascade is valid. This is in PTHR10300. Someone needs to say whether or not it should pass the taxon checks or not. (and fix accordingly) Use this to test (Taxon 559292 is yeast) http://owlservices.berkeleybop.org/isClassApplicableForTaxon?format=txt&idstyle=obo&id=GO:0070884&taxid=NCBITaxon:559292
The same is true for the annotation of YCK1 (aka P23291) to GO:0016055 (Wnt signaling pathway) in family PTHR11909. Taxon checks are showing this as valid, so, if it is not possible biologically, then the fix has to be there.

This issue seems to span more than one ticket.
The calcineurin pathway is conserved from yeast to humans. In humans, calcineurin acts with NFAT in the immune system. In yeast and other fungi, calcineurin functions in stress responses but NFAT does not exist in yeast nor does an immune cell function.  It is the conjugation of calcineurin and NFAT as a signalling pathway that creates a problem for propagation to yeast. Calcineurin is fine, it is NFAT that does not apply to an organism without an immune system.

Thanks @dianeoinglis .
Based on your feedback, I’d restrict ‘calcineurin-NFAT signaling cascade’ to ‘only in Metazoa’.
Do we need to add a separate term for ’calcineurin pathway’? If yes, could you please suggest a definition, placement and references.
As for GO:0016055 (Wnt signaling pathway), I’d also restrict it to ‘only in Metazoa’ (see https://en.wikipedia.org/wiki/Wnt_signaling_pathway). Though, I’m wondering if the taxon rule should be applied to its parent ‘cell-cell signaling by wnt’ instead. Asking @dosumis as he added ‘cell-cell signaling by wnt’.
Thanks,
Paola

The long term solution should be to axiomatize the pathways based on the ancestral proteins involved and inherit the taxon constraints from those. This may seem a bit circular but I think that's OK, we just end up coordinating the engineering of the signaling part of GO with curating the families.
@rebeccafoulger and I started on this a while ago, x-signaling.owl in the extensions directory

Yes, I do think a separate term for ’calcineurin pathway' or possibly a better term is "calcineurin signalling." Calcineurin regulates quite a few responses in yeast and mammals and the singular "pathway" seems restrictive. "Calcineurin signalling" allows for multiple responses to calcineurin. I can definitely suggest a definition, suggest placement and include references.

Opened new ticket for NTR,
https://github.com/geneontology/go-ontology/issues/12591

Note for self: the new term from https://github.com/geneontology/go-ontology/issues/12591 is
GO:0097720 calcineurin-mediated signaling

Hi @selewis,
As a quick fix, I restricted ‘calcineurin-NFAT signaling cascade’ and ‘Wnt signaling pathway’ to ‘only in Metazoa’.
I’ll now assign to @cmungall for him to strategize the long-term solution.

@selewis Do we have a systematic solution for this ? Perhaps we need to create new tickets for each pathway ?
(if that's the case we could close this ticket)
Thanks, Pascale


Hangs at startup on MacOS

When I start the MacOS app, which used to work, it just hangs. I think only the updater daemon starts.
my log id: 011b259bee32c67b1427e51c

Sent you some ideas over Keybase chat, but it's a permissions issues:
ERROR: open /Users/<your username>/Library/Application Support/Keybase/keybase.leveldb/LOCK: permission denied

Huh, did a
chmod -R u+rw Library/Application Support/Keybase
and now it's happy. Wonder what happened to the permissions.

Great, glad that fixed it (our error reporting could be better of course).
Maybe you tried to run keybase as root?

No, I owned the files but something made them all read-only, perhaps the migration process from the disk on my old dead mac to the current one.


[code-infra] Update runners from node 18 to 20

related https://github.com/mui/mui-x/pull/14466

Should we also update this?
https://github.com/mui/material-ui/blob/e3e28b657075acfa0b0a288b604b8546fe2425b2/.github/workflows/ci.yml#L33-L36

Thanks @ZeeshanTamboli, I missed GHA entirely 😅
https://github.com/mui/material-ui/pull/43593


Add list and delete cli command options

Having the ability to list repositories and packages would be a big help, as would having the ability to delete packages.  I'd like to request this as an enhancement.

The option to delete packages already exists using the package-delete command. We'll look into the other two requests. Thanks @jsabo.

I am looking at the jfrog cli documentation: https://www.jfrog.com/confluence/display/RTF/JFrog+CLI.
There doesn't seem to be a command to delete an artifact?

Delete, move and copy artifacts commands will be added in our next release.
The commands are already commited to dev  branch, you are welcome to clone, build and use it.
Thanks @tmds.

@tmds, version 1.2.0 has been released and includes move, copy and delete commands.

@romangurevitch super!

looking forward to 1.2.0 showing up in homebrew

Sure @jonathan-luminal. We submitted the pull request for 1.2.0 to homebrew today.

Can you use any of the following to get a listing of what's in artifactory or bintray?
$ jfrog --version
jfrog version 1.2.0
$ jfrog rt --help
NAME:
   jfrog rt - Artifactory commands

USAGE:
   jfrog rt command [command options] [arguments...]

COMMANDS:
   config, c	Configure Artifactory details
   upload, u	Upload files
   download, dl	Download files
   move, mv	Move files
   copy, cp	Copy files
   delete, del	Delete files
   help, h	Shows a list of commands or help for one command

OPTIONS:
   --help, -h	show help

Hi @jonathan-luminal,
Yes, you can use:
jfrog rt

to see the list of available Artifactory commands and
jfrog bt

for the Bintray commands.
It recommended to use the online documentation for details about the commands arguments.

@eyalbe4 @jonathan-luminal is asking for a command to list artifacts, which would be very useful.
The output of this command could be lines in the form [repository_name]/[repository_path]. For example, you can then take the output, pipe it to grep and then use it as input for the delete command.
A --recurse can allow you to list recursively.
For efficiency reasons it may be handy to support providing a filter as part of the list command if the server supports such a list query.

Got it @tmds. Thanks.
We are planning to support list (search) of artifacts in future releases.

@jonathan-luminal, version 1.2.0 of jfrog-cli-go is available in homebrew.

@jonathan-luminal and @tmds, version 1.4.0 of the CLI adds a new search command for Artifactory.

if you want a plain list, you could just use jq:
JFROG_CLI_LOG_LEVEL=ERROR jfrog rt search repository/*.html | jq .[].path --raw-output

The JFROG_CLI_LOG_LEVEL environment variable was added in version 1.4.1. it will be soon available in homebrew as well. Closing this issue.


Native crash "RN version 0.62.x" with 3rd party SDK

Running react native sample with our face recognition SDK crashes the app. Our SDK uses libc++_shared.so.
Description
If I create new project and with our SDK, init with System.loadLibrary("iface") in Activity onCreate - app crashes, crash log is attached. If I call System.loadLibrary("iface") in Application onCreate before everything else app does not crash. It also does not crash if I load System.loadLibrary("c++_shared") instead in Application onCreate.
React Native version:
info Fetching system and libraries information...
System:
OS: Windows 10 10.0.18363
CPU: (8) x64 Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Memory: 7.01 GB / 31.88 GB
Binaries:
Node: 12.16.1 - C:\Program Files\nodejs\node.EXE
Yarn: Not Found
npm: 6.13.4 - C:\Program Files\nodejs\npm.CMD
Watchman: Not Found
SDKs:
Android SDK:
Android NDK: 21.1.6352462
IDEs:
Android Studio: Version  <IP_ADDRESS> AI-192.7<IP_ADDRESS>92135
Languages:
Java: javac 11 - C:\Program Files\Java\jdk-11\bin\javac.EXE
Python: 3.6.0 - C:\Users\pmolcany\AppData\Local\Programs\Python\Python36\python.EXE
npmPackages:
@react-native-community/cli: Not Found
react: 16.11.0 => 16.11.0
react-native: 0.62.0 => 0.62.2
npmGlobalPackages:
react-native: Not Found
Steps To Reproduce
Provide a detailed list of steps that reproduce the issue.

Setup the face recognition SDK with instructions below
System.loadLibrary("iface") in Activity onCreate
crash.txt

Expected Results
App should not crash
Snack, code example, screenshot, or link to a repository:
To load the face recognition SDK into project add repository in build.gradle allprojects
maven {
            url 'http://maven.innovatrics.com/releases'
        }

And add dependency to app/build.gradle
implementation 'com.innovatrics.android:dot:2.16.2'

App does dot crash with react native 0.61.5

hey, just asking... Was this solved somehow or just ignored ?