Favour user fn response codes for triggers, with caveats

Currently, this is a little unclear and there is a problem with user
fns returning error responses which we can not differentiate from our
service errors.
This changes breaks the status selection logic out and adds some tests that spell out
how we want this fiddly bit to work. The offering here is: user fn
status returned, if set, but modify 5xx to 502. If no response code
from user fn, use one the service has deemed appropriate. Default to
200 if neither of them have commented.

Seems good to me.

@rdallman I think you should take a glance at this change, you to @zootalures.
This seems like the right way to go, but is one of those changes that modifies mental models.

I think I’d rather trigger status codes didn’t propagate if the function was in error. I see this as nesting of “what did the service say?” takes precedence over “what did the FDK say” takes precedence over “what did the user code say”
To a certain extent I’m wondering if propagating any headers (possibly including partial ones that the function set when it was still on the happy path) is right

As @zootalures pointed out, we've already seen an example of this happening where FDK emits a 500, but depending on the failure type & timing of events, that could easily end up being a 502 due to IO failure detected by the agent. We should also keep in mind both invokes and trigger cases where we freely read and propagate the http status code from FDK (user-side). The status codes in general I think need a context, that's why I was suggesting inserting a header to show origin. For example:

User errors due to I/O/parsing(*) are flagged as 502 => origin=function (502 is punched in by agent.go)
If  we ever read a http status code from FDK => origin=function
everything else => origin=service

Then we can use this for stats as a tag, etc. Or even include it in http headers in responses to clarify where the http code originated from.
(*) We'll need to carefully cherry pick some of these, as some cases may be due to service issue, eg. docker failure. Today, UDS I/O failure or docker-exit is considered a 502 by agent code.

related #1314

FYI: #1337

#1337  has superseded these ideas.


Gitignore shared schemes

Checking shared schemes in to Swift packages vends those schemes to consumers. Ie, if I make a new Xcode project and add a dependency on xmlwrangler, xmlwrangler will start appearing in my list of schemes, despite the fact I never really need to build or test xmlwrangler directly. This is undesirable.
For developers of the xmlwrangler package, Xcode auto-creates an xmlwrangler scheme when you open the package. If you need to persist customizations to the scheme, I believe there are alternatives, such as adding a workspace to the package and locating schemes there.
See discussion here, including some unofficial guidance from the Swift team: apple/swift-collections#155
Some Swift packages ignore their whole .swiftpm directory, but I leave that to your discretion. This PR is narrower, simply untracking shared schemes.

First of all thanks for a) opening a PR instead of just an issue and b) explaining the reasoning behind it in detail!
I'm a bit torn by this, though. One the one hand, I see the reason behind it and the rather simple solution this provides. On the other hand, this is technically an Xcode bug IMHO. Showing schemes of (non-local) dependencies doesn't make any sense to me.
Since I usually stick quite closely to the auto-generated .gitignore, I checked the SPM repo for issues/PRs regarding this and found these:

https://github.com/apple/swift-package-manager/pull/2200
https://github.com/apple/swift-package-manager/pull/2209
https://github.com/apple/swift-package-manager/pull/2674

They're all related to ignoring the whole .swiftpm directory, or at least the complete xcode directory beneath it, though.

Back to the change at hand, I noticed that one thing would go "missing" after this.
I did set up the test plans for our Swift F/OSS components (cf85375 in this repo), to e.g. get parallel testing by default on developer machines. The xctestplan still exists after this PR, but it's no longer used, since there's no scheme referencing it.
With shared schemes:

W/o shared schemes (after this PR):

Since currently it's only parallel testing on developer machines, that's not too bad of a loss, but I'd like to keep the developer setup as straightforward as possible. Namely being able to do xed . in the package directory (or opening it manually with Xcode) and being ready to go.
Adding a separate Xcode workspace would work in theory, but also requires additional steps for getting started since Xcode prefers packages over xcworkspaces. So xed . will open the package instead of the xcworkspace.
Any ideas on this?

In either way, I'll open a radar regarding the shared schemes of dependencies showing up in Xcode. Not that I'm really expecting this to change, though. 🙃

Let's add this for now. I've modified it to include the complete .swiftpm directory. I'll remove the directory from git in a separate commit.
I'll keep an eye on Apple's handling of the directory to see if anything changes here.


Added logging for hubs.

Made this in relation to (https://github.com/samemory/homebridge-eufy-security/issues/20) so we can determine if the client library is returning all hubs.


Fix mission download

Whoops :sweat_smile:

Oops. Do we need to create a patch release after this?


Downgraded kfp-tekton to supported version

LGTM, but I don't have time to test and confirm. Merge away!


is support react-native or has demo

is support react-native or has demo?

found any solution for react-native expo ?


高并发时报错

[mysql] 2016/06/14 08:45:09 statement.go:27: invalid connection

这个是驱动报的错误。

I will close this issue. Please feel free to reopen it if you have any further information.


Add native_inline_images request parameter

The native_inline_images request parameter is missing in the list of valid options for /post, /post/edit and /post/reblog.

:100:

:100:

@epochiero I know this pull request is quite old, but could you fix the merge conflicts and sign the CLA?

Thank you for submitting this pull request, however I do not see a valid CLA on file for you. Before we can merge this request please visit https://yahoocla.herokuapp.com/ and agree to the terms. Thanks! :smile:


brew.sh analytics default enabled

Describe the bug
https://docs.brew.sh/Analytics is enabled by default
What did you expect to happen?
As it is customary in Fedora, analytics should be disabled by default. https://docs.brew.sh/Analytics
Output of rpm-ostree status
No response
Hardware
No response
Extra information or context
No response

I don't think this will be in scope for UBlue.
Creating a ujust script to disable telemetry, maybe.  Default disabling them, I'm not sure.
As long as the collected metrics are anonymous and the data gathered is relevant, we prefer to keep the settings stock and be responsible open-source software (OSS) consumers. Telemetry plays a crucial role in helping projects understand how their software is being used and where they should prioritize their efforts.

My opinion, take it or leave it :)
Telemetry plays crucial role to projects if the user actually uses it. Brew comes pre-installed, yet nothing that I can see uses it, it's opaque and if it wasn't for scrolling through dmesg I wouldn't even know that it was there. It is not beneficial to any project trying to understand the user counts, having something preinstalled without the knowledge of the user, essentially merely polluting the metrics. At a glance someone might think "oh look, we have 54k users this month" but in reality it was probably "oh look, 54k people happen to have this on their system, out of which only 12k know about it and actually use it." Now i don't know how they gather metrics, but it does run update on a timer which prints the telemetry warning, so I'd assume that they have even such an inactive pre-installed oblivious user logged as a number in their little spreadsheet. Just like a metric of "this month this many people installed our brew product" probably wasn't exactly accurate either, if it came pre-installed.

The brew maintainers ask that the analytics not be changed downstream, but given we're pre installing it may be worth looping them into this conversation to see what they recommend.
We install brew because it solves for the CLI what flatpak and distrobox do for GUI applications. One less reason to layer things.

We install brew because it solves for the CLI what flatpak and distrobox solve for GUI applications. One less reason to layer things.

Homebrew on the host doesn't really solve it, see this forum thread. 😬


Add test for fuzzy matching on strings with /'s in API

This revision originally started life as a workaround, which lead to the
discovery of CVE-2016-7040. Since
https://github.com/ManageIQ/manageiq/pull/12822, which addressed the
security issues, has obviated the need for a workaround on the API side,
this revision serves only to prove that this feature of the API works
properly, preventing future regressions.
Closes https://github.com/ManageIQ/manageiq/issues/11035
Thanks to @rberecka who originally paired with me on this issue!
@miq-bot add-label api, test
@miq-bot assign @abellotti

still weird/unrelated failures.  Maybe rebase/repush ?

@abellotti troubling, but I believe the failures on the build are legit. It looks like creating the tags in my test is changing the world in an unexpected way such that cleaning them up is not sufficient. Possibly a uniqueness validation issue with the factory? I will try to find out what's going on here....suffice to say do not merge! :wink:

From this one test ?  troubling indeed since FactoryGirl.create(:tag, :name => "/managed/...") is used in other places too like tag_specs.rb, etc.

This depends on https://github.com/ManageIQ/manageiq/pull/13365

@abellotti now that https://github.com/ManageIQ/manageiq/pull/13365 has been merged, I've rebased and this should be :green_heart:

🙏  for 🍏

Yeay, 🍏


Label suggestion

Following are some of the annotations used by "spring-projects/spring-boot".
ideal-for-contribution
first-timers-only

there are already tags for beginner easy pick and good first issue
at some point we have to decide whether standardization is better or accessibility to more labels

Maybe have one single option in the UI, which searches for issues using any of the labels in GitHub repos?
ie a mapping:
beginner:[beginner, easy pick, ideal-for-contribution, ...]
good first issue: [good first issue, first-timers-only, ...]

That way potential contributors that use issuehub.io don't need to know (or care) that different projects use different labels


made players factorio global

This adds an on_load event to make a local reference to the global players table.
Adds a migration script to add an empty global players table
changes all event.player_index to local pindex in move_key
should fix issue #1

Not ready! just found a bug with smoothwalking and teleportation, which seems like it should be unrelated... but apparently it is related...

The autocalibration pull request includes all changes from this one so I'm closing this one.


Cannot host web server

I tried launching AstroLauncher for the first time and it did not work. I kept getting an error that a process is already using your ___ port. My console log is
12:54:22 - INFO   'AstroLauncher - Unofficial Dedicated Server Launcher v<IP_ADDRESS>'
12:54:22 - INFO   'If you encounter any bugs please open a new issue at:'
12:54:22 - INFO   'https://github.com/ricky-davis/AstroLauncher/issues'
12:54:22 - INFO   'To safely stop the launcher and server press CTRL+C'
12:54:23 - INFO   'Starting a new session'
12:54:23 - INFO   'Attempting to validate Playfab Certs'
12:54:24 - CRITICAL 'A process is already using your Server Port ( 7777 UDP )'
12:54:24 - CRITICAL 'A process is already using your Console Port ( 1234 TCP )'
12:54:24 - CRITICAL 'A process is already using your Web Port ( 5000 TCP )'

@ricky-davis you helped this person on discord and it can be closed.

The solutions on the discord did not help. I restarted my computer multiple times and I looked at my firewall settings and I didn't see anything that was using those ports.

Those errors mean something is already taking those ports, for all 3 ports to be taken, that means that most likely a second copy of the server is running already. If there's not, it's possible your setup is breaking AstroLauncher's process/port detection. Try changing the ports used in AstroServerSettings.ini, Launcher.ini, and Engine.ini, and if it's still throwing errors, something is fundamentally wrong with your setup.


With no network connection present, dies after timeout

terminating with uncaught exception of type std::__1::bad_weak_ptr: bad_weak_ptr
!!! Kill PID: 1, SIG: 6 - IOT trap / ABORT

	**** CPU 0 PANIC: ****
 	Killing a process doesn't make sense in IncludeOS. Panic.
	Heap is at: 0x607000 / 0xffdffff  (diff=0xf9d8fff)
	Heap usage: 941 / 256785 Kb (0.37%)

[0]   11ef50 + 0x149: panic
[1]   11f120 + 0x04b: kill
[2]   269c20 + 0x01a: _kill_r
[3]   2660e0 + 0x103: default_terminate_handler()
[4]   263510 + 0x008: std::__terminate(void (*)())
[5]   262e40 + 0x0ad: __cxa_throw
[6]   12e960 + 0x17d: net::tcp::Connection::clean_up()
[7]   12ba40 + 0x0db: net::tcp::Connection::rtx_timeout()
[8]   2ad500 + 0x01f: spec::inplace<16u, 8u, void>::inplace<delegate<void (), spec::inplace, 16u, 8u>::delegate<net::tcp::Connection>(net::tcp::Connection*, void (net::tcp::Connection::*)())::'lambda'(), delegate<void (), spec::inplace, 16u, 8u>::delegate<net::tcp::Connection>(net::tcp::Connection*, void (net::tcp::Connection::*)())::'lambda'()>(net::tcp::Connection&&)::'lambda'(std::__1::aligned_storage<16u, 8u>::type&)::__invoke(std::__1::aligned_storage<16u, 8u>::type&)

Easiest way to replicate is to run any service without bridge present

From the backtrace it looks like maybe it's this call to shared_from_this().
From http://en.cppreference.com/w/cpp/memory/enable_shared_from_this/shared_from_this:

Effectively executes std::shared_ptr<T>(weak_this), where weak_this is the private mutable std::weak_ptr<T> member of enable_shared_from_this.

Not really sure how that's possible...

This was probably fixed when we realized how shared_from_this actually worked


[libmodplug] [ffmpeg] fix libmodplug .pc file for static windows build

What does your PR fix?
Fixes the libmodplug .pc file on windows (user32 is needed). Patch has been posted upstream here: https://github.com/Konstanty/libmodplug/pull/59 I've enabled windows static builds for the ffmpeg modplug feature as well.

Which triplets are supported/not supported? Have you updated the CI baseline?
Libmodplug now additionally supports all static windows triplets.

Does your PR follow the maintainer guide?
Yes, to the best of my knowledge.

Yes.

file(INSTALL ${SOURCE_PATH}/COPYING DESTINATION ${CURRENT_PACKAGES_DIR}/share/{PORT} RENAME copyright)

@NancyLi1013 Are you aware that the RENAME argument is not a documented feature? If it is meant to be used even more in vcpkg, a documentation request should be made at Kitware.

Could you please update
file(COPY ${SOURCE_PATH}/COPYING DESTINATION ${CURRENT_PACKAGES_DIR}/share/libmodplug)
file(RENAME ${CURRENT_PACKAGES_DIR}/share/libmodplug/COPYING ${CURRENT_PACKAGES_DIR}/share/libmodplug/copyright)

Done.

Are you aware that the RENAME argument is not a documented feature? If it is meant to be used even more in vcpkg, a documentation request should be made at Kitware.

Yes, that would be great.

file(INSTALL ${SOURCE_PATH}/COPYING DESTINATION ${CURRENT_PACKAGES_DIR}/share/{PORT} RENAME copyright)

@NancyLi1013 Are you aware that the RENAME argument is not a documented feature? If it is meant to be used even more in vcpkg, a documentation request should be made at Kitware.

Thanks for pointing this out. @dg0yt Currently, there is no any documentation that declares how we should use RENAME in handling license. We suggest to handle like this is just to reduce codes.
You can also see this here.

LGTM, thanks again for your fixes. @mcmtroffaes

Merged master & bumped port version.


Fix destructuring of connect() parameters

Have to wrap requestLogin and requestWebView in an object.
This way the {} destructuring of connect() parameters works.
I get this error with the current version:

"openUrlAndWaitForCallback is not a function"

@pierresegonne I've reverted the yarn.lock file change and added support for Token in the Playground now.

LGTM! I'll merge it :) Good job !


point to updated spack branch for ufs-related changes

@AlexanderRichert-NOAA error on macos-gcc for ufs-weather-model build:
[+] /Users/runner/work/spack-stack/spack-stack/envs/default/install/apple-clang/13.0.0/gftl-shared-1.5.0-jhxhcwu
==> Installing parallel-netcdf-1.12.2-gclitwdvpta75oalspcm74and2ww44ry
==> No binary for parallel-netcdf-1.12.2-gclitwdvpta75oalspcm74and2ww44ry found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/3e/3ef1411875b07955f519a5b03278c31e566976357ddfc74c2493a1076e7d7c74.tar.gz
==> No patches needed for parallel-netcdf
==> parallel-netcdf: Executing phase: 'autoreconf'
==> parallel-netcdf: Executing phase: 'configure'
==> Error: ProcessError: Command exited with status 2:
    '/Users/runner/work/spack-stack/spack-stack/cache/build_stage/spack-stage-parallel-netcdf-1.12.2-gclitwdvpta75oalspcm74and2ww44ry/spack-src/configure' '--prefix=/Users/runner/work/spack-stack/spack-stack/envs/default/install/apple-clang/13.0.0/parallel-netcdf-1.12.2-gclitwd' '--with-mpi=/Users/runner/mpi' 'SEQ_CC=/Users/runner/work/spack-stack/spack-stack/spack/lib/spack/env/clang/clang' '--enable-cxx' '--enable-fortran' 'CFLAGS=-fPIC' 'CXXFLAGS=-fPIC' 'FCFLAGS=-fPIC' 'FFLAGS=-fPIC' '--enable-relax-coord-bound' '--enable-shared' '--enable-static' '--disable-silent-rules'

1 error found in build log:
     130    checking for C++ compiler vendor... clang
     131    checking base compiler command in MPICXX wrapper... not found
     132    checking for MPI_File_close... yes
     133    checking whether MPI C++ compiler redefines SEEK_SET ... no
     134    checking whether /Users/runner/mpi/bin/mpif77 exists and is executa
            ble... yes
     135    checking whether /Users/runner/mpi/bin/mpif90 exists and is executa
            ble... yes
  >> 136    /Users/runner/work/spack-stack/spack-stack/cache/build_stage/spack-
            stage-parallel-netcdf-1.12.2-gclitwdvpta75oalspcm74and2ww44ry/spack
            -src/configure: line 23095: syntax error near unexpected token `('
     137    /Users/runner/work/spack-stack/spack-stack/cache/build_stage/spack-
            stage-parallel-netcdf-1.12.2-gclitwdvpta75oalspcm74and2ww44ry/spack
            -src/configure: line 23095: `     ## test no != "$_LT_TAGVAR(hardco
            de_shlibpath_var, F77)" &&'

See build log for details:
  /Users/runner/work/spack-stack/spack-stack/cache/build_stage/spack-stage-parallel-netcdf-1.12.2-gclitwdvpta75oalspcm74and2ww44ry/spack-build-out.txt

See https://github.com/NOAA-EMC/spack-stack/actions/runs/3527458921/jobs/5916545108

@AlexanderRichert-NOAA I would like to merge this PR (and the spack PR) after https://github.com/NOAA-EMC/spack-stack/pull/408 - that PR has its spack submodule PR merged already and just requires a final test on an HPC (doing now). CI tests are ok. You'll need the CI test bug fixes from https://github.com/NOAA-EMC/spack-stack/pull/408 for your's to work.

Sounds good to me.
On Tue, Nov 29, 2022 at 1:34 PM Dom Heinzeller @.***>
wrote:

@AlexanderRichert-NOAA https://github.com/AlexanderRichert-NOAA I would
like to merge this PR (and the spack PR) after #408
https://github.com/NOAA-EMC/spack-stack/pull/408 - that PR has its
spack submodule PR merged already and just requires a final test on an HPC
(doing now). CI tests are ok. You'll need the CI test bug fixes from #408
https://github.com/NOAA-EMC/spack-stack/pull/408 for your's to work.
—
Reply to this email directly, view it on GitHub
https://github.com/NOAA-EMC/spack-stack/pull/407#issuecomment-1331344127,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ATVT32CYNJV435RJ6NCX6YTWKZZGNANCNFSM6AAAAAASIJTWHA
.
You are receiving this because you were mentioned.Message ID:
@.***>

One of the CI tests seem to have failed for no obvious reason: https://github.com/NOAA-EMC/spack-stack/actions/runs/3579347796/jobs/6023518324 - I am rerunning the test, and if successful I'll run the other canceled jobs manually, too.

@AlexanderRichert-NOAA Something is wrong with this PR, I don't see the submodule pointer update for your spack PR. I see only the .gitmodules change.

The issue with the CI tests is that spack now builds two versions of netcdf (shared) that only differ in their parallel-netcdf variant:
spec: netcdf-c<EMAIL_ADDRESS>build_system=autotools patches=de556da arch=darwin-bigsur-ivybridge
spec: netcdf-c<EMAIL_ADDRESS>build_system=autotools patches=de556da arch=darwin-bigsur-ivybridge

I am seeing the same problem on my macOS with apple-clang, and on Cheyenne with Intel. Each time when building from the skylab-dev template.

I am seeing the same problem on my macOS with apple-clang, and on Cheyenne with Intel. Each time when building from the skylab-dev template.

Hmm yep, that makes sense, I should have seen this coming. I think the play is to just take base-env out of ufs-weather-model-env and just have all the (non-python) UFS-related packages in one place, that way no conflicts over particular needs about netcdf variants etc. I'll update this shortly.

I am seeing the same problem on my macOS with apple-clang, and on Cheyenne with Intel. Each time when building from the skylab-dev template.

Hmm yep, that makes sense, I should have seen this coming. I think the play is to just take base-env out of ufs-weather-model-env and just have all the (non-python) UFS-related packages in one place, that way no conflicts over particular needs about netcdf variants etc. I'll update this shortly.

I think I have a better solution in place, I messaged you in slack but I can also post it here. We can tag up quickly in google meet to discuss this if you like.

I think we need to remove almost all the variant specifiers in base-env, except the +/~shared logic. I’ll try that and send you a PR.

Please have a look at https://github.com/NOAA-EMC/spack/pull/203/commits/c33b8ece8973a5750dd0cdd0af9f138051340424 in conjunction with https://github.com/NOAA-EMC/spack-stack/pull/416/commits/73a71f805700cf8c6225a7ba48320993143bb073. I think this is all you need on top of your PR.

10:33
I am testing this now. If that works, how about the following suggestion: We review each other’s PRs and have other people review them individually. but then we only focus on merging https://github.com/NOAA-EMC/spack/pull/203 and https://github.com/NOAA-EMC/spack-stack/pull/416 . This will automatically mark all other PRs that I have pulled in as merged. We’ll only need to test (CI + manual testing) https://github.com/NOAA-EMC/spack-stack/pull/416 and make sure it does what it should.

@AlexanderRichert-NOAA @mathomp4 I am happy with this PR and the associated spack PR. Please open this PR up for review, and if @mathomp4 could take another look at https://github.com/NOAA-EMC/spack/pull/192, please?

@AlexanderRichert-NOAA I merged the spack PR. Please point your spack submodule pointer in this spack-stack PR back to the head of noaa-emc jcsda_emc_spack_stack (correct hash is 377817d) and revert the .gitmodules changes. No need to run the CI tests again.


Add more contextual info in buffer errors

For example if ser/deser fails we only get this: panicked at 'assertion failed: self.remaining() >= dst.len()'. Would be helpful if the serialisation id could be given in the error message.

@adamhass

Okay I think that some of these assertions have to be made in the low level buffer code/the BufMut interface which doesn't return errors and doesn't have access to the serialisation ID's. The panic (should be) caused by trying to send a NetMessage that doesn't fit inside a BufferChunk at all.
WIth the next PR I'm doing (adding Buffer-Chaining) I'll add some assertions in the serialisation method to try and add assertions in the serialisation method such that it would panic before trying to put it into the buffer, if the message is too big.

I seem to get this when receiving a message. And it shouldn't be due to too large message, the message is just 8 bytes.
14:     0x7f89165de581 - <kompact_benchmarks::bench::atomic_broadcast::client::Client as kompact::actors::Actor>::receive_network::hea79e8942f89f761


Wrapdb relaunching

Now that wrapdb is back up (thanks for all who fixed it), maybe we should do some work on relaunching it and build back some trust. Maybe something like:

write a blog post/mailing list post about how the crash went down and how it was eventually fixed
explain about the new site reliability features such as mirroring, maybe ask for companies to donate us some virtual servers (or whatever it is the system requires) to get extra mirrors
organize a "wrap day" to properly launch the new site

The concept of "wrap day" is just to have one day when people come together, create new wrap files, submit them to the db, review them etc. We would be there to help newbies with their questions and so on. Hopefully this would provide us with more people who want to help with both creating and rewiewing wraps.

Why is a crash or hosting a thing?
Isn't the vcpkg model 'quite good' where the index is a repo that can be git pulled, and the search is then a local operation etc?  Then all this site reliability stuff is github's problem.

@jpakkane still working on actually making the proxy robust, will take a bit more time.
It's basically an untested code which I started on Tim's machine. I don't think it's ready to be deployed by people outside of the core meson team. Things to do here: remove standalone mode, add test coverage for the proxy mode, add configuration for uwsgi server and create package for at least one distro, possibly docker as well.
@jmansion the idea is to avoid lock-in if we have to migrate away from github.


Browservice on termux

I tested on proot-distro but doesn't seem to work so anyone has way to compile it on termux

For Android? I don't think you can do it, you had to use Windows, Linux or macOS for this one, since there's no packages for the dependencies that browservice needs for Android.

In what way did it not work? I think that I have also tried to compile and run it in PRoot (running it on Termux without PRoot is probably much more difficult); I may remember wrong but I think it ran correctly if I set the flag --chromium-args=no-sandbox for Browservice to disable the Chromium sandbox. However, using Chromium without the sandbox is a big security risk; if you disable the sandbox, you should only browse sites that you absolutely trust. Getting the sandbox (which relies heavily on kernel-level features typically disabled on Android) to work on Termux seems unlikely, so I don't see a feasible way of getting Browservice to work for general-purpose browsing on Termux.


Download IEDBD and process it into an AnnData object

I added a function to download data from the IEDB and process it into an AnnData object. This can be used as an additional epitope database for the query function.

[ ] mention in the tutorial that IEDB exists as alternative to vdjdb

Hi Markus,
thanks a lot for this PR and for your message!
The current problem is indeed just formatting. You can fix it with the black formatter. There's also some instructions in the scirpy developer docs.
Let me know if you are stuck.
Best,
Gregor

btw, don't worry about that failing conda test. I'm not sure yet what's the problem, but it is not your fault.

Hi Gregor,
thanks for your suggestions, I updated my request accordingly.
Regarding your two questions, I will have a closer look and then come back to you.
Best,
Markus

Ah, the formatting again, will fix it!

Good morning Gregor,
regarding your questions:
Around 2.6 % of all entries are BCRs, I do not filter them out when processing the data. When performing the actual query, I would assume that we do not need to worry that TCR CDR3 sequences from the query wrongly match with the BCR ones in the dataset. Within the dataset, none of the TCR and BCR CDR3 sequences match.
8 % of entries have a receptor ID that is present more than once, however most of these receptors still differ in other aspects. From these 8%, around 25% have the same chain1 CDR3, chain2 CDR3, antigen organism, antigen and cell type, we could think about removing these, because the rest would then have the same receptor ID, but would still differ in one of these essential aspects.
From all the entries, around 1% are completely duplicated, I can remove them right at the beginning.

Ok, getting there. I found another issue, but already fixed it myself: IEDB does only specify "light" chains for BCRs, but scirpy needs to know if they are IGL (lambda) or IGK (kappa) loci. To fix this, I re-used a function to infer the locus from the gene names that I already used in other IO modules.
I also renamed the tcr_table to iedb_df, because it does not only contain TCRs.
Here's the chain_qc report for iedb now. Does this look reasonable?

pairing
receptor type

Great! Thank you for your help!

We are live!
pip install scirpy==0.12


Implemtation Question

Question
Dear All,
first thanks for providing this free plugin for SW6 - i am about to implement CDN bunny CDN, but evenif i configured and tested the API, the upload to bunny does not get triggered. Could you help me with this simply question. Thanks Stefan

Thanks - yes i was referring to existing and possibly new images.  It looks like i may need professional help to get this running. In case you offer  (paid) CDN implementation services  as well, let me know. Thanks

Sadly, not possible while I don't have any business :-(


docs: Add link to Gitter room

Let's go with Gitter for now. It works and is quite popular among open-source projects.


args to custom run command not working correctly on windows (spaces?)

Version of VSCode: 1.11.2
Version of the extension: 0.3.14
OS: Windows 10
Description:
I have a custom run configuration like this:
    "rust.customRunConfigurations": [
        {
            "title": "PTRPI campaign",
            "args": ["--", "C:\\Users\\radix\\Google Drive\\RPG\\FolioRPG\\PTCampaigns", "eberron.yaml"]
        },
        {
            "title": "release",
            "args": [ "--release" ]
        }
    ],

Since the latest update, I can't use it any more. It seems that the command is now being run in PowerShell, and my guess is that it's not escaping spaces correctly (e.g. in the "Google Drive" path).
Windows PowerShell
Copyright (C) 2016 Microsoft Corporation. All rights reserved.

Loading personal and system profiles took 545ms.
C:\Users\radix\Projects\pandt [master ≡ +0 ~2 -0 !]> $ENV:RUST_BACKTRACE="1"
C:\Users\radix\Projects\pandt [master ≡ +0 ~2 -0 !]> $ENV:CARGO_INCREMENTAL="1"
>>
C:\Users\radix\Projects\pandt [master ≡ +0 ~2 -0 !]> cd "c:\Users\radix\Projects\pandt\ptrpi"
>>
C:\Users\radix\Projects\pandt\ptrpi [master ≡ +0 ~2 -0 !]> cargo run -- C:\Users\radix\Google Drive\RPG\FolioRPG\PTCampaigns eberron.yaml
>>
    Finished dev [unoptimized + debuginfo] target(s) in 0.1 secs
     Running `C:\Users\radix\Projects\pandt\target\debug\ptrpi.exe C:\Users\radix\Google Drive\RPG\FolioRPG\PTCampaigns eberron.yaml`
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { repr: Os { code: 3, message: "The system cannot find the path specified." } }',
 src\libcore\result.rs:859
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: core::result::Result<std::fs::File, std::io::error::Error>::unwrap<std::fs::File,std::io::error::Error>
             at C:\projects\rust\src\libcore\result.rs:737
   1: ptrpi::load_app_from_path
             at .\src\main.rs:194
   2: ptrpi::main
             at .\src\main.rs:213
error: process didn't exit successfully: `C:\Users\radix\Projects\pandt\target\debug\ptrpi.exe C:\Users\radix\Google Drive\RPG\FolioRPG\PTCampaigns eberron.yam
l` (exit code: 101)
C:\Users\radix\Projects\pandt\ptrpi [master ≡ +0 ~2 -0 !]>
>>

Output of the "Rust logging" channel:
DEBUG: Configuration: Rustup: create: sysroot=C:\Users\radix\.rustup\toolchains\nightly-x86_64-pc-windows-msvc
DEBUG: Configuration: Rustup: updateComponents: this.components=["cargo-x86_64-pc-windows-msvc (default)","rls-x86_64-pc-windows-msvc (installed)","rust-analysis-x86_64-pc-windows-msvc (installed)","rust-docs-x86_64-pc-windows-msvc (default)","rust-src (installed)","rust-std-aarch64-apple-ios","rust-std-aarch64-linux-android","rust-std-aarch64-unknown-fuchsia","rust-std-aarch64-unknown-linux-gnu","rust-std-arm-linux-androideabi","rust-std-arm-unknown-linux-gnueabi","rust-std-arm-unknown-linux-gnueabihf","rust-std-arm-unknown-linux-musleabi","rust-std-arm-unknown-linux-musleabihf","rust-std-armv7-apple-ios","rust-std-armv7-linux-androideabi","rust-std-armv7-unknown-linux-gnueabihf","rust-std-armv7-unknown-linux-musleabihf","rust-std-armv7s-apple-ios","rust-std-asmjs-unknown-emscripten","rust-std-i386-apple-ios","rust-std-i586-pc-windows-msvc","rust-std-i586-unknown-linux-gnu","rust-std-i686-apple-darwin","rust-std-i686-linux-android","rust-std-i686-pc-windows-gnu","rust-std-i686-pc-windows-msvc","rust-std-i686-unknown-freebsd","rust-std-i686-unknown-linux-gnu","rust-std-i686-unknown-linux-musl","rust-std-mips-unknown-linux-gnu","rust-std-mips-unknown-linux-musl","rust-std-mips64-unknown-linux-gnuabi64","rust-std-mips64el-unknown-linux-gnuabi64","rust-std-mipsel-unknown-linux-gnu","rust-std-mipsel-unknown-linux-musl","rust-std-powerpc-unknown-linux-gnu","rust-std-powerpc64-unknown-linux-gnu","rust-std-powerpc64le-unknown-linux-gnu","rust-std-s390x-unknown-linux-gnu","rust-std-sparc64-unknown-linux-gnu","rust-std-wasm32-unknown-emscripten","rust-std-x86_64-apple-darwin","rust-std-x86_64-apple-ios","rust-std-x86_64-pc-windows-gnu","rust-std-x86_64-pc-windows-msvc (default)","rust-std-x86_64-rumprun-netbsd","rust-std-x86_64-unknown-freebsd","rust-std-x86_64-unknown-fuchsia","rust-std-x86_64-unknown-linux-gnu","rust-std-x86_64-unknown-linux-musl","rust-std-x86_64-unknown-netbsd","rustc-x86_64-pc-windows-msvc (default)",""]
DEBUG: Configuration: Rustup: updatePathToRlsExecutable: rlsInstalled=true
DEBUG: Configuration: Rustup: updatePathToRlsExecutable: rlsPath=C:\Users\radix\.cargo\bin\rls.EXE
DEBUG: runInRlsMode: rlsPath=C:\Users\radix\.cargo\bin\rls.EXE
DEBUG: runInRlsMode: env={"RUST_SRC_PATH":"C:/Users/radix/.multirust/toolchains/stable-x86_64-pc-windows-msvc/lib/rustlib/src/rust/src/"}
DEBUG: runInRlsMode: args=[]
DEBUG: runInRlsMode: revealOutputChannelOn=3
DEBUG: Language Client Manager: start

It may be related to #218
I actually don't know.
@BusyJay, can you answer if the problems are similar?

as a workaround I was able to get it working by changing my configuration to include quotes around the path:
            "args": ["--", "\"C:\\Users\\radix\\Google Drive\\RPG\\FolioRPG\\PTCampaigns\"", "eberron.yaml"]

so maybe vscode-rust should quote all arguments automatically? I don't know if this would cause problems elsewhere though.

@radix,
@BusyJay pointed to NodeJS issue.
It's not my issue and I am not going to write any workaround.

I don't really know the details that well, but it doesn't look like exactly the same issue? it seems the nodeJS bug is about when the command has a space, but I don't think the command here has a space.

I suppose NodeJS behave the same disregarding whether it is an executable path or an argument

I do actually think it is easy to avoid, so I am going to open a PR to fix this.

I was wrong. It is not easy)


[Docs] Missing users module

Fixed in https://github.com/scottwestover/Unofficial-LiveEngage-API-Wrapper/commit/b31bbffb9a9d3bf5045699a5c45014a94f7dd92b


Dask compatibility

This PR introduces a full dask compatible version. It builds on branch v0.2, but has further improvements on performance and structure.
closes #61
solves #48
User effective changes

Choosing a module (e.g. module='sarah'), will only retrieve variables which are provided by the module.
Choosing more than one module is supported now (e.g. module=['sarah', 'era5'])
layout in conversion functions must be a xr.DataArray
atlite.Cutout takes the argument chunks, which serves as a default for chunking
atlite now default to chunking when loading or preparing a cutout
all conversion functions are automatically using dask
class Window is removed due to replacement by dask

Main changes under the hook

datasets module only handle features that the underlying data provides. So, sarah.py is strongly reduced to sarah data handling functionality only.
the get_data  in all modules returns data in a standard format, always xr.Dataset with chunkes dask.arrays. No delayed dask objects. Using xr.Dataset with dask.arrays performs better and allows to directly apply standard xarray functions on the dataset "skeleton" (as setting attributes, renaming, etc).
loading features are coordinated by the data.py module, it looks which of the modules in Cuotut.module provides the requested feature and gets it from there. If more than one module provides a feature (or to be more precise: a variable, e.g. influx_diffuse) it gets it from the first  module in the module list.

Remaining issues

[x] clarify small deviations in heat_demand calculation

Thanks for reviewing so fast! I will comment where there are question or things are unclear, otherwise I'll commit the suggestions or add them remotely.

Alright, I'm done so far. Three things small remaining points on the discussion list:

rename argument module into modules?
non-exploded Series (so Series of lists) for available_features and prepared_features?
gc.collect in prepare function due to possible file handling errors?

I'd say we can unworried discuss those things later after merging (as I'm already spending too much time with this, i'd be happy to wrap up).


Feature Request: Improved view manipulation

When viewing the build area, and with 'Rotate' selected, it is almost possible to do all of the things one needs to do without having to toggle modes to 'move camera' or 'move viewpoint'.
The middle button wheel scroll  is mapped to zoom in/out: good
left drag is mapped to rotate: good
there are two mappings left: the right mouse drag, and middle drag. if either one of these remaining ones was mapped in 'rotate' mode to 'pan-- ie, move viewpoint', then most all things you need would be available in the main choice.
I would recommend that holding down the scroll wheel while dragging is pan, while holding down left mouse is rotate.
This is not a huge deal, but i think it would make manipulating views much more convienient.

No need to change mode. Use the Keys for this:
"alt" key : Move selected object.
"shift" key : Move view point.
"strg" key : Rotate.
"+" / "-" : Zoom in/out (focus must be in the 3D window).
Mouse scroll wheel : Zoom in/out.
Using the buttons for these sounds like a good idea at least if you have them. Will test how it works.


[TEST] Resurrect IT tests that were missing resources and not being

executed.

:white_check_mark: Build finished in 10m 45s
Build command:
mvn clean verify -B -e -Daudit -amd -pl plugins/pur/core

:ok_hand: All tests passed!
Tests run: 280, Failures: 0, Skipped: 0    Test Results

:information_source:  This is an automatic message

:white_check_mark: Build finished in 7m 2s
Build command:
mvn clean verify -B -e -Daudit -amd -pl plugins/pur/core

:ok_hand: All tests passed!
Tests run: 280, Failures: 0, Skipped: 0    Test Results

:information_source:  This is an automatic message


How to configure options such as hover and data lebel?

Bug report
Codepen
I have 3 issue Please refer to this codepen:

I try to disable mouse-hover behaviour for donut chart by using config, following this [#issue 322] but its not working. Hover still working whenever mouse on top of someting(https://github.com/apexcharts/apexcharts.js/issues/322)
How to make label at centre to fit in white space only? not overlapping with the graph? Is it possible to put datalabel on the right of its segment?
How to limit data label percentage to no decimal?

use
tooltip : { enabled:false }


More fields

@f3r and @ggharry
user can now add checkboxes, radio and description
creates a preview page

There is a bug I did not manage o fix...
when you modify a content, the input field should be already filled by the informations already saved before. It works up to the point where you save a new 'label' for a different content, it then show the last information saved instead of the one contained in the object...


XPC-20740:GetActive() seg faults in very very corner case once in a while

When a cache.Refresh() followed by GetActive() is done on a deleted object, it occasionally results in a segfault like below. The root cause is the entry is not validated before getInvalidated() is performed on it
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x15ce765]
....
....
/usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/Comcast/goburrow-cache.(*entry).getInvalidated(...)
/Users/soggu464/go/pkg/mod/github.com/!comcast/goburrow-cache@v1.0.3/policy.go:90
github.com/Comcast/goburrow-cache.(*localCache).GetActive(0xc000110780,
/Users/soggu464/go/pkg/mod/github.com/!comcast/goburrow-cache@v1.0.3/local.go:198 +0x65…
$ go build -v
$ go test -v -bench . -race
=== RUN TestBloomFilter
--- PASS: TestBloomFilter (0.04s)
=== RUN TestSum
--- PASS: TestSum (0.00s)
=== RUN TestCache
--- PASS: TestCache (0.00s)
=== RUN TestMaximumSize
--- PASS: TestMaximumSize (0.22s)
=== RUN TestRemovalListener
--- PASS: TestRemovalListener (0.00s)
=== RUN TestClose
--- PASS: TestClose (0.00s)
=== RUN TestLoadingCache
--- PASS: TestLoadingCache (0.00s)
=== RUN TestCacheStats
--- PASS: TestCacheStats (0.00s)
=== RUN TestExpireAfterAccess
--- PASS: TestExpireAfterAccess (0.00s)
=== RUN TestExpireAfterWrite
--- PASS: TestExpireAfterWrite (0.00s)
=== RUN TestRefreshAterWrite
--- PASS: TestRefreshAterWrite (0.00s)
=== RUN TestGetIfPresentExpired
--- PASS: TestGetIfPresentExpired (0.00s)
=== RUN TestLoadingAsyncReload
--- PASS: TestLoadingAsyncReload (0.00s)
=== RUN TestLoadingRefresh
--- PASS: TestLoadingRefresh (0.00s)
=== RUN TestCloseMultiple
--- PASS: TestCloseMultiple (0.00s)
=== RUN TestLRU
--- PASS: TestLRU (0.00s)
=== RUN TestLRUWalk
--- PASS: TestLRUWalk (0.00s)
=== RUN TestSegmentedLRU
--- PASS: TestSegmentedLRU (0.00s)
=== RUN TestSLRUWalk
--- PASS: TestSLRUWalk (0.00s)
=== RUN TestCountMinSketch
--- PASS: TestCountMinSketch (0.00s)
=== RUN TestStatsCounter
--- PASS: TestStatsCounter (0.00s)
=== RUN TestTinyLFU
--- PASS: TestTinyLFU (0.00s)
goos: darwin
goarch: amd64
pkg: github.com/Comcast/goburrow-cache
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkSame
benchmark_test.go:95: total: 10000 (12.778221ms), hits: 9999 (99.99%), misses: 1 (0.01%), evictions: 0
benchmark_test.go:95: total: 933829 (1.321641988s), hits: 933826 (100.00%), misses: 3 (0.00%), evictions: 0
BenchmarkSame-12<PHONE_NUMBER> ns/op 0 B/op 0 allocs/op
BenchmarkUniform
benchmark_test.go:95: total: 10000 (53.745031ms), hits: 4713 (47.13%), misses: 5287 (52.87%), evictions: 4423
benchmark_test.go:95: total: 218217 (815.208703ms), hits: 109108 (50.00%), misses: 109109 (50.00%), evictions: 101458
benchmark_test.go:95: total: 320929 (1.340897932s), hits: 160362 (49.97%), misses: 160567 (50.03%), evictions: 149627
BenchmarkUniform-12<PHONE_NUMBER> ns/op 139 B/op 3 allocs/op
BenchmarkUniformLess
benchmark_test.go:95: total: 10000 (17.117974ms), hits: 9482 (94.82%), misses: 518 (5.18%), evictions: 0
benchmark_test.go:95: total: 674713 (1.330440863s), hits: 674200 (99.92%), misses: 513 (0.08%), evictions: 0
BenchmarkUniformLess-12<PHONE_NUMBER> ns/op 27 B/op 0 allocs/op
BenchmarkCounter
benchmark_test.go:95: total: 10000 (73.527256ms), hits: 0 (0.00%), misses: 10000 (100.00%), evictions: 9428
benchmark_test.go:95: total: 161356 (1.249213548s), hits: 0 (0.00%), misses: 161356 (100.00%), evictions: 160789
BenchmarkCounter-12<PHONE_NUMBER> ns/op 345 B/op 8 allocs/op
BenchmarkExponential
benchmark_test.go:95: total: 10000 (18.385768ms), hits: 9990 (99.90%), misses: 10 (0.10%), evictions: 0
benchmark_test.go:95: total: 649167 (1.022836388s), hits: 649153 (100.00%), misses: 14 (0.00%), evictions: 0
BenchmarkExponential-12<PHONE_NUMBER> ns/op 0 B/op 0 allocs/op
BenchmarkZipf
benchmark_test.go:95: total: 10000 (46.818616ms), hits: 6693 (66.93%), misses: 3307 (33.07%), evictions: 2759
benchmark_test.go:95: total: 249249 (1.188586142s), hits: 180622 (72.47%), misses: 68627 (27.53%), evictions: 67536
BenchmarkZipf-12<PHONE_NUMBER> ns/op 100 B/op 2 allocs/op
BenchmarkHotspot
benchmark_test.go:95: total: 10000 (40.920259ms), hits: 7987 (79.87%), misses: 2013 (20.13%), evictions: 1463
benchmark_test.go:95: total: 288040 (958.924426ms), hits: 239559 (83.17%), misses: 48481 (16.83%), evictions: 46996
benchmark_test.go:95: total: 360103 (1.197184314s), hits: 299741 (83.24%), misses: 60362 (16.76%), evictions: 58592
BenchmarkHotspot-12<PHONE_NUMBER> ns/op 53 B/op 1 allocs/op
BenchmarkSumInt
BenchmarkSumInt-12 100000000 11.46 ns/op 0 B/op 0 allocs/op
BenchmarkSumString
BenchmarkSumString-12 49283776 26.69 ns/op 0 B/op 0 allocs/op
BenchmarkGetSame
BenchmarkGetSame-12<PHONE_NUMBER> ns/op 0 B/op 0 allocs/op
BenchmarkCacheSegment
BenchmarkCacheSegment-12 3256952 362.1 ns/op 1 B/op 0 allocs/op
BenchmarkCountMinSketchReset
BenchmarkCountMinSketchReset-12 29988 39724 ns/op 0 B/op 0 allocs/op
PASS
ok github.com/Comcast/goburrow-cache 20.295s
goburrow-cache work_sree goburrow-cache)
$

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


⚠ Bofrost.lu has degraded performance

In 06b1275, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1588 ms

Resolved: Bofrost.lu performance has improved in a22e978.


Fixed Issue - #1534

Pull Request Type
What kind of change does this PR introduce?
[ X ] Solved an issue/task
[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Documentation content changes
[ ] Other... Please describe:
What is the current behavior?

Issue Number: #1534
What is the new behavior?
Created a contact form in index.html
Other information

Merged

This is an automated message from Fork, Commit, Merge [BOT].

Thank you for your contribution! Your pull request has been merged. The files have been reset for the next contributor.
What's next?
If you're looking for more ways to contribute, I invite you to check out our forkcommitmerge.io website repository. The repository contain real issues that you can help to resolve or create some of your own if you see any bugs or want to add new features to the site, etc. You can also check out the Influences section in the README to find more projects similar to this one.
Also please leave a star to this project if you feel it helped you, i would really appreciate it.
I look forward to seeing your contributions!


Running selective unit tests documentation

Fixes #1489
@mairaw @BillWagner This seems like a good approach:

It doesn't clutter up the options on the dotnet test command with more --filter info. It places the option details in a section at the bottom of the topic out of the way titled Filter option details. It keeps the option details in that new section focused on the option details and doesn't include the full-blown examples.
It creates a doc over in the testing area to hold the remaining content with the lengthy examples. I took the original content (giving the original author credit) and reorganized the content to my liking.
I link to the testing area doc from the dotnet command --filter option and from the Filter option details section.

What da ya tinks?
If the structure of all of this looks fine, then my only remaining question is whether or not the testing area doc should be more of a walkthrough tutorial or should it be left as it is ... as more of a reference doc. I think it's ok as a ref doc. It's fairly straightforward the way Satya wrote it with some SillyRex:tm: changes.
btw- Rendering the table cell content that contains pipes as <code> HTML markup. I can't escape a pipe in a table cell.

@GuardRex,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by .NET Foundation. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot

One of my intentions on adding this doc here is that we'd replace the aka.ms link that is used on the CLI tools to go here instead of the CLI repo (before it was to a blog post). So that's why I was planning to have the topic both covering the conceptual part as well as the examples.
@piotrpMSFT @livarcocc any thoughts on this?

@rpetrusha Updates complete. I found a few other little things, too. I did some additional 🌻table beautification 🌻; so in addition to checking the %7C table rendering, we'll need to look at how well these other tables render before we merge.

For this one, if we want to replace the link to https://aka.ms/vstest-filtering with the work being done in this PR, would we send them to the new topic or the dotnet test page?

@mairaw Does it depend on the context of the link?
Command-related :point_right: docs/core/tools/dotnet-test.md
Unit testing-related :point_right: docs/core/testing/index.md
... or perhaps both if both are relevant?

see the output of dotnet test --help @GuardRex

@mairaw ah ... I thought you meant there was more than one. Shouldn't that go to the docs/core/testing/selective-unit-tests.md because it pertains to the filtering expression that would be used with the --filter option?

@rpetrusha I stripped out the other <code> blocks in favor of markdown inline code with | pipes. If that doesn't work on this pass, then I'll add back the <code> blocks to see if that will fix it.

@rpetrusha Looks like that got it. https://review.docs.microsoft.com/en-us/dotnet/articles/core/testing/selective-unit-tests?branch=pr-en-us-1776

@GuardRex, yes, this looks good. It's ready to merge.


Impossible to build on systems with case insensitive file systems (macOS)

The clash between STAT.hs and Stat.hs is a problem for macOS.

I've renamed STAT to solve this. Can you see if builds for you now?

Yes, that resolves the issue!


Modifying cerberus specs to latest upstream

This PR is to incorporate changes from the merge of below PR upstream:
https://github.com/cloud-bulldozer/cerberus/pull/119 - Adding custom-check failures to slack notification

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: Rajalakshmi-Girish
To complete the pull request process, please assign mkumatag after the PR has been reviewed.
You can assign the PR to them by writing /assign @mkumatag in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/cc @mkumatag

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: mkumatag, Rajalakshmi-Girish
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [mkumatag]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@Rajalakshmi-Girish Please deploy these specs into IKS cluster and see how it goes.


fix pendingComponent not shown when pendingMs is 0

pendingConponent is not shown when pendingMs is 0.
Is that a bug or intended behavior?
I am using Number.EPSILON in the meantime.

Can you please provide a reproducer on e.g. codesandbox?


Publish updated package to PyPI

Tasks

[x] Bump version up to 0.5.0
[x] Publish as jsontableschema
[ ] Update the packages for jtskit and json-table-schema o point to the new package going forward.

Not exactly sure how this works, but we either a) get an actual redirect (unlikely) or b) publish an update to those packages that tells the user where the new package is.

For the outdated packages, perhaps we could do something like this https://pypi.python.org/pypi/pypi-alias.

ok.

A new version of json-table-schema has been uploaded PyPi to 'redirect' to jsontableschema. I'll do the same with jtskit once the PR for goodtables to upgrade jtskit to jsontableschema has been merged (https://github.com/okfn/goodtables/pull/62).

@brew thanks i merged GT now.


boards: update nrf51_ble400 configurations

nrf51_ble400 core board upgrades chip to nRF51822_QFAC
Signed-off-by: Freddie Yang<EMAIL_ADDRESS>

@carlescufi As requested in issue #47272

@anangl I'd like you to take a look at this one

@carlescufi Could you kindly remind me if new PR is what you asked?

No, this is fine now.


Level Design: obstacles generating variety

Create 3-15 unique obstacle terrain blocks

Not ASAP task, closed


add: release notes for v0.33.1 and v0.32.2

⚠ IMPORTANT ⚠: This is a public repository. Make sure to not disclose:

[x] personal data beyond what is necessary for interacting with this Pull Request;
[x] business confidential information, such as customer names.

Unsure if we want the v0.32.2 release notes above the ones for v0.33.0.
Thoughts @aarnq?

I kept the chronological order, but I can move it if that looks better

I think the chronological order is nicer


Why doest it auto load tags after I update fields (which have nothing to do with tags)?

I'm using Rails 5.0.0 with Ruby 2.3.0.
I noticed a strange thing: when I update a record, its tags will always automatically reload.
Let's say... I have a Video model and an instance of it, called video.
class Video < ApplicationRecord
  acts_as_commentable
end

Here are what I tested:
video = Video.select(:id, :name, :shares_count).find(2)

Video Load (0.8ms)  SELECT  `videos`.`id`, `videos`.`name`, `videos`.`shares_count` FROM `videos` WHERE `videos`.`id` = 2 LIMIT 1

video.update_attribute(:shares_count, 16)

(0.4ms)  BEGIN
SQL (0.8ms)  UPDATE `videos` SET `shares_count` = 16, `updated_at` = '2017-03-31 08:01:44' WHERE `videos`.`id` = 2
ActsAsTaggableOn::Tagging Load (0.7ms)  SELECT `taggings`.* FROM `taggings` WHERE `taggings`.`taggable_id` = 2 AND `taggings`.`taggable_type` = 'Video'
(12.7ms)  COMMIT

Why does it reload tags? It also happens when I update other fields, which have nothing to do with tags.
And I don't add any callbacks associating with tags.
The amazing thing is that, when I use increment! (which may not use transaction), it doesn't query tags:
video.increment! :shares_count

SQL (13.8ms)  UPDATE `videos` SET `shares_count` = COALESCE(`shares_count`, 0) + 1 WHERE `videos`.`id` = 2

I happens to me too. Did you find an explanation?

This is happening to me on models that aren't even using tags and aren't directly related to models that have tags. It's killing performance in production. What's happening here?

I just noticed today and it is affecting our performance in production also :(


⚠ Cioos National CKAN has degraded performance

In fbe316a, Cioos National CKAN (https://catalogue.cioos.ca/) experienced degraded performance:

HTTP code: 200
Response time: 11297 ms

Resolved: Cioos National CKAN performance has improved in d2126de after 4 minutes.


Run benchmarks just on roaring

Benchmarks are run on different implementations of bitmaps. The first goal was to see how roaring performs in comparison with concise, wah and ewah. However, it might be useful now to have a way to run benchmarks only on roaring to assess performance tweaks on the code.

Done.
Passing the system property "-DBITMAP_TYPES=ROARING_ONLY" to the command line will do the job.


Regenerate and encrypt SauceLabs keys

In the .travis.yml file, your SauceLabs keys are open to the public. Regenerate and encrypt it so no-one else can use it - http://docs.travis-ci.com/user/environment-variables/
- SAUCE_USERNAME='legomushroom'
- SAUCE_ACCESS_KEY='6982f2ea-fd37-453f-9cc6-58ea67254b66'

:+1:

You are right, guys. Totally forgot about that, thanks for the alert.

Good day.
I've added new keys as env variables in repo settings.
Didn't encrypt them cuz, as I got it, encrypted vars will not be added to pull requests, which I still want to run tests for.
Will update .travis.yml file on master branch on the next merge, the key was regenerated anyways.
I'm closing the issue, if you think this is not secure enough, please reopen.
Thanks for the heads up!

Yeah, that's correct.
If you'd have used the travis gem to encrypt them in your .travis.yml file, that would have been exactly the same. But it's encrypted now which is the main thing


How to stop all the notifications

when  I launch any exe I get .... Checking for updates... Game mode ON
can you tell me how to disable this notifications ?
or at least give me the name of this app to block from my notification manager

@SychoAN
For disable "Game mode ON/OFF" disable GameMode in LW settings:

"Checking for updates... "
Does this also happen every time you run the EXE?

@SychoAN For disable "Game mode ON/OFF" disable GameMode in LW settings:
"Checking for updates... " Does this also happen every time you run the EXE?

But I don't wanna disable gamemode...Just the notification
and checking for updates happens every day and I get like 3 notifications beside game mode

@SychoAN
That particular update check cannot take place every day, because it is configured to check every three days. if you did not change the verification parameters yourself in ~/LuxWine/bin/lwrap
Anyway, here's what you can do:
Disable notify for LW container (~/LuxWine/bin/lwrun):
echo DONT_NOTIFY=1 >> ~/LuxWine/bin/Run.rcfg

Disable notify for LW (~/LuxWine/bin/lux-wine):
echo 'NO_NOTIFY=1' >> ~/LuxWine/bin/Run.rcfg

Disable gamemode config in LW container (It prescribes sending notifications about its start and stop):
echo 'BUWRAP_ARGS+=(--bind /dev/null /etc/gamemode.ini)' >> ~/LuxWine/bin/Run.rcfg

Didn't work still sends Gamemode ON...maybe I should logout and login

Didn't work still sends Gamemode ON...maybe I should logout and login

You just need to restart everything related to LW

@SychoAN
Also check that there are no lines related to sending notifications in your gamemode config in the user's home directory


Refactor of sfmath.h and movement of most back into OIIO fmath.h

There was a huge amount of duplication between sfmath.h and OIIO's
fmath.h, with the ones here having only very minor modifications. With
almost no exceptions, I was able to port the fixes here back to
fmath. This now eliminates the redundant copies on the OSL side.

Remove sfm::ifloor in favor of OIIO::ifloor
Remove sfm::clamp in favor of improved OIIO::clamp
Remove sfm::madd in favor of the improved OIIO::madd
Remove sfm::lerp, bilerp, trilerp in favor of the fixed OIIO fmath version.
Remove sfm::bit-cast in favor of the improved OIIO fmath version
Remove sfm::negate in favor of OIIO fast_neg, add Dual fst_neg
Remove sfm:log functions in favor of improved OIIO fmath.
Remove unnecessary sfm::absi
Eliminate most svm transcendentals, use improved OIIO ones instead
Rearrange safe_fmod
Remove sfm::absf which did nothing but call std::abs
Keep sfm::isinf, but only for Intel

The requisite changes should be in both OIIO master ("2.2.2-dev" and later)
and the current release branch 2.1.12 and later. It should also build against
earlier versions of OIIO, but they won't have all the performance additions
for the vectorized loops.

@AlexMWells This all builds and passes tests for me. I hope I didn't break too much on your end, the code you haven't submitted yet will probably need a bunch of changes to redirect back to the fmath versions.

Direction is good, just would be nice if there they could be conditionally exist if an older OIIO is there.

That seems like the wrong approach to me. I would rather avoid the duplication and just bump the OIIO requirement to be new enough to have the updated fmath. I've already backported it to last month's 2.1 release. In fact, I think it will build properly with even older versions, but only be fully performant in SIMD mode if it's a new enough fmath.

Is this acceptable, @AlexMWells ? The current release OIIO 2.1.12 incorporates all the changes you need, there is no need to be on oiio master if you don't want to be.

Yeah, will be fine, can work around locally if need be, there is history to refer to if needed as well.


WIP: Workspaces

Warnings

:warning:

Please add a changelog.md entry for your changes.

  Generated by :no_entry_sign: dangerJS

Agreed yeah. Let’s revisit once it’s landed - it’s awesome but probably not
terrible either.
On Thu, Mar 1, 2018 at 11:45 Yen-Wei Liu<EMAIL_ADDRESS>wrote:

@bishwei commented on this pull request.
In packages/gestalt/src/Icon/Icon.js
https://github.com/pinterest/gestalt/pull/22#discussion_r171672573:

@@ -58,6 +60,8 @@ export default function Icon(props: IconProps) {
);
}

+Icon.icons = IconNames;
not bundle size, just pointing it out since it does seem like something
that is specifically done for testing. my first thought when i looked at
this was "why is this here" and had to go look around for how it was used
so, that's why i raised it.
definitely not a concern for this pr. happy to take this as a follow up
action item for myself as well
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pinterest/gestalt/pull/22#discussion_r171672573, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAACzsyiQxt9xL-xzSkGmWUBPpbgG-Kaks5taE_HgaJpZM4SOKYL
.


Logredactor feature

Description
This PR aims to add a new cp feature - logredactor to cp-ansible for 7.2 release.
Log Redactor plugin for log4j is used to configure regular expression patterns (redaction rules) to identify and redact specific patterns of sensitive information from your logs, before they are delegated to other appenders and emitted.
README of logredactor repo for more background.
Summary of changes:

Defined common variables in variables/defaults/main.yml - logredactor_enabled, logredactor_rule_path_local, logredactor_rule_path, logredactor_rule_url, logredactor_policy_refresh_interval
Corresponding variables for specific components were defined in the <component-roles>/defaults/main.yml
All these variables were also commented with their description, which was forwarded to VARIABLES.md
Added task configure_logredactor.yml in common role which is used by all components for configuring logredactor based on the component variables. This task goes through the <component-name>_logredactor_logger_specs_list and adds the redactor appender to corresponding logger. This then defines the redactor configuration.
Added tasks in <component-name>/tasks/main.yml for all components which calls out configure_logredactor.yml and a couple of more tasks (copying rule files, restart component to apply change) for enabling logredactor.
Added test coverage of logredactor on one of existing molecule scenario archive-plain-rhel. This enables logredactor for all components based on a rule file and checks the logs for redacted word.

Fixes # (ANSIENG-1001)
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[x] This change requires a documentation update - will raise a doc ticket post review

How Has This Been Tested?
Tested locally on a couple of scenarios by using a custom rules file and checking logs of all component for redaction. Have added the test for logredactor in archive-plain-rhel scenario.
https://jenkins.confluent.io/job/cp-ansible-on-demand/210/ - archive-plain-rhel
Test Configuration:
Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation - doc on repo has been updated
[ ] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[ ] Any dependent changes have been merged and published in downstream modules
[ ] Any variable changes have been validated to be backwards compatible

As we discussed, I have opened a jira for the refactoring task - https://confluentinc.atlassian.net/browse/ANSIENG-1322?atlOrigin=eyJpIjoiMjg0YmQ5ZDRlMzg3NGQ5NWE1Mjk3NjQ5MTFlZTg4ZmIiLCJwIjoiaiJ9
We will do it in a separate PR

As we discussed, I have opened a jira for the refactoring task - https://confluentinc.atlassian.net/browse/ANSIENG-1322?atlOrigin=eyJpIjoiMjg0YmQ5ZDRlMzg3NGQ5NWE1Mjk3NjQ5MTFlZTg4ZmIiLCJwIjoiaiJ9 We will do it in a separate PR

Okey. I was thinking for only tags in log related tasks. If thats not much of work can we consider adding in this one ?

My bad, I understood the wrong thing. Have added the tags.
Ran a couple of more tests post the changes - https://jenkins.confluent.io/job/cp-ansible-on-demand/238/ , https://jenkins.confluent.io/job/cp-ansible-on-demand/239/


Former Name field

The old_name=* key is very commonly used in OSM. This issue tracks adding a Former Name field, as originally proposed in openstreetmap/iD#1554.
In theory at least, this key indicates a former name of something that is still observable, for example due to a ghost sign, or that locals still use to refer to the feature out of habit. In practice, it’s also quite common to shunt a name from name=* to old_name=* whenever the name changes, regardless of the circumstances. (I’m as guilty of history hoarding as anyone else. Despite my preference for moving these details to OpenHistoricalMap, sometimes it’s just a matter of pragmatism or triage.)
Due to the varied uses of this key, I left it out of #215, which was already bogged down by back-and-forth about more straightforward keys. If we add this field, we’d also have to be comfortable with the potential for local history buffs to enthusiastically fill in obscure historical details that will then be more difficult to move to OHM for licensing reasons.

I support.
If we have loc_name, nat_name, official_name, reg_name, short_name we should also have old_name which is used more often than these ones.

I wonder…

is this a key that should be added globally, like in in #215 or can we scope the usage a reasonable set of presets
is the OSM Wikidata item description and Wiki page clear on the usage that is described in this post? I did not research this myself, yet. Is the "observable" part a strong requirement? For "local name" it is not (right?).
what is the relation from old name to local name?

If we add this field, we’d also have to be comfortable with the potential for local history buffs to enthusiastically fill in obscure historical details that will then be more difficult to move to OHM for licensing reasons.

It would still be a "moreFields" field, so people still have to look for it and I always how the also research what they are doing in this scenario. So I am less concerned with this once the docs are good.


Fix 404 page issue on payment complete

I want to merge this change because... it backport fix to 3.0 from https://github.com/mirumee/saleor-storefront/pull/1098

Screenshots

Pull Request Checklist

[ ] All visible strings are translated with proper context.
[ ] All data-formatting is locale-aware (dates, numbers, and so on).
[ ] The changes are tested.
[ ] The code is documented (docstrings, project documentation).
[ ] Changes are mentioned in the changelog.

Test Environment Config

API_URI=https://qa.staging.saleor.cloud/graphql/

Task linked: SALEOR-4425 Stripe plugin issue


Add history of sent emails per craftsman

As I user, I want to know what emails the craftsman already received, so I can formulate an appropriate next email.
At the moment it is still unclear whether emails are sent usually to many craftsman (would favor one view for all emails) or usually only to some single craftsman (would favor a view per craftsman).

Probably does not fully answer user's need. Lets investigate further.

resolved in #612


Update connection string url in cloudsql sample

You might wish to update the readme a bit to explain what is going on here, and what they need to do, LGTM after that.

LGTM - @shun-fan Could you update the README to mention the revised string.


Merge branch 'develop'

Adding DS28EA00 lib
lwow_device_ds28ea00.zip

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Additional conditionals

I added a couple of useful helpers: isLess and isBetween.
{#ifBetween total 100 1000}
10% discount applied
{/ifBetween}
{ifLess 1000 total}
20% discount applied
{/ifLess}

Thank you for the PR!
In general it looks good for me, but I've commented some problems in your commits.

Sorry, I've implemented

ifLess
ifBetween
ifBetweenLeftClosed
ifBetweenRightClosed
ifBetweenClosed
ifMore

by myself. I'll release a new version of library soon.

Done. All the helpers available since v1.1.0


Redraw issue

I use fish shell that will display possible autocompletions. In terminal, it doesn't always draw the screen correctly. Here is a screenshot after completion where the last couple ghost characters were left over from the suggested completion. Note, this doesn't always happen. Normally everything is redrawn correctly.

Currently using 0.3.2171.0

I'd bet money that this is due to us incorrectly guessing the widths of the powerline chars. Looks like they're all being treated as double wide, but I'd imagine they're actually only 1 wide in the buffer.
Could you repro that scenario with the same font set in WSL via conhost.exe (not the Windows Terminal), and paste what it looks like?

I can not reproduce via conhost.exe nor via the terminal in VSCode. It's purely a Windows Terminal issue.

@zadjii-msft playing with this a bit more in windows terminal, it only appears to happen when the powerline has "git status" icons in it. The font that is used for that, doesn't display those symbols in conhost.

Could I get a screenshot with the up and down arrows in conhost? they look like maybe the one we're guessing is 2 wide instead of 1 wide:
These guys:

And here they are in VSCode...

Yea that'll do it.
This is probably some combination of issues, #2066 might be the one that ends up fixing it.
@miniksa might also have thoughts on the character being rendered as 2 wide

@zadjii-msft Let me know if you need anything else from me.

On mine it is not rendering poweline characters correctly.
VSCode:

Windows Terminal:

@zadjii-msft I have some rare case. Characters have some strange random gaps (they are not always on the same spot) but it is like pixel or two wide not whole character.

This is almost certainly #633 #635, and will be helped by #2066.


Frontend sentry not reporting errors causing 500s

Description

The production frontend service has a regular but a very small number of 500s, less than 30 per day and often enough, none at all in a given day, typically. However, whatever errors cause these 500s are never reported in Sentry.
Additional context

As you can see, we do have 500s, yet these never appear in Sentry.
It may be helpful to look through Nuxt logs around the time that 500 happens to see if anything reveals itself as far as reproduction or why Sentry might not be receiving these errors.
cc @WordPress/openverse-frontend I've marked this high because it could mean that Sentry is not working, which means our main method of observability for frontend stability isn't reliable right now.

It's not much, but these were the only errors I was able to find after the latest deployment. For some reason they didn't have timestamps which I can't seem to figure out:
WARN  There was an error fetching media providers for audio: read ECONNRESET
 ERROR  Range Not Satisfiable
  at createHttpError (/home/node/node_modules/.pnpm/send@0.18.0/node_modules/send/index.js:984:7)
  at SendStream.error (/home/node/node_modules/.pnpm/send@0.18.0/node_modules/send/index.js:270:31)
  at SendStream.send (/home/node/node_modules/.pnpm/send@0.18.0/node_modules/send/index.js:666:19)
  at onstat (/home/node/node_modules/.pnpm/send@0.18.0/node_modules/send/index.js:725:10)
  at FSReqCallback.oncomplete (node:fs:203:5)
  at FSReqCallback.callbackTrampoline (node:internal/async_hooks:130:17)

I also confirmed Sentry is working locally by enabling it, hardcoding the DSN, and pushing some events with $sentry.captureException(...).

Here's two examples, one SSR one client-side. These two were triggered with throw new Error() to make sure that works too:

https://openverse.sentry.io/share/issue/d30dc852709e422c87ef6dffba9862f7/
https://openverse.sentry.io/share/issue/d30dc852709e422c87ef6dffba9862f7/

In production, I also see that the frontend tasks log ✔ Sentry reporting is enabled (client side: enabled, server side: enabled) and have the correct environment variables in their task definitions.
Ah! But I just noticed, in prodution for some reason Sentry appears to actually be disabled, despite all of those things 🤔 window.$nuxt.$sentry.captureException is a noop:
const apiMethods = ["init", "vueRouterInstrumentation", "attachErrorHandler", "createTracingMixins", "FunctionToString", "Hub", "InboundFilters", "Scope", "addBreadcrumb", "addGlobalEventProcessor", "captureEvent", "captureException", "captureMessage", "configureScope", "createTransport", "getCurrentHub", "getHubFromCarrier", "makeMain", "setContext", "setExtra", "setExtras", "setTag", "setTags", "setUser", "startTransaction", "withScope", "BrowserClient", "makeFetchTransport", "makeXHRTransport", "defaultStackParser", "eventFromException", "eventFromMessage", "close", "flush", "forceLoad", "lastEventId", "onLoad", "showReportDialog", "wrap", "Replay", "makeBrowserOfflineTransport", "onProfilingStartRouteTransaction", "BrowserProfilingIntegration", "GlobalHandlers", "TryCatch", "Breadcrumbs", "LinkedErrors", "HttpContext", "Dedupe"]

/** @type {import('@nuxt/types').Plugin} */
export default function (ctx, inject) {
    const SentryMock = {}
    apiMethods.forEach(key=>{
        SentryMock[key] = _=>_
    }
    )

    // Inject mocked sentry to the context as $sentry (this is used in case sentry is disabled)
    inject('sentry', SentryMock)
    ctx.$sentry = SentryMock
}

I have to sign off, but this is quite odd. Hopefully there's something really simple we're missing.

If SENTRY_DSN isn't present at build time then webpack is going to cut it out of the build :confused:

The Nuxt Sentry module resolves Sentry DSN at build time. The runtime config doesn't matter at all. We need to change how we define the configuration to get it to work with runtime configuration: https://sentry.nuxtjs.org/sentry/runtime-config

Okay, I think I've got a working fix in the associated PR.


Add XDG_CONFIG_DIRS, close #29

Signed-off-by: Nicholas Malcolm<EMAIL_ADDRESS>

The XDG Base Directory Specification says that the XDG_CONFIG_DIRS environment variable should default to /etc/xdg if not set. This is an issue because confuse attempts to create these directories in config_dir() on line 955 if they do not already exist but on most systems /etc/xdg can only be written to by root so it raises an OSError. Should this behavior be added to fully conform to the spec and if so what should be done about the error?

Looks perfect; thank you! But instead of duplicating those two lines in the two functions, could you please put them into a little function of their own?
(Also, do we need to worry about escaping :? Probably not, but just thought I'd ask…)

I did move this to a separate function and the : does not need to be escaped. Any thoughts on my comment above or did you want to pull as it is now?

Looks excellent! Thank you again!!


send_email: Support add_context for remote servers.

612f2c73d663 started passing add_context to
send_custom_server_email, but did not make it make use of it.
Also add the hostname as a built-in value, since that is most likely the most useful property.

Fixes:

Screenshots and screen captures:

Self-review checklist

[ ] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[ ] Explains differences from previous plans (e.g., issue description).
[ ] Highlights technical choices and bugs encountered.
[ ] Calls out remaining decisions and concerns.
[ ] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[ ] Each commit is a coherent idea.
[ ] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[ ] Visual appearance of the changes.
[ ] Responsiveness and internationalization.
[ ] Strings and tooltips.
[ ] End-to-end functionality of buttons, interactions and flows.
[ ] Corner cases, error conditions, and easily imagined bugs.

Looks good, marked to merge once CI passes.


TF2 Keras OOM Training ImageNet with MobileNet V2

Click to expand!
Issue Type
Bug
Source
binary
Tensorflow Version
tf 2.9.2
Custom Code
No
OS Platform and Distribution
centos rhel fedora
Mobile device
No response
Python version
3.8.12
Bazel version
No response
GCC/Compiler version
No response
CUDA/cuDNN version
11.6
GPU model and memory
4x Tesla V100 16G
Current Behaviour?
As part of the effort of validating the official pretrained Keras models, I tried to measure the accuracy and try training MobileNet V2 with ImageNet2012. The measure accuracy is lower than expected, which is documented separately. The training also seems problematic with OOM errors even when the batch size is reduced to 32.
The experiment was performed on a 4-GPU node with TF2/Keras. The ImageNet 2012 dataset is prepared using tfds. The multi-GPU training used tf.distribute.MirroredStrategy(). Please see the code and log for more details.
Standalone code to reproduce the issue
import tensorflow as tf
from tensorflow import keras
import tensorflow_datasets as tfds
import numpy as np
import os
import time

from tensorflow.keras.applications.mobilenet_v2 import MobileNetV2
from tensorflow.keras.applications.mobilenet_v2 import preprocess_input, decode_predictions
from tensorflow.keras.preprocessing import image

tf.keras.backend.set_image_data_format(
    data_format='channels_last'
)

# ## MobileNet V2 Smoke Test

mbv2 = MobileNetV2(weights='imagenet')
img  = image.load_img('./dog.jpeg', target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)

preds = mbv2.predict(x)
print('Predicted:', decode_predictions(preds, top=3)[0])

# ## Prepare ImageNet Train & Validation

# Get imagenet labels
labels_path = tf.keras.utils.get_file('ImageNetLabels.txt','https://storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt')
imagenet_labels = np.array(open(labels_path).read().splitlines())

data_dir_val  = '/home/le_user/imagenet_dataset/'
write_dir_val = '/home/le_user/imagenet_dataset_tfds_full'

# Construct a tf.data.Dataset
download_config_val = tfds.download.DownloadConfig(
    extract_dir=os.path.join(write_dir_val, 'extracted'),
    manual_dir=data_dir_val)

download_and_prepare_kwargs_val = {
    'download_dir': os.path.join(write_dir_val, 'downloaded'),
    'download_config': download_config_val,
}

def resize_with_crop(image, label):
    i = image
    i = tf.cast(i, tf.float32)
    i = tf.image.resize_with_crop_or_pad(i, 224, 224)
    i = tf.keras.applications.mobilenet_v2.preprocess_input(i)
    return (i, label)

def resize_with_crop_v3(image, label):
    i = image
    i = tf.cast(i, tf.float32)
    i = tf.image.resize_with_crop_or_pad(i, 224, 224)
    i = tf.keras.applications.mobilenet_v3.preprocess_input(i)
    return (i, label)

ds = tfds.load('imagenet2012',
               data_dir=os.path.join(write_dir_val, 'data'),
               split=['train', 'validation'],
               shuffle_files=True,
               download=False,
               as_supervised=True,
               download_and_prepare_kwargs=download_and_prepare_kwargs_val)

AUTOTUNE = tf.data.AUTOTUNE
BATCH_SIZE_PER_REPLICA = 128
NUM_GPUS = strategy.num_replicas_in_sync

# ## Multi-GPU MB-V2 Validation & Training
ds_val_parallel = ds[1].map(resize_with_crop)
ds_val_parallel = ds_val_parallel.batch(batch_size=BATCH_SIZE_PER_REPLICA * NUM_GPUS)
ds_val_parallel = ds_val_parallel.cache().prefetch(buffer_size=AUTOTUNE)

ds_train_parallel = ds[0].map(resize_with_crop)
ds_train_parallel = ds_train_parallel.batch(batch_size=BATCH_SIZE_PER_REPLICA * NUM_GPUS)
ds_train_parallel = ds_train_parallel.cache().prefetch(buffer_size=AUTOTUNE)

strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
    mbv2_train_parallel = keras.applications.MobileNetV2(include_top=True,
                                                        weights='imagenet',
                                                        classifier_activation='softmax')
    mbv2_train_parallel.trainable = True
    mbv2_train_parallel.compile(optimizer='adam',
                                loss=keras.losses.SparseCategoricalCrossentropy(from_logits=False),
                                metrics=['accuracy'])
start_time = time.time()
result_parallel = mbv2_train_parallel.evaluate(ds_val_parallel)
print(f"--- {strategy.num_replicas_in_sync}-GPU eval took {(time.time() - start_time)} seconds ---")

print(dict(zip(mbv2_train_parallel.metrics_names, result_parallel)))

mbv2_train_parallel.fit(
    x=ds_train_parallel,
    validation_data=ds_val_parallel,
    epochs=20
)

Relevant log output
2.9.2
2.9.0
channels_first
channels_last
1/1 [==============================] - 2s 2s/step
Predicted: [('n02109961', 'Eskimo_dog', 0.35159874), ('n02114548', 'white_wolf', 0.13579218), ('n02110063', 'malamute', 0.033763986)]
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2', '/job:localhost/replica:0/task:0/device:GPU:3')

INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2', '/job:localhost/replica:0/task:0/device:GPU:3')

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

98/98 [==============================] - 56s 394ms/step - loss: 1.7855 - accuracy: 0.6155
--- 4-GPU eval took 56.09924674034119 seconds ---
{'loss': 1.7854773998260498, 'accuracy': 0.6154599785804749}

Epoch 1/20
INFO:tensorflow:batch_all_reduce: 158 all-reduces with algorithm = nccl, num_packs = 1

INFO:tensorflow:batch_all_reduce: 158 all-reduces with algorithm = nccl, num_packs = 1

INFO:tensorflow:batch_all_reduce: 158 all-reduces with algorithm = nccl, num_packs = 1

INFO:tensorflow:batch_all_reduce: 158 all-reduces with algorithm = nccl, num_packs = 1

 120/2503 [>.............................] - ETA: 19:42 - loss: 1.9443 - accuracy: 0.5517

---------------------------------------------------------------------------
ResourceExhaustedError                    Traceback (most recent call last)
/tmp/ipykernel_103014/1560377782.py in <cell line: 1>()
----> 1 mbv2_train_parallel.fit(
      2     x=ds_train_parallel,
      3     validation_data=ds_val_parallel,
      4     epochs=20
      5 )

~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)
     65     except Exception as e:  # pylint: disable=broad-except
     66       filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67       raise e.with_traceback(filtered_tb) from None
     68     finally:
     69       del filtered_tb

~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     52   try:
     53     ctx.ensure_initialized()
---> 54     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     55                                         inputs, attrs, num_outputs)
     56   except core._NotOkStatusException as e:

ResourceExhaustedError: Graph execution error:

5 root error(s) found.
  (0) RESOURCE_EXHAUSTED:  Failed to allocate memory for the batch of component 0
	 [[{{node MultiDeviceIteratorGetNextFromShard}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

	 [[RemoteCall]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

	 [[IteratorGetNextAsOptional]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

	 [[group_deps/_681]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

  (1) RESOURCE_EXHAUSTED:  Failed to allocate memory for the batch of component 0
	 [[{{node MultiDeviceIteratorGetNextFromShard}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

	 [[RemoteCall]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

	 [[IteratorGetNextAsOptional]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

	 [[div_no_nan_1/_655]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

  (2) RESOURCE_EXHAUSTED:  Failed to allocate memory for the batch of component 0
	 [[{{node MultiDeviceIteratorGetNextFromShard}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

	 [[RemoteCall]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

	 [[IteratorGetNextAsOptional]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

	 [[div_no_nan/ReadVariableOp/_612]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

  (3) RESOURCE_EXHAUSTED:  Failed to allocate memory for the batch of component 0
	 [[{{node MultiDeviceIteratorGetNextFromShard}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

	 [[RemoteCall]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

	 [[IteratorGetNextAsOptional]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

	 [[cond/output/_14/_116]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

  (4) RESOURCE_EXHAUSTED:  Failed to allocate memory for the batch of component 0
	 [[{{node MultiDeviceIteratorGetNextFromShard}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

	 [[RemoteCall]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

	 [[IteratorGetNextAsOptional]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_77642]

@lcmeng
This issue seems to be Keras issue.  Please post this issue on keras-team/keras repo. as Keras development is fully moving to github.com/keras-team/keras. All issues and PRs related to keras will be addressed in that repo.
To know more see this TF forum discussion ;
https://discuss.tensorflow.org/t/keras-project-moved-to-new-repository-in-https-github-com-keras-team-keras/1999
Thank you!

@sushreebarsa Thank you for your response. It turned out to be a tfds issue most likely. I commented out the line ds_train_parallel = ds_train_parallel.cache().prefetch(buffer_size=AUTOTUNE), which then prevented the OOM from happening. It could be that cache() conflicts with shuffle_files=True in tfds.load(...). I also noticed some other issues with the dataset code, such as the order of augmentation and cache() and batch().
I wonder if there's some tfds expert who can advise the best practice or pointers to real-world examples. Thanks.

@lcmeng You are just running out of memory. Can you please use distributed mirror strategy as show here to make sure all the GPUs are used correctly. Thanks!!

@gowthamkpr Thanks for your reply. The mirror strategy has been used in the original post.
strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
    mbv2_train_parallel = keras.applications.MobileNetV2(include_top=True,
                                                        weights='imagenet',
                                                        classifier_activation='softmax')
    mbv2_train_parallel.trainable = True
    mbv2_train_parallel.compile(optimizer='adam',
                                loss=keras.losses.SparseCategoricalCrossentropy(from_logits=False),
                                metrics=['accuracy'])

I think the issue is what I commented 24 days ago above. But please correct me if I was wrong. Thanks.

@lcmeng Can you please try batch size of 32 and let me know if you still run into OOM error? Thanks!!


Add Support For Watermark

It would be useful to be able to include a watermark overtop the stream.

You can create a watermark. framework supports visual effects system (CIFilter).
see also https://github.com/shogo4405/lf.swift/issues/110


I can't find the UPIOT plugin from the package control!

Before report an issue, please make sure it hasn't been reported before.
Operating system
Board used
Description of problem
Error(s) in ST Console? (View > Show Console)
Steps to Reproduce
Actual Results
Expected Results
Additional info
You can edit the text above according to your need

hi gepd：
thks for your share ,but i  can't find the UPIOT plugin from the package control as your install steps.

Another question please! Does the UPIOT can support STM32F05?, and I just find the EPS32 and ESP8266 in the  subdiretory folder path  'board' .
Thanks ,
you can contract me via Email<EMAIL_ADDRESS>

hi @ghcyguohai this plugin isn't yet in the package control repository, you can follow this steps to install it: https://github.com/gepd/uPiotMicroPythonTool#setup
I'll search about the STM32F05 and I'll let you know about it, it shouldn't be hard to add it.

hi gepd;
My honour from your reply. and I will try it  later. Sorry for the error mcu name"STM32F05",It is "STM32F405"

I cannot find uPIOT either. Does this support Sublime Text 3? Thanks!


There is no "load earlier messages" button

There is no "load earlier messages" button in my component. Any clue?

Code:
<GiftedMessenger
    ref={(c) => this._GiftedMessenger = c}
    ...
    loadEarlierMessagesButton={true}
    loadEarlierMessagesButtonText="Load"
    ...
/>

Hello, maybe it's because of https://github.com/FaridSafi/react-native-gifted-messenger/blob/master/GiftedMessenger.js#L224
Did you set initial "messages" prop with data ?

Hey guys.
This is my render method:
  ...
  ref: (c) ->
    @_GiftedMessenger = c

  render: ->
    <GiftedMessenger
      style={styles.chatContainer}
      ref={@ref}

      messages={[]}
      handleSend={@handleSend}
  ...

This solved it for me:
  ...
  componentDidMount: ->
    @_GiftedMessenger.setState(allLoaded: false)
    @loadMsgs()
  ...

please re-open if you are still experimenting this issue with 0.1.0