Poll selector

This PR adds support for using the Unix poll API as a backend.
Why this is required
There are some operating systems out there which only support poll (or other API's). The main challenge here was to track the file descriptors, as the poll syscall requires all file descriptors at call site. This is solved by keeping a mutex around the state of the selector and thus tracking everything added/removed from it.
The main motivation for this PR is to enable support for the esp-idf and it's growing eco system, eventually allowing tokio to fully run on these devices. As the esp-idf only supports poll, mio (and by this extent the full feature set of tokio) is unable to run on it. Additionally this benefits OS such as Haiku (see https://github.com/tokio-rs/mio/issues/1472), which also only provide a poll based API.
Does this work?
The short answer is: Yes, it should.
The long answer:
the polling crate already uses this, and in fact, the implementation here is largely a copy-and-paste of the one provided by polling. There are some notable differences whatsoever:

mio additionally needs cloning support, this is achieved by keeping everything in an Arc
mio (and tokio) used to assume edge triggering, but poll is level triggered (may be a breaking change!)

What did change
In addition to a new a new polling backend, a feature called force-old-poll was added in order to allow further testing of the poll implementation (this feature is required to force use of the poll API, whenever the OS provides kqueue or epoll).
It also now really is necessary to deregister event sources, else stale events may be fired (the general rules seems to always have been do deregister event sources, tokio's PollEvented for example does so as well as all mio examples).
The mio Waker also gained a new method called did_wake, which needs to be called after a task has been woken up in order to reset the event source (and not wake the task again when polling). This requires a very small patch in tokio which I have ready.
Effect on downstream crates
Probably none, but this is still a breaking change (see the notes about Waker and the edge vs. level triggered situation).
Since the poll implementation will virtually not be used anywhere other than new systems, no existing code should break on existing systems. Still, the API did change and does have new requirements.
I expect most downstream crates to properly deregister event sources before destruction, but this now is absolutely required to not run into weird race conditions. Additionally, every crate using mio Wakers now needs to call did_wake after being woken by the Waker.
Notes
This needs a lot of testing and feedback from the tokio and mio maintainers. Also thanks to @Kestrer, who implemented large portions of this code for the polling crate.

Before starting to work on large things like this, please open an issue first explain why you need this.

This started out as an experiment to see if it even was possible - turns out, it is and works pretty well. I thought I'd PR that for a review - that being said, if you still want me to open an issue I can do so (though I guess that would mostly be a copy'n paste of the PR description)

I have to wonder: should we also support the select() libc function, for the systems that don't comply with POSIX?

I thought about this as well, my reasoning for not using select directly was its limitation of 1024 fd's. A Selector using select would look mostly the same as this one.
Fun fact: the platform I'm specifically targetting with this PR implements poll using select and just starts screaming when you pass in too many file descriptors (see https://github.com/espressif/esp-idf/blob/master/components/newlib/poll.c#L13). For a proper Unix system using poll is still better than select.
So if you need this right now, you can also just implement poll this way :smile:

Also just mentioned a couple of time you copy and pasted this from the polling crate? If so, we can't accept that without a copyright notice. So, we'll likely simply not accept the code.

Yes. I haven't yet added a copyright notice, because I'm not exactly sure how it is supposed to look. Both libraries use the MIT license (polling allows you to choose either Apache or MIT), but neither have a copyright header in any of their files. Do you want a proper copyright notice with a fully blown header or rather a statement, that the code has been taken from polling?

Yes. I haven't yet added a copyright notice, because I'm not exactly sure how it is supposed to look. Both libraries use the MIT license (polling allows you to choose either Apache or MIT), but neither have a copyright header in any of their files.

That's because all files have the same copyright, so it's not needed to add one to each file.

Do you want a proper copyright notice with a fully blown header or rather a statement, that the code has been taken from polling?

No, I'm afraid it likely means this code will not be accepted.

Well, as the sole contributor to polling’s poll.rs implementation I do give you permission to add this code to Mio if you want.

Well, as the sole contributor to polling’s poll.rs implementation I do give you permission to add this code to Mio if you want.

That solves that then. 👍

An update on this PR: I have implemented tracking fd's in userspace and hopefully removed all breaking API changes. The poll selector now uses the same technique as the Win32 implementation for edge based triggering. This in return also means, that it behaves the same way.
I found one case where mio wasn't following its own docs, according to which one needs to ensure WouldBlock is returned before further readiness events can be received (epoll and kqueue don't have that problem, but the Win32 implementation and poll do actually suffer from this limitation).
I think this PR in general is ready for another code review pass - I'm still figuring out a CI solution (turns out mio went from crashing on haiku to crashing haiku, so my next attempt for CI will be Solaris [which also only has poll]).
The force-old-poll feature will be removed once I confirm this to be working on an OS which has only poll.

An update on this PR: I have implemented tracking fd's in userspace and hopefully removed all breaking API changes. The poll selector now uses the same technique as the Win32 implementation for edge based triggering. This in return also means, that it behaves the same way.

👍

I found one case where mio wasn't following its own docs, according to which one needs to ensure WouldBlock is returned before further readiness events can be received (epoll and kqueue don't have that problem, but the Win32 implementation and poll do actually suffer from this limitation).

To be clear the docs say that:

there is no guarantee that another readiness event will be delivered

Basically, without properly draining the readiness events Mio doesn't guarantee you'll get another event, but the underlying OS might return a readiness event any way. Note however that for Windows we control that behaviour and we're thinking about changing it (see https://github.com/tokio-rs/mio/issues/1611), basically following epoll and kqueue.

I think this PR in general is ready for another code review pass - I'm still figuring out a CI solution (turns out mio went from crashing on haiku to crashing haiku, so my next attempt for CI will be Solaris [which also only has poll]).

Wait... we crash haiku? Any way any OS with just poll (and nothing better) will do. I'll try a do review pass this weekend.

The force-old-poll feature will be removed once I confirm this to be working on an OS which has only poll.

👍

I'm still figuring out a CI solution (turns out mio went from crashing on haiku to crashing haiku

Is this with a nightly build or the beta? If you have a reliable reproducer then please open a ticket with Haiku and we will investigate (and probably fix within days.)

@Janrupf are you still working on this?

Not really - the implementation itself was (and still is) working, but I have no way to proof it using a CI pipeline or something along those lines (at least not with the constraints previously set by the maintainers). There are now a bunch of merge conflicts as you can see.
I'm using an implementation of this in a production environment where it has been running on a few hundred embedded devices without any issues, but that's about as good as it gets. If you have interest on picking this up or getting it merged, I'm willing to help.

We test it in polling by using an environment variable to conditionally compile it for Linux, see here.

We test it in polling by using an environment variable to conditionally compile it for Linux, see here.

The problem with that is that I'm not willing to maintain this implementation on any OS that supports something "better", e.g. epoll on Linux (better might be debatable here, but let's not). Having 1 implementation per OS is enough already.

On the following month I'll try implementing CI for one of the poll-only OSes then.

Hey, I've managed to compile and run the tests using qemu, but it takes a lot longer to run. Even if we optimize and cache in a lot of different ways, like building our own haiku image with rust pre-installed and running cargo vendor before copying the library into the VM, I think it would still take at least 15 minutes to run , while also being a lot more complex. See 1 and 2. The first one was hanging and I cancelled, just like it happened in my local (and fast) qemu, so I suppose it's an implementation issue.
From the issue #1682, it looks like you're willing to reconsider unsing a cfg exclusively to run tests on linux? We could have a mio_unsupported_force_poll cfg which would switch the implementation to poll to be used just to test the poll implementation, like here. This way we could also make this PR a lot simpler, as it would only implement poll, not full Haiku support. If there is interest on merging Haiku support in the future it can be done by someone else.

@pheki (see first part of your comment below)

From the issue #1682, it looks like you're willing to reconsider unsing a cfg exclusively to run tests on linux? We could have a mio_unsupported_force_poll cfg which would switch the implementation to poll to be used just to test the poll implementation, like here. This way we could also make this PR a lot simpler, as it would only implement poll, not full Haiku support. If there is interest on merging Haiku support in the future it can be done by someone else.

I've documented my thoughts on it in https://github.com/tokio-rs/mio/pull/1684, but yes let's continue with a mio_unspported_force_poll_poll flag, which we can use in the CI. But note that this means poll will not be officially supported for anything but (I think) Haiku.

Hey, I've managed to compile and run the tests using qemu, but it takes a lot longer to run. Even if we optimize and cache in a lot of different ways, like building our own haiku image with rust pre-installed and running cargo vendor before copying the library into the VM, I think it would still take at least 15 minutes to run , while also being a lot more complex. See 1 and 2. The first one was hanging and I cancelled, just like it happened in my local (and fast) qemu, so I suppose it's an implementation issue.

I still like to see some kind of Haiku support in the CI if we want to officially support it, even if it's just a cargo check and then we test the actual implementation on Linux with the cfg flag (not ideal, but it will have to do). We can only run it for merges into the master branch so that the prs aren't blocked on slow CI.
Thank you for looking to this.

I'm interested in throwing my hat in the ring here to try to help get this over the line (and with luck I can get some other esp-rs folks onboard!).  Before I get too far though I want to double check that I understand the issues, which AFAICT comes down to two missing requirements:

Automated tests covering a specific platform that is intended to (not just can) use the poll() code path introduced in this PR
A reliable story around ownership/maintenance for one or more of the platform in (1).

If this understanding is correct, I think the major issue we're facing right now is that the kinds of platforms that want the poll support aren't well set up for efficient, low cost automation which of course makes (1) difficult.  Separately, (2) is complicated by the fact that these platforms don't tend to be stable over long periods of time and many dozens of such platforms have come and gone while Linux persists.  This doesn't mean they don't deserve support, it just means that the overall community of people is likely going to be a tiny fraction of Linux.  Combine that with the fact that the Rust community is a small fraction of developers as a whole and in practice you can never expect to find a large group of committed maintainers.  An unfortunate situation for both tokio maintainers and the communities that would benefit from their great work.  I think we can find good solutions and move forward though, hopefully others agree :)
Let's take a closer look at (1) first.  As an example, esp-idf requires a lot of extra build support and tools (e.g. https://github.com/esp-rs/embuild) as well as a somewhat unsupported qemu fork (https://esp-rs.github.io/book/tooling/simulating/qemu.html) to run the code.  I'm confident that we could get this off the ground to run in github's CI, but I doubt very much that we'd ever be able to get the performance good enough to run within a couple of minutes.  Realistically I expect this will be 10+ minutes as @pheki has discovered with Haiku as well.  Most of the embedded community chooses instead to write platform agnostic code that can be tested on the host, as is the case with this PR being able to run on Linux, then move the rest to run on real hardware that requires significant overhead and cost to maintain in a CI infrastructure.  Even companies with deep pockets do it this way just simply because it can become a real money pit to try to get $6 devices to ever be as robust and reliable as Linux rackmount servers (trust me, I tried to do this at Meta for quite some time hehe).
While solutions do exist, I do think we need to maybe reexamine how flexible the requirements really are.  For example, we are already in the business of pragmatic support for platforms like Android and iOS despite being unable to test those reliably in GitHub CI as well for much the same reasons.  I'd propose that we're relying on the fact that a good number of supported targets are "close enough" to FreeBSD, Linux, Windows, and OS X that testing on those should be sufficient to cover the rest.  The real issue with (1) then is that there are no platforms currently that actually would use the poll code path and so the tests for this code feel arbitrary and contrived to use Linux just because it technically supports poll.  I agree here (and actually I agree with everything @Thomasdezeeuw has been saying), but consider for a moment that what we're really testing here is that Linux, a POSIX-compliant platform, should work the same as any other POSIX-compliant platform, in the same way we're saying that epoll on any obscure configuration of Linux should work the same as our desktop or rackmount server Linux configurations.  Definitely not literally true, but true enough to say we've done our due diligence by relying on reliable, stable standards and doing automated testing against a well known and correct implementation.  This actually makes the case stronger for using Linux with poll instead of, say, esp-idf's newlib poll which might be buggy and not a good proxy for, say, Haiku.
For (2), I personally think we can find strong partners in the embedded community, and in particular esp-rs which is a large and growing community with committed maintainers across a number of components.  I initially assumed that the gap to get tokio support for esp32 was so large that it wouldn't be practical even technically but honestly @Janrupf 's work will likely inspire others to step up and commit to getting this work landed and maintain it going forward, myself included.
So I guess all that novel was really to say, what exactly do I need to do to get this landed?  I see the PR is missing some of the cfg changes you proposed, and a few things could use tidying per your review.  Is there anything else that's insufficient?  What more do we need to convince tokio maintainers that we've got the right long term support?  I know that for example Rust async has a big push from Espressif as evidenced by Scott Mabin's blog posts: https://mabez.dev/blog/posts/esp-rust-30-06-2023/.  Probably not a stretch to imagine I could convince the esp-rs maintainers to commit to having tokio support on the roadmap...

Hi @jasta, I just want to add as a note that I'm in fact using this with tokio on the ESP32. I created a support PR on the Tokio repository some time ago, which should contain the changes neccessary to also get tokio to run this. Once mio works as expected it's only a matter of compiling Tokio with the right features.
And thanks a bunch for the long comment, I really appreciate it!

I'm interested in throwing my hat in the ring here to try to help get this over the line (and with luck I can get some other esp-rs folks onboard!). Before I get too far though I want to double check that I understand the issues, which AFAICT comes down to two missing requirements:
1. Automated tests covering a specific platform that is intended to (not just can) use the poll() code path introduced in this PR

Ideally the tests don't need to be modified (except for maybe adding a cfg(target_os = "haiku") or similar). Because that means the behaviour for the poll(2) implementation is the same as the on the other platforms. Since Haiku isn't supported by rustup (yet) it's ok to add a mio_unsupported_force_poll_poll flag to force the use of the poll(2) implementation so we can test on Linux on the CI. See https://github.com/tokio-rs/mio/pull/1684 for docs and https://github.com/tokio-rs/mio/pull/1685 for an example.

2. A reliable story around ownership/maintenance for one or more of the platform in (1).

This is pretty important, but mostly for Haiku or other OS(s) that I don't use. Basically if something breaks on Haiku I want to ask somewhere with at least the OS running to help. With the mio_unsupported_force_poll_poll I should be able to reproduce most issues on Linux, but not the one specific to Haiku (or other OS that uses the poll(2) implementation).

If this understanding is correct, I think the major issue we're facing right now is that the kinds of platforms that want the poll support aren't well set up for efficient, low cost automation which of course makes (1) difficult. Separately, (2) is complicated by the fact that these platforms don't tend to be stable over long periods of time and many dozens of such platforms have come and gone while Linux persists. This doesn't mean they don't deserve support, it just means that the overall community of people is likely going to be a tiny fraction of Linux. Combine that with the fact that the Rust community is a small fraction of developers as a whole and in practice you can never expect to find a large group of committed maintainers. An unfortunate situation for both tokio maintainers and the communities that would benefit from their great work. I think we can find good solutions and move forward though, hopefully others agree :)

This is spot on, we all have limited time so for me (other another Mio maintainer) to spend hours trying to setup a Haiku (or other) machine just to fix an issue is quite a tough ask. That is why (2) is important, for people with Haiku already running it can be a 2 minute check to at least reproduce an issue.

Let's take a closer look at (1) first. As an example, esp-idf requires a lot of extra build support and tools (e.g. https://github.com/esp-rs/embuild) as well as a somewhat unsupported qemu fork (https://esp-rs.github.io/book/tooling/simulating/qemu.html) to run the code. I'm confident that we could get this off the ground to run in github's CI, but I doubt very much that we'd ever be able to get the performance good enough to run within a couple of minutes. Realistically I expect this will be 10+ minutes as @pheki has discovered with Haiku as well. Most of the embedded community chooses instead to write platform agnostic code that can be tested on the host, as is the case with this PR being able to run on Linux, then move the rest to run on real hardware that requires significant overhead and cost to maintain in a CI infrastructure. Even companies with deep pockets do it this way just simply because it can become a real money pit to try to get $6 devices to ever be as robust and reliable as Linux rackmount servers (trust me, I tried to do this at Meta for quite some time hehe).
While solutions do exist, I do think we need to maybe reexamine how flexible the requirements really are. For example, we are already in the business of pragmatic support for platforms like Android and iOS despite being unable to test those reliably in GitHub CI as well for much the same reasons. I'd propose that we're relying on the fact that a good number of supported targets are "close enough" to FreeBSD, Linux, Windows, and OS X that testing on those should be sufficient to cover the rest. The real issue with (1) then is that there are no platforms currently that actually would use the poll code path and so the tests for this code feel arbitrary and contrived to use Linux just because it technically supports poll. I agree here (and actually I agree with everything @Thomasdezeeuw has been saying), but consider for a moment that what we're really testing here is that Linux, a POSIX-compliant platform, should work the same as any other POSIX-compliant platform, in the same way we're saying that epoll on any obscure configuration of Linux should work the same as our desktop or rackmount server Linux configurations. Definitely not literally true, but true enough to say we've done our due diligence by relying on reliable, stable standards and doing automated testing against a well known and correct implementation. This actually makes the case stronger for using Linux with poll instead of, say, esp-idf's newlib poll which might be buggy and not a good proxy for, say, Haiku.

Agreed, testing on Linux with mio_unsupported_force_poll_poll would be good enough. Of course this assumes that the Linux poll(2) implementation works exactly like the implementation found in Haikue/esp-idf/etc., which is a big if, but it's (I think) the best we can realistically do right now.

For (2), I personally think we can find strong partners in the embedded community, and in particular esp-rs which is a large and growing community with committed maintainers across a number of components. I initially assumed that the gap to get tokio support for esp32 was so large that it wouldn't be practical even technically but honestly @Janrupf 's work will likely inspire others to step up and commit to getting this work landed and maintain it going forward, myself included.

👍

So I guess all that novel was really to say, what exactly do I need to do to get this landed? I see the PR is missing some of the cfg changes you proposed, and a few things could use tidying per your review. The CI checks are currently failing so obviously we gotta fix that. Is there anything else that's insufficient? What more do we need to convince tokio maintainers that we've got the right long term support? I know that for example Rust async has a big push from Espressif as evidenced by Scott Mabin's blog posts: https://mabez.dev/blog/posts/esp-rust-30-06-2023/. Probably not a stretch to imagine I could convince the esp-rs maintainers to commit to having tokio support on the roadmap...

From my end:

Add the mio_unsupported_force_poll_poll flag as discussed above.
Use the mio_unsupported_force_poll_poll to the poll(2) implementation on Linux. Maybe FreeBSD as well so we have at least two different poll(2) implementations we test, so that we don't depend on Linux's specific behaviours/bugs in poll(2)?
No changes to the tests, ideally at least, so that we know Mio stays cross platform. Exception here for things like add cfg(target_os = "haiku").
And of course a rebase + passing CI.

I think that about it?
(note that the list might not seem to long, but I only recently agreed to the mio_unsupported_force_poll_poll flag, this was mainly stuck on getting Haiku or poll(2) implementations specific support in the CI)

Ok I'm going to work on the technical aspects of this first:

Use mio_unsupported_force_poll_poll
Rebase + get CI checks passing generally
Add a CI configuration for the tests in GitHub Actions and make sure all that passes

Once I'm ready I'll submit a new PR and then start socializing the maintenance issues a bit with the esp-rs folks.  I'm feeling really good that we can get this over the line soon-ish!

I noticed a couple of bugs in the poll implementation while doing this work that result directly from inconsistencies to how the Windows implementation works.  For example, when poll yields POLLHUP the edge-triggered IoSourceState doesn't work properly and then the caller just repeatedly gets spammed POLLHUP because the fd is not deregistered internally.  The Windows implementation solves this by using a Mutex and shared state between IoSourceState and the Selector, so I think I'll just further copy that behaviour.
This will fix a failing test I'm seeing locally in tcp_stream.rs:tcp_reset_close_event

Only one issue left to fix and I could use your help @Janrupf.  Specifically I bumped into: https://github.com/tokio-rs/tokio/issues/5866 which it looks like you should've hit as well unless you were only testing recv_from and send_to APIs or anything else not covered by PollEvented.

Only one issue left to fix and I could use your help @Janrupf. Specifically I bumped into: tokio-rs/tokio#5866 which it looks like you should've hit as well unless you were only testing recv_from and send_to APIs or anything else not covered by PollEvented.

Also see https://github.com/tokio-rs/mio/issues/1611. We could also decide that reading less then buf.len() bytes should trigger reregistering, matching what epoll and kqueue do already (with edge triggers as used by Mio).

FWIW, a (very basic, FDs/PROC only) implementation of kqueue is being worked on and should make its way into Haiku nightly builds before too long, hopefully removing the need for poll there.
Above, however, it was noted that some versions of this change caused Haiku to crash. Is that reproducible? Do you have a reproducer, or even just a screenshot of the crash?

FWIW, a (very basic, FDs/PROC only) implementation of kqueue is being worked on and should make its way into Haiku nightly builds before too long, hopefully removing the need for poll there.

Nice. We still need the poll(2) based implementation for ESP-IDF, see https://github.com/tokio-rs/mio/pull/1687 and https://github.com/tokio-rs/mio/pull/1692.

Above, however, it was noted that some versions of this change caused Haiku to crash. Is that reproducible? Do you have a reproducer, or even just a screenshot of the crash?

FWIW, a (very basic, FDs/PROC only) implementation of kqueue is being worked on and should make its way into Haiku nightly builds before too long, hopefully removing the need for poll there.

Nice. We still need the poll(2) based implementation for ESP-IDF, see #1687 and #1692.

I would also hope that Solaris would be able to use poll(2) too. See #1152.

Superseded by #1687, thanks to everyone who continued this!

Thanks @Janrupf for the implementation!


Add sonatype settings and apply spotless check

According to the doc:
Important. Users registered in Sonatype after 24 February 2021 need to customize the following URLs:

nexusPublishing {
    repositories {
        sonatype {  //only for users registered in Sonatype after 24 Feb 2021
            nexusUrl.set(uri("https://s01.oss.sonatype.org/service/local/"))
            snapshotRepositoryUrl.set(uri("https://s01.oss.sonatype.org/content/repositories/snapshots/"))
        }
    }
}

we need to add these settings for release.

Thanks you Jiefan, LGTM.
A minor nit: maybe bump the gradle-nexus plugin version from 1.0.0 to 1.1.0?


cpp for c++ code samples, edits

Use cpp for c++ code examples so they get syntax highlighting
Edit for correct standard MS notation for code literal, keyword.
Make headings parallel and aligned with the defined terms wrt the inclusion/explicit instantiation models

@Paul-Dempsey,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot


Deploy celo core contracts release 4 to staging

Expected Behavior
Release 4 deployed on staging
Current Behavior
Release 4 not deployed

deduping in favor of https://github.com/celo-org/celo-monorepo/issues/7820


Ubuntu 16.04 Installation Operation not permitted

SQL Operations Studio Version:

Steps to Reproduce:
Operating System: Ubuntu-x64 16.04.3
sudo apt-get install libxss1
sudo apt-get install libgconf-2-4
sudo apt-get install libunwind8
sudo dpkg -i /home/tom/Downloads/sqlops-linux-0.27.3.deb
(Reading database ... 263102 files and directories currently installed.)
Preparing to unpack sqlops-linux-0.27.3.deb ...
Unpacking sqlops (0.27.3-1522127969) ...
dpkg: error processing archive sqlops-linux-0.27.3.deb (--install):
unable to open '/usr/share/sqlops/resources/app/node_modules/jquery-ui/themes/base/selectmenu.css.dpkg-new': Operation not permitted
Processing triggers for bamfdaemon (0.5.3~bzr0+16.04.20180209-0ubuntu1) ...
Rebuilding /usr/share/applications/bamf-2.index...
Processing triggers for desktop-file-utils (0.22-1ubuntu5.1) ...
Processing triggers for gnome-menus (3.13.3-6ubuntu3.1) ...
Processing triggers for mime-support (3.59ubuntu1) ...
Errors were encountered while processing:
sqlops-linux-0.27.3.deb

@tomchavakis are you still hitting this issue on current releases?  We do test Ubuntu every release and I haven't been able to repro this yet.

@kburtram I am still hitting this issue on the latest release (sqlops-linux-0.31.4.deb) at Ubuntu 16.04.
Now I use 18.04 and everything is fine.
For the issue purposes I have just tested again at a VM using Ubuntu 16.04 and now I am getting the following error:
sudo dpkg -i sqlops-linux-0.31.4.deb
(Reading database ...
(Reading database ... 263550 files and directories currently installed.)
Preparing to unpack sqlops-linux-0.31.4.deb ...
Unpacking sqlops (0.31.4-1531879093) ...
dpkg: error processing archive sqlops-linux-0.31.4.deb (--install):
unable to open '/usr/share/sqlops/resources/app/out/vs/workbench/parts/debug/browser/media/clear-repl-inverse.svg.dpkg-new': Operation not permitted
Processing triggers for bamfdaemon (0.5.3~bzr0+16.04.20180209-0ubuntu1) ...
Rebuilding /usr/share/applications/bamf-2.index...
Processing triggers for desktop-file-utils (0.22-1ubuntu5.1) ...
Processing triggers for gnome-menus (3.13.3-6ubuntu3.1) ...
Processing triggers for mime-support (3.59ubuntu1) ...
Errors were encountered while processing:
sqlops-linux-0.31.4.deb

Closing as this related to old ADS build and older version of Ubuntu.  Please let us know if this problem persists with current ADS and Ubuntu versions.  Thanks!


XFAIL SDKDependencies test in Windows.

Windows paths attack again. Some paths from the dependency tracker have mixed folder separators, so comparing textually fail. Additionally some paths seems to be extended paths (\?) which seems not to be liked by LLVM path utilities.
Windows builds started failing after introducing #27635 in https://ci-external.swift.org/job/oss-swift-windows-x86_64/1631/. From the failed test:
$ "s:\\swift\\bin\\swiftc.exe" [...snip...] "C:\jenkins\workspace\oss-swift-windows-x86_64\swift\test\ModuleInterface\ModuleCache\SDKDependencies.swift"
# command stderr:
<unknown>:0: error: fatal error encountered during compilation; please file a bug report with your project and the crash log

<unknown>:0: note: Unexpected path for swiftinterface file:

S:\swift\test-windows-x86_64\ModuleInterface\ModuleCache\Output\SDKDependencies.swift.tmp\my-sdk\ExportedLib.swiftinterface

The module <-> path mapping we have is:

SdkLib <-> S:\swift\test-windows-x86_64\ModuleInterface\ModuleCache\Output\SDKDependencies.swift.tmp/my-sdk\SdkLib.swiftinterface

This should fix the comparison between the two paths to use the same kind of slashes in both sides.

@swift-ci please test Windows platform

@swift-ci please smoke test

Seems that the fix is not enough, but in the following case, the matching is even more confussing:
$ "s:\\swift\\bin\\swiftc.exe" "-frontend" "-target" "x86_64-unknown-windows-msvc" "-module-cache-path" "S:\\swift\\swift-test-results\\x86_64-unknown-windows-msvc\\clang-module-cache" "-swift-version" "4" "-typo-correction-limit" "10" "-typecheck" "-I" "S:\swift\test-windows-x86_64\ModuleInterface\ModuleCache\Output\SDKDependencies.swift.tmp/my-sdk" "-sdk" "S:\swift\test-windows-x86_64\ModuleInterface\ModuleCache\Output\SDKDependencies.swift.tmp/my-sdk" "-module-cache-path" "S:\swift\test-windows-x86_64\ModuleInterface\ModuleCache\Output\SDKDependencies.swift.tmp/MCP" "-emit-dependencies-path" "S:\swift\test-windows-x86_64\ModuleInterface\ModuleCache\Output\SDKDependencies.swift.tmp/dummy.d" "-track-system-dependencies" "-emit-loaded-module-trace-path" "S:\swift\test-windows-x86_64\ModuleInterface\ModuleCache\Output\SDKDependencies.swift.tmp/trace.json" "C:\jenkins\workspace\swift-PR-windows\swift\test\ModuleInterface\ModuleCache\SDKDependencies.swift"
# command stderr:
<unknown>:0: error: fatal error encountered during compilation; please file a bug report with your project and the crash log

<unknown>:0: note: Unexpected path for swiftinterface file:

S:/swift/test-windows-x86_64/ModuleInterface/ModuleCache/Output/SDKDependencies.swift.tmp/my-sdk/ExportedLib.swiftinterface

The module <-> path mapping we have is:

ucrt <-> S:/swift/test-windows-x86_64/ModuleInterface/ModuleCache/Output/SDKDependencies.swift.tmp/MCP/3JO5MDX5AF7SW/ucrt-37M3AHJZWWF6Y.pcm

_Builtin_stddef_max_align_t <-> S:/swift/test-windows-x86_64/ModuleInterface/ModuleCache/Output/SDKDependencies.swift.tmp/MCP/3JO5MDX5AF7SW/_Builtin_stddef_max_align_t-2D3F05T99HALZ.pcm

visualc <-> \??\s:\swift\lib\swift\windows\x86_64\SwiftOnoneSupport.swiftinterf

SdkLib <-> S:/swift/test-windows-x86_64/ModuleInterface/ModuleCache/Output/SDKDependencies.swift.tmp/my-sdk/SdkLib.swiftinterface

ExportedLib <-> \??\s:\swift\lib\swift\windows\x86_64\SwiftOnoneSupport.swiftm

SwiftOnoneSupport <-> s:/swift/lib/swift/windows/x86_64/SwiftOnoneSupport.swiftmodule

SomeCModule <-> \??\s:\swift\lib\swift\windows\x86_64\SwiftOnoneSupport.swiftinterfa

Swift <-> s:/swift/lib/swift/windows/x86_64/SwiftOnoneSupport

SwiftShims <-> \??\s:\swift\lib\swift\windows\x86_64\SwiftOnoneSupport.swiftinterfa

Seems that some paths are not actually handled by llvm::sys:path, which is a pity (I also don't know if it is an artifact of the output or what, but those paths also look truncated).
The failure in the Linux test also show that convert_to_slash is not as no-op as the documentation says it is, but that failure wasn't happening in my machine.
I will have to figure out what's happening there.

The lengths are not even consistent. 😕 Moreover, the some of the paths seem very wrong?
SomeCModule <-> \??\s:\swift\lib\swift\windows\x86_64\SwiftOnoneSupport.swiftinterfa

Swift <-> s:/swift/lib/swift/windows/x86_64/SwiftOnoneSupport

I think you might want to consider XFAILing the test on Windows for now and investigate it later, so it at least unblocks the Windows CI.

Repourposed the PR to xfail the test for the time being.
@swift-ci please test Windows platform

@swift-ci please smoke test

@varungandhi-apple: those path prefixes look like the extended length prefix for Windows paths (ref: https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file#maximum-path-length-limitation), but it is not completely right, and also seems to be missing some characters at the end. I don't know if that might be the result of convert_to_slashes not dealing with the original prefix. (which looks like \\?\).

those path prefixes look like the extended length prefix for Windows paths (ref: https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file#maximum-path-length-limitation),

I don't think that's the case. Or at least, I don't think that explains the whole issue around the paths getting truncated. If you look at two of the strings:
\??\s:\swift\lib\swift\windows\x86_64\SwiftOnoneSupport.swiftm - 62 chars

\??\s:\swift\lib\swift\windows\x86_64\SwiftOnoneSupport.swiftinterfa - 68 chars

Admittedly, I don't understand what the \??\ is doing but these are still way shorter than 255 chars. Moreover, they have different lengths.


Replace unsafe usage in fastanvil by const-init

I was doing a quick audit of my dependencies and noticed this unnecessary unsafe usage. To make an array of !Copy but const-constructible types you just need to create a const value to use as the prototype in the array literal.

Looks great. Always good to get rid of some unsafe. :) Thank you.


User Authentication and Roles

See #3.

I may end up moving roles from #3 to a separate PR that handles the relationship between Users and Boxes (#5).

Using Devise for authentication for now (may end up rolling my own auth in the future).
Closed for now :rocket:


MAX_ROLLUP_THREADS cleanup

I noticed that the MAX_ROLLUP_THREADS configuration variable is still present in several places, even though it has been replaced with MAX_ROLLUP_READ_THREADS and MAX_ROLLUP_WRITE_THREADS since rax-release-v1.0.1956.
Therefore I've changed these occurrences with the new variants. You might want to verify the default values though as I don't have much experience with them, I've used the same values as in CoreConfig.java

Coverage remained the same at 75.088% when pulling 2d96a59dac0f2c9e4797ea4e3614e13906bf5d1e on Klarrio:max_rollop_threads_rename_refactor into 5a0433dda2f4ce9d6dfddefa44823fd7f9d8b2a3 on rackerlabs:master.


[bounty] improve query speed

query are slow when data is big
things we could do:

use fts for keyword filtering
tweak indexing, index regularly
other

/bounty 200
plz suggest what you are going to do first

i did it


Fancy Zones frequently forgets applied settings

I have FancyZones configured with custom zones. It frequently forgets that I have applied the custom zones and reverts to the default. And now, it has overlayed my monitor 2 zones on top of my monitor 1 zones with no way to fix it as they're on the wrong screen and will not drag over.
ℹ Computer information

PowerToys version: 0.25
PowerToy Utility: FancyZones
Running PowerToys as Admin: No
Windows build number: Windows 10 version 2004 (OS Build 19041.630

📝 Provide detailed reproduction steps (if any)

… Set custom Zones and apply
… Wait
… Use zones to snap windows

✔ Expected result
Custom set zones exist and can be used
❌ Actual result
Zones are now set to the default.

@gordol1
the fix will be available in the next release.
Closing as duplicate of https://github.com/microsoft/PowerToys/issues/7790


adding handler for createReviews err, fixing misspelling, removing un…

Fixing the ignored err returned from createReviews.
If a user makes a typographic error when creating the model attributes, the data connector will return an error because the attribute names in create-sample-model.js do not match the model's attribute names. This error will be ignored because it is not handled by the callback.
Also fixing a misspelling and removing unnecessary commas.

LGTM, but I would like @superkhau to take a quick look before merging just to confirm.

LGTM


HA problem of WNs

execute run failed, err execute query q11 failed Error 1105: other error for mpp stream: Code: 49, e.displayText() = DB::Exception: Check status.ok() failed: Failed to fetch all pages from s8_t2739_3460_3_4494205696178457, status=14, message=Connection reset by peer, e.what() = DB::Exception

other error for mpp stream: Code: 0, e.displayText() = DB::Exception: Receiver state: ERROR, error message: Exchange receiver meet error : Code: 0, e.displayText() = DB::Exception: Receiver state: ERROR, error message: Exchange receiver meet error : Code: 0, e.displayText() = DB::Exception: EstablishDisaggTask Failed14: Connection reset by peer, e.what() = DB::Exception,, e.what() = DB::Exception,, e.what() = DB::Exception

We have added more detail logging and some timeout about this problem. Will dive deep into it if a similar issue happens again.

https://github.com/pingcap/tiflash/pull/8795
https://github.com/pingcap/tiflash/pull/8807


Run this in a container?

Has anyone managed to run this in a container (e.g. custom baked Homeassitant / Node-red)?
I have been trying to get Bluetooth and pycalima working however I'm getting permission errors (probably not a pycalima issue) but thought if someone are already running this in a container it would save me many hours of troubleshooting.
I can't wait to get my fan into Home Assistant!
Thanks!

Not much for containers, running a few for applications I want isolated. I think there's loads of resources on getting Bluetooth supported in containers of Hass. However, I don't myself. If anyone else wants to chime in please do so! This is useful and related information!

OK, thanks!
I got it running in my self-baked node-red image that pulls your repo and makes it available inside the container.
Let me know if there's interest to publish this for the public.

OK, thanks!
I got it running in my self-baked node-red image that pulls your repo and makes it available inside the container.
Let me know if there's interest to publish this for the public.

Im very interest :), im running Home Assistant (Supervisor) with Node-Red

This is the docker file I used to make the container
FROM nodered/node-red:1.0.6-2-12-minimal-arm32v6

USER root

#RUN /bin/sh -c apk add --no-cachel glib-dev
RUN apk add glib-dev

RUN echo "**** install Python ****" && \
    apk add --no-cache python3

# Dependencies
RUN apk add \
  --no-cache \
  #-- virtual \
  --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing \
  --repository http://dl-cdn.alpinelinux.org/alpine/edge/main

#RUN apk add glib-dev

#RUN apk add --no-cache python3-pip libglib2.0-dev

RUN if [ ! -e /usr/bin/python ]; then ln -sf python3 /usr/bin/python ; fi

RUN echo "**** install pip ****" && \
    python3 -m ensurepip && \
    rm -r /usr/lib/python*/ensurepip

RUN pip3 install --no-cache --upgrade pip setuptools wheel && \
    if [ ! -e /usr/bin/pip ]; then ln -s pip3 /usr/bin/pip ; fi

#RUN pip3 -v install bluepy
RUN git clone https://github.com/IanHarvey/bluepy.git /usr/bluepy
RUN apk add make g++
RUN pip3 install git+https://github.com/IanHarvey/bluepy.git
#& cd bluepy && python setup.py build && sudo python setup.py install

RUN git clone https://github.com/PatrickE94/pycalima.git /usr/pycalima
RUN pip3 install git+https://github.com/PatrickE94/pycalima.git

# User configuration directory volume
VOLUME ["/data"]
EXPOSE 1880

# Environment variable holding file path for flows configuration
ENV FLOWS=flows.json

CMD ["npm", "start", "--", "--userDir", "/data"]


Adding new block types

As a player, I want more block types available to me so that I have more options when building stuff in minecraft

adding new types of blocks such as water, ore, etc
how to handle visuals of the new block type
understand the process of adding a new block type into the program


feat: Manage disconnection

About this PR

Allow external clients to disconnect.
Linked to https://github.com/lenra-io/devtool/pull/212
Technical highlight/advice
Handle a full disconnect flow by call the revoke endpoint and go to the logout endpoint.
How to test my changes
Run the exemple project :
cd example
lenra build
lenra start
npm i
npm run build
npm start

Go to http://localhost:3030/
Checklist

[x] I didn't over-scope my PR (except a small one for login popup)
[x] My PR title matches the commit convention
[x] I did not include breaking changes
[x] I made my own code-review before requesting one

I included unit tests that cover my changes

[ ] 👍 yes
[x] 🙅 no, because they aren't needed
[ ] 🙋 no, because I need help

I added/updated the documentation about my changes

[ ] 📜 README.md
[ ] 📕 docs/*.md
[ ] 📓 docs.lenra.io
[x] 🙅 no documentation needed

@taorepoara  I don't understand how to test your PR with the devtool PR. I have no disconnect button and I have nothing more in the json routes by running the example.
Can you add more detail about how to run this or maybe we could have a meeting on discord, it would be faster.


Pangeo training material for Big data geosciences

We have:

pangeo: Pangeo ecosystem 101 for everyone - Introduction to Xarray Galaxy Tools
pangeo-notebook: Pangeo Notebook in Galaxy - Introduction to Xarray

The first one (pangeo 101) is meant to be used by anyone and does not require any programming skills (using Galaxy Tools) and shows what is Pangeo and its community and how to use Xarray tools in Galaxy.
The second one (pangeo notebook) makes use of Pangeo JupyterLab interactive tool and is an introduction to Xarray for those who have basic Python programming skills.

Amazing! Thank you Anne! Really top! I made a first rapid review and PR to nordicESMhub repo bit I think I made something wrong like 1 pr on main branch and another on the good pangeo one.... Don't hesitate if you have doubts on how to manage it ;)

The second one (pangeo notebook) makes use of Pangeo JupyterLab interactive tool and is an introduction to Xarray for those who have basic Python programming skills.

Fyi @annefou there is a new format you can opt-in to using, that generates the ipynb files automatically. You can see it in action here: https://training.galaxyproject.org/training-material/topics/data-science/ anything tagged jupyter-notebook and rmarkdown-notebook have these files automatically generated from their GTN content, if that's interesting to you

Fyi @annefou there is a new format you can opt-in to using, that generates the ipynb files automatically.

Wow!!! This is so cool!!! I was looking for something like that!!! I definitely want it.
Thank you so much!

Oh, I even wrote documentation! https://training.galaxyproject.org/training-material/topics/contributing/tutorials/create-new-tutorial-content/tutorial.html#automatic-jupyter-notebooks

Do not use the built in citation system

is also outdated, citations work now.

presenter notes have been added for both Pangeo tutorials.

@annefou I can't seem to push to your branch, can you please make a commit like:
From 318ef39adac638acf0980ce350c0a0e7ecd36c03 Mon Sep 17 00:00:00 2001
From: Helena Rasche<EMAIL_ADDRESS>Date: Tue, 8 Feb 2022 10:55:57 +0100
Subject: [PATCH] fix missing citations section

---
 _layouts/slides-plain.html | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/_layouts/slides-plain.html b/_layouts/slides-plain.html
index 52015e3b01..8411f2dca8 100644
--- a/_layouts/slides-plain.html
+++ b/_layouts/slides-plain.html
@@ -86,3 +86,8 @@ This material is the result of a collaborative work. Thanks to the <a href="http

 This material is licensed under the Creative Commons Attribution 4.0 International License.

+
+{% if page.cited %}
+
{{locale['references']| default: "References" }}

+{% bibliography --cited %}
+{% endif %}
--
2.25.1

adding those lines to the bottom of the _layouts/slides-plain.html. I suspect that fixes the build errors.

I've written https://github.com/galaxyproject/training-material/pull/3161 in support of this, it'll add support for the citations working properly in the automatic videos. Otherwise, fantastic! From the technical side it looks ok

Awesome!
So to summarize: do I need to add what you suggested earlier or should I wait for your PR to be merged and then update my branch?
Many thanks!

You need to add what I suggested in https://github.com/galaxyproject/training-material/pull/3147#issuecomment-1032422137, that will let this PR pass.
After that, one of the other maintainers will need to merge my PR, that will let your videos get built correctly when they do get built
And last we'll merge your PR

Normally I'd make all these commits directly to your branch but for whatever reason I couldn't push directly.

Normally I'd make all these commits directly to your branch but for whatever reason I couldn't push directly.

I am not sure why you can't. I usually allow maintainers to update and I don't see the "box" to tick/untick now.

Ok, since you've fixed the auth, I'll just add all of those commits into this PR and it'll be much simpler! Thanks @annefou :)

Thank you so much @hexylena !!!!

failing now due to a dependency missing in the video dry run, I'll look into it.

Wow! This is green!!! Thank you!

I think we'll want to make changes to the intro section. Right now it still says "best viewed in JupyterLab" but that's not appropriate here, and the intro section has some potential issues like the tool link box breaking the notebook currently.
Currently there is a snippet that is auto-included if it's a notebook, defaulting to jupyter.
I think the best approach is that I move the preamble into a snippet just for this tutorial, and then I update the auto-include jupyter tip box, to instead have your setup instructions.
Then they'll see at the top something like:
Tip: best viewed in Pangeo Jupyterlab
  create new history
  upload data
  starting pangeo jupyter
``

but that'll take a bit of work. I think it's useful for the long run to support the other jupyter flavours though.

In the meantime Climate folks can start reviewing :)

Ok, I got that working. I think from a technical standpoint, it's ready to go. The setup instructions (history, run this tool, etc) are moved into a snippet (preamble.md) and I've included the final instructions to download the GTN notebook into the Pangeo + start it. (We cannot just load the notebook into Galaxy, the html of the GTN notebooks is unacceptable, sadly. So we...work around it by starting the notebook, and then downloading the ipynb + switching to it.)

Fine from a technical standpoint. If anyone is up to review content/language it would be good :)

Is there anything we can do on our side (pangeo & Climate) to help?

Fine from a technical standpoint. If anyone is up to review content/language it would be good :)

Is there anything we can do on our side (pangeo & Climate) to help?

I (not me personally ;) ) can try something!

Yeah, if anyone just wants to follow the instructions and say if it works / if it's appropriate  for climate scientists/etc :)

The comments included in code blocks can't be read easily; I'm not sure if it would be possible to modify the color palette @hexylena?

It's a known issue @gallardoalba , we should definitely fix it, there's some other colour schemes we can try (but in this tutorial the majority of people will not be reading the gtn material, and will instead be using the notebook.)

What do you think @gallardoalba

And for admin training where we use diffs very heavily.

Perfect @hexylena!

@gallardoalba Thanks a lot for your very careful review. This is really awesome! I have committed most of your changes and I will make the remaining requested changes/improvements.

thank you @yvanlebras and Solenne! I'll update the few pending comments (from previous review too). Many thanks for reviewing this material!

Ok. So I think I took into account all your comments. Thanks a lot for your review!

Amazing! I will try to test this final version now and validate it! If you think @annefou you can test mine ;) https://github.com/galaxyproject/training-material/pull/3152 this can be amazing !!!! Have a nice week-end!

Amazing! I will try to test this final version now and validate it! If you think @annefou you can test mine ;) #3152 this can be amazing !!!! Have a nice week-end!

Cool. Yes I can review your training material! Thanks.

Really sorry... now I have the dataset, I have an error message:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/data/jwd/main/041/778/41778869/tmp/ipykernel_354/4122857281.py in <module>
----> 1 dset = xr.open_dataset("CAMS-PM2_5-20211222.netcdf")

/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, backend_kwargs, *args, **kwargs)
    477
    478     if engine is None:
--> 479         engine = plugins.guess_engine(filename_or_obj)
    480
    481     backend = plugins.get_backend(engine)

/srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/backends/plugins.py in guess_engine(store_spec)
    150         )
    151
--> 152     raise ValueError(error_msg)
    153
    154

ValueError: did not find a match in any of xarray's currently installed IO backends ['netcdf4', 'h5netcdf', 'scipy', 'cfgrib', 'pydap', 'rasterio', 'zarr']. Consider explicitly selecting one of the installed engines via the ``engine`` parameter, or installing additional IO dependencies, see:
http://xarray.pydata.org/en/stable/getting-started-guide/installing.html
http://xarray.pydata.org/en/stable/user-guide/io.html

typing this dset = xr.open_dataset("CAMS-PM2_5-20211222.netcdf") ...

my jupyter notebook FYI https://3525516ba8d111f5-3742e8a717c04bddb0a50d763b550537.interactivetoolentrypoint.interactivetool.ecology.usegalaxy.eu/ipython/lab/tree/Untitled.ipynb

my jupyter notebook FYI https://3525516ba8d111f5-3742e8a717c04bddb0a50d763b550537.interactivetoolentrypoint.interactivetool.ecology.usegalaxy.eu/ipython/lab/tree/Untitled.ipynb

I am not sure why. It usually happens when the type of the file is not set to netcdf but h5.

ok, I retest from start and it is ok now! I can go further! THANK YOU !

hey hey! Done!

Amazing tuto! Thank you Anne!!!!!

What a cool tutorial!

whoo!! so awesome! :tada:
(currently there seems to be a problem with rendering the slides video, but we are working on it!)

(currently there seems to be a problem with rendering the slides video, but we are working on it!)

Let me know if there is anything to do on my side.

@annefou nah, there was a small bug in the video generation. but the video's are up now :) The pronunciation of "Pangeo" is a bit off tho, so we wil look into teaching it how to pronounce it

Cool! That's awesome!
I find a few "dots" that cut sentences and sometimes it is very odd. I guess sentences were far too long. I have started to note precisely when it happens for the first video. Let me know how I can fix these issues.
in the first pangeo video:

1:55 there is a dot at the end and it should be removed (probably my fault)!
must be scalable ... current and future challenges of big data ...  e.g. no dot beteen future and challenges.
2:08 we should also remove the dot after use cases e.g. use cases as well as ...
2:20 remove the dot after be e.g. cannot be tackled separately.
2:30 remove dot after define e.g. developers can define priorities for future...
3:22 remove dot after interface e.g. user interface with many functions...
3:54 remove dot after Galaxy e.g. from Galaxy Tools can be useful.

We have similar issues in the second pangeo video (for pangeo-notebook.
Also netCDF is not pronounced correctly. I think I should have written net CDF or net-CDF (I forgot about it).
Thanks!
How can I fix these small issues?

Also netCDF is not pronounced correctly. I think I should have written net CDF or net-CDF (I forgot about it).

You can add these in bin/ari-map.yml, Keep writing netCDF in your slides (better for screen readers/etc), and then the ari-map will map those terms to the way to pronounce them.

I guess sentences were far too long

Ahh I see what happened, you didn't use bullet points, so they were treated as individual lines. Until now most people have used bullet points or at least had a full sentence on a single line, rather than wrapping which is what's causing the error.
If you rearrange the subtitles so an entire line of text is a single line in the file, this will fix it.


Tagline

Issue #94
It should be all good to go.

@jmripper Fix merge conflict and then I will review!

@erinkelley27  should be all good to go now!

@erinkelley27 yep I can add it in

@jmripper What is going on with the font here?

@erinkelley27 Not sure I understand...I copied the exact front over from the mock and its pulling in the font Playfair Display from the CSS variable?

Here's the mobile version. I think the subtitle needs to be Raleway.

@erinkelley27 that's super weird! This is what it looks like for me.


Blank lines output when translating file

I just started using trans a few minutes ago. Initially, it worked fine. But for some reason, it just stopped all of a sudden. Here's the output you asked for in a previous similar problem:
➜ trans -D fr: dejeuner
  31 bytes > HTTP/1.1 429 Too Many Requests
  36 bytes > Date: Mon, 25 May 2020 15:45:03 GMT
  17 bytes > Pragma: no-cache
  39 bytes > Expires: Fri, 01 Jan 1990 00:00:00 GMT
  51 bytes > Cache-Control: no-store, no-cache, must-revalidate
  24 bytes > Content-Type: text/html
  30 bytes > Server: HTTP server (unknown)
  21 bytes > Content-Length: 2189
  20 bytes > X-XSS-Protection: 0
  18 bytes > Connection: close
   1 bytes >
  63 bytes > <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
   6 bytes >
 385 bytes > <meta http-equiv="content-type" content="text/html; charset=utf-8"><meta name="viewport" content="initial-scale=1"><title>http://translate.googleapis.com/translate_a/single?client=gtx&ie=UTF-8&oe=UTF-8&dt=bd&dt=ex&dt=ld&dt=md&dt=rw&dt=rm&dt=ss&dt=t&dt=at&dt=gt&dt=qca&sl=fr&tl=en&hl=en&q=dejeuner</title>
 177 bytes >
  30 bytes >

  68 bytes >

  29 bytes >

 222 bytes > Our systems have detected unusual traffic from your computer network.  Please try your request again later.  Why did this happen?

 116 bytes >

 730 bytes > This page appears when Google automatically detects requests coming from your computer network which appear to be in violation of the Terms of Service. The block will expire shortly after those requests stop.

This traffic may have been sent by malicious software, a browser plug-in, or a script that sends automated requests.  If you share your network connection, ask your administrator for help — a different computer using the same IP address may be responsible.  Learn more

Sometimes you may see this page if you are using advanced terms that robots are known to use, or sending requests very quickly.
  10 bytes >

   0 bytes >
 310 bytes > IP address: <IP_ADDRESS>
Time: 2020-05-25T15:45:03Z
URL: http://translate.googleapis.com/translate_a/single?client=gtx&ie=UTF-8&oe=UTF-8&dt=bd&dt=ex&dt=ld&dt=md&dt=rw&dt=rm&dt=ss&dt=t&dt=at&dt=gt&dt=qca&sl=fr&tl=en&hl=en&q=dejeuner

   6 bytes >

   6 bytes >

   7 bytes >
   7 bytes >
content = '''
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

<meta http-equiv="content-type" content="text/html; charset=utf-8"><meta name="viewport" content="initial-scale=1"><title>http://translate.googleapis.com/translate_a/single?client=gtx&ie=UTF-8&oe=UTF-8&dt=bd&dt=ex&dt=ld&dt=md&dt=rw&dt=rm&dt=ss&dt=t&dt=at&dt=gt&dt=qca&sl=fr&tl=en&hl=en&q=dejeuner</title>

Our systems have detected unusual traffic from your computer network.  Please try your request again later.  Why did this happen?

This page appears when Google automatically detects requests coming from your computer network which appear to be in violation of the Terms of Service. The block will expire shortly after those requests stop.

This traffic may have been sent by malicious software, a browser plug-in, or a script that sends automated requests.  If you share your network connection, ask your administrator for help — a different computer using the same IP address may be responsible.  Learn more

Sometimes you may see this page if you are using advanced terms that robots are known to use, or sending requests very quickly.

IP address: <IP_ADDRESS>
Time: 2020-05-25T15:45:03Z
URL: http://translate.googleapis.com/translate_a/single?client=gtx&ie=UTF-8&oe=UTF-8&dt=bd&dt=ex&dt=ld&dt=md&dt=rw&dt=rm&dt=ss&dt=t&dt=at&dt=gt&dt=qca&sl=fr&tl=en&hl=en&q=dejeuner

'''
tokens = ["<!DOCTYPE", "html", "PUBLIC", "\"-//W3C//DTD HTML 4.01 Transitional//EN\"", ">", "", "<meta", "http-equiv=", "\"content-type\"", "content=", "\"text/html; charset=utf-8\"", "><meta", "name=", "\"viewport\"", "content=", "\"initial-scale=1\"", "><title>http:", "", "
", "

", "
", "Our", "systems", "have", "detected", "unusual", "traffic", "from", "your", "computer", "network", ".", "Please", "try", "your", "request", "again", "later", ".", "Why", "did", "this", "happen?

", "
", "This", "page", "appears", "when", "Google", "automatically", "detects", "requests", "coming", "from", "your", "computer", "network", "which", "appear", "to", "be", "in", "violation", "of", "the", "Terms", "of", "Service", ".", "The", "block", "will", "expire", "shortly", "after", "those", "requests", "stop", ".", "

This", "traffic", "may", "have", "been", "sent", "by", "malicious", "software", ",", "a", "browser", "plug-in", ",", "or", "a", "script", "that", "sends", "automated", "requests", ".", "If", "you", "share", "your", "network", "connection", ",", "ask", "your", "administrator", "for", "help", "&mdash", "

", "IP", "address:", "67.190", ".84", ".172", "
Time:", "2020", "-0", "5", "-25", "T", "15", ":", "45", ":", "0", "3", "Z
URL:", "http:", "
", "
", "", ""]
ast = {
"0"	"software"
"1"	"plug-in"
"2"	"connection"
"3"	""
}
-- display original text & phonetics

-- display major translation & phonetics

Running without the -D flag yields nothing similarly:
➜ trans fr: dejeuner

Reading through the actual message, I see:
Our systems have detected unusual traffic from your computer network.  Please try your request again later.  Why did this happen?

and
This page appears when Google automatically detects requests coming from your computer network which appear to be in violation of the Terms of Service. The block will expire shortly after those requests stop.

This traffic may have been sent by malicious software, a browser plug-in, or a script that sends automated requests.  If you share your network connection, ask your administrator for help — a different computer using the same IP address may be responsible.  Learn more

Sometimes you may see this page if you are using advanced terms that robots are known to use, or sending requests very quickly.

I'm trying to translate an entire document, so I'm not sure if there's a better way of doing this. The file is 1380 lines long (though with a lot of blank lines in it). The command I'm using is trans :ru file://VSTGRusOCR.txt -o VSTGRusOCR_trans.txt.

Same for me, although I'm translating 10 requests for 10 words and 10-15 seconds inbetween requests. Maybe it's a regional issue with Google translate API?

Same for me, I failed to get any kind of output, for any test input.

Is it a google issue? I kept running into this problem when my input file is more than 200 lines etc.

It happens when you try to translate same word again and again (4-6 times for me). One way to fix this is by using vpn.


Error with running external language model (Transformer XL)

Hi every body,
I want to use external language model for ASR. according to documentation found in openseq2deq  page, it must be launched in 2 steps:

run inference to dump logits to pickle.
run beam search decoder with specific parameters.

I could successfully dump logits to pickle in step 1
But when i run scripts/decoder.py it gets me this Error:
No module named 'ctc_decoders'
I search a lot to find this script or module but no result!!
any help please.
Thank you
Arash

Can you please run ./scripts/install_decoders.sh?

Thank you @vsl9  I'll run it.

@arashdehghani Hello, have you run ./scripts/install_decoders.sh ?And have you fixed the problem above?
After I run  ./scripts/install_decoders.sh, I still got the same error no module named 'ctc_decoders' as you.
If you solved it, please tell the result ,thank you

Can you please run ./scripts/install_decoders.sh?
@vsl9 Hi, I have run ./scripts/install_decoders.sh but still get the same error with @arashd
File "scripts/decode.py", line 11, in
from ctc_decoders import Scorer
ImportError: No module named 'ctc_decoders'

Could you please tell the solutions, is there any step I forgot to run , or should i change scripts/decode.py ?

Hi @xw1324832579  this script is expired.
@vsl9  can you please update it for other OpenSeq2Seq users?

Can you upload the log from install_decoders.sh? Are you sure it completed successfully?

Hi @blisc  this is the log i get from running install_decoders.sh:
`arman@arman-pc:~/Projects/ASR/test_english/OpenSeq2Seq-master$ ./scripts/install_decoders.sh

git clone https://github.com/PaddlePaddle/DeepSpeech
Cloning into 'DeepSpeech'...
remote: Enumerating objects: 18, done.
remote: Counting objects: 100% (18/18), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 2448 (delta 4), reused 9 (delta 1), pack-reused 2430
Receiving objects: 100% (2448/2448), 915.03 KiB | 596.00 KiB/s, done.
Resolving deltas: 100% (1572/1572), done.
cd DeepSpeech
git checkout a76fc69
Note: checking out 'a76fc69'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b
HEAD is now at a76fc69 Merge pull request #247 from ericxk/patch-1

cd ..
mv DeepSpeech/decoders/swig_wrapper.py DeepSpeech/decoders/swig/ctc_decoders.py
mv DeepSpeech/decoders/swig ./decoders
rm -rf DeepSpeech
cd decoders
sed -i s/.decode('utf-8')//g ctc_decoders.py
sed -i s/.decode("utf-8")//g ctc_decoders.py
sed -i s/name='swig_decoders'/name='ctc_decoders'/g setup.py
sed -i s/-space_prefixes[i]->approx_ctc/space_prefixes[i]->score/g decoder_utils.cpp
sed -i s/py_modules=['swig_decoders']/py_modules=['ctc_decoders', 'swig_decoders']/g setup.py
chmod +x setup.sh
./setup.sh
Cloning into 'kenlm'...
remote: Enumerating objects: 13149, done.
remote: Total 13149 (delta 0), reused 0 (delta 0), pack-reused 13149
Receiving objects: 100% (13149/13149), 5.24 MiB | 581.00 KiB/s, done.
Resolving deltas: 100% (7533/7533), done.

Download and extract openfst ...
--2019-07-01 07:33:09--  http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz
Resolving www.openfst.org (www.openfst.org)... <IP_ADDRESS>
Connecting to www.openfst.org (www.openfst.org)|<IP_ADDRESS>|:80...
`
it seems the download links are out of date. we try to use other existing download links instead but it doesn't work!

Hi @blisc this is the log i get from running install_decoders.sh:
`arman@arman-pc:~/Projects/ASR/test_english/OpenSeq2Seq-master$ ./scripts/install_decoders.sh

git clone https://github.com/PaddlePaddle/DeepSpeech
Cloning into 'DeepSpeech'...
remote: Enumerating objects: 18, done.
remote: Counting objects: 100% (18/18), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 2448 (delta 4), reused 9 (delta 1), pack-reused 2430
Receiving objects: 100% (2448/2448), 915.03 KiB | 596.00 KiB/s, done.
Resolving deltas: 100% (1572/1572), done.
cd DeepSpeech
git checkout a76fc69
Note: checking out 'a76fc69'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b
HEAD is now at a76fc69 Merge pull request #247 from ericxk/patch-1

cd ..
mv DeepSpeech/decoders/swig_wrapper.py DeepSpeech/decoders/swig/ctc_decoders.py
mv DeepSpeech/decoders/swig ./decoders
rm -rf DeepSpeech
cd decoders
sed -i s/.decode('utf-8')//g ctc_decoders.py
sed -i s/.decode("utf-8")//g ctc_decoders.py
sed -i s/name='swig_decoders'/name='ctc_decoders'/g setup.py
sed -i s/-space_prefixes[i]->approx_ctc/space_prefixes[i]->score/g decoder_utils.cpp
sed -i s/py_modules=['swig_decoders']/py_modules=['ctc_decoders', 'swig_decoders']/g setup.py
chmod +x setup.sh
./setup.sh
Cloning into 'kenlm'...
remote: Enumerating objects: 13149, done.
remote: Total 13149 (delta 0), reused 0 (delta 0), pack-reused 13149
Receiving objects: 100% (13149/13149), 5.24 MiB | 581.00 KiB/s, done.
Resolving deltas: 100% (7533/7533), done.

Download and extract openfst ...
--2019-07-01 07:33:09-- http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz
Resolving www.openfst.org (www.openfst.org)... <IP_ADDRESS>
Connecting to www.openfst.org (www.openfst.org)|<IP_ADDRESS>|:80%7C<IP_ADDRESS>%7C:80)... failed: Connection timed out.
Retrying.
--2019-07-01 07:35:21-- (try: 2) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz
Connecting to www.openfst.org (www.openfst.org)|<IP_ADDRESS>|:80%7C<IP_ADDRESS>%7C:80)... failed: Connection timed out.
Retrying.
--2019-07-01 07:37:33-- (try: 3) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz
Connecting to www.openfst.org (www.openfst.org)|<IP_ADDRESS>|:80%7C<IP_ADDRESS>%7C:80)... failed: Connection timed out.
Retrying.
--2019-07-01 07:39:47-- (try: 4) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz
Connecting to www.openfst.org (www.openfst.org)|<IP_ADDRESS>|:80%7C<IP_ADDRESS>%7C:80)... failed: Connection timed out.
Retrying.
--2019-07-01 07:42:01-- (try: 5) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz
Connecting to www.openfst.org (www.openfst.org)|<IP_ADDRESS>|:80%7C<IP_ADDRESS>%7C:80)... failed: Connection timed out.
Retrying.
--2019-07-01 07:44:17-- (try: 6) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz
Connecting to www.openfst.org (www.openfst.org)|<IP_ADDRESS>|:80%7C<IP_ADDRESS>%7C:80)... failed: Connection timed out.
Retrying.
--2019-07-01 07:46:34-- (try: 7) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz
Connecting to www.openfst.org (www.openfst.org)|<IP_ADDRESS>|:80%7C<IP_ADDRESS>%7C:80)... failed: Connection timed out.
Retrying.
--2019-07-01 07:48:50-- (try: 8) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz
Connecting to www.openfst.org (www.openfst.org)|<IP_ADDRESS>|:80%7C<IP_ADDRESS>%7C:80)... failed: Connection timed out.
Retrying.
--2019-07-01 07:51:08-- (try: 9) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz
Connecting to www.openfst.org (www.openfst.org)|<IP_ADDRESS>|:80%7C<IP_ADDRESS>%7C:80)... failed: Connection timed out.
Retrying.
--2019-07-01 07:53:26-- (try:10) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz
Connecting to www.openfst.org (www.openfst.org)|<IP_ADDRESS>|:80%7C<IP_ADDRESS>%7C:80)... failed: Connection timed out.
Retrying.
--2019-07-01 07:55:46-- (try:11) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz
Connecting to www.openfst.org (www.openfst.org)|<IP_ADDRESS>|:80%7C<IP_ADDRESS>%7C:80)... failed: Connection timed out.
Retrying.
--2019-07-01 07:58:06-- (try:12) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz
Connecting to www.openfst.org (www.openfst.org)|<IP_ADDRESS>|:80%7C<IP_ADDRESS>%7C:80)... failed: Connection timed out.
Retrying.
--2019-07-01 08:00:31-- (try:13) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz
Connecting to www.openfst.org (www.openfst.org)|<IP_ADDRESS>|:80%7C<IP_ADDRESS>%7C:80)... failed: Connection timed out.
Retrying.
--2019-07-01 08:03:01-- (try:14) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz
Connecting to www.openfst.org (www.openfst.org)|<IP_ADDRESS>|:80%7C<IP_ADDRESS>%7C:80)... failed: Connection timed out.
Retrying.
--2019-07-01 08:05:25-- (try:15) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz
Connecting to www.openfst.org (www.openfst.org)|<IP_ADDRESS>|:80%7C<IP_ADDRESS>%7C:80)... failed: Connection timed out.
Retrying.
--2019-07-01 08:07:49-- (try:16) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz
Connecting to www.openfst.org (www.openfst.org)|<IP_ADDRESS>|:80%7C<IP_ADDRESS>%7C:80)... failed: Connection timed out.
Retrying.
--2019-07-01 08:10:12-- (try:17) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz
Connecting to www.openfst.org (www.openfst.org)|<IP_ADDRESS>|:80%7C<IP_ADDRESS>%7C:80)... failed: Connection timed out.
Retrying.
--2019-07-01 08:13:05-- (try:18) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz
Connecting to www.openfst.org (www.openfst.org)|<IP_ADDRESS>|:80%7C<IP_ADDRESS>%7C:80)... failed: Connection timed out.
Retrying.
--2019-07-01 08:15:29-- (try:19) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz
Connecting to www.openfst.org (www.openfst.org)|<IP_ADDRESS>|:80%7C<IP_ADDRESS>%7C:80)... failed: Connection timed out.
Retrying.
--2019-07-01 08:17:49-- (try:20) http://www.openfst.org/twiki/pub/FST/FstDownload/openfst-1.6.3.tar.gz
Connecting to www.openfst.org (www.openfst.org)|<IP_ADDRESS>|:80%7C<IP_ADDRESS>%7C:80)... failed: Connection timed out.
Giving up.
tar (child): openfst-1.6.3.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
Cloning into 'ThreadPool'...
remote: Enumerating objects: 82, done.
remote: Total 82 (delta 0), reused 0 (delta 0), pack-reused 82
Unpacking objects: 100% (82/82), done.
Install decoders ...
sh: 1: swig: not found
running install
error: can't create or remove files in install directory
The following error occurred while trying to add or remove files in the
installation directory:
[Errno 13] Permission denied: '/usr/local/lib/python3.6/dist-packages/test-easy-install-23436.write-test'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
/usr/local/lib/python3.6/dist-packages/

Perhaps your account does not have write access to this directory? If the
installation directory is a system-owned directory, you may need to sign in
as the administrator or "root" account. If you do not have administrative
access to this machine, you may wish to choose a different installation
directory, preferably one that is listed in your PYTHONPATH environment
variable.
For information on other options, you may wish to consult the
documentation at:
https://setuptools.readthedocs.io/en/latest/easy_install.html
Please make the appropriate changes for your system and try again.
`
it seems the download links are out of date. we try to use other existing download links instead but it doesn't work!

@arashdehghani I think you need to install swig before running install_decoders.sh

@arashdehghani, there are two issues:

the script can't download openfst-1.6.3.tar.gz; please check your internet connection, firewall settings and make sure that you can open http://www.openfst.org/
as @shiv6146 mentioned, please install swig: sudo apt-get install swig or conda install swig

@vsl9  @shiv6146  Thanks a lot for your helps.


Cannot reproduce Zero-shot Video-QA (MSVD)

As I mentioned in a previous issue, I am unable to reproduce the MSVD Video-QA results, which are significantly different from the reported results in the paper displayed below.

Although you suggested referring to this issue, it is NOT relevant to my case because I tested this using your codebase (and, I checked it more than twice).
I am sharing my inferenced results on MSVD Video-QA link predicted with the shared checkpoint you provided Video-LLaVA-7B.
Could you please share your predicted results on MSVD Video-QA as well?

Issue 37 did not encounter trouble on MSVD, perhaps you can refer to it.

If you're doing multi-GPU inference, it may be due to this line, which should not be commented out, otherwise q/a pairs are incorrectly aligned in the resulting json files:
https://github.com/PKU-YuanGroup/Video-LLaVA/blob/e93f4927eaa926ed8450b481fde95c994ed23d2d/videollava/eval/video/run_inference_video_qa.py#L122
I was getting a similar 33% accuracy, but after uncommenting that line I can reproduce the paper results (70%)

If you're doing multi-GPU inference, it may be due to this line, which should not be commented out, otherwise q/a pairs are incorrectly aligned in the resulting json files:
https://github.com/PKU-YuanGroup/Video-LLaVA/blob/e93f4927eaa926ed8450b481fde95c994ed23d2d/videollava/eval/video/run_inference_video_qa.py#L122
I was getting a similar 33% accuracy, but after uncommenting that line I can reproduce the paper results (70%)

Awesome!

If you're doing multi-GPU inference, it may be due to this line, which should not be commented out, otherwise q/a pairs are incorrectly aligned in the resulting json files:
https://github.com/PKU-YuanGroup/Video-LLaVA/blob/e93f4927eaa926ed8450b481fde95c994ed23d2d/videollava/eval/video/run_inference_video_qa.py#L122
I was getting a similar 33% accuracy, but after uncommenting that line I can reproduce the paper results (70%)

Should this be uncommented for single-GPU inference too? This sounds like a pretty major bug.


Tags Aren't Correct Width

Any idea what's going on here? Feel free to ask me questions for clarification. Using SnapKit for sizing / positioning of the tag fields. I have 4 separate tag fields on the same page one below the other.

Figured Out That I Needed To Call The Following Before Calling addTags().
view.setNeedsLayout()
view.layoutIfNeeded()

This could be done automatically. Thanks for opening an issue.

Actually, this is done automatically.


More specific error classes

I'd be wonderful to be able to dispatch errors by class rather than by checking message they pass.
Currently:
when 401
  raise StreamApiResponseException, error_message(response, 'Bad feed')
when 403
  raise StreamApiResponseException, error_message(response, 'Bad auth/headers')
when 404
  raise StreamApiResponseException, error_message(response, 'url not found')

Having:
UrlNotFound = Class.new(StreamApiResponseException)

when 404
  raise UrlNotFound, error_message(response, 'url not found')

would be backwards compatible and enable responding differently when matching Stream::UrlNotFound === error in application codebase.

fixed by #142, thanks @groyoh


KAFKA-5167: Release state locks in case of failure

Call for review @enothereska @bbejeck @guozhangwang @dguy

This must be back ported to 0.11.0 branch.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5746/
Test PASSed (JDK 8 and Scala 2.12).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5760/
Test PASSed (JDK 7 and Scala 2.11).

Updated. It did leave createStreamTask and createStandbyTask as protected for now and created a JIRA (https://issues.apache.org/jira/browse/KAFKA-5551). Both method are use in many test cases and it's a larger refactoring.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5864/
Test PASSed (JDK 7 and Scala 2.11).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5849/
Test PASSed (JDK 8 and Scala 2.12).

Can we do a system test of this? Sounds pretty critical.

@enothereska What do we gain by system tests that is not covered by the unit tests? If there is value, I am happy to add a system test.

I wish all unit tests caught all problems. I find it useful to kick a system test in the background. Just to make it clear: I don't mean all system tests, just kafkatest/tests/streams/

I believe @enothereska is suggesting that we run the system tests, not write a new one.

Ah. "do a test" sounded like adding a new test.
Did trigger https://jenkins.confluent.io/job/system-test-kafka-branch-builder/951/

One EOS test failed (we know it's flaky). Did retrigger. Passed:
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/953/

LGTM.

Merged to trunk and 0.11.0.


RunEx: NullReferenceException with "parameters" parameter null

Hi,
First of all, thanks for this project.
The RunEx method takes a "parameters" parameter string[], but crashes if it's null. If you've read the documentation for RegisteredTask.RunEx this is very confusing, as null is a valid value (meaning "no parameters).
Here's the first line that uses the parameter without checking it for null: https://github.com/dahall/TaskScheduler/blob/2b2e8eb9932fac8111d4021bf348eb1c2125acd2/TaskService/Task.cs#L1292
Thanks again!

The RunEx method uses a params parameter for the last argument that accepts 0 or more strings. If you are passing null then you are effectively passing it as the first string value parameter, which is not valid. I have put in some error checking to check for that condition and provide a more descriptive exception.


NCL_native_1.py

Closes #303
NCL Plot: https://www.ncl.ucar.edu/Applications/Images/native_1_lg.png
Current state of plot

Hi all. It looks like this was merged before I had a chance to fix those typos. Should I open a new PR for that or can I make those changes here?

I think you can click 'revert changes' and it should reopen the PR (or at least create a new one). If that doesn't work, I would just make a new PR with the changes and we can get it reviewed/merged again pretty quickly

Hi all. It looks like this was merged before I had a chance to fix those typos. Should I open a new PR for that or can I make those changes here?

I think you can click 'revert changes' and it should reopen the PR (or at least create a new one). If that doesn't work, I would just make a new PR with the changes and we can get it reviewed/merged again pretty quickly

Hi all. It looks like this was merged before I had a chance to fix those typos. Should I open a new PR for that or can I make those changes here?

Hi @hCraker you don't need to do anything since I merged it and then corrected those typos in the next merge of another PR. Check the code in main branch and you will see they are corrected. Let me know if I missed anything though

Hi @hCraker you don't need to do anything since I merged it and then corrected those typos in the next merge of another PR. Check the code in main branch and you will see they are corrected. Let me know if I missed anything though

@erogluorhan I see that you fixed the ones that @anissa111 pointed out. There is also a typo in line 12 for the word accommodate.

Hi @hCraker you don't need to do anything since I merged it and then corrected those typos in the next merge of another PR. Check the code in main branch and you will see they are corrected. Let me know if I missed anything though

@erogluorhan I see that you fixed the ones that @anissa111 pointed out. There is also a typo in line 12 for the word accommodate.

Yeah, I only corrected those Anissa found. Please create a simple new PR that handles typo corrections.


Mark all as read context menu

Fixes #2158
The reason why it's failing to show context menu for "Mark All as Read" in accounts is because its expecting a content view and since it doesn't have one from UIInteraction its failing.
So I did some refactoring and added content view from both index paths or straight away from any UIView.
I did test all the contextual menus on MasterFeedController just to make sure I don't break anything.
Screenshot

Thanks @rizwankce.  Nice work!


Fixed the default Stratis REST API port

Changed the default API port for the Stratis daemon from 37220 to 37221. It still uses the 37220 in Bitcoin mode.

Thanks for redoing this @andrasfuchs !


Nick: Watch interm.in

Nick requests the watch of the watch_keyword interm\.in. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

interm\.in has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by tripleee in Charcoal HQ


Dynamic update support

It looks like that the context is not automatically updated while editing. Until I reload the file (so that GPS query can refresh), nvim-gps fails to show the context for newly added scopes.

It would be great if we can support dynamic update of the context information powered by treesitter!

It should update dynamically. As soon as you exit insert mode, nvim-gps asks treesitter for new information to show. It seems that for some reason your treesitter isn't providing latest information 🤔
https://user-images.githubusercontent.com/43147494/152925448-9175dd66-e895-4ca2-a2a6-c02015036ae1.mp4

Hmm interesting. I am new to treesitter, so have no concrete ideas yet on how to debug this. FYI, my treesitter parser and plugins are up-to-date (and neovim is nightly-master). I have nothing special in my config, and :checkhealth nvim-treesitter seems fine.
The TSPlayground window also doesn't update and gets out of sync, so this might be related to the treesitter or its config itself.

@SmiteshP Is treesitter-playground as well automatically updating as you change the text?

@SmiteshP Is treesitter-playground as well automatically updating as you change the text?

Yep, playground should also update on the fly. So this means the issue is definitely with treesitter. You could try opening a issue on nvim-treesitter repo, they should be able to help.

Surprisingly, nvim-treesitter/nvim-treesitter#2492 says that currently the module highlight is required to make on-the-fly update work properly.

@SmiteshP So from the discussion nvim-treesitter/nvim-treesitter#2492 (-> nvim-treesitter/playground#64), it seems calling the parser to update the syntax tree for the current buffer (i.e., parser:parse(buf, lang)) is the responsibility of downstream treesitter plugins. Apparently both playground and nvim-gps missed that (and many other treesitter plugins probably also aren't aware of that).
I can confirm that adding the following lines somewhere in M.get_location() makes the syntax tree is updated, despite highlight { enabled = false }.
	-- Request treesitter parser to update the syntax tree.
	local parser = ts_parsers.get_parser(vim.fn.bufnr(), filelang)
	parser:parse()

But this would be too excessive, frequent call because get_location() is usually called from statusline. A better way would be having some autocmd (like InsertLeave autocmd) set up, but I'm not sure this is something nvim-gps is responsible for or users are supposed to have it configuring treesitter. Either way, if I get convinced, I will make a PR for nvim-gps.

@SmiteshP So the conclusion is that ensuring the parse tree is up-to-date is the module's responsibility, which is however often subsumed when the treesitter highlight module is enabled. I submitted a PR #70 that fixes this bug, even when the highlight module is absent.

Oh interesting! I did use to have an InsertLeave autocommand to update the parser but at some point I realised everything works fine without it too, so I removed it. 😆


Start a documentation site

Probably using Eleventy? At least that's the static site generator I'm the most familiar with and have had (mostly) good experience.

yes yes yes


outbound-networking: More permissive allow-all

Two changes here that were regressions in LHC's understanding of allow-all:

insecure:allow-all allows http and https
*:* implies self

Should backport for 2.0 if merged

Accidentally pushed to main...


Fluid Cows Spawn on pollition blocks from Pollution of the Realms

Running this mod with the mod Polution of the Realm, Fluid cows spawn on the pollution blocks generated by this mod, but since they have no hitbox, the cows just fall to their death from y =196, so there is a constant rain of falling, dying cows when running these mods together

so there is a constant rain of falling, dying cows when running these mods together
AHHAHAHAHAH


Enable support for boolean values

Describe the bug
Note: I'm using typescript for the configuration, which is supported in tailwindcss v3.3.2
If i have a tailwind configuration with a boolean value, then I get the following error message.

/.../.pnpm/tailwindcss-themer@3.1.0_tailwindcss@3.3.2/node_modules/tailwindcss-themer/lib/utils/themeUtils.js:49
                        throw new Error(`unusable value found in config on path "${pathSteps.join('.')}"`);

It happens when add the following configuration to tailwind theme
theme: {
  container: {
    center: true,
  },
}

Your minimal, reproducible example
https://stackblitz.com/edit/vitejs-vite-d1g2vm?file=tailwind.config.ts
Steps to reproduce

Go to tailwind.config.ts in provided stackblitz link
Have a look at the center: true. It break because the only allowed values are string or numbers

Expected behavior
I would expect the center: true will work with booleans, so it works with the tailwindcss documentation
Link to tailwindcss documentation for centering containers:
https://tailwindcss.com/docs/container#centering-by-default
How often does this bug happen?
None
Screenshots or Videos
No response
Platform

all platforms

tailwindcss-themer version
v3.1.0
Additional context
No response

Thanks for opening this!
I see your issue, and unfortunately, I don't think that value is something that this plugin can configure. This plugin can only theme values that get injected into css rules.
When it comes to container: true/false this configures tailwind to generate
/* when false or undefined */
.container {
  width: 100%
}

/* when true */
.container {
  width: 100%;
  margin-right: auto;
  margin-left: auto
}

Because this config value isn't a single css rule, this plugin doesn't support that. I can update the docs to reflect accordingly.
I'm open to ideas and pull requests if you think otherwise though! Let me know what you think.
For now, I'll close this as expected behavior.


Implement MSFTextStyle for macOS

Platforms Impacted

[ ] iOS
[x] macOS

Description of changes
At WWDC 2020, Apple revealed that they are creating default text styles for macOS.
New HIG Guidelines here, and developer documentation here
Let's add a new enum MSFTextStyle (similar to the one we already have on iOS) for macOS, which just calls the Apple API for macOS 11+, and mimics the style for older OS's.
Let's also add textStyle as a property for existing controls, and as a param to their initializers.
Apple's WWDC Talk

Our new Text Styles

Pull request checklist
This PR has considered:

[ ] Light and Dark appearances
[ ] VoiceOver and Keyboard Accessibility
[ ] Internationalization and Right to Left layouts
[ ] Different resolutions (1x, 2x, 3x)
[ ] Size classes and window sizes (iPhone vs iPad, notched devices, multitasking, different window sizes, etc)

Microsoft Reviewers: Open in CodeFlow

Well that's fun, CI failed. Perhaps because this depends on Xcode 12 beta because I reference macOS 11.0 API's?

Worth noting that these are the styles in FURN, and we should map these eventually...
https://developer.microsoft.com/en-us/fluentui#/controls/crossplatform/text

Closing as this in't needed at this time


duckdb: run unit tests

Description of changes
This PR runs duckdb's test suite as an install check.
Running the tests takes about 3m30s on my machine, so I'm happy to close this
PR if that adds too much time to the build.
Unfortunately, it doesn't look like there's a way to get more output during the
test phase than what I've already got here, which isn't much.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

On macos 10.15:
builder for '/nix/store/aszdg3bx3pn5kpsvky4s03xzvhacx04n-duckdb-0.3.4.drv' failed with exit code 134; last 10 log lines:
  -- Installing: /nix/store/jc88vdpkq2370mqikiv9cq55b2gfal61-duckdb-0.3.4/lib/libduckdb_fastpforlib.a
  post-installation fixup
  strip is /nix/store/fjd0p8rk93db76l7wnk2jxh32x9yadfw-cctools-binutils-darwin-949.0.1/bin/strip
  stripping (with command strip and flags -S) in /nix/store/jc88vdpkq2370mqikiv9cq55b2gfal61-duckdb-0.3.4/lib  /nix/store/jc88vdpkq2370mqikiv9cq55b2gfal61-duckdb-0.3.4/bin
  patching script interpreter paths in /nix/store/jc88vdpkq2370mqikiv9cq55b2gfal61-duckdb-0.3.4
  running install tests
  dyld: Library not loaded: libduckdb.dylib
    Referenced from: /private/tmp/nix-build-duckdb-0.3.4.drv-0/source/build/test/unittest
    Reason: image not found
  /nix/store/3kqqqy9gkl25r3g631yikv08l6wgsblx-stdenv-darwin/setup: line 1383: 62109 Abort trap: 6           $PWD/test/unittest exclude:'*test_slow' exclude:'Test file buffers for reading/writing to file' exclude:'[test_slow]' exclude:'test/common/test_cast_hugeint.test' exclude:'test/sql/copy/csv/test_csv_remote.test' exclude:'test/sql/copy/parquet/test_parquet_remote.test'

Setting
  export DYLD_LIBRARY_PATH="$out/lib:$DYLD_LIBRARY_PATH"

before starting the tests solves it for me.

Interestingly, darwin-aarch64 gives me
Filters: exclude:*test_slow exclude:Test file buffers for reading/writing to file exclude:[test_slow] exclude:test/common/test_cast_hugeint.test exclude:test/sql/copy/csv/test_csv_remote.test exclude:test/sql/copy/parquet/test_parquet_remote.test test/sql/aggregate/aggregates/test_kurtosis.test test/sql/aggregate/aggregates/test_skewness.test test/sql/function/list/aggregates/skewness.test
No test cases matched 'exclude:*test_slowexclude:Test file buffers for reading/writing to fileexclude:[test_slow]exclude:test/common/test_cast_hugeint.testexclude:test/sql/copy/csv/test_csv_remote.testexclude:test/sql/copy/parquet/test_parquet_remote.testtest/sql/aggregate/aggregates/test_kurtosis.testtest/sql/aggregate/aggregates/test_skewness.testtest/sql/function/list/aggregates/skewness.test'
===============================================================================
No tests ran

@risicle Can you give this another go on the darwins?

I don't have a darwin machine spun up at the moment - let's let ofborg do the work...


ForceRemap issue with ntdll

I'm trying to remap ntdll in my process. The issue is specific to x86 process on an x64 host and specific to ntdll (you'll see why).
Trying to call any Nt* func. Example here: (left on a real ntdll, right is the mapped one)

Following the pointer in the edx here:

Now you can see what's the issue, the pointer to the wow64cpu.dll is not relocated properly, however I've got no idea what should relocate it at all. Is it the job of the loader to relocate it at all? I know I could manually just overwrite and "hack" it, but I'm curious what should do it and if it's possible to fix it without specifying that "IF ntdll THEN patch THAT". Or is it something very custom for ntdll?
My mapping code if it helps:
	Process thisProc;
	thisProc.Attach(GetCurrentProcessId());

	auto callback = [](CallbackType, void*, Process&, const ModuleData&)
	{
		return LoadData(MT_Manual, Ldr_None);
	};

	auto image = thisProc.mmap().MapImage(L"C:\\Windows\\SysWOW64\\ntdll.dll", ForceRemap, callback);
	if (!image)
	{
		std::wcout << L"Mapping failed with error 0x" << std::hex << image.status
			<< L". " << Utils::GetErrorDescription(image.status) << std::endl << std::endl;
	}
	else
		std::wcout << L"Successfully mapped, unmapping\n";

	// Now just find the export you need and call it.

@DarthTon Do you have any idea what should be the responsible for this?

Wow64Transition is initialized once by wow64.dll!CpupResolveReverseImports during process initialization. So I'm afraid you need to update the address manually and a special case for ntdll mapping.

Ahh, so that's what resolves it. Thank you! I'm still amazed by your knowledge on PE internals :D