[fix][broker] Fix recentlyJoinedConsumers to address the out-of-order issue

Motivation
The Key_Shared subscription type has the following issues.

Key_Shared subscription has out-of-order cases because of the race condition of the recently joined consumers feature.
Consider the following flow.

Assume that the current read position is 1:6 and the recently joined consumers is empty.
Called OpReadEntry#internalReadEntriesComplete from thread-1.
Then, the current read position is updated to 1:11 (Messages from 1:6 to 1:10 have yet to be dispatched to consumers).
Called PersistentStickyKeyDispatcherMultipleConsumers#addConsumer from thread-2.
Then, the new consumer is stored to recently joined consumers with read position 1:11.
Called PersistentDispatcherMultipleConsumers#trySendMessagesToConsumers from thread-5.
Then, messages from 1:6 to 1:10 are dispatched to consumers. From the recently joined consumers feature, the new consumer can receive messages from 1:6 to 1:10. However, it is not expected.
For example, if existing consumers have some unacked messages and disconnect, it causes out of order in some cases.

Key_Shared subscription has a redundant process.
The stuckConsumers feature was introduced from https://github.com/apache/pulsar/pull/7553 .
However, it can't fix the issue entirely because it doesn't consider the range changes.
After this commit, https://github.com/apache/pulsar/pull/10762 was introduced. It fixes the issue.

Modifications

Store last sent position instead of read position to recently joined consumers.
Updating read position, then dispatching messages, and adding new consumer are not exclusive.
I used the last send position to get a position without any new exclusion features.

Keep the last sent position per key.
When I introduced the last sent position, I noticed a new concern. Consider the following flow.

Assume that the entries has the following messages,

msg-1, key: key-a, position: 1:1
msg-2, key: key-a, position: 1:2
msg-3, key: key-a, position: 1:3
msg-4, key: key-b, position: 1:4
msg-5, key: key-b, position: 1:5
msg-6, key: key-b, position: 1:6
the dispatcher has two consumers (c1 messagesForC is 1, c2 messageForC is 1000), and the selector will return c1 if key-a and c2 if key-b.

Send msg-1 to c1 and msg-4 - msg-6 to c2.

So, the current last sent position is 1:6.
c1 never acknowledge msg-1.

Add new consumer c3, and the selector will return c3 if key-a.
Send msg-2 - msg-3 to c3 because 1:2 and 1:3 are less than the last sent position, 1:6.
Disconnect c1.
Send msg-1 to c3.
Now, c3 receives messages without expected order about key-a.

To avoid this issue, I introduce the last sent positions feature.

Remove redundant features due to the addition of the last sent positions feature.

Remove this condition because the last sent positions feature restricts part of messages.
New consumers can receive any new messages with the message key that is not in the last sent position.
https://github.com/apache/pulsar/blob/35e9897742b7db4bd29349940075a819b2ad6999/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java#L323-L336

Remove this behavior because the last sent positions has positions per key.
If some key is stuck at a certain point in time, from that point, new consumers store the same information about the key to the recentlyJoinedConsumers.
https://github.com/apache/pulsar/blob/35e9897742b7db4bd29349940075a819b2ad6999/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java#L365-L399

Remove this behavior because this calculation is moved to LastSentPositions#compareToLastSentPosition.
https://github.com/apache/pulsar/blob/35e9897742b7db4bd29349940075a819b2ad6999/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java#L400-L408

Remove the stuckConsumers feature.

Reconstruct some consumer stats fields related to Key_Shared.
This is a breaking change. However, if this PR is merged, the existing field which is removed in this PR is no longer needed.

Verifying this change

[ ] Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

Added unit test to ensure the behavior of last sent positions feature works properly

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

[ ] Dependencies (add or upgrade a dependency)
[ ] The public API
[ ] The schema
[ ] The default values of configurations
[ ] The threading model
[ ] The binary protocol
[ ] The REST endpoints
[ ] The admin CLI options
[x] The metrics
[ ] Anything that affects deployment

Documentation

[ ] doc
[x] doc-required
[ ] doc-not-needed
[ ] doc-complete

Matching PR in forked repository
PR in forked repository: https://github.com/equanz/pulsar/pull/1

@poorbarcode
Thank you for your clarification.

[1] Yes.
[2] Partially yes. However, I think the main point of this PR is the change in the timing of getting the position.
https://github.com/apache/pulsar/blob/27880e42c03eabd02d3d5f93f97988f6d03a5d90/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java#L450-L458

@equanz

Currently, the position is the readPosition. The readPosition is updated when the read is completed, even if messages are not dispatched. In this PR, I will change the position to the last sent position.

Scenario-2

time
process: add consumer
process: delivery messages to client

1
read position is 1:6

2
Add new consumer into the selector

3

Read entries 1:6 ~ 1:10 complete

4

Set read position 1:11

5
Add the new consumer into recently joined consumers

6
(Highlight)The max read-position of the new consumer is 1:11, but the exact correct value is 1:6

7

Choose consumer by the selector

8

Delivery entries 1:6 ~ 1:10 to all consumers(includes old consumers and the new consumer)

This scenario is what you are saying, right?[Q-1]
If yes[Q-1], we have two solutions to fix it:

Use last send position instead of last read position(what this PR does)
Find a way to make the lock works for Scenario-1[scenario-1], and add a lock for set read position for Scenario-2.

I also think solution-1 is better, but declaring the data structure of lastSentPosition as a Map is an improvement, right[Q-2]?
If yes[Q-2], could you split this PR into two PR: one typed "fix" and another one typed "improve"? And I think the PR which typed "improve" should be submitted with a PIP because the mechanism of Key_Shared dispatcher is so complicated now we need PIPs to trace the changes of it.
[scenario-1]: https://github.com/apache/pulsar/pull/20179#pullrequestreview-1400545109

@poorbarcode

[Q-1] Yes.

[Q-2] When I created this PR, I introduced Map structure because it was necessary. For more details, please see the description("2. Keep the last sent position per key.").
I just noticed we could avoid introducing a Map structure if we implemented the last sent position feature, like the mark delete position and the individually deleted messages feature. In other words,

The position is already scheduled to be sent.
All positions less than or equal to it are already scheduled to be sent.
Manage individually sent positions to update the position as expected.

Consider the following flow.

Assume that the entries has the following messages,

msg-1, key: key-a, position: 1:1
msg-2, key: key-a, position: 1:2
msg-3, key: key-a, position: 1:3
msg-4, key: key-b, position: 1:4
msg-5, key: key-b, position: 1:5
msg-6, key: key-b, position: 1:6
the dispatcher has two consumers (c1 messagesForC is 1, c2 messageForC is 1000), and the selector will return c1 if key-a and c2 if key-b.

Send msg-1 to c1 and msg-4 - msg-6 to c2.

So, the current last sent position is 1:1 and the individually sent positions is [[1:3, 1:6]] (list of closed intervals. not list of list).
c1 never acknowledge msg-1.

scenario A

Add new consumer c3, and the selector will return c3 if key-a.
Can't send msg-2 - msg-3 to c3 because 1:2, and 1:3 are greater than the last sent position, 1:1.
Disconnect c1.
Send msg-1 - msg-3 to c3.Now, c3 receives messages with expected order about key-a.

scenario B

c1 messagesForC is back to 999.
Send msg-2 - msg-3 to c1

So, the current last sent position is 1:6, and the individually sent positions is [].

My motivation is not to improve Key_Shared but fix Key_Shared.
Therefore, if the current approach is not accepted as a fix, I introduce it and remove the feature by Map per key.

@eolivelli
Thank you for your comments. I'll check it.
By the way,  do you have any comments about the following concern from @poorbarcode ?

By the way, if there are a huge number of keys, will the constructor LastSentPositions cost too much memory?

If the current approach is not approved, I create the next PR as in https://github.com/apache/pulsar/pull/20179#issuecomment-1539314184 instead of addressing your comments in this PR.

@poorbarcode
What do you think about the following comment?
https://github.com/apache/pulsar/pull/20179#issuecomment-1539314184

I think we can't split easily between fix and improvement in the current approach.

For more details, please see the description("2. Keep the last sent position per key.").

I think we need to implement a new feature like https://github.com/apache/pulsar/pull/20179#issuecomment-1539314184 to fix without introducing Map structure.
My primary motivation is not to improve Key_Shared but fix Key_Shared.
Therefore, if the current approach is not approved as a fix, I introduce it and remove the feature by Map per key.

I opened a new PR https://github.com/apache/pulsar/pull/21953.

https://github.com/apache/pulsar/pull/21953 was merged.


Configure Netlify CMS

To make it easier for non devs to add or edit content, I want to try out netlify CMS.

Running Netflify on GitHub pages is not directly supported, as there is a backend that needs to be run on an actual server. We need to deploy to netlify to make it really easy, or find somewhere to host the backend.

So I tried deploying the site to netlify, in order to make it easy to use the netlify cms.
This was so easy and netlify really seems to be an awesome platform to host websites. I think this will be the way to go.
This even solves the issue of review branches with preview deployments just out of the box.
The only concern is, that the free version of netlify only supports 300 free minutes of build time per month. But since currently a build only takes 30s, this means about 600 commits per month. I doubt that this will be a limitation that hurts us any time soon.


TechDocs triggers vulnerable tags warnings in browser console

Expected Behavior
No browser warnings

Current Behavior
Browser console when viewing techdocs shows:

Your allowedTags option includes, style, which is inherently
vulnerable to XSS attacks. Please remove it from allowedTags.
Or, to disable this warning, add the allowVulnerableTags option
and ensure you are accounting for this risk.

Stacktrace
at Reader (webpack-internal:///../../plugins/techdocs/src/reader/components/Reader.tsx:43:19)
at article
at Content (webpack-internal:///../core/src/layout/Content/Content.tsx:53:3)
at div
at Page (webpack-internal:///../core/src/layout/Page/Page.tsx:45:17)
at TechDocsPageWrapper (webpack-internal:///../../plugins/techdocs/src/reader/components/TechDocsPageWrapper.tsx:32:3)
at TechDocsPage (webpack-internal:///../../plugins/techdocs/src/reader/components/TechDocsPage.tsx:31:90)
at Route (webpack-internal:///../../node_modules/react-router/index.js:34:673)
at Routes (webpack-internal:///../../node_modules/react-router/index.js:37:804)
at AppRoutes
at div
at SidebarPage (webpack-internal:///../core/src/layout/Sidebar/Page.tsx:56:89)
at Root (webpack-internal:///./src/components/Root/Root.tsx:100:17)
at Route (webpack-internal:///../../node_modules/react-router/index.js:34:673)
at Routes (webpack-internal:///../../node_modules/react-router/index.js:37:804)
at SignInPageWrapper (webpack-internal:///../core-api/src/app/App.tsx:245:18)
at Router (webpack-internal:///../../node_modules/react-router/index.js:36:23)
at BrowserRouter (webpack-internal:///../../node_modules/react-router-dom/index.js:63:23)
at AppRouter (webpack-internal:///../core-api/src/app/App.tsx:256:26)
at CssBaseline (webpack-internal:///../../node_modules/@material-ui/core/esm/CssBaseline/CssBaseline.js:65:31)
at WithStyles (webpack-internal:///../../node_modules/@material-ui/styles/esm/withStyles/withStyles.js:61:31)
at ThemeProvider (webpack-internal:///../../node_modules/@material-ui/styles/esm/ThemeProvider/ThemeProvider.js:42:24)
at AppThemeProvider (webpack-internal:///../core-api/src/app/AppThemeProvider.tsx:78:29)
at AppContextProvider (webpack-internal:///../core-api/src/app/AppContext.tsx:31:31)
at ApiProvider (webpack-internal:///../core-api/src/apis/ApiProvider.tsx:40:24)
at Provider (webpack-internal:///../core-api/src/app/App.tsx:191:25)
at App
at AppContainer (eval at ES6ProxyComponentFactory (webpack-internal:///../../node_modules/react-hot-loader/dist/react-hot-loader.development.js), <anonymous>:14:7)
at HotExportedApp (eval at ES6ProxyComponentFactory (webpack-internal:///../../node_modules/react-hot-loader/dist/react-hot-loader.development.js), <anonymous>:14:7)

Possible Solution

Steps to Reproduce

Start example app up and click view techdocs.
you don't need to actually have it configured

Step 1
Step 2
...

Context

Your Environment

NodeJS Version (v12):
Operating System and Version (e.g. Ubuntu 14.04):
Browser Information:

cc @emmaindal

@timja Thanks for reporting! 💯   My guess is that this warning is because we sanitize the html but allow the style tag to be able to use the generated styles from mkdocs. We will discuss this further!


Fix crashes in shadow server

@freerdp-bot would you mind test this PR and see if it compiles cleanly, thank you.

It looks good to me, +1.
@xhaakon it would be really great, if you could write a unitary test (complementing an existing one or creating a new one) that would verify your correction.

As hardening suggested. If you could add a unit test that reproduces this problem that would be great.

This also fixes the race condition I pasted at https://github.com/FreeRDP/FreeRDP/pull/2395#issuecomment-77742784

@bmiklautz I've extended TestSynchBarrier to cover also the case of barrier's early deletion. Please let me know if you'd rather prefer a test in some different format.

@freerdp-bot test


Flaky Test: TestClusterJoinAndReconnect/TestJoinLeave flake

Per https://app.circleci.com/pipelines/github/prometheus/alertmanager/3414/workflows/c8f0b621-4406-438c-96ee-be09c7a45d92/jobs/15139 seems to be flaky but really hard to reproduce.
It took me a few retries before I saw it failing:
🙇 -> go test -run "TestClusterJoinAndReconnect/TestJoinLeave" -count 300
--- FAIL: TestClusterJoinAndReconnect (0.02s)
    --- FAIL: TestClusterJoinAndReconnect/TestJoinLeave (0.00s)
        cluster_test.go:104:
                Error Trace:    /Users/gotjosh/go/src/github.com/prometheus/alertmanager/cluster/cluster_test.go:104
                Error:          Not equal:
                                expected: 2
                                actual  : 1
                Test:           TestClusterJoinAndReconnect/TestJoinLeave
FAIL
exit status 1
FAIL    github.com/prometheus/alertmanager/cluster      249.449s

This test fails pretty regularly during autopkgtest in the prometheus-alertmanager Debian package also. It seems to less reliable on non-amd64 archs. Also, what strikes me as curious is how quickly it fails. Something is failing very early, but the tests don't log enough detail to ascertain what.
arm64:
152s --- FAIL: TestClusterJoinAndReconnect (0.84s)
152s     --- PASS: TestClusterJoinAndReconnect/TestJoinLeave (0.80s)
152s     --- PASS: TestClusterJoinAndReconnect/TestReconnect (0.00s)
152s     --- PASS: TestClusterJoinAndReconnect/TestRemoveFailedPeers (0.00s)
152s     --- PASS: TestClusterJoinAndReconnect/TestInitiallyFailingPeers (0.00s)
152s     --- FAIL: TestClusterJoinAndReconnect/TestTLSConnection (0.04s)

armhf:
152s --- FAIL: TestClusterJoinAndReconnect (1.01s)
152s     --- PASS: TestClusterJoinAndReconnect/TestJoinLeave (0.81s)
152s     --- PASS: TestClusterJoinAndReconnect/TestReconnect (0.00s)
152s     --- PASS: TestClusterJoinAndReconnect/TestRemoveFailedPeers (0.00s)
152s     --- PASS: TestClusterJoinAndReconnect/TestInitiallyFailingPeers (0.00s)
152s     --- FAIL: TestClusterJoinAndReconnect/TestTLSConnection (0.19s)

i386:
251s --- FAIL: TestClusterJoinAndReconnect (0.96s)
251s     --- PASS: TestClusterJoinAndReconnect/TestJoinLeave (0.81s)
251s     --- PASS: TestClusterJoinAndReconnect/TestReconnect (0.00s)
251s     --- PASS: TestClusterJoinAndReconnect/TestRemoveFailedPeers (0.00s)
251s     --- PASS: TestClusterJoinAndReconnect/TestInitiallyFailingPeers (0.00s)
251s     --- FAIL: TestClusterJoinAndReconnect/TestTLSConnection (0.15s)

s390x:
163s --- FAIL: TestClusterJoinAndReconnect (0.82s)
163s     --- PASS: TestClusterJoinAndReconnect/TestJoinLeave (0.80s)
163s     --- PASS: TestClusterJoinAndReconnect/TestReconnect (0.00s)
163s     --- PASS: TestClusterJoinAndReconnect/TestRemoveFailedPeers (0.00s)
163s     --- PASS: TestClusterJoinAndReconnect/TestInitiallyFailingPeers (0.00s)
163s     --- FAIL: TestClusterJoinAndReconnect/TestTLSConnection (0.02s)

Compare to a successful test, where TestTLSConnection nearly always takes between 0.8s and 0.85s:
149s === RUN   TestClusterJoinAndReconnect/TestTLSConnection
149s --- PASS: TestClusterJoinAndReconnect (1.61s)
149s     --- PASS: TestClusterJoinAndReconnect/TestJoinLeave (0.80s)
149s     --- PASS: TestClusterJoinAndReconnect/TestReconnect (0.00s)
149s     --- PASS: TestClusterJoinAndReconnect/TestRemoveFailedPeers (0.00s)
149s     --- PASS: TestClusterJoinAndReconnect/TestInitiallyFailingPeers (0.00s)
149s     --- PASS: TestClusterJoinAndReconnect/TestTLSConnection (0.80s)

Still seeing this from time to time.


Spongebob

Logo for spongebob

https://github.com/traderjoe-xyz/joe-tokenlists/pull/127/commits/90915330c278e130785edbceed60f82efcafa6eb


FIX: rename export mutator fail because of duplicated entry bug

CC @fitzgen in https://github.com/bytecodealliance/wasm-tools/pull/349

Addressed :)


Native API Hook - retrieve trampoline bypass address

This is an enhancement request, follow up from https://github.com/EasyHook/EasyHook/issues/152, to expose "OldProc" member in an accessible manner.

Hi @spazzarama Can you kindly tell an estimated date when the 2.8 milestone/version would be released??

@spazzarama , Can you please comment on the estimated release date of 2.8 milestone ??

@rramk I've started changes for 2.8. I should have them completed within a few weeks.

@rramk there is a pre-release build with this fix on appveyor if you want to try it.
https://ci.appveyor.com/project/spazzarama/easyhook/build/0.0.31/artifacts
Check the unit test for usage:
https://github.com/EasyHook/EasyHook/blob/develop/Test/EasyHook.Tests/LocalHookTests.cs#L120

@spazzarama, the fix works with the pre-release 2.7.6251.0 binaries. Thank you. So, by when can I expect the 2.8 stable binaries?

@rramk thanks for checking out the fix. I'm not sure exactly when I will get a 2.8 release done, as there are quite a few tasks for that release. I'll look at doing a new 2.7 release with this change instead as it is fairly minor.

That would be great. Thanks. So, Can I know the expected date for the 2.7 release with this fix?

Nuget released


Add libx11-xcb-dev key

Please add the following dependency to the rosdep database.
Package name:
libx11-xcb-dev
Package Upstream Source:
https://gitlab.freedesktop.org/xorg/lib/libx11
Purpose of using this:
Needed by gz_ogre_next_vendor
Distro packaging links:
Links to Distribution Packages

Debian: https://packages.debian.org/

https://packages.debian.org/bookworm/libx11-xcb-dev

Ubuntu: https://packages.ubuntu.com/

https://packages.ubuntu.com/noble/libx11-xcb-dev

Fedora: https://packages.fedoraproject.org/

https://packages.fedoraproject.org/pkgs/libX11/libX11-devel/

Arch: https://www.archlinux.org/packages/

https://archlinux.org/packages/extra/x86_64/libx11

Gentoo: https://packages.gentoo.org/

https://packages.gentoo.org/packages/x11-libs/libX11

macOS: https://formulae.brew.sh/

skipped

Alpine: https://pkgs.alpinelinux.org/packages

https://pkgs.alpinelinux.org/package/edge/main/x86_64/libx11

NixOS/nixpkgs: https://search.nixos.org/packages

https://github.com/NixOS/nixpkgs/blob/nixos-23.11/pkgs/servers/x11/xorg/default.nix#L1121

openSUSE: https://software.opensuse.org/package/

skipped

rhel: https://rhel.pkgs.org/

skipped

The bot thinks we can satisfy this key on opensuse and rhel as well, so we should probably do that.

Done in https://github.com/ros/rosdistro/pull/40281/commits/1c2827da0f92d80ceebfbf3a9443cb3a6ff14a14


Install issue on DLAMI images, conflict with PyYAML

wandb has a dependency conflict when installing on AWS Deep Learning images -- DLAMI v23
You can get arround it with 'pip install wandb --ignore-installed', but also perhaps wandb could relax PyYAML version requirement to make life easier (ie, I can't put wandb in requirements.txt because of this)
(pytorch_p36) ubuntu@ip-172-31-28-233:~$ pip install wandb
Collecting wandb
  Using cached https://files.pythonhosted.org/packages/6a/d1/af8371f39d9383f4f1e9ba76c8894f75c01d5eddf4ec57bd45952fefab74/wandb-0.8.3-py2.py3-none-any.whl
Collecting watchdog>=0.8.3 (from wandb)
Requirement already satisfied: psutil>=5.0.0 in ./anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from wandb) (5.4.5)
Collecting backports.tempfile>=1.0 (from wandb)
  Using cached https://files.pythonhosted.org/packages/b4/5c/077f910632476281428fe254807952eb47ca78e720d059a46178c541e669/backports.tempfile-1.0-py2.py3-none-any.whl
Requirement already satisfied: requests>=2.0.0 in ./anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from wandb) (2.20.0)
Requirement already satisfied: sentry-sdk>=0.4.0 in ./anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from wandb) (0.9.5)
Requirement already satisfied: six>=1.10.0 in ./anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from wandb) (1.11.0)
Collecting shortuuid>=0.5.0 (from wandb)
Collecting gql>=0.1.0 (from wandb)
Requirement already satisfied: subprocess32>=3.5.3 in ./anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from wandb) (3.5.4)
Collecting GitPython>=1.0.0 (from wandb)
  Using cached https://files.pythonhosted.org/packages/fe/e5/fafe827507644c32d6dc553a1c435cdf882e0c28918a5bab29f7fbebfb70/GitPython-2.1.11-py2.py3-none-any.whl
Requirement already satisfied: docker-pycreds>=0.4.0 in ./anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from wandb) (0.4.0)
Requirement already satisfied: nvidia-ml-py3>=7.352.0 in ./anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from wandb) (7.352.0)
Requirement already satisfied: Click>=7.0 in ./anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from wandb) (7.0)
Requirement already satisfied: python-dateutil>=2.6.1 in ./anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from wandb) (2.7.3)
Collecting PyYAML>=4.2b4 (from wandb)
Requirement already satisfied: argh>=0.24.1 in ./anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from watchdog>=0.8.3->wandb) (0.26.2)
Collecting pathtools>=0.1.1 (from watchdog>=0.8.3->wandb)
Collecting backports.weakref (from backports.tempfile>=1.0->wandb)
  Using cached https://files.pythonhosted.org/packages/88/ec/f598b633c3d5ffe267aaada57d961c94fdfa183c5c3ebda2b6d151943db6/backports.weakref-1.0.post1-py2.py3-none-any.whl
Requirement already satisfied: urllib3<1.25,>=1.21.1 in ./anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from requests>=2.0.0->wandb) (1.23)
Requirement already satisfied: certifi>=2017.4.17 in ./anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from requests>=2.0.0->wandb) (2019.3.9)
Requirement already satisfied: idna<2.8,>=2.5 in ./anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from requests>=2.0.0->wandb) (2.6)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./anaconda3/envs/pytorch_p36/lib/python3.6/site-packages (from requests>=2.0.0->wandb) (3.0.4)
Collecting graphql-core>=0.5.0 (from gql>=0.1.0->wandb)
  Using cached https://files.pythonhosted.org/packages/f1/88/a4a7bf8ab66c35b146e44d77a1f9fd2c36e0ec9fb1a51581608c16deb6e3/graphql_core-2.2-py2.py3-none-any.whl
Collecting promise>=0.4.0 (from gql>=0.1.0->wandb)
Collecting gitdb2>=2.0.0 (from GitPython>=1.0.0->wandb)
  Using cached https://files.pythonhosted.org/packages/da/30/a407568aa8d8f25db817cf50121a958722f3fc5f87e3a6fba1f40c0633e3/gitdb2-2.0.5-py2.py3-none-any.whl
Collecting rx>=1.6.0 (from graphql-core>=0.5.0->gql>=0.1.0->wandb)
  Using cached https://files.pythonhosted.org/packages/33/0f/5ef4ac78e2a538cc1b054eb86285fe0bf7a5dbaeaac2c584757c300515e2/Rx-1.6.1-py2.py3-none-any.whl
Collecting smmap2>=2.0.0 (from gitdb2>=2.0.0->GitPython>=1.0.0->wandb)
  Using cached https://files.pythonhosted.org/packages/55/d2/866d45e3a121ee15a1dc013824d58072fd5c7799c9c34d01378eb262ca8f/smmap2-2.0.5-py2.py3-none-any.whl
thinc 6.12.1 has requirement msgpack<0.6.0,>=0.5.6, but you'll have msgpack 0.6.0 which is incompatible.
tensorflow 1.13.1 has requirement protobuf>=3.6.1, but you'll have protobuf 3.5.2 which is incompatible.
tensorboard 1.13.1 has requirement protobuf>=3.6.0, but you'll have protobuf 3.5.2 which is incompatible.
docker-compose 1.24.0 has requirement PyYAML<4.3,>=3.10, but you'll have pyyaml 5.1.1 which is incompatible.
Installing collected packages: PyYAML, pathtools, watchdog, backports.weakref, backports.tempfile, shortuuid, rx, promise, graphql-core, gql, smmap2, gitdb2, GitPython, wandb
  Found existing installation: PyYAML 3.12
Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
You are using pip version 10.0.1, however version 19.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(pytorch_p36) ubuntu@ip-172-31-28-233:~$ echo $?

The proximal cause is SageMaker and wandb both adding restrictions on pyyyaml version.
However, the real culprit seems to be pip rather than wandb.
Version restrictions can be satisfied, but since pip is not a real package manager, it can't figure out the right version of pyyaml and just guesses 5.1.1 which is wrong. Moving community to pipenv or conda may eventually fix this

Solution on user side:
pip install --upgrade --force-reinstall pip==9.0.3
pip install -U pyyaml --disable-pip-version-check
pip install --upgrade pip
pip install wandb

added in https://github.com/wandb/client/pull/384

Reopning because I think this issue affects every Amazon Deep Learning AMI user, so it could use a workaround from wandb side

Version 0.8.5 no longer pins the pyyaml version.  @yaroslavvb wandb should install cleanly on AMI's now.


Saved Athena Queries

I am not seeing any saved Athena Queries in the "Master" or "Compliance" account. Do they get populated or deployed after the compliance engine runs the first time with rule populated in there?

I validated that these queries do populated after a while once the config check is run.


Updatedfiles

added table of contents in part 3(issue #25) and part 5(issue #27).

Sure, will create new pr.


Design and content tweaks to Optimization page

Tugboat has finished building the preview for this pull request!
Link:

https://pr189-pmvf57dhj8uuvgrdlpqaowoqbslgppy8.tugboat.qa

Dashboard:

https://dashboard.tugboat.qa/613acc62baa70d18919ce3ca

@dcaryll Looks like the bullets below Pattern 3 got a little screwed up, which I have a fix for. Otherwise, ready for review.

Tugboat has finished building the preview for this pull request!
Link:

https://pr189-pmvf57dhj8uuvgrdlpqaowoqbslgppy8.tugboat.qa

Dashboard:

https://dashboard.tugboat.qa/613acc62baa70d18919ce3ca

Tugboat has finished building the preview for this pull request!
Link:

https://pr189-pmvf57dhj8uuvgrdlpqaowoqbslgppy8.tugboat.qa

Dashboard:

https://dashboard.tugboat.qa/613acc62baa70d18919ce3ca

Tugboat has finished building the preview for this pull request!
Link:

https://pr189-pmvf57dhj8uuvgrdlpqaowoqbslgppy8.tugboat.qa

Dashboard:

https://dashboard.tugboat.qa/613acc62baa70d18919ce3ca

Tugboat has finished building the preview for this pull request!
Link:

https://pr189-pmvf57dhj8uuvgrdlpqaowoqbslgppy8.tugboat.qa

Dashboard:

https://dashboard.tugboat.qa/613acc62baa70d18919ce3ca

Tugboat has finished building the preview for this pull request!
Link:

https://pr189-pmvf57dhj8uuvgrdlpqaowoqbslgppy8.tugboat.qa

Dashboard:

https://dashboard.tugboat.qa/613acc62baa70d18919ce3ca


Add an example of importing a CSV to the tuning cookbook

Description of the feature request:
Add an example of importing a CSV using pandas.read_csv to the tuning cookbook: https://github.com/google-gemini/cookbook/blob/main/quickstarts/Tuning.ipynb
AI Studio accepts csv files as datasets for tuning jobs. The tuning cookbook currently demonstrates creating a tuned model using an list of dictionaries. This is a request for an additional sample demonstrating importing a training dataset from a local csv file using pandas.read_csv, to mirror the functionality of importing a csv file through AI Studio.
What problem are you trying to solve with this feature?
Help users understand how to tune a model using data from a CSV.
Any other information you'd like to share?
No response

This could also be a new notebook based on the existing tuning notebook.

I don't think this necessarily needs a new guide, at least not for Python. genai.create_tuned_model() can take a CSV file handle or string filename as the training_data arg.
So this
genai.create_tuned_model(
    training_data=[
        { ... }, { ... }, etc
    ], ...
)

becomes
genai.create_tuned_model(
    training_data='my_file.csv', ...
)

It supports JSON files too.

hi

Fixed in #300


TensorFlow engine not found when exported to a runnable jar file

Description
We have a large project in which we use djl to allow users to load acoustic deep learning models. The djl library is packaged within a single dependency (jdl4pam) generated from a seperate project. During development everything was working great, however, when we packaged the project into a jar file, the TensorFlow based classifiers stopped working returning "Deep learning engine not found" exceptions. Pytorch models were unaffected.
Expected Behavior
It should make no difference running a project from an IDE (in this case Eclipse) or packaging it into a jar file.
Error Message
Exception in thread "main" java.lang.IllegalArgumentException: Deep learning engine not found: TensorFlow
at ai.djl.engine.Engine.getEngine(Engine.java:150)
at ai.djl.Model.newInstance(Model.java:82)
at testjfrog.TestJFrog.main(TestJFrog.java:32)
How to Reproduce?
I've created a minimal project  (master branch) which imports a tensorflow "***.pb" file and loads it. The program accepts a path as an argument so any TensorFlow model can be loaded. The available engines are [Tensorflow Pytorch] if run from Eclipse. However, if this is exported to a jar and run from powershell in Windows the only available engine is Pytorch and a the program will throw an exception.
Interestingly if the TensorFlow engine is added before the jdl4pam dependency then the opposite occurs, Tensorflow becomes the only available engine and Pytorch is no longer available.
i.e. this works for Tensorflow models
<!-- https://mvnrepository.com/artifact/ai.djl.tensorflow/tensorflow-engine-->
	<dependency>
		<groupId>ai.djl.tensorflow</groupId>
		<artifactId>tensorflow-engine</artifactId>
		<version>0.11.0</version>
	</dependency>

	 <dependency>
		<groupId>org.jamdev</groupId>
		<artifactId>jdl4pam</artifactId>
		<version>0.0.87</version>
	</dependency>

but it should work using just the jdl4pam dependency.
Steps to reproduce

Import the minimal example Maven project into eclipse
Download the TensorFlow model from here.
Run the project in Eclipse.
Export the project as .jar file and run in powershell e.g. java -jar test_jdl4pam.jar "C:\Users\Jamie\Desktop\Right_whales_DG\model_lenet_dropout_input_conv_all\saved_model.pb"
The project will run fine in Eclipse but will return an exception when run from the jar file. In eclipse the available models will be [PyTorch, TensorFlow] and from the jar only [Pytorch].

What have you tried to solve it?
I updated jdl4pam to the latest version of the djl library. I played around with Maven dependencies and found where the issue is (see above) but cannot explain why this occurs?
Environment Info
Windows 10.
Version	10.0.19042 Build 19042
Java 14
(This issue also occurs on MacOS)

@macster110 You also need include tensorflow-native-auto as dependency. See: http://docs.djl.ai/tensorflow/tensorflow-engine/index.html#installation

@macster110 You also need include tensorflow native-auto as dependency. See: http://docs.djl.ai/tensorflow/tensorflow-engine/index.html#installation
All the required dependencies are in the jdl4pam library. I only have the tensorflow-engine dependency as an example to show how you can resolve this issue. But, this is not an ideal solution because it should just work when all enquired djl dependencies are bundled with jdl4pam.

@macster110
I cloned your project: git@github.com:macster110/jpam.git, but I'm not able to build the project. The dependency isn't configured properly.

I noticed you include both pytorch and tensorflow eingine, both the jar file contains a file named:
META-INF/services/ai.djl.engine.EngineProvider

When you merge the two jar file into single fat jar, one of the file get override (depends on your merge policy).
If Pytorch one get preserved, TensorFlow one will be lost.
The only workaround if you manually merge this file and added into your fat jar.

Thanks @frankfliu. I'll check out what's going on with jpam.
Does this mean that there is no easy way to bundle two engines in one jar using  djl? If this is the case, what is the recommended way to structure and then export a project using multiple engines and djl?

The Java service loader is a common pattern and there seem to be other solutions existing. A quick search turned up:

https://stackoverflow.com/questions/42540485/how-to-stop-maven-shade-plugin-from-blocking-java-util-serviceloader-initializat
https://stackoverflow.com/questions/47310215/merging-meta-inf-services-files-with-maven-assembly-plugin
https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer

The Java service loader is a common pattern and there seem to be other solutions existing. A quick search turned up:
* https://stackoverflow.com/questions/42540485/how-to-stop-maven-shade-plugin-from-blocking-java-util-serviceloader-initializat

* https://stackoverflow.com/questions/47310215/merging-meta-inf-services-files-with-maven-assembly-plugin

* https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer

Thanks @zachgk . After the diagnosis of the issue from @frankfliu I managed to get the test project working by exporting in Eclipse and selecting "package required libraries into generated jar" instead of "extract packaged libraries into generated jar".
The MAVEN shade plugin should be able to deal with this but I have been unsuccessful so far in getting it to work. I will post the solution here when I do.
Thanks again for all the help on this.

For reference there are three solutions to this question.
Export the jar file without extracting libraries
In Eclipse IDE right click on the project -> Export. In the Runnable JAR File Speciifcation dialog select Package required libraries int generated jar instead of the default Extract required libraries into generated jar.
Note that this solution is not ideal for larger projects.
Manually change the META-INF/services/ folder in the jar file

Create a jar file.
Unzip it using 7-zip or equivalent.
Navigate to the META-INF/services folder
Open the ai.djl.engine.EngineProvider file (e.g. in Sublime)
Replace the contents of the files with the desired engines...for example to run both Pytorch and Tensorflow the text should read

ai.djl.pytorch.engine.PtEngineProvider
ai.djl.tensorflow.engine.TfEngineProvider

-Save the file and re-zip the contents of the jar file. Rename the zip file to .jar. Everything should work.
Let Maven do it all for you
Add the Maven shade plugin with the following settings and the jar will be created properly. Note that you must use Maven to generate the jar i.e. set the goal to package shade:shade
	<!-- Maven Shade plugin - for creating the uberjar / fatjar -->
		<!-- see http://maven.apache.org/plugins/maven-shade-plugin/index.html for
			details -->
		<plugin>
			<groupId>org.apache.maven.plugins</groupId>
			<artifactId>maven-shade-plugin</artifactId>
			<version>3.2.1</version>
			<configuration>
				<transformers>
					<transformer
						implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
				</transformers>
			</configuration>
			<executions>
				<execution>
					<phase>package</phase>
					<goals>
						<goal>shade</goal>
					</goals>
					<configuration>
						<filters>
							<filter>
								<artifact>*:*</artifact>
								<excludes>
								 	<exclude>PamModel/*.*</exclude>  <!--  don't include files in the PamModel folder -->
									<exclude>META-INF/*.SF</exclude> <!-- get rid of manifests from library jars -->
									<exclude>META-INF/*.DSA</exclude>
									<exclude>META-INF/*.RSA</exclude>
								</excludes>
							</filter>
						</filters>
					</configuration>
				</execution>
			</executions>
		</plugin>


Websockets does not connect sometimes

I'm encountering a problem when sometimes my client connects to websockets API and sometimes not. I can't see anything from cloudwatch blog. Could that be an issue of the websockets gateway? Unfortunately I couldn't find anything about the error message.

Probably a websockets gateway limit, sorry for bothering


Local dev installation fails due to explicit version number in setup.cfg

Installing via pip install -e . fails due to an explicit version number in the setup.cfg, as per https://github.com/invl/retry/issues/8
This can be addressed by commenting out the version number in setup.cfg, but looks to be associated with extracting the semantic version number from the repo.
Possible symptom of misconfigured automatic extraction of git version number.

Hey team! Please add your planning poker estimate with Zenhub @ccarouge @SeanBryan51

We might have to rework the changes I made when implementing the --version command line flag in #55. In that PR, the SSOT of the version number is defined in setup.cfg so that I could get the version number in the source code, and get the version number in meta.yaml.
It seems if I comment out the version number from setup.cfg and then run conda build -c conda-forge . in the project root, the benchcab version in the local channel is None:
$ conda search --use-local benchcab
Loading channels: done
# Name                       Version           Build  Channel
benchcab                        None           py_92  scratch/tm70/sb8430/conda/bld

It would be great if we could get the version number that is inferred by pbr into meta.yaml somehow. I'll look into it.

@SeanBryan51 Should we rather look at this from the viewpoint of #152 ? So a complete redesign of the setup using .toml instead of just fixing this particular aspect of it? I would assume a good solution for #152 would fix this issue as well.

@ccarouge yep I agree. This issue can probably wait until we get around to #152 as it only effects us (developers).

Discussed at meeting: will investigate versioneer

@bschroeter suggested using grayskull for automatically publishing pypi packages to conda

Versioneer (vendored mode) looks promising, the version string inferred by versioneer can be found by conda (see here for the implementation):
$ pip install --user -q . && pip list | grep benchcab
benchcab           3.0.0+12.g7c06481
$ conda render . | grep version
  version: 3.0.0+12.g7c06481

To move to versioneer, we will need to deprecate support for PBR so that we can use the versioneer version string instead of the PBR version string. In the above changes, this meant using the vanilla setuptools setup.cfg options instead of the additional options provided by PBR.


Contribute this to Apache CouchDB?

@klaemo would you be up for contributing this to Apache CouchDB proper, so we don’t have to maintain our own Dockerfile? Development could continue as usual, just on apache/couchdb instead of here.

or we could create apache/couchdb-docker repository as well to keep container for 1.x and have separate release cycles. But in any case I'm +1 on this (:

Does Dockerfile have to live on the top level? If not, we can do a sub-repo on our end. If yes, we can also import the 1.x version into our 1.x.x branch

:+1:

I have no problem donating this to Apache CouchDB. Does this mean migrating to git.couchdb.org and JIRA, though? because that'd be... :cry: I'd prefer apache/docker-couchdb on github, because I want to keep the barrier to entry low.
@janl the Dockerfiles can live anywhere, but there are some restrictions with docker hub's automated build system pertaining to tags/versions.
Also, once the migration is done, we can try to get the docker image promoted to an official image on docker hub. That way couchdb will be on par with other databases in that regard.
How do you propose going forward?

Yay, this is good news :)
The main repo for all Apache stuff is git.apache.org/, but all of this is mirrored to github.com/apache/ and you can run Pull Requests against that as usual (just no issues or merge button).
So for this, it would mean either apache/couchdb (if we put the Dockerfile into the main, top-level repo) or apache/couchdb-docker if it should get its own repo.
Since you are the maintainer, you get to pick :)
One thing to consider: from 2.0 (current master) onwards, people check out apache/couchdb and then run ./configure which in turn fetches all the other couchdb-* repos that are required to stand up CouchDB. So if you wanted to to run CouchDB in Docker, you’d have to fetch couchdb-docker on your own. If you are used to fetching the main repo, having the Dockerfile in there might be preferable. Or we leave a note in there, for people to go loo in couchdb-docker. OR you’d have to run ./configure before you could get to the Dockerfile at src/couchdb-docker, but then you’d also wait for all other modules to fetch which you don’t necessarily need.
That said, if you download a regular CouchDB release tarball, src/couchdb-docker would already exist, so less of an issue there.
Aaanyway, happy to chat in person / IRC to clear things up, but ultimately, this is your contribution and we should do what feels right to you :)
I’m +1 on making the image official :)

From wednesday on I'll have more free time to get this rolling.
We should definitely have a chat about the details :)

@criloz since you've contributed to this repo, I have to ask: are you willing to sign the CLA? https://www.apache.org/licenses/icla.txt

@klaemo sure where can I send the documents.?

@criloz<EMAIL_ADDRESS>

@criloz did you have a chance to send the file(s)? I couldn’t find anything under criloz. If you used a real name, could you tell me? (privately at<EMAIL_ADDRESS>if you prefer), so I can match it up with the ASF records.
After some more revisiting, I think we only need the CLA from you, but not the grant. Sorry for the confusion.

Update: The CouchDB dev community has accepted docker-couchdb as a contribution, now we have to get some paperwork done, that you don’t have to concern yourselves with. I’ll keep you posted.

:heart:

@janl just checking in, what's the status here?

thanks for the nudge! :)

https://github.com/klaemo/docker-couchdb/pull/27

Ok, I started the paperwork, we should be good to go in 72 hours :)

This time it was me slacking, due to being busy. I just responded to the corresponding email. We're on our way! Slowly, but surely :D

\o/

I'm ready to rock! ;) What's next?

@janl / @klaemo - what's the status with moving this into the apache repo?  There's some stuff that can be done to reduce the size of the 2.0-dev image, but it probably makes sense to submit the patches directly to the apache repo?  https://github.com/klaemo/docker-couchdb/issues/32

@snowch I'm waiting for more information from the CouchDB folks. It's kind of confusing currently.

wait, who’s waiting for who? :D

We're in some kind of limbo. Partly because I didn't get around to migrating everything properly, partly because Apache and Github aren't the best of friends. :D
There is apache/couchdb-docker which mirrors apache git. Unfortunately there is no straightforward way to migrate this repo to the apache on, so I push to both (that's what @jo is doing with nano as well afaik). Also apache/* repos don't allow github issues. So, that's pretty much a no-go for me.
I've opened a PR on apache/couchdb to update the Dockerfile there. Need to finish this.
I'm currently favoring a 2-tier approach:

apache/couchdb/Dockerfile: for development and testing only (getting the needed deps, building couchdb locally from source etc)
klaemo/docker-couchdb and/or  apache/couchdb-docker: for running 1.x, 2.x and master quickly. should all be available on docker hub for quick downloadable images (is already the case, except for master). Ideally, I get them to accept it as an official image.

So, yeah...it's mainly on me to get my act together ;-) But if you have any advice on the repo migration I'd be happy :)

IIRC @robertkowalski experimented with complete Github repo transfer to ASF with stars, issues and everything. Not sure how this ended, however.

still waiting for the new github api that asf infra needs / wants

I just submitted CouchDB to the official-images repo. If the review goes well, CouchDB will have an official Docker image soon (like pretty much every database out there). That means users will be able to docker run couchdb instead of docker run klaemo/couchdb.
You can check the status here.
I submitted version 1.6.1 as well as for 2.0-dev. That should make it pretty easy for people to give 2.0 a spin. Of course, I'll add future releases as they come out.
Next step: get the development Dockerfile in apache/couchdb sorted out.

Update: ASF infra is not yet ready to import GitHub repos wholesale, and keeping GitHub issues is another thing that needs some work. We’ll continue this process, once ASF infra is ready.

Status update from my side:

this repo acts as official docker image source for now
PouchDB started using our docker images in their CI setup :tada:
the PR for the dev docker setup is in good shape now imho (https://github.com/apache/couchdb/pull/340)
the official docker image PR is still waiting for review (https://github.com/docker-library/official-images/pull/1288)

Actually, this is done 🎉


Bot error in spoiler season background task

()
{}
[]

Reported on discordbot by discord user--------------------------------------------------------------------------------
DatabaseException
Failed to execute CREATE TABLE IF NOT EXISTS _new_cache_card (idINTEGER NOT NULL PRIMARY KEY AUTO_INCREMENT ,nameVARCHAR(190) NOT NULL ,mana_costVARCHAR(190) ,cmcREAL ,powerVARCHAR(190) ,toughnessVARCHAR(190) ,loyaltyVARCHAR(190) ,typeVARCHAR(190) NOT NULL ,textLONGTEXT NOT NULL ,search_textLONGTEXT NOT NULL ,image_nameVARCHAR(190) ,handINTEGER ,lifeINTEGER ,starterINTEGER ,positionINTEGER NOT NULL ,card_idINTEGER ,layoutVARCHAR(190) NOT NULL ,namesLONGTEXT ,legalitiesLONGTEXT ,pd_legalBOOLEAN ,bugs LONGTEXT , FOREIGN KEY(card_id) REFERENCES card(id)) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci AS SELECT u.idASid, CASE WHEN layout = 'double-faced' OR layout = 'flip' THEN GROUP_CONCAT(CASE WHEN u.position = 1 THEN face_name ELSE '' END SEPARATOR '') WHEN layout = 'meld' THEN GROUP_CONCAT(CASE WHEN u.position = 1 OR u.position = 2 THEN face_name ELSE '' END SEPARATOR '') ELSE GROUP_CONCAT(face_name SEPARATOR ' // ' ) END AS name, CASE WHEN layout = 'flip' THEN GROUP_CONCAT(CASE WHEN u.position = 1 THEN mana_cost ELSE '' END SEPARATOR '') ELSE GROUP_CONCAT(u.mana_cost SEPARATOR '|') END AS mana_cost, CASE WHEN layout = 'split' OR layout = 'aftermath' THEN SUM(u.cmc) WHEN layout = 'meld' THEN SUM(CASE WHEN u.position = 1 OR u.position = 2 THEN cmc ELSE 0 END) ELSE SUM(CASE WHEN u.position = 1 THEN u.cmc ELSE 0 END) END AS cmc, GROUP_CONCAT(CASE WHEN u.position = 1 THEN u.powerELSE '' END SEPARATOR '') ASpower, GROUP_CONCAT(CASE WHEN u.position = 1 THEN u.toughnessELSE '' END SEPARATOR '') AStoughness, GROUP_CONCAT(CASE WHEN u.position = 1 THEN u.loyaltyELSE '' END SEPARATOR '') ASloyalty, CASE WHEN layout = 'meld' THEN GROUP_CONCAT(CASE WHEN u.position = 1 OR u.position = 2 THEN type ELSE '' END SEPARATOR '') ELSE GROUP_CONCAT(CASE WHEN u.position = 1 THEN type ELSE '' END SEPARATOR '') END AS type, GROUP_CONCAT(u.text` SEPARATOR '
') AS text, GROUP_CONCAT(u.search_text SEPARATOR '
') AS search_text, GROUP_CONCAT(CASE WHEN u.position = 1 THEN u.image_name ELSE '' END SEPARATOR '') AS image_name, SUM(CASE WHEN u.position = 1 OR (u.position = 2 AND u.layout = 'meld') THEN u.hand ELSE NULL END) AS hand, SUM(CASE WHEN u.position = 1 OR (u.position = 2 AND u.layout = 'meld') THEN u.life ELSE NULL END) AS life, SUM(CASE WHEN u.position = 1 OR (u.position = 2 AND u.layout = 'meld') THEN u.starter ELSE NULL END) AS starter, SUM(CASE WHEN u.position = 1 OR (u.position = 2 AND u.layout = 'meld') THEN u.position ELSE NULL END) AS position, SUM(CASE WHEN u.position = 1 OR (u.position = 2 AND u.layout = 'meld') THEN u.card_id ELSE NULL END) AS card_id, u.layout AS layout, GROUP_CONCAT(face_name SEPARATOR '|') AS names, legalities, u.pd_legal AS pd_legal, bugs
FROM (
SELECT c.id, c.layout, f.mana_cost, f.cmc, f.power, f.toughness, f.loyalty, f.type, f.text, f.search_text, f.image_name, f.hand, f.life, f.starter, f.position, f.card_id, f.name AS face_name,
pd_legal,
legalities
FROM
card AS c
INNER JOIN
face AS f ON c.id = f.card_id
LEFT JOIN (
SELECT
cl.card_id,
SUM(CASE WHEN cl.format_id = 21 THEN 1 ELSE 0 END) > 0 AS pd_legal,
GROUP_CONCAT(CONCAT(fo.name, ':', cl.legality)) AS legalities
FROM
card_legality AS cl
LEFT JOIN
format AS fo ON cl.format_id = fo.id
GROUP BY
cl.card_id
) AS cl ON cl.card_id = c.id
GROUP BY
f.id
ORDER BY
f.card_id, f.position
) AS u
LEFT JOIN (
SELECT
cb.card_id,
GROUP_CONCAT(CONCAT(cb.description, '|', cb.classification, '|', cb.last_confirmed, '|', cb.url, '|', cb.from_bug_blog, '|', cb.bannable) SEPARATOR 'SEPARATOR') AS bugs
FROM
card_bug AS cb
GROUP BY
cb.card_id
) AS bugs ON u.id = bugs.card_id
WHERE u.id IN (SELECT c.id FROM card AS c INNER JOIN face AS f ON c.id = f.card_id WHERE (1 = 1))
GROUP BY u.id
with[]because of(1452, 'Cannot add or update a child row: a foreign key constraint fails (discordbot._new_cache_card, CONSTRAINT _new_cache_card_ibfk_1 FOREIGN KEY (card_id) REFERENCES card (id))')`
Stack Trace:

  File "run.py", line 120, in <module>
    run()
  File "run.py", line 17, in run
    bot.init()
  File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/bot.py", line 202, in init
    client.init()
  File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/bot.py", line 32, in init
    self.run(configuration.get('token'))
  File "/home/discord/.local/lib/python3.6/site-packages/discord/client.py", line 563, in run
    loop.run_forever()
  File "/usr/lib64/python3.6/asyncio/base_events.py", line 422, in run_forever
    self._run_once()
  File "/usr/lib64/python3.6/asyncio/base_events.py", line 1432, in _run_once
    handle._run()
  File "/usr/lib64/python3.6/asyncio/events.py", line 145, in _run
    self._callback(*self._args)
  File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/bot.py", line 186, in background_task_tournaments
    await self.background_task_spoiler_season()
  File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/bot.py", line 135, in background_task_spoiler_season
    oracle.insert_scryfall_card(c, True)
  File "/home/discord/Penny-Dreadful-Discord-Bot/magic/oracle.py", line 134, in insert_scryfall_card
    multiverse.update_cache()
  File "/home/discord/Penny-Dreadful-Discord-Bot/magic/multiverse.py", line 273, in update_cache
    db().execute(create_table_def('_new_cache_card', card.base_query_properties(), base_query()))
  File "/home/discord/Penny-Dreadful-Discord-Bot/shared/database.py", line 45, in execute
    [n, _] = self.execute_anything(sql, args, False)
  File "/home/discord/Penny-Dreadful-Discord-Bot/shared/database.py", line 60, in execute_anything
    raise DatabaseException('Failed to execute `{sql}` with `{args}` because of `{e}`'.format(sql=sql, args=args, e=e))

()
{}
[]

Reported on discordbot by discord user--------------------------------------------------------------------------------
DatabaseException
Failed to execute CREATE TABLE IF NOT EXISTS _new_cache_card (idINTEGER NOT NULL PRIMARY KEY AUTO_INCREMENT ,nameVARCHAR(190) NOT NULL ,mana_costVARCHAR(190) ,cmcREAL ,powerVARCHAR(190) ,toughnessVARCHAR(190) ,loyaltyVARCHAR(190) ,typeVARCHAR(190) NOT NULL ,textLONGTEXT NOT NULL ,search_textLONGTEXT NOT NULL ,image_nameVARCHAR(190) ,handINTEGER ,lifeINTEGER ,starterINTEGER ,positionINTEGER NOT NULL ,card_idINTEGER ,layoutVARCHAR(190) NOT NULL ,namesLONGTEXT ,legalitiesLONGTEXT ,pd_legalBOOLEAN ,bugs LONGTEXT , FOREIGN KEY(card_id) REFERENCES card(id)) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci AS SELECT u.idASid, CASE WHEN layout = 'double-faced' OR layout = 'flip' THEN GROUP_CONCAT(CASE WHEN u.position = 1 THEN face_name ELSE '' END SEPARATOR '') WHEN layout = 'meld' THEN GROUP_CONCAT(CASE WHEN u.position = 1 OR u.position = 2 THEN face_name ELSE '' END SEPARATOR '') ELSE GROUP_CONCAT(face_name SEPARATOR ' // ' ) END AS name, CASE WHEN layout = 'flip' THEN GROUP_CONCAT(CASE WHEN u.position = 1 THEN mana_cost ELSE '' END SEPARATOR '') ELSE GROUP_CONCAT(u.mana_cost SEPARATOR '|') END AS mana_cost, CASE WHEN layout = 'split' OR layout = 'aftermath' THEN SUM(u.cmc) WHEN layout = 'meld' THEN SUM(CASE WHEN u.position = 1 OR u.position = 2 THEN cmc ELSE 0 END) ELSE SUM(CASE WHEN u.position = 1 THEN u.cmc ELSE 0 END) END AS cmc, GROUP_CONCAT(CASE WHEN u.position = 1 THEN u.powerELSE '' END SEPARATOR '') ASpower, GROUP_CONCAT(CASE WHEN u.position = 1 THEN u.toughnessELSE '' END SEPARATOR '') AStoughness, GROUP_CONCAT(CASE WHEN u.position = 1 THEN u.loyaltyELSE '' END SEPARATOR '') ASloyalty, CASE WHEN layout = 'meld' THEN GROUP_CONCAT(CASE WHEN u.position = 1 OR u.position = 2 THEN type ELSE '' END SEPARATOR '') ELSE GROUP_CONCAT(CASE WHEN u.position = 1 THEN type ELSE '' END SEPARATOR '') END AS type, GROUP_CONCAT(u.text` SEPARATOR '
') AS text, GROUP_CONCAT(u.search_text SEPARATOR '
') AS search_text, GROUP_CONCAT(CASE WHEN u.position = 1 THEN u.image_name ELSE '' END SEPARATOR '') AS image_name, SUM(CASE WHEN u.position = 1 OR (u.position = 2 AND u.layout = 'meld') THEN u.hand ELSE NULL END) AS hand, SUM(CASE WHEN u.position = 1 OR (u.position = 2 AND u.layout = 'meld') THEN u.life ELSE NULL END) AS life, SUM(CASE WHEN u.position = 1 OR (u.position = 2 AND u.layout = 'meld') THEN u.starter ELSE NULL END) AS starter, SUM(CASE WHEN u.position = 1 OR (u.position = 2 AND u.layout = 'meld') THEN u.position ELSE NULL END) AS position, SUM(CASE WHEN u.position = 1 OR (u.position = 2 AND u.layout = 'meld') THEN u.card_id ELSE NULL END) AS card_id, u.layout AS layout, GROUP_CONCAT(face_name SEPARATOR '|') AS names, legalities, u.pd_legal AS pd_legal, bugs
FROM (
SELECT c.id, c.layout, f.mana_cost, f.cmc, f.power, f.toughness, f.loyalty, f.type, f.text, f.search_text, f.image_name, f.hand, f.life, f.starter, f.position, f.card_id, f.name AS face_name,
pd_legal,
legalities
FROM
card AS c
INNER JOIN
face AS f ON c.id = f.card_id
LEFT JOIN (
SELECT
cl.card_id,
SUM(CASE WHEN cl.format_id = 21 THEN 1 ELSE 0 END) > 0 AS pd_legal,
GROUP_CONCAT(CONCAT(fo.name, ':', cl.legality)) AS legalities
FROM
card_legality AS cl
LEFT JOIN
format AS fo ON cl.format_id = fo.id
GROUP BY
cl.card_id
) AS cl ON cl.card_id = c.id
GROUP BY
f.id
ORDER BY
f.card_id, f.position
) AS u
LEFT JOIN (
SELECT
cb.card_id,
GROUP_CONCAT(CONCAT(cb.description, '|', cb.classification, '|', cb.last_confirmed, '|', cb.url, '|', cb.from_bug_blog, '|', cb.bannable) SEPARATOR 'SEPARATOR') AS bugs
FROM
card_bug AS cb
GROUP BY
cb.card_id
) AS bugs ON u.id = bugs.card_id
WHERE u.id IN (SELECT c.id FROM card AS c INNER JOIN face AS f ON c.id = f.card_id WHERE (1 = 1))
GROUP BY u.id
with[]because of(1406, "Data too long for column 'type' at row 19")`
Stack Trace:

  File "run.py", line 120, in <module>
    run()
  File "run.py", line 17, in run
    bot.init()
  File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/bot.py", line 202, in init
    client.init()
  File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/bot.py", line 32, in init
    self.run(configuration.get('token'))
  File "/home/discord/.local/lib/python3.6/site-packages/discord/client.py", line 563, in run
    loop.run_forever()
  File "/usr/lib64/python3.6/asyncio/base_events.py", line 422, in run_forever
    self._run_once()
  File "/usr/lib64/python3.6/asyncio/base_events.py", line 1432, in _run_once
    handle._run()
  File "/usr/lib64/python3.6/asyncio/events.py", line 145, in _run
    self._callback(*self._args)
  File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/bot.py", line 186, in background_task_tournaments
    await self.background_task_spoiler_season()
  File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/bot.py", line 135, in background_task_spoiler_season
    oracle.insert_scryfall_card(c, True)
  File "/home/discord/Penny-Dreadful-Discord-Bot/magic/oracle.py", line 134, in insert_scryfall_card
    multiverse.update_cache()
  File "/home/discord/Penny-Dreadful-Discord-Bot/magic/multiverse.py", line 273, in update_cache
    db().execute(create_table_def('_new_cache_card', card.base_query_properties(), base_query()))
  File "/home/discord/Penny-Dreadful-Discord-Bot/shared/database.py", line 45, in execute
    [n, _] = self.execute_anything(sql, args, False)
  File "/home/discord/Penny-Dreadful-Discord-Bot/shared/database.py", line 60, in execute_anything
    raise DatabaseException('Failed to execute `{sql}` with `{args}` because of `{e}`'.format(sql=sql, args=args, e=e))

And five more times since then.

The issue here is that we have an overly long bug?

These are still coming in.
I think the issue is a "spoiler" we are scraping from scryfall.
I get an error on local seemingly trying to deal with a (double-faced token?) card "Soldier // Solider". Not sure if that's the same thing or not.

I have these things in "face":
| Bog Humbugs                    |
| Rampaging Brontodon            |
| Immortal Phoenix               |
| Vengeant Vampire               |
| Angler Turtle                  |
| Angelic Guardian               |
| Avatar of Growth               |
| Goblin Goliath                 |
| Rot Hulk                       |
| Inspired Sphinx                |
| Militant Angel                 |
| Voja                           |
| Saproling                      |
| Elemental                      |
| Centaur                        |
| Wurm                           |
| Saproling                      |
| Saproling                      |
| Elf Knight                     |
| Soldier                        |
| Goblin                         |
| Saproling                      |
| Insect                         |
| Goblin                         |
| Soldier                        |
| Weird                          |
| Goblin                         |
| Soldier                        |
| Soldier                        |
| Soldier                        |
| Soldier                        |
| Soldier                        |
| Soldier                        |
| Soldier                        |
| Soldier                        |

Prod is similar:
| Rampaging Brontodon                                                                                |
| Immortal Phoenix                                                                                |
| Vengeant Vampire                                                                                |
| Angler Turtle                                                                                |
| Angelic Guardian                                                                                |
| Avatar of Growth                                                                                |
| Goblin Goliath                                                                                |
| Rot Hulk                                                                                |
| Inspired Sphinx                                                                                |
| Militant Angel                                                                                |
| Vraska, Golgari Queen Emblem                                                                                |
| Ral, Izzet Viceroy Emblem                                                                                |
| Elf Knight                                                                                |
| Insect                                                                                |
| Bird Illusion                                                                                |
| Voja                                                                                |
| Weird                                                                                |
| Inspiring Commander                                                                                |
| Bog Humbugs                                                                                |
| Voja                                                                                |
| Saproling                                                                                |
| Elemental                                                                                |
| Centaur                                                                                |
| Wurm                                                                                |
| Saproling                                                                                |
| Saproling                                                                                |
| Elf Knight                                                                                |
| Soldier                                                                                |
| Goblin                                                                                |
| Saproling                                                                                |
| Insect                                                                                |
| Goblin                                                                                |
| Soldier                                                                                |
| Weird                                                                                |
| Goblin                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |
| Soldier                                                                                |

I see a new layout of double_faced_token which I suspect.

As spoiler season code goes directly to scryfall I don't think it uses the layout checking of multiverse.layouts.

Ok I'm misremembering how this works a little. We do actually store tokens, planes, etc. in card.

The problem on prod is that we try to form a very long name involving the word Soldier a lot.
MariaDB [discordbot]> SELECT id, card_id, name FROM face WHERE id > 18900 AND name = 'Soldier';
+-------+---------+---------+
| id    | card_id | name    |
+-------+---------+---------+
| 42510 |   42071 | Soldier |
| 56841 |   56218 | Soldier |
| 56846 |   56220 | Soldier |
| 56849 |   56222 | Soldier |
| 56850 |   56222 | Soldier |
| 56851 |   56222 | Soldier |
| 56852 |   56222 | Soldier |
| 56853 |   56222 | Soldier |
| 56854 |   56222 | Soldier |
| 56855 |   56222 | Soldier |
| 56856 |   56222 | Soldier |
| 56857 |   56222 | Soldier |
| 56858 |   56222 | Soldier |
| 56859 |   56222 | Soldier |
| 56860 |   56222 | Soldier |
| 56861 |   56222 | Soldier |
| 56862 |   56222 | Soldier |
| 56863 |   56222 | Soldier |
| 56864 |   56222 | Soldier |
| 56865 |   56222 | Soldier |
| 56866 |   56222 | Soldier |
| 56867 |   56222 | Soldier |
| 56868 |   56222 | Soldier |
| 56869 |   56222 | Soldier |
| 56870 |   56222 | Soldier |
| 56871 |   56222 | Soldier |
| 56872 |   56222 | Soldier |
| 56873 |   56222 | Soldier |
| 56874 |   56222 | Soldier |
| 56875 |   56222 | Soldier |
| 56876 |   56222 | Soldier |
| 56877 |   56222 | Soldier |
| 56878 |   56222 | Soldier |
| 56879 |   56222 | Soldier |
| 56880 |   56222 | Soldier |
| 56881 |   56222 | Soldier |
| 56882 |   56222 | Soldier |
+-------+---------+---------+
37 rows in set (0.02 sec)

The word soldier 34 times is longer than 192 chars.
So the question is why did this get added 34 times under the same card_id?
What should we be doing with tokens with the same name as another token re: cards?

The logic in spoiler season job is:
                try:
                    oracle.valid_name(c['name'])
                    await asyncio.sleep(1)
                except InvalidDataException:
                    oracle.insert_scryfall_card(c, True)

oracle.insert_scryfall_card does a bit of data massaging and then calls multiverse.insert_card. Once you've done that although there's an in-memory lookup it's only for things inserted by this process during this run and then bop you're in the db. So if oracle.valid_name throws an exception you are pretty likely to get a new entry in the db. This whole thing feels pretty fragile and unchecked and should maybe be made more robust.
For now I'm going to delete all the Soldiers and see what happens.

MariaDB [discordbot]> DELETE FROM card_subtype WHERE card_id = 56222;
Query OK, 8 rows affected (0.01 sec)

MariaDB [discordbot]> DELETE FROM face WHERE card_id = 56222;
Query OK, 34 rows affected (0.00 sec)

MariaDB [discordbot]> DELETE FROM card WHERE id = 56222;
Query OK, 1 row affected (0.00 sec)


Meditron stops after the first line of answer

Hello all,
I've tried Meditron with "ollama run meditron" and after that, I've asked "what are the symptoms of Kawasaki disease?".
The answer started with one line definition of Kawasaki disease and stopped after that.
I've tried with different questions, but results were only one liners.
What could be the reason for that?
Best,
Orkut

The short answers are probably related to what the data the model was trained on looked like. You might be able to get longer answers if you're looking for something more verbose by increasing the temperature and modifying your prompt.
>>> /set parameter temperature 1.2
Set parameter 'temperature' to '1.2'

>>> what is kawasaki disease, answer at length with details?
The disease is named after a Japanese pediatrician who first described it in 1967. It causes
inflammation and swelling of blood vessels throughout the body, especially the coronary
arteries. The name mucocutaneous lymph node syndrome comes from this finding, which is when an
examiner looks underneath a patient's arm at the bones in their forearm to see enlarged lymph
nodes (swollen immune system cells) as well as other redness and swelling of the skin. The term
vasculitis describes inflammation or irritation of blood vessels, which is the actual cause of
this condition.
The illness most often happens in children aged 6 months to 8 years old; however, it can happen
in older people and teenagers too. Boys are more likely than girls to be affected. There's no
cure for Kawasaki disease; however, medicines such as high doses of aspirin help lower the
chance of heart problems after the illness.
Kawasaki disease happens when a person has an infection with the virus known as coronavirus,
which is common and causes cold-like symptoms but can sometimes cause severe respiratory and
gastrointestinal illnesses too. There are many types of viruses that cause similar illnesses,
including coxsackievirus A9 and adenovirus 12.
In Kawasaki disease, the coronavirus makes the lining of blood vessels weak and damaged so they
bleed easily. When a person's heart muscles have no support from strong blood vessels, it is
easier for them to be injured or damaged during a heart attack or other problems that lead to
the death of heart cells (myocardial infarction).

Just a warning that this seems like it could make the model more likely for the model to hallucinate.


Merge changes for the next release into the main branch (2)

What type of PR is this?
/kind documentation
What this PR does / why we need it:
Complete commits missed in #1109
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?

NONE

/assign @tenzen-y

/label tide/merge-method-merge


Loss calculation

https://github.com/NM512/dreamerv3-torch/blob/4e50f302cdfaca1c8104f203376844f82c635a4e/models.py#L147
Just wondering, but shouldn't kl_loss be included in the sum() call?
From the paper:
$$L(\phi) = E_{q\phi} \left[\sum_{t=1}^T (\beta_\text{pred}L_\text{pred}(\phi) + \beta_\text{dyn}L_\text{dyn}(\phi) + \beta_\text{rep}L_\text{rep}(\phi))\right]$$

Nevermind. I just checked this again and apparently I was wrong.


Entity serialization is slow

Do you want to request a feature or report a bug?
Bug
What is the current behavior?
store.get / Entity.load are currently slow because serializing even simple entities by allocating and writing to WASM memory is implemented in an inefficient way. We tracked this down to AscHeap::asc_new / AscHeap::raw_new causing a lot of small memory allocations, each of which take ~3-5ms. For a reasonably sized entity, this ends up at around 20-40 allocations and 150-200ms (locally, debug build).
What is likely taking so long under the hood is memory.grow() calls inside wasmi.
If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.
Clone and deploy any subgraph that uses Entity.load and follow the logs. You should see Store get messages that are fast, but Done processing Ethereum event messages that indicate the handler takes a long time (~500-1500ms):
Aug 24 20:20:25.758 DEBG Store get finished, time: 1ms, id: 0xd4fd252d7d2c9479a8d616f510eac6243b5dddf9, type: Account, block_hash: 0x3669b543ae19ed0a56c50f80f66e67d6dc3004e62bc3d848ecc2b95b42d253ee, block_number: 4161317, subgraph_id: Qmd1WDLn9QzX8tJYJyYh8AdCCjqqg3oqE8fDs4bQyjad7d, component: SubgraphInstanceManager
Aug 24 20:20:25.845 DEBG Store get finished, time: 3ms, id: 0xd4fd252d7d2c9479a8d616f510eac6243b5dddf9-0xe41d2489571d322189246dafa5ebde1f4699f498, type: AccountBalance, block_hash: 0x3669b543ae19ed0a56c50f80f66e67d6dc3004e62bc3d848ecc2b95b42d253ee, block_number: 4161317, subgraph_id: Qmd1WDLn9QzX8tJYJyYh8AdCCjqqg3oqE8fDs4bQyjad7d, component: SubgraphInstanceManager
Aug 24 20:20:26.101 INFO Done processing Ethereum event, waiting_ms: 0, total_ms: 1331, handler: handleTransfer, signature: Transfer(indexed address,indexed address,uint256), block_hash: 0x3669b543ae19ed0a56c50f80f66e67d6dc3004e62bc3d848ecc2b95b42d253ee, block_number: 4161317, subgraph_id: Qmd1WDLn9QzX8tJYJyYh8AdCCjqqg3oqE8fDs4bQyjad7d, component: SubgraphInstanceManager

What is the expected behavior?
The expected behavior is that each handler takes only a few milliseconds, even with store.get / Entity.load involved. The serialization of an entity to WASM should ideally only take <5ms overall.

Also you could thy this PR which improve detection when memory.grow should call


Update authorization rule

What is the purpose of the change
XXXXX
Brief changelog
XX
Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist.
[ ] Run `mvn clean compile --batch-mode -DskipTests=false -Dcheckstyle.skip=false -Drat.skip=false -Dmaven.javadoc.skip=true to make sure basic checks pass.

Codecov Report

Merging #1061 (99b21e2) into refactor-with-go (a08d6b2) will increase coverage by 3.16%.
The diff coverage is 98.55%.

:exclamation: Current head 99b21e2 differs from pull request most recent head 0adb033. Consider uploading reports for the commit 0adb033 to get more accurate results

@@                 Coverage Diff                  @@
##           refactor-with-go    #1061      +/-   ##
====================================================
+ Coverage             60.62%   63.79%   +3.16%
====================================================
  Files                    27       29       +2
  Lines                  2131     2323     +192
====================================================
+ Hits                   1292     1482     +190
- Misses                  742      744       +2
  Partials                 97       97

Impacted Files
Coverage Δ

pkg/authority/rule/authorization/rule.go
96.21% <97.41%> (+5.73%)
:arrow_up:

pkg/authority/rule/authentication/definition.go
100.00% <100.00%> (ø)

pkg/authority/rule/authentication/rule.go
96.21% <100.00%> (-0.27%)
:arrow_down:

pkg/authority/rule/authorization/definition.go
100.00% <100.00%> (ø)

... and 2 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Button "Go"

Hello, when I click on that "GO" button, the character doesn't change trajectory and  appears in the console #img. Is it normal or has it not yet been implemented?

Yes, sorry about that, not implemented for Necrobot.

but it will be implemented later? if not, where i can download mobbot? https://github.com/PocketMobsters/MobBotTemp is disabled for me.

This is not planned (at least not by me) on neither Necrobot ofr Mobbot.
It commes from another project of mine, sorry.


Add option to disable translation button

Hopefully, this fix addresses the over-indexing by the Algolia crawler.
For reference: https://t.me/c/1322374388/5994

This is very, very cool stuff

#1014 will fix CI


Add Conda environment module

This adds a very very simple module (based off the code for the virtual_env_module) to highlight the name of the Conda environment you are in.
For the unaware: Conda environments are very similar to virtualenvs but they are a different beast as they can also handle non-Python software, or even the version of Python itself. They are very powerful.

This would be a very cool addition as I use conda environments instead of virtualenv.

👍 I was literally just about to make this PR.

Thanks for the input and apologies for the long delay in replying.
I'm reworking the way custom modules/segments canbe added.
My original idea for pureline was to keep it a simple single file system. This premise however, excludes adding a lot of extra code, meaning functionality which would be useful to some is excluded.
The new system will allow unlimited plugin files to be 'sourced' into pureline, while keeping the core simple and small. I will try to work your code in soon.

No problem Chris, this probably is all volunteer work that you have to fit around your other work.

Thanks for incorporating the changes @chris-marsh 🎉

Its good to have more choice for people to use :)
I was thinking ... what about the conda and virtual env code being merged to be a python_section?
Something like;
if conda_env set
... do this
else if virtual_env set
...do that?

You can but I'd like to point out that conda and virtualenv are not mutually exclusive.
You probably wouldn't use virtualenv inside a conda env, but you can, so 👍 for combining the code, but don't do "if conda, else if virtualenv", because you can mix both and then pureline will be omitting one in that scenario. It should detect both, not one or the other.


How can a Code Action insert text that contains replacement placeholders/tabstops?

I'm familiar with snippets/completions being able to include tokens that the user can tab-between, however I'm struggling to find if it's possible to do something similar with a code action.
For example, this code action allows you to wrap a widget around some selected code:

The word widget is intended to be replaced by the user with the name of the class that they're wrapping this with.
Currently my code action executes a command which calls vs.workspace.applyEdit, however it doesn't look like WorkspaceEdit has support for placeholders like this.
Is there another way I can handle this? Eg. can I somehow get Code to do a replacement that can contain tokens/placeholders?

Looks like this is possible via TextEditor.insertSnippet.


RealmSwift object without final modifier

Goals
My objects Realm and Result conform to a protocol to abstract these two objects. Then, I would like writing some tests to test the implementation of these protocols. My idea was to extend Realm and Result and check if the method of super are called. Unfortunately, I'm not able to do it since they are final.
Expected Results
I want to be able to extend the objects.
Actual Results
The classes are final
Code Sample
protocol RealmType {
	func objects<T: Object>(type: T.Type) -> RealmResultType
}

extension Realm: RealmType {

	func objects<T: Object>(type: T.Type) -> RealmResultType {
                // I want to test that the protocol implementation calls this `objects` method
		return objects(type) as RealmResultType
	}
}

Thanks for using Realm.
For various internal reasons, it's not feasible for us to make those types non-final, nor do we intend to in the future.
I suggest you test your protocol by creating a dummy object that doesn't inherit from Realm or Results but has the correct methods, and ensure that the methods on that dummy object are called.


Having issue with changing the folders.

    "message" => "09-05 20:40:58.723 [pool-16-thread-52] INFO  c.a.c.Controller Try # 1 for doc 789456 feed abcdef took 18 ms",
   "timestamp" => "2018-09-05T20:40:58.723Z"

}
{
"querytime_ms" => 22,
"@timestamp" => 2018-09-05T20:40:58.723Z,
"javaclass" => "c.a.c.Controller",
"loglevel" => "INFO",
"@version" => "1",
"host" => "testing_host",
"context" => "pool-16-thread-52",
"message" => "09-05 20:40:58.723 [pool-16-thread-52] INFO  c.a.c.Controller (On try # 1) Doc 789456 clustered into feed abcdef and joined 1 clusters. Total time: 22 ms. Updating DB!",
"timestamp" => "2018-09-05T20:40:58.723Z"
}
-[2018-09-24T06:15:56,455][WARN ][logstash.agent           ] stopping pipeline {:id=>"main"}
./test.sh: line 72: "1537769757" - "1537769741" : syntax error: operand expected (error token is ""1537769757" - "1537769741" ")
TEST SUMMARY
✔ passed - All tests

Thanks for filing this issue. Can you please clarify the folder changes that were applied?

This should be resolved by https://github.com/agolo/logstash-test-runner/pull/5


Is that support for spring boot 3?

I am getting this message when trying with spring boot 3.0.4:
An attempt was made to call a method that does not exist. The attempt was made from the following location:
com.github.valb3r.letsencrypthelper.tomcat.TomcatWellKnownLetsEncryptChallengeEndpointConfig.matchesCertFilePathAndPassword(TomcatWellKnownLetsEncryptChallengeEndpointConfig.java:252)

The following method did not exist:
'java.lang.String org.apache.tomcat.util.net.SSLHostConfig.getCertificateKeystoreFile()'

The calling method's class, com.github.valb3r.letsencrypthelper.tomcat.TomcatWellKnownLetsEncryptChallengeEndpointConfig, was loaded from the following location:
jar:file:/app.jar!/BOOT-INF/lib/letsencrypt-helper-tomcat-0.2.5.jar!/com/github/valb3r/letsencrypthelper/tomcat/TomcatWellKnownLetsEncryptChallengeEndpointConfig.class

The called method's class, org.apache.tomcat.util.net.SSLHostConfig, is available from the following locations:
jar:file:/app.jar!/BOOT-INF/lib/tomcat-embed-core-10.1.5.jar!/org/apache/tomcat/util/net/SSLHostConfig.class

The called method's class hierarchy was loaded from the following locations:
org.apache.tomcat.util.net.SSLHostConfig: jar:file:/app.jar!/BOOT-INF/lib/tomcat-embed-core-10.1.5.jar!/

Supported now


(If yes, list here)
Screenshot
(Link above will always be the most up-to-date version.)

@Connor1892  I hope this is in a work plan page that you are referring. Cz we have filters in all the grids.
Should we save it in the browser level or user level.
If we save it in the browser level, the user keep seeing this filters as long as they are using the same machine and same browser - Just front end changes are required
If it is in the user level, they keep seeing it from where ever they login(no matter what computer or what browser) - this needs database level changes and api development to create/retrieve the filter as they move between pages

Browser level pls. user level stuff won't be happening until EPIC.centre.
we need it for the My Workplans page as well as for workplan pages.


✨ Added more functions to query builder

TODO:

[ ] Add the date and time, and record functions
[ ] Add doc comments for the new additions
[ ] Cleanup and standardize some of the naming conventions


The team sidebar is not working after clicking on post profile picture.

Summary
The team sidebar is not working after clicking on post profile picture.
Environment Information

Device Name:
OS Version: Android 7.1.1
Mattermost App Version: 1.6.1
Mattermost Server Version: 4.7

Steps to reproduce

Join multiple teams to use team sidebar.
Click on any profile picture in channel view.
Close user profile modal, try to use team sidebar.

Expected behavior
Should works as expected.
Observed behavior
Select team button and swiping are not working.

Hi @cometkim
Thanks for your feedback,
Please try upgrading to the latest version of the app as well as the Mattermost server (v4.9 released today - April 16th) to see whether this fixes the issue for you?

@lindy65 It looks already fixed in a previous version 👍
now it looks fine to me.

Perfect @cometkim 👍


lv_timer_t compiler issue

There are a few callback functions like auto hud_update = [](lv_timer_t *timer) in the example.cpp (line 104) and application.cpp that throw compiler errors due to timer being an 'incomplete type'.

Hi, yes you are right, thanks for letting me know.
The issue had to do with changes made to v9.2.0 which was released last week, I updated the example.

awesome! thank you for putting this together and maintaining it for the community!
You clearly know what you're doing - any suggestions on how to use PIO in VSCode for the T-Display-S3 to do live debugging? I'm aware that https://docs.platformio.org/en/latest/boards/espressif32/lilygo-t-display-s3.html#debugging says the device isn't ready for it, but I wanted to see if you had any suggestions. thanks again!!


Change lines container height based on TextEditorElement dimensions

After merging the tiles PRs, I have overlooked that the lines/line-numbers containers were still using scrollHeight as their height. This is clearly wrong and, although this seemed to have no effect performance-wise, @nathansobo found it had an impact while benchmarking stuff with chrome://tracing.
To fix this, I have introduced a state.content.maxHeight that's calculated using TextEditorElement's bounding client rect and that gets updated every time its dimensions change.
Sorry about this but hopefully we can get it fixed pretty soon with this PR.
/cc: @atom/feedback for :eyes:

@as-cii I hypothesized that it was a problem but it's not clear that it was. The memory estimate for large files was pretty massive, but I couldn't find a way to determine actual memory consumption. Regardless this seems prudent. Why hit the GPU with any large layers than necessary?

Agreed! :+1: (and I am truly sorry I haven't caught this early on :disappointed_relieved:)
Please, merge at will when/if Travis becomes :green_heart: (I know it's having some issues today).

Thanks @as-cii :clapper:


Add support for PackagesSpec

This PR add the support for generic packages to Wave container request

@pditommaso @marcodelapierre please review


Completely Documented Pallet Fragments

So, I think this PR can be merged? It's OK for me.
Please, squash all the commits before merging.

CI is failing


Kafka upgrade to 1.0.0

kafka upgrade
kafka upgrade again

Can one of the admins verify this patch?

Pull Request Test Coverage Report for Build 3332

0 of 0   changed or added relevant lines in 0 files are covered.
5 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.002%) to 24.66%

Files with Coverage Reduction
New Missed Lines
%

core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java
5
77.81%

Totals

Change from base Build 3330:
0.002%

Covered Lines:
15461

Relevant Lines:
62697

💛  - Coveralls


getLatestLocation() always returns null, subscribeToLocationUpdates never fires

I am encountering the behaviour described in the above title for certain models on samsung devices. I can't tell if it is something wrong I'm doing with the library or it is the device.
I have properly followed the guidelines to install this library in my react-native application. The user is prompted to grant the location permission and he grants it to the app. Here is the initialisation and the subscription ode:
import RNLocation from 'react-native-location';
...

    RNLocation.requestPermission({
      ios: 'whenInUse',
      android: {detail: 'coarse'},
    }).then((granted) => {
      if (granted) {
        RNLocation.getLatestLocation().then((loc) => {
          Alert.alert(JSON.stringify(loc)); // <--- this returns null
        });
        RNLocation.subscribeToLocationUpdates((locations) => { // <-- this never fires
          const {latitude, longitude} = locations[0];
          Alert.alert(`${JSON.stringify({latitude, longitude})}`);
        });
      }
    });

package.json:
"react": "16.11.0",
 "react-native": "0.62.2",
"react-native-location": "^2.5.0"

build.gradle
buildToolsVersion = "28.0.3"
minSdkVersion = 16
compileSdkVersion = 28
targetSdkVersion = 28
googlePlayServicesVersion  = "17.0.0"

My code works perfectly well in an old Samsung J7 model (2015):
Android 7.0 (Nougat)
API level 24

But the location returned is not correctly in a Samsung J2 core (2018):
Android 8.1 (Oreo)
Api level 27

In both cases, the location permission is properly granted and gps location is enabled in both devices.
Thank you for your library. I hope you may help me to find out what I may do wrong or where I should start looking at.

Update on my issue: I was able to solve it, by changing the initialisation for android:
from coarse to fine. Now it is working as expected in the 2 models. I would just want to know what is the difference between these 2 settings and why it works in the second case (Apologies if this is not the right forum for this discussion). Thank you.

According to android docs you can see:
Google Dev Location Docs

Android offers two location permissions: ACCESS_COARSE_LOCATION and ACCESS_FINE_LOCATION. The permission you choose determines the accuracy of the location returned by the API. You only need to request one of the Android location permissions, depending on the level of accuracy you need:
android.permission.ACCESS_COARSE_LOCATION – Allows the API to use WiFi or mobile cell data (or both) to determine the device's location. The API returns the location with an accuracy approximately equivalent to a city block.
android.permission.ACCESS_FINE_LOCATION – Allows the API to determine as precise a location as possible from the available location providers, including the Global Positioning System (GPS) as well as WiFi and mobile cell data.

@seyaobey I am sorry, but I still get this error even I twitched setting to 'fine' in android.
Can you figure it out why?

@seyaobey I think the issue is you are making those two requests at the same time.
RNLocation.getLatestLocation().then((loc) => {
          Alert.alert(JSON.stringify(loc)); // <--- this returns null
        });
        RNLocation.subscribeToLocationUpdates((locations) => { // <-- this never fires
          const {latitude, longitude} = locations[0];
          Alert.alert(`${JSON.stringify({latitude, longitude})}`);
        });

This code is dispatching both the getLatestLocation and subscribeToLocationUpdates at the same time. I was doing something similar. You can use async methods and call those two sequentially and it should resolve this issue.

Don't use this repo because of this issue. Please refer to: https://www.npmjs.com/package/react-native-get-location It works fine.


nixos/xkb: extract from xserver

Description of changes
I did not test this yet.
@Shawn8901 @alois31 can you help me review this, again? That would be lovely.
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

@jtojnar @Shawn8901 @K900 can you give another round of review? I want to finally get this merged.


Feature: investigate using git2go

Type of feature
🍕 Feature
Current behavior
Currently, the proof-of-concept uses git-go, which is a pure implementation of git APIs in Go. Some of the critical performance bottlenecks (primarily cloning, fetching, pulling repos) have been abit lackluster.
Suggested solution
It may be beneficial to investigate and look into using git2go: https://github.com/libgit2/git2go which are Go bindings for lib2git, the git APIs implemented in C designed to be portable and fast. While the Go APIs we use would be almost the same, this could complicate our build process since we'd also need to build in the C bindings for lib2git.
Additional context
At this early stage (and especially before we enter any kind of production use case) this is a good time to get wild and try a few different things.
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Contributing Docs

[X] I agree to follow this project's Contribution Docs

After some investigating, there doesn't seem to be too many crazy optimizations we get from using git2go. And we'd have to take on a few build time dependencies on some compiled C code which could really slow us down.
In the future, if we really need to change things up for performance, we should look into a re-write with gitoxide, a pure rust implementation of git protocols that looks really fast: https://github.com/Byron/gitoxide
But for now, this gets us pretty far!


Is it possible to add swipe from left to right?

Hello,
@freeze-frames Thank you very much for you great work!
Could you tell me is it possible to add swipe from left to right with the same behavior it works from right to left?

Do you mean that you need to swipe an item both from right to left and left to right?

Do you mean that you need to swipe an item both from right to left and left to right?

Yes, exactly!
I've tried to do something like this to have menu from both sides:
<LinearLayout>

	<FrameLayout>
		<!-- Left side menu items -->
	</FrameLayout>

	<TextView/>

	<FrameLayout>
		<!-- Right side menu items -->
	</FrameLayout>

</LinearLayout>

But without luck. Maybe there is an option to reach the goal?

No. Currently it only supports right side menu to a item layout in left-to-right direction and the left side menu you placed above has been just treated as a normal view (not a menu), so support for both side menus will need to take some time to expand and debug the view.

Oh, thank you anyway. Unfortunately I'm newbie android developer but I'll try to understand how it works so may be I can add the function. Could you give me any clue where to start?

onInterceptTouchEvent() -> onTouchEvent()
Most of the remaining functions are used within these two, which you can also take a look just along the way.

onInterceptTouchEvent() -> onTouchEvent()
Most of the remaining functions are used within these two, from which you can choose some useful ones to take a look at just along the way.

onInterceptTouchEvent() -> onTouchEvent()
Most of the remaining functions are used within these two, from which you can choose some useful ones to take a look at just along the way.

Thank you!

@lzls I can't use the library doesn't see me in my project


Typo

Oh, Microsoft. You're usually always so perfect!

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

 I'll catch that now.