[feature] Created register menu group functionality

Currently I have made a function to register menu groups. Also I am showing registered menu groups in the menu. User can navigate throughout the website very easily using the menu.
Note: In this commit designing was not my concern.

closes #189
closes #190
closes #193

Coverage decreased (-0.4%) to 99.097% when pulling e3b7192eb4788b2fea5c1765fb1475c9d58b6b59 on niteshsinha17:gsoc/new-navigation-menu-and-ux-improvements into 388b8e20ccb77e0d5f50b6825973a1c8e9d53187 on openwisp:master.

Functionality Added

Registration of multiple menu groups.
Functionality to add new items in the default menu group. I have divided the menu into two sections i.e custom and default.
Now it is possible to add icons to the group.
It logs a warning when using old register menu items functionality.
Now we can pass preference in which order we want to show menu groups.
It also logs errors when we apply different preferences to the same group and when we use the same name for the groups multiple times.
We can now register both model link and link of another site.

I do not understand this.

I am dividing menu to sections. One of them is default group that will be provided by Utils itself. Anyone can add more links too it.
If you don't want this then I can remove it.

I took for granted that we'd implement icons with CSS since CSS is more powerful and can give us more control

Okay I will change this.

Is ordering implemented? Keep in mind we need to be able to specify the position of each link.
Can icons be defined also at link level (not only group level)?

Yes, we can implement both. I will update it again for this.

@nemesisdesign and @atb00ker Thanks for waiting. I have updated it again.

Now we can add icons to each link level too.
We can also provide position of each link.
For groups, I am using a preference list so that if users skip this then we don't run any extra piece of code to achieve ordering in groups.
I have added extra checks so that if user provide wrong values then we can throw error to make them easy to understand whats wrong they have done.
Using f-strings

@atb00ker
Thanks, I will correct it soon. Also started designing in a separate branch and will push soon.

That's great! @niteshsinha17

@nemesisdesign and @atb00ker Its ready.
How should we proceed now?

Hey, your last comment on this PR is 3 days old, if something is getting stale, please remember to ping someone, it's entirely possible to miss pings on GitHub!

I was going to remind you tomorrow. I thought you were waiting for the results.

Could you please elaborate the question? Proceed in context of this PR?

I was thinking to start merging them into some branch if things are ok.

Could you please include changes in REAME to explain how can a user register menu group / use the new functionality.

Sure,  I will add this. I am noting every point so that we can open issues to solve them.

Could you please include changes in REAME to explain how can a user register menu group / use the new functionality.

@atb00ker Added in README, you can check now.

Added tests. It is ready for testing.

@nemesisdesign  and @atb00ker
I have updated this PR as you have suggested.

Two things in the screenshot:

Docs section (and code) don't have icons!
I think there are 2 things registered in the auth group.

        auth_list = {
            2: {
                'label': 'Add user',
                'model': 'auth.User',
                'name': 'changelist',
                'icon': 'add-icon',
            },
            1: {
                'model': 'auth.User',
                'name': 'add',
                'icon': 'edit',
                'label': 'Edit User',
            },
        }

I am a superuser, yet I don't see the second one. I am missing something?

I found the issue. There was an indentation problem which was stoping the registration of other item. I will include a test to detect this. Thanks

Update of latest changes:

Removed register_menu_items section from readme.
I think Model Link is a better name. So changed name of Menu Item.
Double checked spellings. I hope nothing is left now.
I was using value error to raise exceptions in many places. I think it was not good so changed it to ImproperlyConfigured.
Added one more test.
Fixed problems in readme.

Updated task list here https://github.com/openwisp/openwisp-utils/issues/189#issue-894593577

Hi,
I pulled this again and tried, but this is not fixed: #175 (review)
This PR was in ready for review tab,

@atb00ker  It's working correctly. I fixed it last time. for icons, let's do it later in https://github.com/openwisp/openwisp-utils/issues/206 or https://github.com/openwisp/openwisp-utils/pull/185.

LGTM!
I think we should move forward, if we face issues later, we will tackle them accordingly.
@niteshsinha17 also, have you tried breaking tests as Fed has asked in the mailing list?

Yes

@nemesisdesign Thanks for your suggestions.  I have followed your suggestions and have done some more improvements from myside.


DDO-2445 Refactor Lead Time Metrics To Support Multi Replica Sherlock

Problem: Right now with the way Sherlock's accelerate metrics implementation works, the metrics would report incorrectly if we run more than one Sherlock instance.
For the case of the deploy frequency metric this can be solved with just a minor adjustment to the Prometheus query. In iterator logic this would look like:
sherlock_instances
    .iterate()
    .map(delta(sherlock_deploy_frequency{service="blah",env="blah}))
    .sum()

However for lead times it is a little more complex. With the event driven lead time metric as we currently have it only the sherlock replica that serves the request will be aware of the updated lead time, the others will still have the old lead time which will cause weird looking data in grafana
Solution: Refactor the lead time metric implementation so that rather than responding in an event based  manner on requests to a particular endpoint. Sherlock will instead run a concurrent process which maintains a cache of the most recent lead time records in the db. The cache is flushed and rebuilt on some interval. The metrics write interval and cache flush interval are both changeable via Sherlock config.
This implementation uses time.Tickers to coordinate this.
There is potential that this could cause a bit of jitter in the lead time time series data in grafana if the tickers are out of sync in different sherlock instances. However this can be solved by a combination of playing with the timer configs and some smoothing on the prometheus side. Millisecond accuracy is not a requirement of accelerate metrics.

Thanks, for the thumb. Just realized I have a major bug here. Will fix.


Diffrent color schema and UI with VS Code

Environment:

Azure Data Studio Version: 1.39.1
VS Code 1.73.1
OS Version: Ubuntu 20.04.5

Steps to Reproduce:

Run Azure Data Studio and VS Code
Open Source Control tab
Change file(s) in the same repo
See the difference in UI and colors

Example:
On left - VS Code
On right - Azure Data Studio

ADS doesn't keep up with the very latest changes in VS Code - we're usually a few months behind at least. So differences are expected. If there's any specific fixes/improvements you'd like to see please open a separate issue detailing exactly what you'd like to see brought over and we can look into cherry picking stuff as needed.

We don't have the exact same codebase as VS Code so this is not unexpected.  We will pickup more recent vscode changes through an on-going process that lags behind a bit.


Added Translate for getDoc and getList data.

There are a lot of things coming in at once :D
@mainul94 can u please clean this PR, maybe include just those things related to translation ?

@faztp12 alright.

There are a lot of things coming in at once :D
@mainul94 can u please clean this PR, maybe include just those things related to translation ?

PR #43


PERMISSION_DENIED: Request had insufficient authentication scopes.

I have a problem making a request just like the provided Quickstart. I keep getting the error:
PERMISSION_DENIED: Request had insufficient authentication scopes.
I am connecting from my App Engine project with my default service account, which I have added as a read user in the Analytics property I would like to connect to. The client library should automatically make sure the authorization is done correctly.
any idea why this happens?

FYI, this is still not working in my case:
The Quick Start is not working. I have tried to add a role to the service account, but there is no role that gives it the scopes /auth/analytics or /auth/analytics.readonly. If someone could tell me which role this is, that would be very kind.
The Quick Start JSON Credentials is working. I am just retrieving my Application Default Credentials (the standard service worker for my App Engine Application) and applying one of the scopes.
So i think my service worker does not have the correct roles automatically. So maybe anyone can see which role this should be. I have searched for it for many hours.


Error: Failed to authenticate SSH session: Unable to extract public key from private key

System information

node version: 8.10.0
npm or yarn version: yarn ver 1.12.3
OS/version/architecture: Mac OS X Mojave 10.14.2
Applicable nodegit version: 0.23.0

$ node -v
yarn -v # (or yarn -v)
node -e "console.log(process.platform)"
node -e "console.log(require('os').release())"
node -e "console.log(process.arch)"
v8.10.0
1.12.3
darwin
18.2.0
x64

When running nodegit I ran into this error message:
Error: Failed to authenticate SSH session: Unable to extract public key from private key.

After much debugging I eventually landed on this stack exchange article which ultimately provided a work around to this problem:
https://security.stackexchange.com/questions/143114/what-is-the-difference-between-pem-format-to-dsa-rsa-ecc-might-i-confuse-pem-w
Specifically Mac OSX Mojave's ssh-keygen will generate newer style OpenSSH key pairs by default, as these keys more more secure. However, these keys may not be backwards compatible. They do not seem to be backwards compatible with the ssh lib that nodegit uses.
You can tell the kind of keypair you are generating by looking at the preamble of the private key file. If it begins with
-----BEGIN OPENSSH PRIVATE KEY-----

then your keypair is the newer style OpenSSH key that is not currently compatible with nodegit.
To create the older style PEM keypair that is compatible with nodegit you can include the following args in your ssh-keygen command: -m PEM. You can tell this is a PEM keypair that is compatible with nodegit because it's private key preamble begins with:
-----BEGIN RSA PRIVATE KEY-----

I think ultimately nodegit will probably want to upgrade the ssh lib to a version that is compatible with the newer OpenSSH key format as these keys are more secure and are becoming the default types of keys that are generated from ssh-keygen.

was going to be its own issue as could not find anything but then searched on "-m PEM" and found this issue. So also adding my report for extra context

OpenSSH private key format is not supported (infinite loop; invalid credentials)
System information

node version: v10.16.0 and v12.7.0
yarn version: 1.17.3
OS/version/architecture: linux ubuntu 19.04 / 4.18.0-25-generic / x64
Applicable nodegit version: 0.24.3 and 0.25.0-alpha.16
SSH version: OpenSSH_7.9p1 Ubuntu-10, OpenSSL 1.1.1b  26 Feb 2019

When using a recently generated (via ssh-keygen) private key, NodeGit fails on the credentials (infinite loop; invalid credentials).  Unlike the OP I just got an infinite loop.
This seems to be due to the underlying libraries not supporting the OpenSSH private key format. This format is what current (from OpenSSH 7.8) ssh-keygen generates by default. i.e.
-----BEGIN OPENSSH PRIVATE KEY-----
XyZ...
-----END OPENSSH PRIVATE KEY-----

To get a supported (OpenSSL PEM format) format I had to create a new key with:
ssh-keygen -m PEM

The above commands generates a key with a working format (OpenSSL PEM format)
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,2EFA5019B068691036BC8466920A87D8

XyZ...
-----END RSA PRIVATE KEY-----

Added that key to the remote and it works..
Note: also tried with the node v12 alpha (0.25.0-alpha.16). Also failed.
Conclusion is that the underlying libraries need to be updated?
OpenSSH 7.8 Changelog

write OpenSSH format private keys by default instead of using OpenSSL's PEM format. The OpenSSH format, supported in OpenSSH releases since 2014 and described in the PROTOCOL.key file in the source distribution, offers substantially better protection against offline password guessing and supports key comments in private keys. If necessary, it is possible to write old PEM-style keys by adding "-m PEM" to ssh-keygen's arguments when generating or updating a key.

Ref https://serverfault.com/questions/939909/ssh-keygen-does-not-create-rsa-private-key


Fix broken @see blocks.

@see lines try to link the whole line to the name, it tries to lookup these URLs as names in the API and fails:
@see name description
https://www.tensorflow.org/jvm/api_docs/java/org/tensorflow/DeviceSpec

Thanks @MarkDaoust! Before merging, is your intention is to generate the documentation for 0.4.1 that we released last week? If so, the doc must be generated from ther0.4 branch, just want to check if we need to merge this PR in that branch instead.

I was just planning on "eventual consistency", let's not make this more complicated than it needs to be.

Well, looks like it will be more complicated than we were hoping for anyway, because Google's Java style checker did not liked it 😄 (see failing status in GitHub)
what about keeping @see and just passing the straight URL, with no markup nor label?


Can't get it to install on Raspberry Pi

Wanting to install arm32 openjdk-16.0.1+9
This is what I'm doing:

I don't know why it isn't working. May be a stupid question, but definitely need help

In step 6 you need to replace the <codename> in the snippet below with the actual code name of your distro.
echo "deb [signed-by=/usr/share/keyrings/adoptopenjdk-archive-keyring.gpg] https://adoptopenjdk.jfrog.io/adoptopenjdk/deb <codename> main" | sudo tee /etc/apt/sources.list.d/adoptopenjdk.list
That codename snippet is usually recorded in /etc/os-release and can be extracted on Debian by running cat /etc/os-release | grep VERSION_CODENAME | cut -d = -f 2 and on Ubuntu by running cat /etc/os-release | grep UBUNTU_CODENAME | cut -d = -f 2.

In the instructions, for step 6 you need to replace the <codename> place holder with the actual code name of your distro.
That codename is usually recorded in /etc/os-release and can be extracted on:

Debian by running cat /etc/os-release | grep VERSION_CODENAME | cut -d = -f 2
Ubuntu by running cat /etc/os-release | grep UBUNTU_CODENAME | cut -d = -f 2.

THANK YOU. I glazed right over that


Please ignore. This is for testing

@bazel-io cherry-pick
commits: https://github.com/iancha1992/bazel/commit/a2b775467c1b36bd1c935d35770916105cd7102d, https://github.com/iancha1992/bazel/commit/79a0dc276cc81717ac2951d575e19f6a54d71e43, https://github.com/iancha1992/bazel/commit/98d5d5f6980ec8513dc5c0ee95fcabe3b80beb47
reviewers: iancha1992
teams: team-ExternalDeps

Cherry-picked in https://github.com/bazelbuild/bazel/pull/19395, which includes the following commit(s): https://github.com/iancha1992/bazel/commit/a2b775467c1b36bd1c935d35770916105cd7102d, https://github.com/iancha1992/bazel/commit/79a0dc276cc81717ac2951d575e19f6a54d71e43. It also has 1 failed commit(s): https://github.com/iancha1992/bazel/commit/98d5d5f6980ec8513dc5c0ee95fcabe3b80beb47 (Cherry-pick was attempted, but there may be merge conflict(s). Please resolve manually.)The failed commit(s) are NOT included in the PR.


docs(defaults): list of default header values

Fixes #226

Codecov Report

Merging #237 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #237   +/-   ##
=======================================
  Coverage   98.97%   98.97%
=======================================
  Files           6        6
  Lines         391      391
  Branches      117      117
=======================================
  Hits          387      387
  Misses          2        2
  Partials        2        2

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 560d5d6...66c9720. Read the comment docs.

will make appropriate changes as suggested :+1:


Terminal Using new Treesitter group names

As of Oct 15, nvim-treesitter removed the TS* highlight groups. The new groups begin with @. See:
https://github.com/nvim-treesitter/nvim-treesitter/issues/2293#issuecomment-1279974776
This PR changes the monokai TS* group names to the @ names

I've found several group names not matching the ones used in nvim-treesitter (see suggestions above).
Please, let me know if it's not the right way to interact with the PR / repo.

Thanks for catching that, I applied the fixes to my branch.


[BUG] RetryExponential in EventHub can return invalid values

Describe the bug
EventHub RetryExponential logic returns values smaller than 0 which throws exceptions when trying to send data through the EventHub
Exception or Stack Trace
One or more errors occurred. (The value needs to translate in milliseconds to -1 (signifying an infinite timeout), 0 or a positive integer less than or equal to Int32.MaxValue. (Parameter 'delay'));   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
To Reproduce
Define a RetryPolicy with the following parameters:
Code Snippet
       var client = EventHubClient.CreateFromConnectionString(connectionString);
        client.RetryPolicy =new RetryExponential(new TimeSpan(0,0,0,0,10),new TimeSpan(0,0,0,0,500),10  );

// If the Send fails, the error occurs due to the retry,
         client.SendAsync(new EventData(new byte[]{}));

The exception happens because we're doing a Task.Delay of a negative value here.
You can also use this test to check that the value returned by the retry policy in smaller than 0.
        [Fact]
        [DisplayTestMethodName]
        public void RetryPolicySmallerThanZeroRetryInterval()
        {
            int retryCount = 0;
            RetryPolicy retry = new RetryExponential(new TimeSpan(0, 0, 0, 0, 10), new TimeSpan(0, 0, 0, 0, 500), 10);

            TimeSpan? firstRetryInterval =
                retry.GetNextRetryInterval(new SocketException(), TimeSpan.FromSeconds(60), ++retryCount);
            TestUtility.Log("firstRetryInterval: " + firstRetryInterval);
            Assert.True(firstRetryInterval != null);
            Assert.InRange(firstRetryInterval.Value.TotalMilliseconds,0,TimeSpan.MaxValue.TotalMilliseconds);
        }

Expected behavior
Retry policy should always have positive retry intervals
Setup (please complete the following information):

OS: Windows
IDE : Visual Studio 2019
Version of the Library used:
Built from master

Additional context
Add any other context about the problem here.
Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

[x] Bug Description Added
[x] Repro Steps Added
[x] Setup information Added

@serkantkaraca and @JamesBirdsall:  Would you be kind enough to share your thoughts, please?

Just looked into this one out of curiosity and created this quick PR #9370 for discussion. Still needs minor cleanup if this fix makes sense.

PR approved. Thx for the fix!

Closing this out, since the fix has been merged.


init.sh and update self points to 0.8.7 as latest, while it is 0.8.9 currently

init.sh from the official site has
curl -sSfL https://nim-lang.org/choosenim/stable
which still gives 0.8.7
choosenim update self has the same issue, giving 0.8.7 as latest

Seems fixed.


Update schema attributes in doc to match spec (vr.yaml)

see pull request #147
@Reece I also removed NestedInterval from the vr.yaml and vr.json files. I did not use the make vr.json routine to generate the vr.json. I noticed that after I pushed this request. please make sure that is correct.


Combine services with only one dimension

Combine services with only one dimention into single map in detectDimensionsByService.
At the end, the code is only 16 lines shorter, but it nicely separates routine cases from more complicated ones.

I love that one! <3 - Thanks a lot!

Could you please rebase @jylitalo ? :)

Could you please rebase @jylitalo ? :)

Done.


c++ bindings: add subsystem_vendor/device IDs.

Signed-off-by: Tim Whisonant<EMAIL_ADDRESS>

Coverage increased (+0.04%) to 65.497% when pulling 367f62757c947f7ed5e96c66b0248d8760965e6b on tswhison/cxx_bindings into db9a397b5540c0fb3bb2247caeb2dcdbcd088d76 on master.


[tcplp] remove unused type defines

I agree that removing unused definitions from types.h could be beneficial. You seem to have left caddr_t and c_caddr_t in the code, but I would have removed them. Was that intentional?
I understand the reasoning behind removing __int64_t and __uint64_t, but I think we should leave them in for symmetry (it seems kind of odd to define __int32_t but not __int64_t). Additionally, since this code is derived from FreeBSD, I'd rather minimize the number of changes from the FreeBSD version (except for outright deletions), so I'd rather keep the original definition of u_int64_t, even if it means keeping __uint64_t. That said, these are minor points that won't affect functionality, and there are arguments on both sides, so I don't feel strongly about it.
I agree with the rest of your changes in this pull request.

I agree that removing unused definitions from types.h could be beneficial. You seem to have left caddr_t and c_caddr_t in the code, but I would have removed them. Was that intentional?
I understand the reasoning behind removing __int64_t and __uint64_t, but I think we should leave them in for symmetry (it seems kind of odd to define __int32_t but not __int64_t). Additionally, since this code is derived from FreeBSD, I'd rather minimize the number of changes from the FreeBSD version (except for outright deletions), so I'd rather keep the original definition of u_int64_t, even if it means keeping __uint64_t. That said, these are minor points that won't affect functionality, and there are arguments on both sides, so I don't feel strongly about it.
I agree with the rest of your changes in this pull request.

Hi @samkumar I remove these definitions so that the code can be built by toolchains already have those definitions.  Here's the failure: https://github.com/openthread/ot-br-posix/runs/4309864919?check_suite_focus=true
I found only 64bit types are causing conflicts, so I removed them only. Do you have any suggestions no resolving the failure?

Thanks @bukepo for sharing those details. I didn't realize that the build was failing for platforms that already defined those types.
I did resolve one set of issues in Pull Request #7190, which involves the #include guards causing problems for the build in platforms that have similarly named files. sys/types.h was affected by this (since the #include guard conflicted with other files named sys/types.h). Can you check if the build issues show up with the code in that pull request?
I'll take a closer look at this issue in the next few days.

TCP-related build failures on ot-br-posix are causing the build checks on the submodule updates to fail (e.g. https://github.com/openthread/ot-br-posix/pull/1123).
@samkumar , suggestions on how best to move forward?

@jwhui Thanks for the heads up. I will take a look at those failures now.

I took a look at the build failures and managed to reproduce the issue on my laptop. I managed to solve the build issue by removing the quad definitions (u_quad_t, quad_t, and qaddr_t) from bsdtcp/types.h.
This pull request (which seems to have been revised since I last looked at it) removes additional type definitions, which I entirely support. I checked that the latest version of the TCPlp code (in Pull Request #7190) still builds properly with these changes. I know I said above that I wanted to minimize the changes to FreeBSD code, but in this case I think that it is useful, because in this case, so much has been removed from sys/types.h already that it's probably not worth trying to minimize additional changes. I also like the simplicity of the latest version (defining only the unsigned integrals and nothing else).
I think that the appropriate path forward to resolve the build issues is to merge the current pull request. I will review it ASAP so that that can happen soon.


Added aliases only/except for keepKeys/removeKeys

Would be nice to have shorter aliases for keepKeys/removeKeys

Ohh test fail @iamthefox, get it together mate ;)

@iamthefox you'll need same methods for array.

closing. don't really need those aliases


Kanboard is too great

Expected behaviour
Software should have some flaws.  Should be lacking in features, or broken in some way.
Actual behaviour
Software works perfectly.  Completely satisfies user.  Continues to impress user with newly found features.
Steps to reproduce

Visit kanboard.net
Install
Profit

Conclusion
Just wanted to say thank you for such great software.

:+1:

Subscribe.

Me too

i am chinses，how to create a model? i create like taskCreation, but throw a exception: is not defined ,this is why

@moserensheng  Could you maybe open a new question?

@nerdoc my question is submited    could you help me， thank you

@connorbode Thanks :)

@connorbode is totally right. The bugs seem to be too well hidden. 👻
😱 But this will probably be a wontfix bug.


KIP-557: Add Emit On Change Support

This is the initial draft PR for adding emit on change. For reference of the design document, please see here.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-557%3A+Add+emit+on+change+support+for+Kafka+Streams
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@vvcephei  @cadonna @mjsax  Would you recommend this approach for implementing binary equality? I just want some feedback on this, since it seems there are multiple ways to resolve this issue,

test this please

Thanks for this PR @ConcurrencyPractitioner ! I'll take a look at this asap. Just wanted to acknowledge that it's on my plate.

ok to test

test this please

Just FYI, it looks like the tests couldn't build.

@vvcephei Acknowledged. My intention wasn't really to get a working model of the code. I was intending this to be a draft of what the code might look like. I just want approval on this approach first before I dive in.

@vvcephei It looks like casting the store to MeteredTimestampedKeyValueStore doesn't work, as in some cases, the TimestampedKeyValueStore is in fact TimestampedKeyValueStoreReadWriteDecorator. So we will have to abandon this approach. In this case, we will probably need to consider passing in null serdes instead.

Actually, @vvcephei  after some tinkering around, I think this approach is still feasible. Please take a look! :)

@guozhangwang if you are also available.

@vvcephei Do you think you have time to look this?

@vvcephei @mjsax @guozhangwang Pinging for initial review.

test this please

test this please

test this please

test this please

For posterity, here are the tests:
org.apache.kafka.streams.integration.KTableSourceTopicRestartIntegrationTest.shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosEnabled
org.apache.kafka.streams.integration.KTableSourceTopicRestartIntegrationTest.shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosDisabled
org.apache.kafka.streams.integration.KTableSourceTopicRestartIntegrationTest.shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosDisabled
org.apache.kafka.streams.integration.RestoreIntegrationTest.shouldRestoreStateFromChangelogTopic
org.apache.kafka.streams.integration.RestoreIntegrationTest.shouldRestoreStateFromChangelogTopic
org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.shouldEmitTombstoneWhenDeletingNonJoiningRecords[leftJoin=false, optimization=all, materialized=true, rejoin=true]
org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.joinShouldProduceNullsWhenValueHasNonMatchingForeignKey[leftJoin=false, optimization=all, materialized=true, rejoin=true]
org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.shouldEmitTombstoneWhenDeletingNonJoiningRecords[leftJoin=false, optimization=all, materialized=true, rejoin=false]
org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.joinShouldProduceNullsWhenValueHasNonMatchingForeignKey[leftJoin=false, optimization=all, materialized=true, rejoin=false]
org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.shouldEmitTombstoneWhenDeletingNonJoiningRecords[leftJoin=false, optimization=all, materialized=false, rejoin=true]
org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.joinShouldProduceNullsWhenValueHasNonMatchingForeignKey[leftJoin=false, optimization=all, materialized=false, rejoin=true]
org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.shouldEmitTombstoneWhenDeletingNonJoiningRecords[leftJoin=false, optimization=none, materialized=true, rejoin=true]
org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.joinShouldProduceNullsWhenValueHasNonMatchingForeignKey[leftJoin=false, optimization=none, materialized=true, rejoin=true]
org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.shouldEmitTombstoneWhenDeletingNonJoiningRecords[leftJoin=false, optimization=none, materialized=true, rejoin=false]
org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.joinShouldProduceNullsWhenValueHasNonMatchingForeignKey[leftJoin=false, optimization=none, materialized=true, rejoin=false]
org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.shouldEmitTombstoneWhenDeletingNonJoiningRecords[leftJoin=false, optimization=none, materialized=false, rejoin=true]
org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.joinShouldProduceNullsWhenValueHasNonMatchingForeignKey[leftJoin=false, optimization=none, materialized=false, rejoin=true]
org.apache.kafka.streams.kstream.internals.KTableTransformValuesTest.shouldCalculateCorrectOldValuesIfMaterializedEvenIfStateful
org.apache.kafka.streams.kstream.internals.SuppressScenarioTest.shouldImmediatelyEmitEventsWithZeroEmitAfter
org.apache.kafka.streams.kstream.internals.SuppressScenarioTest.shouldWorkBeforeJoinLeft
org.apache.kafka.streams.kstream.internals.SuppressScenarioTest.shouldWorkBeforeJoinRight
org.apache.kafka.streams.kstream.internals.SuppressScenarioTest.shouldSuppressIntermediateEventsWithTimeLimit

@vvcephei Ack. I took a look at some of the tests, and they aren't trivial to fix. I might need some advice because when we are restoring topics, it looks as if some of the old restored records was dropped. Still need to look into it further.

I've found an anomaly. It looks like when we restore the prior records of a topic (at least for shouldRestoreStateFromChangelogTopic), several of the restored records were dropped. I don't really know the context, so I will do some further investigation.

One extra note. Since I've just merged https://github.com/apache/kafka/pull/8414, You'll need to add <Object, Object> type bounds on the ProcessorContext, or the build will fail with "raw type" warnings.
Sorry for the hassle.

@cadonna Tests has been added.

@vvcephei @cadonna I think I've found the bug for the reason why the restore test fails, and for that matter, why a lot of the prior records are dropped. So I added some log statements, and I found that after the store was constructed, no record was inserted more than once. i.e. a previous version of the same record was already in the store after it was just constructed. Following on this hunch, I realized that when the store is being initialized, we see the below line in the store:
        // register and possibly restore the state from the logs
        maybeMeasureLatency(() -> super.init(context, root), time, restoreSensor);

This was in MeteredKeyValueStore. It turns out that some of the records were probably restored from logs during initialization.  Therefore, this leads me to the following question: if we restarted a Kafka application, and some records were restored, are those records already considered "seen" or put another way, should we send these records to the user after restart? This needs to be ironed out first, because the test would have to be modified accordingly afterwards.

For further confirmation, the test I was debugging was org.apache.kafka.streams.integration.RestoreIntegrationTest.shouldRestoreStateFromChangelogTopic. Anyways, the last few lines of the test looked like this:
        assertTrue(startupLatch.await(30, TimeUnit.SECONDS));
        assertThat(restored.get(), equalTo((long) numberOfKeys - 2 * offsetCheckpointed));

        assertTrue(shutdownLatch.await(30, TimeUnit.SECONDS));
        assertThat(numReceived.get(), equalTo(numberOfKeys));

The number of restored records remained the same, and therefore, the test passed that line. However, at the next assertion, where the latch is only decremented when all numberOfKeys records are received, the test times out. The reason is all records after the last offset checkpointed were injected into the store, and thus why several records were dropped. To confirm my suspicion, I switched numberOfKeys to 2 * offsetCheckpointed which was the number of records which hadn't been restored. The test passed.

Test this please

Test this please

test this please

retest this please

ok to test

Hey @ConcurrencyPractitioner , I have yet to review your new code, but I just looked at that test to unblock you.
That test seems to work thusly:

produce unique records for keys 0 to 10,000 to the changelog topic AND to the input topic.
write checkpoint files indicating that the store already contains the first 1,000 records, in each of the two partitions, so we expect to skip over 2,000 records in total when we restore
start up streams and verify that it restores from the checkpoint to the end of the changelog topic. Since we wrote a total of 10,000 unique records to the two topics, and we checkpointed a total of 2,000 of those, we expect to restore 8,000 records (numberOfKeys - 2*offsetCheckpointed = 10,000 - 2 * 1,000 = 8,000)
Finally, verify that we actually process the 10,000 records that we initially produced to the input topic after the restore finishes. Note, these records are logically independent from the ones that we wrote directly to the changelog, but we happen to write the same set of keys to both. The test fails with your change for this reason: processing those input records is now dropping them as idempotent (because they are: they update the same key with the same value at a later time).

The verification in this test is valid, and the only reason it's failing now is that the updates we're supposed to process in step 4 are now considered idempotent. The fix should be to just make them not idempotent. E.g., I'd add an argument to org.apache.kafka.streams.integration.RestoreIntegrationTest#createStateForRestoration to specify the starting key and the number to generate. So we can pre-load the changelog with keys 0 through 9,999 and then pre-load the input topic with keys 10,000 through 20,000. Then, the test should pass, and it's still verifying what it intended to verify.

Oh, I see. That makes sense! It looks like I misunderstood some of what was going on. Thanks for the help. :)

ok to test

test this please

test this please

retest this please

Just to capture it before Jenkins cleans up the jobs, here are the failing tests:
    org.apache.kafka.streams.integration.KTableSourceTopicRestartIntegrationTest.shouldRestoreAndProgressWhenTopicNotWrittenToDuringRestoration
    org.apache.kafka.streams.integration.KTableSourceTopicRestartIntegrationTest.shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosAlphaEnabled
    org.apache.kafka.streams.integration.KTableSourceTopicRestartIntegrationTest.shouldRestoreAndProgressWhenTopicWrittenToDuringRestorationWithEosDisabled
    org.apache.kafka.streams.integration.RestoreIntegrationTest.shouldRestoreStateFromSourceTopic
    org.apache.kafka.streams.integration.RestoreIntegrationTest.shouldRestoreStateFromChangelogTopic
    org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.shouldEmitTombstoneWhenDeletingNonJoiningRecords[leftJoin=false, optimization=all, materialized=true, rejoin=true]
    org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.joinShouldProduceNullsWhenValueHasNonMatchingForeignKey[leftJoin=false, optimization=all, materialized=true, rejoin=true]
    org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.shouldEmitTombstoneWhenDeletingNonJoiningRecords[leftJoin=false, optimization=all, materialized=true, rejoin=false]
    org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.joinShouldProduceNullsWhenValueHasNonMatchingForeignKey[leftJoin=false, optimization=all, materialized=true, rejoin=false]
    org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.shouldEmitTombstoneWhenDeletingNonJoiningRecords[leftJoin=false, optimization=all, materialized=false, rejoin=true]
    org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.joinShouldProduceNullsWhenValueHasNonMatchingForeignKey[leftJoin=false, optimization=all, materialized=false, rejoin=true]
    org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.shouldEmitTombstoneWhenDeletingNonJoiningRecords[leftJoin=false, optimization=none, materialized=true, rejoin=true]
    org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.joinShouldProduceNullsWhenValueHasNonMatchingForeignKey[leftJoin=false, optimization=none, materialized=true, rejoin=true]
    org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.shouldEmitTombstoneWhenDeletingNonJoiningRecords[leftJoin=false, optimization=none, materialized=true, rejoin=false]
    org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.joinShouldProduceNullsWhenValueHasNonMatchingForeignKey[leftJoin=false, optimization=none, materialized=true, rejoin=false]
    org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.shouldEmitTombstoneWhenDeletingNonJoiningRecords[leftJoin=false, optimization=none, materialized=false, rejoin=true]
    org.apache.kafka.streams.integration.KTableKTableForeignKeyJoinIntegrationTest.joinShouldProduceNullsWhenValueHasNonMatchingForeignKey[leftJoin=false, optimization=none, materialized=false, rejoin=true]
    org.apache.kafka.streams.kstream.internals.KTableTransformValuesTest.shouldCalculateCorrectOldValuesIfMaterializedEvenIfStateful
    org.apache.kafka.streams.kstream.internals.SuppressScenarioTest.shouldImmediatelyEmitEventsWithZeroEmitAfter
    org.apache.kafka.streams.kstream.internals.SuppressScenarioTest.shouldWorkBeforeJoinLeft
    org.apache.kafka.streams.kstream.internals.SuppressScenarioTest.shouldWorkBeforeJoinRight
    org.apache.kafka.streams.kstream.internals.SuppressScenarioTest.shouldSuppressIntermediateEventsWithTimeLimit

Yep, I will see what I can do about this.

Cool. No pressure, just wanted to write it down because it's so frustrating when you click for the details but just get a 404 instead.
By the way, I was thinking I'd do another review pass when the tests are passing; sound good?

Yeah, thats fine with me. :)

@vvcephei I made some changes to the tests so that a couple of them are now passing. But there is some uncertainty that I came across. Specifically, on some tests for SuppressScenarioTest, the same record was inserted (but just with different timestamp). However, could you check to make sure that after my changes, the test is still legitimate? Just wanting some confirmation.

Sure thing, @ConcurrencyPractitioner , I'll take a look at the SuppressScenarioTest when I get a chance.

Hey @ConcurrencyPractitioner , sorry it took so long. It's just again because the test happened to expect idempotent updates to flow through regularly, but not for anything important. Just changing the value of the "tick" record the second time fixes it without breaking anything about the test.
Here's my diff:
--- a/streams/src/test/java/org/apache/kafka/streams/kstream/internals/SuppressScenarioTest.java
+++ b/streams/src/test/java/org/apache/kafka/streams/kstream/internals/SuppressScenarioTest.java
@@ -615,11 +615,11 @@ public class SuppressScenarioTest {
             );

-            inputTopicRight.pipeInput("tick", "tick", 21L);
+            inputTopicRight.pipeInput("tick", "tick1", 21L);
             verify(
                 drainProducerRecords(driver, "output", STRING_DESERIALIZER, STRING_DESERIALIZER),
                 asList(
-                    new KeyValueTimestamp<>("tick", "(null,tick)", 21), // just a testing artifact
+                    new KeyValueTimestamp<>("tick", "(null,tick1)", 21), // just a testing artifact
                     new KeyValueTimestamp<>("A", "(b,2)", 13L)
                 )
             );
@@ -703,11 +703,11 @@ public class SuppressScenarioTest {
             );

-            inputTopicLeft.pipeInput("tick", "tick", 21L);
+            inputTopicLeft.pipeInput("tick", "tick1", 21L);
             verify(
                 drainProducerRecords(driver, "output", STRING_DESERIALIZER, STRING_DESERIALIZER),
                 asList(
-                    new KeyValueTimestamp<>("tick", "(tick,null)", 21), // just a testing artifact
+                    new KeyValueTimestamp<>("tick", "(tick1,null)", 21), // just a testing artifact
                     new KeyValueTimestamp<>("A", "(2,b)", 13L)
                 )
             );

I also took a look at the foreign-key join test, which is actually telling us something awesome: your feature allows us to drop unnecessary tombstones that we'd otherwise send under some conditions.
Anyway, it's complicated, so here's a fix for the test:
--- a/streams/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java
+++ b/streams/src/test/java/org/apache/kafka/streams/integration/KTableKTableForeignKeyJoinIntegrationTest.java
@@ -48,6 +48,7 @@ import java.util.Properties;
 import java.util.function.Function;

 import static java.util.Collections.emptyMap;
+import static java.util.Collections.singletonMap;
 import static org.apache.kafka.common.utils.Utils.mkEntry;
 import static org.apache.kafka.common.utils.Utils.mkMap;
 import static org.apache.kafka.common.utils.Utils.mkProperties;
@@ -371,12 +372,16 @@ public class KTableKTableForeignKeyJoinIntegrationTest {

             // Deleting a non-joining record produces an unnecessary tombstone for inner joins, because
             // it's not possible to know whether a result was previously emitted.
+            // HOWEVER, when the final join result is materialized (either explicitly or
+            // implicitly by a subsequent join), we _can_ detect that the tombstone is unnecessary and drop it.
             // For the left join, the tombstone is necessary.
             left.pipeInput("lhs1", (String) null);
             {
                 assertThat(
                     outputTopic.readKeyValuesToMap(),
-                    is(mkMap(mkEntry("lhs1", null)))
+                    is(leftJoin || !(materialized || rejoin)
+                           ? mkMap(mkEntry("lhs1", null))
+                           : emptyMap())
                 );
                 if (materialized) {
                     assertThat(
@@ -452,12 +457,15 @@ public class KTableKTableForeignKeyJoinIntegrationTest {
             }
             // "moving" our subscription to another non-existent FK results in an unnecessary tombstone for inner join,
             // since it impossible to know whether the prior FK existed or not (and thus whether any results have
-            // previously been emitted)
+            // previously been emitted). HOWEVER, when the final join result is materialized (either explicitly or
+            // implicitly by a subsequent join), we _can_ detect that the tombstone is unnecessary and drop it.
             // The left join emits a _necessary_ update (since the lhs record has actually changed)
             left.pipeInput("lhs1", "lhsValue1|rhs2");
             assertThat(
                 outputTopic.readKeyValuesToMap(),
-                is(mkMap(mkEntry("lhs1", leftJoin ? "(lhsValue1|rhs2,null)" : null)))
+                is(leftJoin
+                       ? mkMap(mkEntry("lhs1", "(lhsValue1|rhs2,null)"))
+                       : (materialized || rejoin) ? emptyMap() : singletonMap("lhs1", null))
             );
             if (materialized) {
                 assertThat(
@@ -469,7 +477,9 @@ public class KTableKTableForeignKeyJoinIntegrationTest {
             left.pipeInput("lhs1", "lhsValue1|rhs3");
             assertThat(
                 outputTopic.readKeyValuesToMap(),
-                is(mkMap(mkEntry("lhs1", leftJoin ? "(lhsValue1|rhs3,null)" : null)))
+                is(leftJoin
+                       ? mkMap(mkEntry("lhs1", "(lhsValue1|rhs3,null)"))
+                       : (materialized || rejoin) ? emptyMap() : singletonMap("lhs1", null))
             );
             if (materialized) {
                 assertThat(

The source topic restart integration test was actually just failing because the tests were polluting each others' topics. This is one way to fix it:
diff --git a/streams/src/test/java/org/apache/kafka/streams/integration/KTableSourceTopicRestartIntegrationTest.java b/streams/src/test/java/org/apache/kafka/streams/integration/KTableSourceTopicRestartIntegrationTest.java
index 3ec239fab9..b42a5852a5 100644
--- a/streams/src/test/java/org/apache/kafka/streams/integration/KTableSourceTopicRestartIntegrationTest.java
+++ b/streams/src/test/java/org/apache/kafka/streams/integration/KTableSourceTopicRestartIntegrationTest.java
@@ -100,6 +100,7 @@ public class KTableSourceTopicRestartIntegrationTest {
     @After
     public void after() throws Exception {
         IntegrationTestUtils.purgeLocalStreamsState(STREAMS_CONFIG);
+        CLUSTER.deleteAllTopicsAndWait(60_000L);
     }

     @Test

Ok, last one. The TransformValuesTest is just another case where the test input data is now considered idempotent, which is fine:
--- a/streams/src/test/java/org/apache/kafka/streams/kstream/internals/KTableTransformValuesTest.java
+++ b/streams/src/test/java/org/apache/kafka/streams/kstream/internals/KTableTransformValuesTest.java
@@ -398,8 +398,8 @@ public class KTableTransformValuesTest {
                 driver.createInputTopic(INPUT_TOPIC, new StringSerializer(), new StringSerializer());

         inputTopic.pipeInput("A", "ignored", 5L);
-        inputTopic.pipeInput("A", "ignored", 15L);
-        inputTopic.pipeInput("A", "ignored", 10L);
+        inputTopic.pipeInput("A", "ignored1", 15L);
+        inputTopic.pipeInput("A", "ignored2", 10L);

         assertThat(output(), hasItems(new KeyValueTimestamp<>("A", "1", 5),
                 new KeyValueTimestamp<>("A", "0", 15),

I hope I didn't step on your toes, @ConcurrencyPractitioner , but I just wanted to make sure that you're unblocked to finish up this PR. Figuring out what's exactly wrong with those tests and whether it's ok can be a bit subtle.

@vvcephei Thanks for all the help! It would've probably taken me quite a while to get through all this by myself. I will update PR shortly. :)

ok to test

test

@vvcephei Included your test changes in the PR. :)

retest this please

test

ok to test

test this please

retest this please

FYI, a recent change is that only committers can kick off the tests. It's mighty inconvenient, but better for security.

@vvcephei  Oh yeah, I was aware of that. Just recently though, I was added to the jenkins whitelist (my handle is listed in .asf.yaml). Supposedly, I will be able to trigger tests after doing that, but it appears that wasn't effective.

@vvcephei  Can you retrigger tests?

Ah, congrats!
In that case, try saying what I said before.

test this please

test this please

On second thought, once the whitelist takes effect, it should just automatically run tests when you push. Maybe it'll only get picked up when you open a new PR?

This one seemed to fail consistently: org.apache.kafka.streams.integration.RestoreIntegrationTest.shouldRestoreStateFromSourceTopic

@vvcephei Alright, will get this one fixed.

@vvcephei Alright, got the test fixed. Turned out I made a change which had broken the test, just that I hadn't caught it until now. This PR should be good for review. :)

test this please

test this please

@vvcephei  Looks like we have some flakies. But I think we can get a review in. :)

Java 11:
org.apache.kafka.streams.examples.wordcount.WordCountDemoTest.testOneWord
org.apache.kafka.streams.examples.wordcount.WordCountDemoTest.testCountListOfWords
were both:
org.apache.kafka.streams.errors.LockException: stream-thread [Test worker] task [0_0] Failed to lock the state directory for task 0_0
and java 8 was only:
kafka.api.PlaintextProducerSendTest.testNonBlockingProducer
So, it looks like an "effectively passing" build to me.
Nevertheless,
retest this please

Just to be sure we don't break the system tests upon merging, I've kicked off the branch-builder: https://jenkins.confluent.io/job/system-test-kafka-branch-builder/3919/console

Whew! System tests passed: http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/2020-04-28--001.1588064884--ConcurrencyPractitioner--EMIT-ON-CHANGE--ddbf2cf/report.html

test this please

test this please

test this please

retest this please

@vvcephei Looks like I forgot to push. You can retrigger the tests after this one.

@vvcephei Alright, all tests passed (looks like there were some flakies) and all comments addressed.

@vvcephei Just pinging. I looked into the flakies, and it turns out they were different test failures for each build. So its the equivalent of an all green.

Retest this please

Sorry, for the delay @ConcurrencyPractitioner . I have been insanely busy. I really appreciate your patience.
I'm giving it a final pass now.

@vvcephei Alright, cool. Got those last comments addressed.

Unrelated test failures:
kafka.api.SslConsumerTest.testCoordinatorFailover
org.apache.kafka.streams.integration.EosBetaUpgradeIntegrationTest.shouldUpgradeFromEosAlphaToEosBeta[true]
kafka.api.ConsumerBounceTest.testClose
org.apache.kafka.connect.mirror.MirrorConnectorsIntegrationTest.testReplication

Hey @ConcurrencyPractitioner , I just pushed a very minor test fix. I'm verifying it locally before merging.

Thanks for the awesome contribution, @ConcurrencyPractitioner !


Canvas not working on Apple devices (desktop and mobile)

Hi - firstly thanks for a great library that allows us to easily interop with the canvas from Blazor!
I've used your library in a game that I've written in Blazor - it's PACMAN at https://pacmanblazor.azurewebsites.net/
The source code is here: https://github.com/SteveDunn/PacManBlazor
It works great everywhere exception Apple devices.  The problem is that I use an off-screen canvas to draw on and then, when ready, draw that canvas to the another visible canvas.
I don't have access to any Apple devices to see what's going on, so I can't provide any more details other than 'it doesn't work on anything Apple!'
Any help would be greatly appreciated!

Hi Steve!
I've just tried your awesome web-application on my MacMini Late 2014. Yeah, I've encountered same problem in Safari browser. However, when I've tried it in Google Chrome (on the same Mac), it works perfect! I can see a maze, enemies and so on. I think, the problem is just in Safari.

Thank you very @CeSiumUA !  Awfully sorry about the extremely slow response!  I was just clearing out my inbox and found an e-mail re. this notification from 6 months ago!  Again, many thanks!

Thank you very @CeSiumUA !  Awfully sorry about the extremely slow response!  I was just clearing out my inbox and found an e-mail re. this notification from 6 months ago!  Again, many thanks!


Standardise SVG CSP

Provisional, needs further r&d/testing:
add_header Content-Security-Policy "default-src 'none'; style-src 'self' 'unsafe-inline'";

@philwareham - additional brain requested…I think I've cracked it.
Based on the above provisional code, default-src 'none' should switch any CSP directive that's not explicitly mentioned to none, and the style-src should open up just that directive…right? That's some future-proofing with the default-src as a catch-all umbrella…right?

Yes, that is generally correct. There are a few directives that don't automatically fallback to using the default-src though if not included - mostly where doing so would cause major breakages.

Can you point me to a list of these, if there is one?

I think the docs linked in my CSP state how a directive falls back (in blue box near top of document). For example: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/base-uri

Done


v.getEdges returns a deleted ghost edge from cache

#665
This definitely applies to 0.4.  Tried the test twice on titan05 and it didn't fail on either run, FWIW...

Given that this doesn't apply to newer versions and 0.4 is no longer supported this is a won't fix.


creating an Sleepy End Device (SED) (CON-467)

Hello,
Hello,
I would like to create a SED. In the Matter specification it is indicated that we have to change the value of the field
SLEEPY_IDLE_INTERVAL  or SLEEPY_ACTIVE_INTERVAL depending on the mode we want to use. However, I couldn't find anything in the sdkconfig and in the esp-matter files (esp-matter/components). Is it possible to do this or is it a feature that will come later.
Most probably I did not look in the right place.
Thanks in advance for your answer.

@glennmullerar It's configurable by CHIP_CONFIG_MRP_LOCAL_ACTIVE_RETRY_INTERVAL and CHIP_CONFIG_MRP_LOCAL_IDLE_RETRY_INTERVAL, the detailed description about these two parameters are here:
https://github.com/project-chip/connectedhomeip/blob/master/src/messaging/ReliableMessageProtocolConfig.h#L52
Could you modify the default values and have a test, let us know if it works for you. We will make it configurable via IDF menuconfig.

Thank you for your answer.
I see that by changing the value the number of steps at the time of the build changes.
However, is there a chip-tool or other command that would allow to know if it is well known as SED by the other devices in the network?

It's could be achieved by the new ICD (Intermittently Connected Devices) feature, and there is a clientmonitoring cluster cmd can be used by chip-tool.
But the feature is currently not supported in esp-matter yet, since the feature is still under developing, we will add the support in short future. Could you let us know which SoC are you using, Wi-Fi or Thread?

I'm trying to make a SED with ESP32-H2 using thread. Is that not supported yet as well? @chshu

@marciogranzotto The H2 sleep mode is not supported yet (IDF and esp-matter), will keep you posted.

What about ESP32C6? Is it possible to create Thread Sleep End Device on ESP32C6 basis?

@chshu is it posible with the C6?

The aim is to reduce the power consumption of a WiFi device.
@glennmullerar If it's a Wi-Fi device, you don't need to wait for the ICD feature, you can just enable the Wi-Fi power save mode (example).

@Gonta01 @marciogranzotto To enable the Thread Sleepy End Device on H2 and C6 for Matter application, the following two tasks need to be done:

Light Sleep support for Thread in esp-idf
ICD feature support in esp-matter

We target to have the two features ready by the end of July.

I’m’also looking for this feature in WiFi and Thread.
I want to see the average power consumption we can expect with esp32-c6 in wifi and thread as sleepy end device.
The objective is to compare nRF52 vs ESP32-C6.
I prefer the esp ecosystem but high power consumption will not be acceptable for my application.

@chshu I see that the Light Sleep support for Thread in esp-idf is already implemented. Is there any news about the ICD/SED feature in esp-matter?
Looking forward for it so I can get my product working!

@LinJiaDeng @caipiblack @marciogranzotto The Thread light sleep on H2 and C6 are all supported in ESP-IDF now, you may first refer to the H2 datasheet for some power consumption data, and evaluate the H2 / C6 power consumption using this example: https://github.com/espressif/esp-idf/tree/master/examples/openthread/ot_sleepy_device.
We are actively adding the ICD support to esp-matter, will posted here once it's ready.

@chshu any news?
If there's anything I can help with on the implementation, I would be happy to

@marciogranzotto The ICD feature itself has been supported in Matter ESP platform (https://github.com/project-chip/connectedhomeip/pull/28359), some sleep tunning work is still under progress, will update here once it's done (target to mid Oct).

The aim is to reduce the power consumption of a WiFi device.
@glennmullerar If it's a Wi-Fi device, you don't need to wait for the ICD feature, you can just enable the Wi-Fi power save mode (example).

Hi @chshu,
I want to check power consumption on ESP32-C6 for Wi-Fi, as you mentioned how to enable Wi-Fi power save mode?

@glennmullerar @marciogranzotto The Matter ICD support on ESP32-H2 is available in esp-matter now, please check the example: https://github.com/espressif/esp-matter/tree/main/examples/sleepy_device

@chshu just for the H2? what about C6?

The example does not compile for the C6 out of the box. I was able to make it compiling after I've removed the espressif/mdns and /insights packages and a few other minor h2->c6 renaming. For HomeKit commissioning I had to comment out the BLE Sleep changes.
Even though, I'm not 100% sure if the example works as expected. I have a cheap USB power meter that says that my C6 still needs 40mA / 200mW which should be way too high. Light sleep on ESP should take 0.8 mA / 4mW.
Can you tell me how to check which mode the ESP32 is currently in and if this behavior is expected?

@marciogranzotto @flashspys Currently the example only supports ESP32-H2 Thread Sleepy device.
C6 Thread is pending on a sleep issue, will be supported once the issue is fixed in IDF.
C6 Wi-Fi support will be added later.

@chshu any news on the C6 sleep issue? Btw, will this be on which version on the idf? I'm currently developing against release/v5.1

@chshu any news on the C6 sleep issue? Btw, will this be on which version of the IDF? I'm currently developing against release/v5.1

@chshu any news on the C6 sleep issue? Btw, will this be on which version of the IDF? I'm currently developing against release/v5.1

The C6 sleep issue has been fixed in IDF, but some compiling issues (as mentioned in https://github.com/espressif/esp-matter/issues/366#issuecomment-1771792028) depend on some changes in other repos, the fixes are under merging.
We will prepare a patch for your testing soon, you may also evaluate the Matter sleep feature on H2 first.

Hi, When can C2/C3 support deep sleep?

[Questions about features added to ESP-IDF v5.1]
Does this mean that the wifi sleep (SED, ICD) function is supported in the ESP32C6 Matter project?
Does this mean that the thread sleep function is supported in the ESP32C6 Matter project?
Does the ESP32C3 Matter project also support wifi sleep (SED, ICD) function?

[Questions about features added to ESP-IDF v5.1]
Does this mean that the WIFI sleep (SED, ICD) function is supported in the ESP32C6 Matter project?

I can not confirm WiFi sleepy, I will have to try later, but I can confirm that Power Management (PM) and ICD server can be enabled on the c6.

Does this mean that the Thread sleep function is supported in the ESP32C6 Matter project?

Yes. After following the batches advised from Espressif, modifying the upstream project chip cmake and idf component files, plus a little bit of tweaking for my project to reenable secure-cert-manager, I was able to make a Matter over Thread Sleepy generic_switch endpoint on the C6 - https://github.com/espressif/esp-matter/assets/5651603/8d2431ba-2cc5-4e57-b3b3-dabd95e6a89f

Does the ESP32C3 Matter project also support WIFI sleep (SED, ICD) function?

I cannot confirm, I don't have a C3, but if WiFi sleep works for the C6 I bet it would work for the C3.

I also applied the patches, using esp-idf commit a36d8bc742 (release/v5.2) and esp-matter commit 96dc30b (release/v1.2).
However my ESP32-C6-DevKitC-1 keeps crashing when running the sleepy_device example:

Logs (click to expand)
Rebooting...
ESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0xc (SW_CPU),boot:0xc (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:2
load:0x40875720,len:0x1804
load:0x4086c410,len:0xde0
load:0x4086e610,len:0x2df8
entry 0x4086c41a
I (25) boot: ESP-IDF v5.2-beta1-226-ga36d8bc742 2nd stage bootloader
I (26) boot: compile time Dec 19 2023 19:37:56
I (27) boot: chip revision: v0.0
I (30) boot.esp32c6: SPI Speed      : 80MHz
I (34) boot.esp32c6: SPI Mode       : DIO
I (39) boot.esp32c6: SPI Flash Size : 4MB
I (44) boot: Enabling RNG early entropy source...
I (49) boot: Partition Table:
I (53) boot: ## Label            Usage          Type ST Offset   Length
I (60) boot:  0 esp_secure_cert  unknown          3f 06 0000d000 00002000
I (68) boot:  1 nvs              WiFi data        01 02 00010000 0000c000
I (75) boot:  2 nvs_keys         NVS keys         01 04 0001c000 00001000
I (82) boot:  3 otadata          OTA data         01 00 0001d000 00002000
I (90) boot:  4 phy_init         RF data          01 01 0001f000 00001000
I (97) boot:  5 ota_0            OTA app          00 10 00020000 001e0000
I (105) boot:  6 ota_1            OTA app          00 11 00200000 001e0000
I (112) boot:  7 fctry            WiFi data        01 02 003e0000 00006000
I (120) boot: End of partition table
I (125) esp_image: segment 0: paddr=00020020 vaddr=42110020 size=38c68h (232552) map
I (226) esp_image: segment 1: paddr=00058c90 vaddr=40800000 size=07388h ( 29576) load
I (241) esp_image: segment 2: paddr=00060020 vaddr=42000020 size=10399ch (1063324) map
I (668) esp_image: segment 3: paddr=001639c4 vaddr=40807388 size=0f284h ( 62084) load
I (698) esp_image: segment 4: paddr=00172c50 vaddr=50000000 size=00048h (    72) load
I (708) boot: Loaded app from partition at offset 0x20000
I (709) boot: Disabling RNG early entropy source...
I (720) cpu_start: Unicore app
W (730) clk: esp_perip_clk_init() has not been implemented yet
I (736) cpu_start: Pro cpu start user code
I (736) cpu_start: cpu freq: 160000000 Hz
I (737) cpu_start: Application information:
I (739) cpu_start: Project name:     sleepy_device
I (745) cpu_start: App version:      1.0
I (749) cpu_start: Compile time:     Dec 19 2023 19:50:01
I (755) cpu_start: ELF file SHA256:  66248d460...
I (761) cpu_start: ESP-IDF:          v5.2-beta1-226-ga36d8bc742
I (767) cpu_start: Min chip rev:     v0.0
I (772) cpu_start: Max chip rev:     v0.99
I (777) cpu_start: Chip rev:         v0.0
I (782) heap_init: Initializing. RAM available for dynamic allocation:
I (789) heap_init: At 40828F40 len 000536D0 (333 KiB): RAM
I (795) heap_init: At 4087C610 len 00002F54 (11 KiB): RAM
I (801) heap_init: At 50000048 len 00003FA0 (15 KiB): RTCRAM
I (808) spi_flash: detected chip: generic
I (812) spi_flash: flash io: dio
W (816) spi_flash: Detected size(8192k) larger than the size in the binary image header(4096k). Using the size in the binary image header.
I (830) sleep: Enable automatic switching of GPIO sleep configuration
I (837) sleep_clock: System Power, Clock and Reset sleep retention initialization
I (845) sleep_clock: Modem Power, Clock and Reset sleep retention initialization
I (853) sleep_sys_periph: Interrupt Matrix sleep retention initialization
I (860) sleep_sys_periph: HP System sleep retention initialization
I (867) sleep_sys_periph: TEE/APM sleep retention initialization
I (874) sleep_sys_periph: UART sleep retention initialization
I (880) sleep_sys_periph: Timer Group sleep retention initialization
I (887) sleep_sys_periph: IO Matrix sleep retention initialization
I (894) sleep_sys_periph: SPI Mem sleep retention initialization
I (901) sleep_sys_periph: SysTimer sleep retention initialization
I (919) pm: Frequency switching config: CPU_MAX: 160, APB_MAX: 80, APB_MIN: 40, Light sleep: DISABLED
I (919) coexist: coex firmware version: e41c5cb
I (923) coexist: coexist rom version 5b8dcfa
I (928) main_task: Started on CPU0
I (931) main_task: Calling app_main()
I (952) pm: Frequency switching config: CPU_MAX: 160, APB_MAX: 160, APB_MIN: 160, Light sleep: ENABLED
I (970) chip[DL]: NVS set: chip-counters/reboot-count = 1685 (0x695)
I (972) chip[DL]: Real time clock set to 946684800 (0100/00/01 00:00:00 UTC)
I (974) BLE_INIT: Using main XTAL as clock source
I (976) esp openthread sleep: Enable ieee802154 light sleep, the wake up source is ESP timer
I (989) OPENTHREAD: Host connection mode none
controller lib commit: [9a23742]
W (996) BLE_INIT: BLE modem sleep is enabled
I (995) phy_init: phy_version 230,c773401,Oct 30 2023,15:07:16
I (1003) BLE_INIT: Modem BLE MAC retention initialization
W (1015) BLE_INIT: Enable light sleep, the wake up source is BLE timer
I (1050) phy: libbtbb version: 7243671, Oct 30 2023, 15:07:30
I (1050) btbb_init: btbb sleep retention initialization
I (1052) ieee802154: ieee802154 mac sleep retention initialization
I (1056) NimBLE: GAP procedure initiated: stop advertising.

I (1065) NimBLE: Failed to restore IRKs from store; status=8

I (1071) CHIP[DL]: BLE host-controller synced
I (1096) OPENTHREAD: OpenThread attached to netif
I(1107) OPENTHREAD:[N] Mle-----------: Role disabled -> detached
I (1119) chip[DL]: OpenThread ifconfig up and thread start
I (1123) chip[DL]: OpenThread started: OK
I (1123) chip[DL]: Setting OpenThread device type to SLEEPY END DEVICE
I (1142) OT_STATE: netif up
I (1580) chip[DL]: Configuring CHIPoBLE advertising (interval 25 ms, connectable)
I (1630) NimBLE: GAP procedure initiated: advertise;
I (1631) NimBLE: disc_mode=2
I (1631) NimBLE:  adv_channel_map=0 own_addr_type=1 adv_filter_policy=0 adv_itvl_min=40 adv_itvl_max=40
I (1639) NimBLE:

BLE assert: line 1420 in function ble_lll_sched_check_remaining_entries, param: 0x1, 0x6000
assert failed: osi_assert_wrapper bt.c:263 (0)
Core  0 register dump:
MEPC    : 0x408004b8  RA      : 0x4080d4e2  SP      : 0x40816e50  GP      : 0x40814864
Stack dump detected
0x408004b8: panic_abort at /Users/marciorodrigues/Olibra/code/matter/esp-idf/components/esp_system/panic.c:472

0x4080d4e2: __ubsan_include at /Users/marciorodrigues/Olibra/code/matter/esp-idf/components/esp_system/ubsan.c:313

TP      : 0x40805be4  T0      : 0x37363534  T1      : 0x7271706f  T2      : 0x33323130
0x40805be4: r_hal_timer_set_exp_tick.part.0 at hal_timer.c.o:?

MHARTID : 0x00000000

Backtrace:

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/marciorodrigues/Olibra/code/matter/.espressif/tools/riscv32-esp-elf-gdb/12.1_20221002/riscv32-esp-elf-gdb/bin/riscv32-esp-elf-gdb-3.9.Awd8AYd
panic_abort (details=details@entry=0x40816ea8 <xIsrStack+1256> "assert failed: osi_assert_wrapper bt.c:263 (0)") at /Users/marciorodrigues/Olibra/code/matter/esp-idf/components/esp_system/panic.c:472
472         *((volatile int *) 0) = 0; // NOLINT(clang-analyzer-core.NullDereference) should be an invalid operation on targets
#0  panic_abort (details=details@entry=0x40816ea8 <xIsrStack+1256> "assert failed: osi_assert_wrapper bt.c:263 (0)") at /Users/marciorodrigues/Olibra/code/matter/esp-idf/components/esp_system/panic.c:472
#1  0x4080d4e2 in esp_system_abort (details=details@entry=0x40816ea8 <xIsrStack+1256> "assert failed: osi_assert_wrapper bt.c:263 (0)") at /Users/marciorodrigues/Olibra/code/matter/esp-idf/components/esp_system/port/esp_system_chip.c:93
#2  0x4081307c in __assert_func (file=<optimized out>, file@entry=0x42120513 "", line=line@entry=263, func=<optimized out>, func@entry=0x421379ec <__func__.3> "", expr=<optimized out>, expr@entry=0x421180dc "") at /Users/marciorodrigues/Olibra/code/matter/esp-idf/components/newlib/assert.c:81
#3  0x40802b8a in osi_assert_wrapper (ln=<optimized out>, fn=<optimized out>, param1=<optimized out>, param2=<optimized out>) at /Users/marciorodrigues/Olibra/code/matter/esp-idf/components/bt/controller/esp32c6/bt.c:263
#4  0x40804cde in ble_lll_sched_check_remaining_entries ()
Backtrace stopped: frame did not save the PC
ELF file SHA256: 66248d460

Can you please advise?

@Pips801 could you tell me which commits of the esp-idf and esp-matter you use to achieve C6 Thread sleepy device? I would greatly appreciate it!

@Pips801 could you tell me which commits of the esp-idf and esp-matter you use to achieve C6 Thread sleepy device? I would greatly appreciate it!

I will have to go check the build environment, but it is IDF v5.1.1 with the latest matter 1.2 repository for esp-matter.
Can you post your sdkconfig for the sleepy c6? I ended up copying most everything from the Sleepy C2 example code and the C2 sdkcondig.
I did also encounter some sporadic crashes like yours with the Bluetooth stack. One thing that I changed, that may help, was changing CONFIG_USE_BLE_ONLY_FOR_COMMISSIONING from y to n in the sdkconfig for the C6. since the panic happens right at a ble-related function in the ble stack.

I've tried with both CONFIG_USE_BLE_ONLY_FOR_COMMISSIONING set to y and to n. I also tested the sleepy_device example, and the light example with the sleepy_device changes applied to it.
I keep getting the same crash on ble_lll_sched_check_remaining_entries, which appears to be on the lower lever bluetooth driver of the ESP32-C6.
I'll try to zip my project later and post it here.

Does this mean that the WIFI sleep (SED, ICD) function is supported in the ESP32C6 Matter project?

No, Wi-Fi ICD is still not supported for our platform. In fact, there is no SED for Wi-Fi devices. The Wi-Fi should wake up to keep the connection per 100ms~500ms

Does this mean that the Thread sleep function is supported in the ESP32C6 Matter project?

Yes, the Thread sleep is supported for ESP32C6 for IDF tag v5.1.2, and you can build the sleepy-device examples for C6 if you have applied the two patches

Does the ESP32C3 Matter project also support WIFI sleep (SED, ICD) function?

No.

Can you please advise?

@marciogranzotto @Pips801 Sorry I forgot to add the phy configuration CONFIG_ESP_PHY_MAC_BB_PD=y to the C6's sdkconfig file in the patch file, please add that configuration option and build the example based on IDF release/v5.1(commit id 47c89441c16e0933474c8dc7d8ca9b3f75d2a1da).

Here is the current picture of C6 with the same default ICD parameters as H2.

Hi All, Both H2 and C6 ICD support is ready, please try out this example: https://github.com/espressif/esp-matter/tree/main/examples/icd_app.
Note that the IDF release/v5.1 branch with the commit id 931eaf7320 or later, should be used for C6 sleep.

@marciogranzotto  Did you manage to reduce esp32 consumption, and what values did you measure?
My measurements are esp32h2 3mA and esp32c6 9mA
this is too much consumption.

Hello,
In icd_app readme (https://github.com/espressif/esp-matter/tree/main/examples/icd_app) it say's:
Note: For ESP32-C6, please use ESP-IDF on branch release/v5.1 with the commit id 931eaf7320.
So:

Why do we have to use this specific commit ?
Is it possible to use newer commits ?
What is the real status of icd_app ? Working ?

How to adjust the 802.15.4 TX Power in  lit-icd-app  ( Matter examples).  I am using a ESP32h2 device as portable switch ( battery backed).  I need to adjust the TX Power to reduce power consumption from the default +20bm to +9dm.


UI Tests

Created the ViewControllerUITests file with various UI tests on the login screen.  I'm open to suggestions for new tests and will create a new branch with tests for the Project List screen when we have that complete.

You need to merge master into the branch

I think it's fixed @JasonRitz


What will genai do if there are same model names but in different LLM adapters?

Bug description
In other LLM libraries you have a type for a specific adapter, and then you specify model name as a string.
However, in genai there is only one parameter - model name.
While in practice it's probably not a big issue, but in my opinion this is a bit wrong.
The main question that came in my mind is: what if both company X and Y release a model under the same name A?
You know, there two OpenAI's: one from OpenAI company, other is Azure...
The other very important thing:
Two main applications to load LLM models locally are: ollama and GPT4All. It's certainly possible to have same model names in both ollama and GPT4All.
I know you don't support GPT4All now, but I noticed in the algorithm for AdapterKind::from_model that if model name doesn't fit into any criteria, then it will be treated as a model from ollama. This might introduce problems if you add GPT4All

Good point. So, here is the current thinking:
1.	We have the ModelIden, which includes both AdapterKind and the model name.
2.	I might add an openai::model_name to have a serialized way to force an adapter kind for a model string (I’m trying to keep the model as &str in exec_chat for simplicity).
3.	There’s a ModelMapper to map a ModelIden to another. This provides a lot of flexibility (building on point 2).
4.	I’m also planning to have an EndpointResolver to resolve a function that provides an endpoint URL. The structure for this is yet to be decided.
5.	I agree that the fallback to the Ollama adapter isn’t ideal, but it's functional for now. I may allow configuration of the fallback AdapterKind at the client level.

Recently I encountered  similar problem, I had fine-tuned model in OpenAI platform, by default they starts with prefix ft:{model-name}:personal:{unique-id}. With current setup genai tried to call some model in Ollama backend, when it should use OpenAI. Some libraries solves this by introducing prefixes for models names, what explicitly states which engine need to be used. Personally I resolved it by providing my own ModelMapper, but I would rather treat this as workaround, not proper solution
    let model_mapper = ModelMapper::from_mapper_fn(|model_iden: ModelIden| {
        Ok(ModelIden::new(AdapterKind::OpenAI, model_iden.model_name))
    });
    let client = Client::builder().with_model_mapper(model_mapper).build();


Document ArtifactView API

This could be included in artifact_transforms.adoc.

This also should cover how to get to the metadata files for custom processing.
It should replace of supplement https://docs.gradle.org/5.6.3/userguide/working_with_dependencies.html#accessing_a_modules_metadata_file in order to answer the use case of #11449

Closing as duplicate of https://github.com/gradle/gradle/issues/26007


Fetch mode fix

My attempt at fixing #49

👍 I can fix the default argument issue. Thanks for contributing this fix! :)


[issue-4364] add attached_images association to workflow copier

Describe your change here.
Solves for this issue
Review checklist

[x] First, the most important one: is this PR small enough that you can actually review it? Feel free to just reject a branch if the changes are hard to review due to the length of the diff.
[ ] If there are any migrations, will they the previous version of the app work correctly after they've been run (e.g. the don't remove columns still known about by ActiveRecord).
[ ] If anything changed with regards to the public API, are those changes also documented in the apiary.apib file?
[x] Are all the changes covered by tests? Think about any possible edge cases that might be left untested.

Overall change looks ok to me. 👌🏼
It looks like ☑ it passed the spec https://github.com/zooniverse/panoptes/blob/master/spec/lib/workflow_copier_spec.rb#L35-L38 which checks to make sure non-excluded attributes are being included. ☑ and it passes your new spec that attached_images be included ☑ passes the spec that any excluded attributes are set to defaults for the copied workflow.
OOC, do you know why deep_cloneable gem's #deep_clone is not including attached images? It feels like it should without us having to explicitly include it. But current behavior in panoptes has proved otherwise. 🤔 (I havent looked too much into this, but was wondering if you have.) @Tooyosi

Yeah, my understanding is it has to be included for it to be copied along, here: https://github.com/moiristo/deep_cloneable?tab=readme-ov-file#association-inclusion

Super sorry I should have caught this sooner when looking through spec.
Are we certain that workflow.attached_images in spec has media (i.e. are we certain workflow.attached_images is not an empty array?) ?
I worry that the way we define media with let and how let gets lazily evaluated, that the media is actually not getting associated with the workflow until invoked (i.e. until we specifically call out media eg. puts media). Maybe let! or instead setting/creating media within the test it copies the attached_images would get media associated to workflow?

Thanks for picking this, somehow it seems like it was good originally but i can see the issue. I have moved the creation into the specific test case and checking the count as both items would not be the exact same records (i.e different id, created date e.t.c)


Enhancements to plugin-mechanism & bugfixes

Enabled plugins to cover multiple object types
Corrected protocol for passing the files-object on de-serialization
Generic plugin based on SOBJ and database-tables only as fallback.

lcl_xml
Minor: fixed #175
Major change: If a name is passed to the table_add, the structure created for the table items needs to have a stable name. Thus, propagated the name to structure/element_add. This renders existing files potentially incompatible => implemented a fallback on reading a table to go for a second try without propagating the name

looks good to me


Rule blocked: CORERULES-576

Rule SEND-130 uses date sorting, the rule being "SESEQ values must be assigned in chronological order within a subject".
The YAML code in the rule editor uses the function "target_is_not_sorted_by", which seems to cause an execution error "string indices must be integers".
I attach the download (Export Artifcats button) from the rule editor.
Or do we need another operator when sorting is for dates?
Rule_editor_export_execution_error_SEND130.zip

Update: I don't get the error "string indices must be integers" anymore, but now get an error "column not found in dataset: sort_order". May however be due to choice of scope. Will discuss next office hours.
All files attached.
Rule_editor_export_execution_error_SEND130_2023-09-15.zip

It looks as there was an error in my script, had "order = asc".
After replacement by "sort_order = asc", there is no error anymore and the rule runs fine.
P.S.: Rule Type: Record Data , Sensitivity: Record.
Same functionality was also tested on rule SEND354. Also there, it works well now.

Covered by #557


Add TypeScript support

Adds types for usage with TypeScript and editor support (e.g. VSCode).
Also results in the following:

[x] bumps version to 1.1
[x] switches default bundle to ESM from CJS
[x] adds a changelog

looks good to me. thanks for doing this


Class table inheritance

Hi,
currently Propel has 2 types of inheritances: single table and concrete.
But the most common inheritance and the one I would like to use is the inheritance that uses join in its children tables.
E.g:
User { id, name } //parent
Company { user_id,  slogan } //child
Person { user_id, sex } //child
That's the normalized version of the schema. It similar to the concrete table, but it won't copy all the parent's field to its children nor the parent fields to its children. Instead it will "join" those tables to access the data.
I had already talked about it in the IRC with others (Jarda mostly) and it seems that it's not just me that want that. Francois didn't really understand what I was asking when we talked, I think I didn't know how to explain it because it is recommended approach in most of the cases when dealing with inheritance.
For reference: http://www.doctrine-project.org/docs/orm/2.0/en/reference/inheritance-mapping.html#class-table-inheritance

DelegateBehavior cannot perform proper PHP inheritance in the generated classes (i.e. using "instanceof" test for a child class on an object generated by a parent query). I think a proper one using PHP extends should be made.


Psyche Engine keeps crashing whenever you open freeplay or Exit a freeplay song

Describe your bug here.
Whenever I hit enter on freeplay, it crashes, and when you exit a song from freeplay or from the chart editor, it also crashes
Command Prompt/Terminal logs (if existing)
No response
Are you modding a build from source or with Lua?
Lua
What is your build target?
Windows
Did you edit anything in this build? If so, mention or summarize your changes.
no

Psych*

i have the same problem

having that EXACT same problem right now.
anytime I specifically go to SNS Retake 2.5, it just crashes. No other mod of mine does that but SNS Retake.
Any clue why????


[WIP] Add Arrow benchmarks

This won't work as-is, as Arrow currently needs a forked version of ASV. I don't know how to integrate it in the configuration files here. See https://github.com/apache/arrow/blob/master/python/README-benchmarks.md

@TomAugspurger

Note the ASV fork is because of https://github.com/airspeed-velocity/asv/pull/626

@TomAugspurger thanks, it looks like i'm not cleaning up the temporary package cache on each build. fix on the way.

Updated the ASV on the benchmarks machine to use https://github.com/pitrou/asv.git@customize_commands
Deploying now.

Hopefully these will start being run and show up at http://pandas.pydata.org/speed/ sometime in the next 24 hours.

The bnechmarks ran, but there was a failure with how the results were collected. Hopefully fixed, and rerunning, now.

http://pandas.pydata.org/speed/arrow/ has the results
http://pandas.pydata.org/speed/arrow/regressions.xml should be an RSS feed with the regressions, but I've had some issues with pandas' in the past. Perhaps updating asv fixed that.

Great! Thanks @TomAugspurger :-)


The result of the code doesn't match the result in the paper

Thank you for releasing the code~
I try to run the multiple choice part of the code and find the result of the code doesn't match the result in the paper.
In my run, the finial MC1 score is 23.869% and the finial MC2 score is 48.73%. However, in your paper, the MC acc of baseline+ITI is 25.9%, which doesn't match the result of the code. While the gap between code result and result in the paper is actually not very large, the gap between result in the paper and baseline acc is also quite small. Therefore, it would be confusing whether the ITI helps the multiple-choice or it damage the multiple-choice.
Looking forward to your reply~

Hi there, I ran exactly this repo with latest commit few days ago and got a 28.8% MC1, averaged from random seed 1 through 5. Hard to deduce what's wrong on your side.

Thank you for your reply~
The problem comes from the setting of my code, which is a little different from the origin one.
I've rerun the code follow your advice~and the result I got is as follows:
gpt-info 68.21%, acc 79.94%,MC acc 26.53%


added utilities for Shape

castShape
intersectsShape
contactShape
containsPoint
projectPoint
intersectsRay
castRay
castRayAndGetNormal

they are on Collider before, and now they are on Shape too, because these methods are duplicated, so i have 'merged' them into a trait and implemented it on SharedShape. please review the changes, thanks.

Thanks!


Rename LedgerSimpleAddressWallet

LedgerSimpleAddressWallet does not use the deprecated "simple address" derivation anymore and the name should reflect that.
Possible options:

LedgerIovWallet
IovLedgerWallet
ideas?

@webmaster128 How specific is it to IOV? Our other wallets are named eg Ed25519HdWallet or Slip10Wallet. Could we name this LedgerWallet?

@willclarktech the Ledger wallet will only be able to connect to the IOV Ledger app. So it can be used for IOV testnet and IOV mainnet only. Also in LedgerSimpleAddressWallet.createIdentity we only pass an address index, which which will translate to the IOV HD path m/44'234'/index.

OK, then my preference is for IovLedgerWallet.

Sounds good


Show a user's total of stars received

Shows the total of stars received on a user's profile.
Example:

I'm not a big fan of vanity metrics like this, but I guess many would want this, so why not. However, it's way too slow. It took 10 seconds to load on my profile. By then, people would have gone elsewhere. Maybe look into using the GraphQL API instead, where you can just extract exactly the information you need.

One of the issues I faced when implementing this is pagination; it loads each page (of 100 repositories max) until none is returned. However, it would be possible to get the number of pages from the public repository count of a user. From there, the repo lists can be fetched asynchronously.
If that doesn't make it much faster, I'll try to find other ways, like the GQL API you mentioned.

I think you should try GraphQL right away. It could potentially let us compute the count server-side, so all we have to do is to fetch the count.

You can borrow some GraphQL request code from here: https://github.com/sindresorhus/refined-github/pull/1399/files#diff-1285b6ab4d237edd45e7c9b6e9412928

I'm not aware of a sum aggregation feature in GraphQL, so it would still require fetching pages of 100 repos sequentially and summing the stargazers of each repo. It does have the advantage of only fetching the necessary field, however.

Relevant GitHub GraphQL forum thread: https://platform.github.community/t/best-way-to-gather-all-stars-a-user-has-received/1068/2

It does have the advantage of only fetching the necessary field, however.

That's a pretty big advantage.

I just tested with GraphQL. The performance is identical to the original implementation, mainly because it relies on a cursor (making the traverse synchronous). I don't know of a way to traverse pages asynchronously because of the cursor system.
So far, the most efficient solution is the asynchronous requests (current impl).

@hkdobrev I might be wrong, but I think this is what the Popular repositories section does (when it's not overriden by the Pinned repositories feature).
I think it would be useful to add a toggle feature to show Popular repositories when the user has pinned repos. This would find the 6 repos with the most stars (using the GraphQL search).

Discarding in favor of #1495


use custom tag instead of native getTag()

Now method show() in 'BottomDialog' use the native 'getTag()' instead of our custom tag. That's not our wish~

yeah, there is a mistake because of my negligence, I solved it by those code:
  public BaseBottomDialog show() {
        show(mFragmentManager, getFragmentTag());
        return this;
    }

Of course you can correct it like this. But the show() method in the base class is wasteful.

yeah, you are the right ! , i will change my code immediately , thanks for your response


Command Snippets

Description of the new feature/enhancement
Command Snippets to Make the Terminal Experience Seamless to Use, This feature works exactly same as Visual Studio Snippets or Other IDE's, In this Case We Have Shortcut to access a snippets flyout toolbar on Any Terminal or We can have different Snippets for Different Terminals.
This Feature Would Make Users More Productive Instead Typing the Same Terminal Commands Again and Again Since the Terminal History Keeps Executed Commands at least amount
https://docs.microsoft.com/en-us/visualstudio/ide/code-snippets?view=vs-2022

Proposed technical implementation details (optional)
Create a Local Data Store and Keeps the Snippets in Check and When User Access the Snippets via Shortcut, We can launch small flyout based WinUI to Shows the Snippets Title and When user selects the Snippets, Terminal would automatically paste the command for that snippets

Thanks for the suggestion! This is actually already being tracked by another issue on our repo - please refer to #1595 for more discussion.
/dup #1595


tf-nightly  version

[Derived]  Tried to append a tensor with incompatible shape to a list. Op element shape: [64,2048] list shape: [1,?]
When I using tf-nightly 2.1.0.dev20191219 to train the model, I got the previous error.
Could you tell me about the version of tf used in your experiments?

This issue seemed to happen with any batch size > 1. Issue fixed in commit https://github.com/noahchalifour/rnnt-speech-recognition/commit/534165e1053deeec8731173b983994f62de0c682


Update Cargo.lock to prevent nightly breakage

See https://github.com/rust-osdev/x86_64/pull/186#issuecomment-700491664
Now that x86_64 has a "good" build, we just need a new lockfile to make things build.
Signed-off-by: Joe Richey<EMAIL_ADDRESS>

Thanks! Could you also update the Cargo.lock files in example-kernel and test-kernel (these are also built in our current test framework).

@phil-opp This needs to be force merged

Thanks! Could you also update the Cargo.lock files in example-kernel and test-kernel (these are also built in our current test framework).

Done

Thanks!

@phil-opp It looks like one of the tests got (inadvertently) cancelled. I don't have permissions to rerun them.

Seems like the Windows job is having some problems downloading QEMU. I already restarted it once after seeing the same issue. Let's try it again…

Seems like the Windows job is having some problems downloading QEMU. I already restarted it once after seeing the same issue. Let's try it again…

Looks like a network failure. The Windows test consistently fails in downloading (the link works just fine for me locally though)

I also just tried it locally and it worked for me too. I'm not sure what we should do about this. I'm inclined to force-merge and publish to unblock the nightly features, but on the other hand it feels not good to publish a new version without a passing CI.

I also just tried it locally and it worked for me too. I'm not sure what we should do about this. I'm inclined to force-merge and publish to unblock the nightly features, but on the other hand it feels not good to publish a new version without a passing CI.

It's building and all the other tests and stuff are passing. The downside here is that without this change, the CI's for all the other projects aren't going to pass. Given that this would be the only change for 0.9.11. I think force merging is OK.

Thanks! Ok, then let's do this.

Published as v0.9.11


Restructure cabal.project and clarify warning when benchmarking in debug mode

This PR makes the following changes:

Restructures cabal.project into cabal.project.release and cabal.project.debug so that we can switch between release/debug builds from the command line rather than by editing a file.
Clarifies and unifies the warning printed when running a benchmark in debug mode.

This is to avoid the need to git stash/apply when switching branches, and to enable an entirely programmatic git bisect for benchmarks.

https://github.com/IntersectMBO/lsm-tree/blob/2e76991bc761a41ac579db324d9199588aa3460d/bench/macro/lsm-tree-bench-lookups.hs#L132 and https://github.com/IntersectMBO/lsm-tree/blob/2e76991bc761a41ac579db324d9199588aa3460d/bench/macro/lsm-tree-bench-bloomfilter.hs#L68 also have these messages, could you change them there too?

https://github.com/IntersectMBO/lsm-tree/blob/2e76991bc761a41ac579db324d9199588aa3460d/bench/macro/lsm-tree-bench-lookups.hs#L132
and
https://github.com/IntersectMBO/lsm-tree/blob/2e76991bc761a41ac579db324d9199588aa3460d/bench/macro/lsm-tree-bench-bloomfilter.hs#L68
also have these messages, could you change them there too?

Should this warning not be in all benchmarks?

https://github.com/IntersectMBO/lsm-tree/blob/2e76991bc761a41ac579db324d9199588aa3460d/bench/macro/lsm-tree-bench-lookups.hs#L132
and
https://github.com/IntersectMBO/lsm-tree/blob/2e76991bc761a41ac579db324d9199588aa3460d/bench/macro/lsm-tree-bench-bloomfilter.hs#L68
also have these messages, could you change them there too?

Should this warning not be in all benchmarks?

https://github.com/IntersectMBO/lsm-tree/blob/2e76991bc761a41ac579db324d9199588aa3460d/bench/macro/lsm-tree-bench-lookups.hs#L132
and
https://github.com/IntersectMBO/lsm-tree/blob/2e76991bc761a41ac579db324d9199588aa3460d/bench/macro/lsm-tree-bench-bloomfilter.hs#L68
also have these messages, could you change them there too?

Should this warning not be in all benchmarks?

Probably, though I think it is currently only missing from https://github.com/IntersectMBO/lsm-tree/blob/main/bench/macro/lsm-tree-bench-wp8.hs

This probably should just use the existing cabal-args fields rather than defining a separate cabal-project field.

Breaking this up into two PRs due to staleness and squashing.


dsm_version parameter not working

Hi,
dsm_version parameter is not accepted by SynologyDrive class and is not passed to SynologySession. SynologySession dsm_version default value is 6 thus the documentation description is wrong, please update the code.

I'll check it tomorrow.

same problem encountered.
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_106/412349273.py in <module>
----> 1 with SynologyDrive(NAS_USER, NAS_PASS, NAS_IP, dsm_version='7') as synd:
      2     pass

TypeError: __init__() got an unexpected keyword argument 'dsm_version'

@wellsleep I'll fix it on Monday.

@schulz @wellsleep You can try 1.0.6.

1.0.6 worked!

@zbjdonald it works, thank you!


testing automatic build generation

What's the intent of this?

Apologies, I meant to target my fork not the main repository.


Add Efficientnet v2 link in README.md

Link to efficientnet v2 paper was not working. So i just added it. A very tiny contribution from my end.

Good catch, thanks!


not working on windows; all details inside

problem
cannot access the site on: <IP_ADDRESS> or herokuwp.local as mentioned on this repo's readme
expected
when I visit <IP_ADDRESS> or herokuwp.local on my browser, that I'm able to view the wordpress site
way to reproduce

vagrant up in the project folder from a cmd.exe window

versions
os: windows 10 pro x64
vagrant: Installed Version: 1.9.0
virtualbox version: 5.1.10
Details
here is a part of the error log which I think is relevant; this log is generated from running vagrant provision (I'd run vagrant up earlier)

Is something wrong with getting php to work?

==> herokuwp: Updating dependencies (including require-dev)
==> herokuwp: Your requirements could not be resolved to an installable set of packages.
==> herokuwp:
==> herokuwp:   Problem 1
==> herokuwp:     - This package requires php ~7.0 but your HHVM version does not satisfy that requirement.

here is the full error log; this log is generated from running vagrant provision (I'd run vagrant up earlier)
https://gist.github.com/dheerajbhaskar/8088ae3da20098ecd19c4bae5b813bbf

@dheerajbhaskar Sorry about the long delay but I finally rebuilt the Vagrant local dev environment in 0172a9d...fabf120fef7879057645d0fb1fd14e659952a23b to fix a bunch of outstanding issues one of which is the php7/hhvm issue you found here. This is now working on Mac OS however I don't have a Windows box to test this on can you give it another shot and see if it works?

Let me check this out.
Also, you're working with a modified WordPress ie there are a lot of
changes to the source to fix one or the other issue. Would you be able to
help with a PR which will get a vanilla WordPress working on heroku so that
WordPress source can be used as is?
On Thu, Jan 26, 2017, 9:22 AM Xiao Yu<EMAIL_ADDRESS>wrote:

@dheerajbhaskar https://github.com/dheerajbhaskar Sorry about the long
delay but I finally rebuilt the Vagrant local dev environment in
0172a9d...fabf120
https://github.com/xyu/heroku-wp/compare/0172a9d...fabf120fef7879057645d0fb1fd14e659952a23b
to fix a bunch of outstanding issues one of which is the php7/hhvm issue
you found here. This is now working on Mac OS however I don't have a
Windows box to test this on can you give it another shot and see if it
works?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/xyu/heroku-wp/issues/73#issuecomment-275302280, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACzvjUgqbNP8_1bE59J5FaRki3rcfpK0ks5rWBiNgaJpZM4LAoHS
.

--
--- Disclaimer --- The information in this mail is confidential and is
intended solely for addressee. Access to this mail by anyone else is
unauthorised. Copying or further distribution beyond the original recipient
may be unlawful. Any opinion expressed in this mail is that of sender and
does not necessarily reflect that of OSW Technologies Pvt Ltd.---

Would you be able to help with a PR which will get a vanilla WordPress working on heroku so that WordPress source can be used as is?

Sure, if you have a PR I'd be happy to take a look.

I'm going to mark this as closed because the new VM should have fixed the original issue here. Please open a new issue or a new PR for any changes to the repo in terms of how WP is installed.


TokenIgnores in the .vale.ini for AsciiDoc are not required.

TokenIgnores in the example .vale.ini for AsciiDoc are not required. This is misleading - they should be removed.
; Ignore attributes definition, id statements
TokenIgnores = (:[^\n]+: [^\n]+), (\[id=[^\n]+)

fixed in https://github.com/redhat-documentation/vale-at-red-hat/pull/575