Building of cumulocity-microservice-templates-metrics fails

looks like a dependency is missing:
[ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /mnt/c/Users/hame/git/cumulocity-microservice-templates/metrics/src/main/java/com/c8y/ms/templates/metrics/service/MeasurementService.java:[5,31] package org.apache.commons.lang does not exist [ERROR] /mnt/c/Users/hame/git/cumulocity-microservice-templates/metrics/src/main/java/com/c8y/ms/templates/metrics/service/MeasurementService.java:[14,38] package com.cumulocity.rest.pagination does not exist [ERROR] /mnt/c/Users/hame/git/cumulocity-microservice-templates/metrics/src/main/java/com/c8y/ms/templates/metrics/service/DeviceService.java:[6,31] package org.apache.commons.lang does not exist [ERROR] /mnt/c/Users/hame/git/cumulocity-microservice-templates/metrics/src/main/java/com/c8y/ms/templates/metrics/service/DeviceService.java:[16,38] package com.cumulocity.rest.pagination does not exist [ERROR] /mnt/c/Users/hame/git/cumulocity-microservice-templates/metrics/src/main/java/com/c8y/ms/templates/metrics/service/MeasurementService.java:[47,21] cannot find symbol   symbol:   variable StringUtils   location: class com.c8y.ms.templates.metrics.service.MeasurementService [ERROR] /mnt/c/Users/hame/git/cumulocity-microservice-templates/metrics/src/main/java/com/c8y/ms/templates/metrics/service/MeasurementService.java:[58,46] cannot find symbol   symbol:   variable RestPageRequest   location: class com.c8y.ms.templates.metrics.service.MeasurementService [ERROR] /mnt/c/Users/hame/git/cumulocity-microservice-templates/metrics/src/main/java/com/c8y/ms/templates/metrics/service/DeviceService.java:[75,132] cannot find symbol   symbol:   variable RestPageRequest   location: class com.c8y.ms.templates.metrics.service.DeviceService [ERROR] /mnt/c/Users/hame/git/cumulocity-microservice-templates/metrics/src/main/java/com/c8y/ms/templates/metrics/service/DeviceService.java:[91,21] cannot find symbol   symbol:   variable StringUtils   location: class com.c8y.ms.templates.metrics.service.DeviceService

has been fixed here:
https://github.com/SoftwareAG/cumulocity-microservice-templates/pull/17


Need option to use system libs for build

Feature request
Please add options for cmake to build HyperHDR with system libs like flatbuffers, mbedtls and qmqtt.
Fedora rules requred unbundle as much as possible.
What problem does this feature solve?
Build HyperHDR for Fedora and add to official repo.
How should this be implemented in your opinion?
Change CMakeLists.txt file.
Are you willing to work on this yourself?
No. I don't know cmake build system.

Hi
flatbuffers and mbedtls are critical core libraries for HyperHDR, each version must be tested before upgrading and I don't see possibilities to rely on system libs here. For example current mbedtls 3.3 causing problems for webos. qmqtt also needed some modification to build configuration to make it work also with Qt6 (its CMakeLists.txt is hardlinked to Qt5) but qmqtt is not critical component and can be disabled cmake configuration option.

Thanks for clarification.

This PR should bring support for Fedora: https://github.com/awawa-dev/HyperHDR/pull/591


Improve docs

The variable containing the magnetURI imo should stay as magnetURI, it's one of the many torrent identifiers.

@feross, please look at this pull request.

@DiegoRBaquero
Some new people who absolutely unfamiliar with WebTorrent might think that magnet URI is the only way to specify torrent in this example.
I've changed the name of variable magnetURI to torrentId to emphasize that users can use not only magnet URIs.
torrentId is more universal variable name than magnetURI.

@feross, please look at this pull request.

Thanks for the suggested doc improvements! I've merged some of your changes in this commit https://github.com/feross/webtorrent/commit/e3803dd2da98da8e1506b826dc7e3048e261776f but the others didn't make sense to me. Thanks again!


create_directory()

ISO paper for this feature: P1164R1
First implemented in: VS 2019 16.0

moved to quest


MySQL "replication client" and "replication slave" privileges are internally stored by MySQL using shorter name

MySQL "replication client" and "replication slave" privileges are internally stored as "repl_client" and "repl_slave".

Fixed up in 4.0.7


Add secure boot hardware compatibility doc

Change-type: patch

TODO: Add a specific section listing tested, working hardware.

Can one of the admins verify this patch?

This document is outdated with the merge of PCR7 sealing.


fix(planner_manager): break while loop when the candidate module fails to generate valid path

Description
Before this PR, the manager registers the highest priority module as candidate moduls even if the module FAILs to generate path. This causes following un/register loop, and some chattering happen.
First cycle

run request module (fails to generate path)
register the request module pointer to candidate_module_opt_

https://github.com/autowarefoundation/autoware.universe/blob/b78b17d6be85611e0686b1c319f3ae0ffe7bd890/planning/behavior_path_planner/src/planner_manager.cpp#L88
Second cycle

request module return ModuleStatus::FAILURE in getCandidateModule()
unregister the module
return First cycle

https://github.com/autowarefoundation/autoware.universe/blob/b78b17d6be85611e0686b1c319f3ae0ffe7bd890/planning/behavior_path_planner/src/planner_manager.cpp#L186-L191
Then, I fixed it in order to break the while loop without registering to candidate_module_opt_.
https://user-images.githubusercontent.com/44889564/230304972-fb0f8d1d-c827-4168-a9eb-0d4719b35a2f.mp4

Tests performed

Use new architecture

https://github.com/autowarefoundation/autoware.universe/blob/b78b17d6be85611e0686b1c319f3ae0ffe7bd890/planning/behavior_path_planner/CMakeLists.txt#L10

Set LC canceling scenario in Psim

https://user-images.githubusercontent.com/44889564/230305603-1023ad1b-1d78-4578-b9dc-fe5ee79f4b2d.mp4

Pre-review checklist for the PR author
The PR author must check the checkboxes below when creating the PR.

[x] I've confirmed the contribution guidelines.
[x] The PR follows the pull request guidelines.

In-review checklist for the PR reviewers
The PR reviewers must check the checkboxes below before approval.

[ ] The PR follows the pull request guidelines.

Post-review checklist for the PR author
The PR author must check the checkboxes below before merging.

[ ] There are no open discussions or they are tracked via tickets.

After all checkboxes are checked, anyone who has write access can merge the PR.

Now, evaluator test is running...

@zulfaqar-azmi-t4 There are still few failed scenarios but no critical regression caused by this PR.

[TIER IV INTERNAL LINK] This PR 1194/1199
[TIER IV INTERNAL LINK] Baseline 1183/1199


Added more channel flags, added documentation for channel flags.

Ignore both the PR name and commit name, typing anything off remote desktop is not easy 💀

Added the rest of the channel flags (that I could find) and documented all of them.

Note: ApplicationShelfConsent is empty since I couldn't find anything about what it actually does.

Comma added


Unicode symbols from another language

I am trying to modify the tacotron model for urdu language. I was modifying the symbols.py file and since urdu is a cursive language, if I remove spaces from between the alphabets, they join together and create garbage. Is it ok to have a spave between each symbol?

@aminbaig you do not need the space between the characters. I understand you have joiners and (matras)? but you can have the single amalgamated string, AFAIK. Devanagari is working out for me.

did it work out? @aminbaig

Thanks Sahar, I have picked up the project again.
Yes it and thank you for your response. I am now having issues training the data as I get OOM error (out of memory). Any idea whats the minimum amount of memory on the GPU will work with this? I am currently using a Geforce 750 TI with 2 GB ram.


New package: Motorola.MobileDrivers version 6.4.0

[ ] Have you signed the Contributor License Agreement?
[ ] Have you checked that there aren't other open pull requests for the same manifest update/change?
[ ] This PR only modifies one (1) manifest
[ ] Have you validated your manifest locally with winget validate --manifest <path>?
[ ] Have you tested your manifest locally with winget install --manifest <path>?
[ ] Does your manifest conform to the 1.4 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: codeflow:open?pullrequest=https://github.com/microsoft/winget-pkgs/pull/113160&drop=dogfoodAlpha

This package also leaves no real executable.

This package also leaves no real executable.

It’s a DRIVER installed by an MSI file


mitmdump: temporarily running out of diskspace corrupts dumpfile

Description
As mentioned in this issue, mitmdump writing to a file can lead to data corruption when disk becomes full.
To resolve this, this PR adds disk usage checks and exits the program when it exceeds a preset threshold.
Checklist

[ ] If this is acceptable, I can write tests for this.

Hello, just wanted to check if this is acceptable

Since stream.add is called from multiple methods in addons/save.py, I added the excpetion handling in io/io.py. Also added logic to detect if the Save addon was initialized for mitmdump or mitmproxy and tested it successfully.

I am a bit confused. Do you mean create a wrapper function for stream.add and add the mitmdump detection logic in that?

We don't want to detect the usage of mitmdump, we specifically want to detect the usage of save_stream_file (which sets self.stream in the Save addon).

For the test, I was thinking of creating a small file system and mounting it on test directory and filing it. But this require root privileges so is that fine or do we need some other method?

I think this is addressed by https://github.com/mitmproxy/mitmproxy/commit/3a5550a09cd40d76acfe71aa45c7a8309525ad51 now. Thanks again! 🍰


custom number types

If I understand the readme correctly (https://github.com/oxfordcontrol/qdldl#including-the-header), there's a bit of a design issue with the custom number types.
At the point when a user redefines these types when including the header, the library has already been compiled using some fixed set of integer types. Using different definitions at include time will just result in linking errors or crashes when compiler finds functions with unexpected argument types. The choice of number types needs to be a compile-time option, e.g., controlled by a config file or cmake config.
Another option is to automatically compile all reasonable combinations of integer/float types and expose them as, e.g., QDLDL_solve_float64_int64. This requires a bit more work but it would allow users to compile a binary that provides multiple precision options at runtime.

I'm not necessarily against doing this, but note that I haved use bool, float and int types in the factorisation to keep the memory footprint small.   If we are to enumerate overall a variety of input types then we should probably replace bool with int to minimize the number of combinations.

I'd agree that there's no real need to let the bool type vary. int is a reasonable choice.

I would be happy to insist that the boolean type should always be of type bool (or _Bool), but that doesn't exist in C89.  The reason for making it user definable was largely to preserve the option to define it as 'int' if when there is no native boolean type.
If it is to be the same type as the other ints (most likely int64), then there is a fairly large memory penalty if you have a really huge matrix with very low density.

Another possibility is to specify the types as CMakeLists.txt options in a qdldl_configure.h.in file as done for osqp here. The same could be done with the bool/int decision for memory usage. What do you think?

Given that C89/C99 support is known at compile time, it's quite reasonable to let this be a config option or just switch on int/bool with macros based on the value of __STDC_VERSION__.

This should be fixed in 2644bc2. CMake flags for doubles and integers + __STDC_VERSION__ check for bools.


workflow: run CI on pushes to main branch, not other branches

Right now, each PR runs CI twice - once because it's a PR, and another time because it comes from a push to a branch that created the PR. See for example https://github.com/tmc/langchaingo/pull/398, causing:

https://github.com/tmc/langchaingo/actions/runs/7094374726
https://github.com/tmc/langchaingo/actions/runs/7094367361

This change makes it so we only run CI once per PR, because there's no need to run it on a branch push.
Now CI runs on either a PR or a push to main

@tmc PTAL - note that this PR runs each of the 3 checks defined in the workflow YAML only once and not twice like other PRs.


Support attaching to bpf LSM hooks

There is interest from a few folks in having bpftrace support attaching to LSM hooks. Not only is it a nice stable way of attaching to security sensitive codepaths, but you can modify the return value easily as well.
Initial work is in https://github.com/iovisor/bpftrace/issues/2716 . Would be nice if someone could pick this up.

+1 on this one.


Update composer.json to target phpstan to v1.5.7

Update composer.json to target newest version of phpstan

fe32177581e7477495af72f654d580aaf3014e1d addresses this issue, however, running phpstan after the update revealed new issues that need to be resolved before this issue can be marked resolved.


Implement RFC fs-1025:  Improve record type inference

Take 2. Rebased on master after all the recent changes.
RFC.

Adding a comment to perhaps kick off a build. This was all green before.

@dotnet-bot test this please

cool! side question: do we need to adjust error messages!?

@forki: I don't think so, but I haven't reviewed the error messages.
The nature of the change is that an ensuing check will now produce an error in fewer cases, because the type it picks during record type inference initially is a better choice. But the ensuing checks are still valid so don't think the error messages there need to be adjusted as a result of this change.
There is a warning produced just above the changed code that is also still valid, for cases like { a with F=3 } where we can't use the "pick the record type with same number of defined fields" trick because not all fields may be specified in the code for record updates. So that warning also remains valid.
I've noticed you added some code to suggest other record types when subsequent errors occur; perhaps you could argue that these may be updated to take the defined field count into account, but that might also be more confusing (i.e. excluding types that the programmer actually wanted to see).
It's entirely possible I've missed something though!

ok let's just assume things will work. And if we see samples where the error message could be better then we will fix them

@kurtschelfthout
Thank you for this contribution.
Kevin


WIP Implement a solana/anchor_lang version of compass

https://github.com/palomachain/paloma/issues/448
Background
A solana/anchor_lang version of compass
Todo

[ ] Some program invocation errors around #[Accounts] structs, which require payer/init annotation
[ ] Cross-program invocation for very specific contracts seems straightforward, for arbitrary contracts I will need to consult an expert
[ ] (optional?) Integrate anchor's js deploy framework

Testing completed

[ ] tbd
[ ] Deploy to solana and invoke other contracts

Can reopen if we get to it.


IncludeOptimizedByPath doesn't work for SqlServerCe.4.0 (Compact)

Setup
EF Plus (version 1.8.4)
.NET Framework 4.0
SqlServerCe 4.0 (Sql Compact)
Case
The following exception is thrown if I want to execute my query with a SQL Compact connection, which has been built with IncludeOptimizedByPath:
Part of the code being executed
var fullQuery = query
                    .IncludeOptimizedByPath("Datastores")
                    .IncludeOptimizedByPath("Datastores.Disks")

Exception thrown
There was an error parsing the query. [ Token line number = 91,Token line offset = 1,Token in error = SELECT ]
SQL Server doesn't have this issue. It only happens while using SQL Compact.

Hello @LeanderClappaert ,
Thank you for reporting. Could you try the same code by disabling the AllowQueryBatch option:
QueryIncludeOptimizedManager.AllowQueryBatch = false;

We are currently re-visiting all features in our new library: http://entityframework-classic.net/
So once it's done here, we will also port the code in this library to include the fix to make it work by default.
Let me know if that work when disabling AllowQueryBatch.
Best Regards,
Jonathan

Hi @JonathanMagnan ,
It works on SQL Compact 4.0 when disabling AllowQueryBatch !
To be sure, I've retested my code also on SQL server and this still works.
--> The fix works.
Thanks,
Kind regards,
Leander

Great :)
I will not recommend to disable this option on SQL Server since you want your queries to run in one database round-trip.
When you disable it, it will make a database run-trip for every query it has to perform.
Best Regards,
Jonathan


[BUG] Trace linking from logs explorer opens a blank page

What is the bug?
Event explorer feature to open traces from logs isn't working as expected. Whenever user clicks on the trace link they land on traces page with a blank screen.
How can one reproduce the bug?
Steps to reproduce the behavior:

Add sample trace data.
Create a logs index with the sample data loaded.
Link on traces link to load trace analytics UI.

Work-around

The current work around is to manually update the link

From:
https://search-endpoint/_dashboards/app/observability-traces#/15d30e4d211d79e10fcaeab97015c90d

To:
https://search-endpoint/_dashboards/app/observability-traces#/traces/15d30e4d211d79e10fcaeab97015c90d

What is the expected behavior?

The URL needs to be fixed for the redirection.
Need to increase test coverage to not have such regression in future.

What is your host/environment?

OS: linux
Version: 2.11
Plugins: dashboards-observability

Do you have any screenshots?

Do you have any additional context?
Add any other context about the problem.

Fixed by external contributor: https://github.com/opensearch-project/dashboards-observability/pull/1390


Update logback-classic to 1.4.12

About this PR
📦 Updates ch.qos.logback:logback-classic from 1.4.4 to 1.4.12
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

🔍 Files still referring to the old version number
The following files still refer to the old version number (1.4.4).
You might want to review and update them manually.
modules/multilanguage-csharp-sdk/Surge.csproj

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "ch.qos.logback", artifactId = "logback-classic" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "ch.qos.logback", artifactId = "logback-classic" }
}]

labels: test-library-update, early-semver-patch, semver-spec-patch, old-version-remains, commit-count:1

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (862ac3a) 60.51% compared to head (5e77ab1) 60.32%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #406      +/-   ##
==========================================
- Coverage   60.51%   60.32%   -0.19%
==========================================
  Files         152      152
  Lines        3277     3277
  Branches      101      105       +4
==========================================
- Hits         1983     1977       -6
- Misses       1294     1300       +6

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Superseded by #407.


regression tests for issues 164 and 576

This PR is to push regression tests for issue #164 and #576 .

I am still reviewing this, but the first issue I am running into is that these tests have the @pytest.mark.regression annotation and these are not being automatically picked up by the test collector.

I am still reviewing this, but the first issue I am running into is that these tests have the @pytest.mark.regression annotation and these are not being automatically picked up by the test collector.

Please run tests marked as regression by passing "-m regression" to the pytest. If your tests are in folder named"tests" your command will be:
python -m pytest -m regression "path to test folder"
This will run only tests marked as regression


Replace EOL spinner.gif

The change proposed replaces the EOL spinner.gif with the drop-in Jelly tag replacement.

Merged, thanks!

Thanks for the swift merge and release!


Large strings cause IncompleteRCommandException

I am passing JSON data from PHP to R as
$rProcess->write("t1<-fromJSON('$ser_j', simplifyVector = TRUE, simplifyDataFrame=TRUE)");
It works fine on relatively short strings $ser_j (tested up to 130-150 3-digit numbers in a string). But for larger strings, it returns an error:

Warning: include(Kachkaev/PHPR/Process/IncompleteRCommandException.php): failed to open stream: No such file or directory in ...

It looks as large strings are truncated and the R command line becomes incomplete.
My questions:

Is there any limit on the size of strings/data being passed to R?
If yes, how would you suggest to pass large data sets? Via MySQL?
How to fix the error of missing IncompleteRCommandException.php? I am using the following namespaces (classes are loaded via spl_autoload_register):

use Kachkaev\PHPR\RCore;
use RKachkaevPHPR\Engine\CommandLineREngine;
use Kachkaev\PHPR\Exception;

Thanks!

Hi @andyG8!
Thanks for reporting all this. Will try to answer:

I believe that yes, there is a limit in how much you can pass to php-r directly. The library is just a thin command line wrapper, so I believe the limit depends on your CLI.

MySQL can be an option, yes. What you can also try is just to ask R to open a specific file by its path. If the R interpreter is on the same machine (which is always the case with CommandLineREngine), both PHP and R have access to the same filesystem.  You can just save your file as JSON and tell R where it is.

Great catch! Looks like IncompleteRCommandException was not referenced correctly. Can you try pulling and checking if the fix worked?

Hi Alexander,
Thank you, the IncompleteCommandException has been fixed. I am using the MySQL option to pass large data sets and it works perfectly fine.
Cheers,
Andy


Cluster Owner is not able to deploy catalog apps

What kind of request is this (question/bug/enhancement/feature request): bug
Steps to reproduce (least amount of steps as possible):

As an admin, create a cluster (rke DO).
Add authentication in Rancher example: Github.
Add another github user(say user1) as the cluster owner for the cluster created in step 1.
As an admin, create a project in the cluster.
As the github user - user1, deploy catalog app in project created in step 4.

Expected Result:
Cluster owner user should be able to create and deploy a catalog app.
Actual Result:
Cluster owner is not able to create a catalog app.
Error - apps.project.cattle.io is forbidden: User "u-75s76v6b2h" cannot create resource "apps" in API group "project.cattle.io" in the namespace "p-hm6x5"
Note:
If the Cluster owner creates a Multicluster app in the project created in Step 4, He is not able to list the app under Apps but he is able to view the app under Workloads.
Error in response - {"baseType":"error","code":"Forbidden","message":"apps.project.cattle.io \"app03-p-b2sv5\" is forbidden: User \"u-75s76v6b2h\" cannot get resource \"apps\" in API group \"project.cattle.io\" in the namespace \"p-b2sv5\"","status":403,"type":"error"}
Other details that may be helpful:
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): rancher:v2.2.2-rc-x12
Installation option (single install/HA): HA

Cluster information

Cluster type (Hosted/Infrastructure Provider/Custom/Imported): RKE cluster DO
Kubernetes version (use kubectl version):

v1.13.5

Docker version (use docker version):

18.9.4

This issue is NOT reproducible in v2.1.8.
This issue is reproducible in v2.2.1.

When a user is added to a cluster, we create a clusterroletemplatebinding, and the controller for crtb creates rolebindings for this user to grant access to them to project level resources in that cluster including apps, from here
But in this case, the project is created in the cluster after the user is added as cluster-owner, and we don't seem to have the corresponding logic on project creation.

I met the same bug with another approach in v2.2.5-rc8, HA setup

as the global admin, enable Github auth
as Github user1, create an RKE Azure cluster
as the global admin, add a project p1 in the cluster
as user1, go to the project p1, try to deploy an app

Results:

the following error shows in the UI

apps.project.cattle.io is forbidden:
User "u-4cnrgx47ty" cannot create resource "apps" in API group "project.cattle.io" in the namespace "p-vwfwx": RBAC: [clusterrole.rbac.authorization.k8s.io "c-877c9-clustermember" not found, clusterrole.rbac.authorization.k8s.io "c-877c9-clusterowner" not found]

I believe it is possible to repro/test this while just using local auth, steps are in https://github.com/rancher/rancher/issues/15819

Verified on master-head - Commit id: 85dedb19b
Steps:

As an admin, create a cluster (rke DO).
Add authentication in Rancher example: AD Auth.
Add another AD user(say user1) as the cluster owner for the cluster created in step 1.
As an admin, create a project in the cluster.
As the AD user - user1, deploy catalog app in project created in step 4.
User is able to deploy a catalog app.


Invoking set_index while or after reading CSV's takes a vert long time to complete

Hi,
If I set index as follows, it seems that on the computation graph many  tasks are added:
df_orig = dd.read_csv(CSV_FILES+ '/zip_LARGE/12/*/*.csv', header=1,
    parse_dates=['Report Time'], names=col_names,dtype=col_types).set_index('Report Time')

And the computation time increased exponentially.
This:
df_orig.set_index(...)
Also seems to be running forever.
What is the best option here?
Using a pure pandas data-frame )after coercion) takes a very short time to complete.

Yes, shuffling in a distributed system is much more expensive than in a shared memory system.  If your data actually fits in memory then I would just use Pandas.  If your data fits on a single machine then I would just use the single-machine threaded scheduler (the default if you don't create a client.)

Closing for now as stale. Feel free to reopen if there's still an issue/question here.


[Feature Request] Tutorials on core modules

Required prerequisites

[X] I have searched the Issue Tracker and Discussions that this hasn't already been reported. (+1 or comment there if it has.)
[ ] Consider asking first in a Discussion.

Motivation
Tutorials on core modules
Solution
No response
Alternatives
No response
Additional context
No response

Are there any reference cases?

Hey @fengju0213  really sorry for the late reply,
here are some tutorial PRs ready for review:
https://github.com/camel-ai/camel/pull/909
https://github.com/camel-ai/camel/pull/908
https://github.com/camel-ai/camel/blob/add_cookbook/cookbooks/agents_with_tools.ipynb
https://github.com/sallyqus/camel/blob/patch-1/docs/key_modules/models.md
https://github.com/camel-ai/camel/pull/914
https://github.com/camel-ai/camel/pull/901
If you have time please check the content that you are interested in, thanks in advance!

finished in https://github.com/camel-ai/camel/commit/db019bbbaf2cbc6b4510b65d0d3d6e620341cc86


Use feature gate for originating identity.

I'd encourage using a separate instance and I missed where you were adding the feature to the flags.


minor changes to docker and token_bucket_rate_limiter

Change the Golang version in comments from 1.8.1 to 1.8.2.
Update argument check in start.sh.
Change the timestamp now in TryConsume in token bucket from entering the function to acquiring the lock. This is to eliminate the waiting time for the lock.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Coverage decreased (-0.04%) to 67.423% when pulling 70c276ccd3350086c999a48cf8cdca1b57605dff on junjieqian:jqian/minorchange into b5c2556b5b366e768e08180e546df76b4af07ed8 on uber:master.

Coverage decreased (-0.05%) to 67.467% when pulling 3526a55d11b4613f2f45ea6e4d59de5a7dcfa48c on junjieqian:jqian/minorchange into a7387a8ffb1e388159f13a8e4465a851404ae541 on uber:master.

Coverage remained the same at 67.449% when pulling 89a73ed93a9ff2b1ced4818ce9105ace9dd301fc on junjieqian:jqian/minorchange into b77af1e43e3ffb218d9df54931b42ffc3732115b on uber:master.

Coverage increased (+0.03%) to 67.489% when pulling 2eaaea5cd9711c803c6f68ce0ea99a0e70f4f6a2 on junjieqian:jqian/minorchange into c06bbbb45055369e1dd9d5b9766ac673ffc66a90 on uber:master.

Coverage increased (+0.05%) to 67.547% when pulling e2c78b609188341d12ab2dbd9eeb85ade46ce09a on junjieqian:jqian/minorchange into 524be31576b7f10660803556b8016f25fbb728b6 on uber:master.

Coverage increased (+0.05%) to 67.641% when pulling 3cd7ff006049e7b75b08ab1cc154646aa731e7b1 on junjieqian:jqian/minorchange into e21f25622ad89920808d7f6a6007c3ee16bb87ad on uber:master.

Coverage decreased (-0.2%) to 67.711% when pulling 66990e28c0ae1795bcb72d4c883c238d157838b7 on junjieqian:jqian/minorchange into 886a2dc0d9beb7badfa20bce499f514b1fc7153f on uber:master.

Coverage decreased (-0.1%) to 67.791% when pulling 40b1d255f9ced298057bd455708dfbc718602c6f on junjieqian:jqian/minorchange into 85ba1dfabf49e36ad1266746b1ba0228ac9c0fb0 on uber:master.

This PR is out of date. We have a dependency on golang/x/ratelimiter now and the Dockerfile has also changed since then. So abandoning this.


API review file filter function is failing due to non-array output from get-ChildItem

Create api review step is failing with an error: "Count" is not property.  This happens when powershell returns an object instead of an array when it finds only one item in the result set.

Thanks @praveenkuttappan !

/check-enforcer override


ruby jmeter installation issue on linux

[root@lK-aM-01 pmi_automation]# gem install ruby-jmeter
Building native extensions.  This could take a while...
ERROR:  Error installing ruby-jmeter:
ERROR: Failed to build gem native extension.
current directory: /usr/local/rvm/gems/ruby-2.2.0/gems/unf_ext-<IP_ADDRESS>/ext/unf_ext

/usr/local/rvm/rubies/ruby-2.2.0/bin/ruby -r ./siteconf20170927-30924-1v0sc4n.rb extconf.rb
checking for main() in -lstdc++... yes
creating Makefile
current directory: /usr/local/rvm/gems/ruby-2.2.0/gems/unf_ext-<IP_ADDRESS>/ext/unf_ext
make "DESTDIR=" clean
current directory: /usr/local/rvm/gems/ruby-2.2.0/gems/unf_ext-<IP_ADDRESS>/ext/unf_ext
make "DESTDIR="
compiling unf.cc
In file included from /usr/local/rvm/rubies/ruby-2.2.0/include/ruby-2.2.0/ruby.h:33:0,
from unf.cc:3:
/usr/local/rvm/rubies/ruby-2.2.0/include/ruby-2.2.0/ruby/ruby.h:107:73: error: size of array ‘ruby_check_sizeof_long’ is negative
typedef char ruby_check_sizeof_long[SIZEOF_LONG == sizeof(long) ? 1 : -1];
^
/usr/local/rvm/rubies/ruby-2.2.0/include/ruby-2.2.0/ruby/ruby.h:111:76: error: size of array ‘ruby_check_sizeof_voidp’ is negative
typedef char ruby_check_sizeof_voidp[SIZEOF_VOIDP == sizeof(void*) ? 1 : -1];
^
In file included from /usr/local/rvm/rubies/ruby-2.2.0/include/ruby-2.2.0/ruby/intern.h:35:0,
from /usr/local/rvm/rubies/ruby-2.2.0/include/ruby-2.2.0/ruby/ruby.h:1710,
from /usr/local/rvm/rubies/ruby-2.2.0/include/ruby-2.2.0/ruby.h:33,
from unf.cc:3:
/usr/local/rvm/rubies/ruby-2.2.0/include/ruby-2.2.0/ruby/st.h:52:93: error: size of array ‘st_check_for_sizeof_st_index_t’ is negative
typedef char st_check_for_sizeof_st_index_t[SIZEOF_VOIDP == (int)sizeof(st_index_t) ? 1 : -1];
^
make: *** [unf.o] Error 1
make failed, exit code 2
Gem files will remain installed in /usr/local/rvm/gems/ruby-2.2.0/gems/unf_ext-<IP_ADDRESS> for inspection.
Results logged to /usr/local/rvm/gems/ruby-2.2.0/extensions/x86-linux/2.2.0/unf_ext-<IP_ADDRESS>/gem_make.out

@lingarajk Have you overcome this error? It looks like your ruby package manager is not able to install unf_ext gem. The issue is not directly related to ruby-jmeter, so I would suggest closing this issue @90kts as non-related to the repository.
First of All, I would suggest to

Re-install that package manually using
 gem install unf_ext -v '<IP_ADDRESS>'
If the installation fails, it means that you don't have something like gcc or kernel sources installed. It mostly depends on distributive. You might need ruby-dev package which will help you to compile native extensions from source. (you would want to dig in that way.)

I would suggest replacing RVM by RBENV. The seconds one has better chance to install it. I have had experience with older ruby versions such as 1.9.3

If you still have questions, please specify OS version, Distro name, RVM -v,


Mobile - Change Menu Text

change "Company" to "My Companies"

change

"project" to "My Projects"
"Finish project" to "my finished projects"
"document" to "my documents"


[WIP] Add connection pool settings for performance tuning.

This pull request includes two new settings that allow users to tune the connection pool used to manage connections to keycloak and the upstream server.
These settings allowed me to tune keycloak-proxy to ~6500 requests a second, sustained for 20 minutes, at which point I was limited by ethernet bandwitdh.
Default settings for net.http are 100 max idle connections, but only 2 max idle connections per host. The net.http library will create and destroy a connection without taking advantage of keepalive when the idle connection pool runs out. Under load this leads to many ephemeral ports in TIME_WAIT state, and the server eventually runs out of ephemeral ports while waiting for these ports to free up. At this point the server will drop requests.
Keep in mind that keycloak-proxy only really talks to the keycloak server and the upstream server, so there are only 2 hosts involved, meaning only 4 connections in the pool will be utilised. Because of this I changed the default max-idle-connections-per-host to 50.
Our situation involved high load spikes - thousands of requests in short periods of time, so we needed to be able to tune this connection pool.
An article I found helpful that describes the issue: http://tleyden.github.io/blog/2016/11/21/tuning-the-go-http-client-library-for-load-testing/
The implementation in this pull requests is minimal intentionally, given the pending move to the keycloak repo. I was thinking it may be better to manage a separate connection pool for the upstream server and the keycloak server (and any other servers the proxy may communicate with). As no-one else has hit load problems I think being able to tune the shared connection pool is likely to meet the need long term.
If you have any feedback or suggestions, please let me know.
Thanks!

Updated readme to reference the correct pull request

Please note: adding validation rules for the two new settings caused quite a number of changes to config_test.go, though the changes are simple. I tried to retain the intent of the test cases, and add new tests for the config validation rules.

much appreciated @jgroffen ... LGTM!


Improve welcome.cr

I fixed it by using a channel_id instead of a guild_id at creating the welcome message. I also added the ability to set the channel where the welcome message should get sended to with !setwelcome (channel).

I don't think the example needs to be this complex. There is an issue though because Discord no longer has the concept of a fixed general channel, but I don't think the way to resolve that would be to add the extra commands, as the example is really just meant to show off that you can use events other than on_message_create.


Feature Request: For non-standard  input dates, allow users to teach SimplePie how to parse them.

See https://github.com/simplepie/simplepie/issues/475 for context.

Done, although I still have some refactoring to do. https://github.com/simplepie/simplepie-ng/commit/03cd2e56998010d704a88509f630784a7244e687


Accessing Query Variables Inside Directive Resolvers

Thank you Apollo/MDG for such incredible tools and work!  I'm a huge admirer.
Use case
Directive resolvers are incredible for permissions, but I cannot figure out how to add an @isOwner permission without access to the query variables.  Say that I only want the owner of a draft post to be able to read the post.
directive @isOwner(type: String) on QUERY

type Query {
    draft(id: ID!): PostPayload @isOwner(type: "Post")
}

And for the directive resolver...
// Prisma exists function, but you can imagine other methods here

const isRequestingUserAlsoOwner = ({ ctx, userId, type, typeId }) =>
    ctx.db.exists[type]({ id: typeId, user: {id: userId}})

// ...

const directiveResolvers {
 // ...
 isOwner: async (next, source, args, ctx) => {
      const user = ctx.response.user
      const isOwner = await isRequestingUserAlsoOwner({ ctx, userId: user.id, type: args.type, typeId: ??? } )
      if (isOwner) {
        return next()
      }
      throw new Error(`Unauthorized, must be owner`)
  }
}

Is there a workaround?  I guess they could be sucked from the request, but that seems clunky -- particularly when the needed value could be sent as a request.body.variables or in the request.query.
I see that the query variables were explicitly left out of the API, so there's probably a good reason for it.  (It's args[1])
https://github.com/apollographql/graphql-tools/blob/80b19b6adad2b138c46f9206db647e07596af6e3/src/schemaGenerator.ts#L704
Thanks for any help for thoughts!

I agree, not having access to the variables severely limits the utility of custom directives.  I was hoping to use them for the following use cases:

authorization of access to data/resources identified in variables (similar to @LawJolla)
include the variable values in error logs

Right now custom directive seem to work well for authentication and authorization against a static set of claims.

@mjdickinson Take a look at this article, it should provide some insight into how to do authorization against a dynamic set of claims: https://medium.com/@lastmjs/advanced-graphql-directive-permissions-with-prisma-fdee6f846044

Looks like this question went the way of Schema Directives!

@lawjolla I haven't been following the latest developments, can you post a link with more info?

@LawJolla , @lastmjs, @mjdickinson   Hey guys, what do you think about doing something like this.
class WithIDDirective extends SchemaDirectiveVisitor {
  visitFieldDefinition(field) {
    const { resovle = defaultFieldResolver } = field;
    field.resolve = async (...args) => {
      const ctx = args[2]; // context
      const { tokenService, Account } = ctx;
      const token = ctx.req.request.headers.authorization; // get the token from request headers

      if (!token) throw new Error("unauthorised");

      const accountID = tokenService.verifyAuthToken(token); // verify and decode the token

      const account = await Account.findById(account).exec(); // find the account
      if (!account) throw new Error("Account does not exist!");

      args[1] = Object.assign({ accountID }, args[1]); // add accountID to the input args

      return resolve.apply(this, args);
    };
  }
}

And then verify the "Ownership" in the actual field resolver ?
Cheers

I know this issue is closed but since this is what came up while I was having this  question I though I'd leave here a solution.
As of today, the SchemaDirectiveVisitor class makes the query variables available at the field's resolver function's argument info as variableValues
Here's a simple example given the context of permissions that checks if the request comes from the owner
class AuthDirective extends SchemaDirectiveVisitor {   visitFieldDefinition(field) {     // destructuring the resolveproperty off offield, while providing a default value defaultFieldResolverin casefield.resolveis undefined.     // destructuring the field'sname` to use in the Error message
const { resolver = defaultFieldResolver, name } = field
field.resolve = async function (source, args, context, info) {

  // the query variables are available at info.variableValues
  const {
    variableValues: { userID }
  } = info

  // checks that the logged in user is the same as the one in the query variable `userID`
  if (context.user.id !== userID)
    throw new ForbiddenError(`Unauthorized field ${name}`)

  // runs if the condition above passes
  const result = await resolver.call(this, source, args, context, info)
  return result
}

}
}`
Simplified for readability

Okay, so I'm finally digging into schema directives. I've been playing around for a bit now, and I still haven't been able to figure out how to add fields to a resolver that were not requested by the user. @Rennos and @antoniojps , your solutions do not provide a way to grab arbitrary fields that the user did not request and use them for authorization (if I'm mistaken, please point it out). So, we're kind of back where we started. I have a working solution that uses directive permissions, but the world is moving on to schema directives. I don't have a working solution there yet, if anyone has insights please let me know. In each field resolver, we need access to any field from the parent object essentially.

@lastmjs That's all done via fragments on the root resolvers.  Remember, the root resolver is called before the directive resolver.  You can take a look here at the index.js resolvers with fragments and the directives.  https://github.com/LawJolla/prisma-auth0-example/tree/master/server/src

Thanks @LawJolla , that makes sense. Though it's still difficult to work with when trying to generalize fragment addition. It would be really nice if schema directives could somehow add fragments to resolvers, since the schema directive itself runs at load time before any resolvers are called (I don't even think there is a concept of a directive resolver with schema directives anymore, you only manipulate the regular resolvers through the directive).

alan345 - here is a way i found to get the context
const fields = type.getFields ()
Object.keys ( fields ).forEach ( f => {
const field = fields [ f ]
f.resolve = async function ( ...args ){
const [ , , ctx ] = args
return await resolve.apply ( this,args)
}
} )


Feature file standards?

Hi @jan-molak,
Greetings!
I have 6 feature files each with 50 to 60 scenarios in my project (using cucumber), should i break the feature file into more detailed functionality or its okay to keep it as it is with respective funtionality tags(@booking, @checkout).
Need suggestions, Thank you
Best Regards,
Abhineet Sharma

Hi Abhineet!
50-60 scenarios per feature seem like a fairly large number, but it's hard to tell without actually seeing the codebase.
Remember, that what you can do is organise feature files into directories, which can make splitting large feature files into smaller ones more manageable.
For example, let's say we test an e-commerce app, and we have our Cucumber test suite structured as follows:
features
  - basket_management.feature
  - checkout.feature
  - product_catalogue.feature
  - search.feature
  - shipping.feature

If you think of an app the scale of Amazon, each one of the above "features" would be quite complex and involve numerous scenarios and a plethora of edge cases.
That's where it helps to introduce the concept of "capabilities":

Capability - something the system enables the user to do (the "What")
Feature - the exact way the system enables you to do it (the "How")

@wakaleo covers this topic in his excellent "BDD in Action"
As you can see, the "high-level" features we defined above align with the idea of capabilities rather well:

basket management - enables the customer to manage their basket: add, remove items, change quantity, remember basket between the sessions etc.
checkout - enables the user to securely pay for the items in their basket, decide on the delivery address and so on
product catalogue - enables the user to browse products by category, brand, etc.
search - enables the user to find the item via full-text search, finding items similar to what the user was looking for, suggesting popular items
shipping - standard delivery, next day delivery, and so on and so forth.

Now that we've described them, we can turn the features those capabilities cover into feature files:
features
  - basket_management
    - remember_basket.feature
    - manage_quantity.feature
    - add_remove_items.feature
  - checkout.feature
    - debit_card_payments.feature
    - paypal_payments.feature
  - product_catalogue
    - products_by_categry.feature
    - products_by_brand.feature
    - featured_products.feature
  - search
    - full-text_search.feature
    - similar_items_search.feature
  - shipping
    - standard_delivery.feature
    - next_day_delivery.feature

This layout can help you keep the feature files simpler, while preserving the bigger picture.
Also, when Serenity produces the report, it will turn the names of the sub-directories storing the feature files into capability names, so no additional configuration is required.
Hope this helps!
Jan


asynchronous throw

What's the best way to test for throw() (great addition btw) asynchronously? using with mocha.
Example...
it('works sync', function() {
  (function() {
    throw {message: 'error'};
  }).should.throw();
});

this works.

it('should throw async', function(done) {
  (function() {
    setTimeout(function() {
      throw {message: 'error'};
    }, 200);
  }).should.throw();
});

AssertionError: expected an exception to be thrown

those callsites will have come and gone so you can't try/catch async stuff, that's why node does callback(err)

also: devthought.com/2011/12/22/a-string-is-not-an-error/
you wont want to throw { message: 'here' }

hey TJ, thanks for the link, yeh I still don't know how to read those stacktraces... I should figure that out.
correct me if I'm wrong, but if you throw new Error doesn't mocha just stop and returns?

what do you mean? you dont know how to read stack traces? but the thing is that exceptions just "unwind" the stack, so when you're in a callback from something async it's not the same stack anymore so try/catch doesn't work

Error
    at Object.<anonymous> (/private/tmp/error.js:2:11)
    at Object.<anonymous> (/dir/errorLoop.coffee:6:4)
    at Module._compile (node.js:462:23)

might be cause I predominantly use coffee-script, but those :2:11 never matches up for me lol... so I usually disregard the error outputs...

ah.. yeah that would be coffeescript for you lol aren't you glad it's so helpful

hah... yes... but coffee has it's perks... !


Incorrect reference for guestbook image in readme

The README for v1 and v2, under Create the guestbook pods, has k8s.gcr.io/guestbook:v3 as image which seems like an old output. The correct one per current manifest is bmcom/guestbook:v1.

WIP


Enhance SLO CRD for Multi-Cluster Support

Description:
We need to enhance our ServiceLevelObjective (SLO) Custom Resource Definition (CRD) to better support multi-cluster deployments with a centralized monitoring solution. The current implementation has limitations when applied across multiple Kubernetes clusters, particularly concerning namespace deletion and recording rule management.
Current Problem:

When integrating the SLO CRD with Helm charts and applying to multiple Kubernetes clusters, namespace and group deletion in one cluster can affect the entire group of rules in the centralized monitoring solution. There is no way we can modify or control recording namespace name.
There's no mechanism to append new reconciled rules from different Kubernetes clusters to the same group without overwriting existing rules for a give namespace.

Desired Solution:

Implement a mechanism to create a new namespace (new attribute in CRD) within the same group when reconciled different Kubernetes clusters in the centralized monitoring solution. By default we can keep namespace and group name same unless it is defined in CRD.
When deleting an SLO object in any given cluster, it should only delete the namespace specific rules instead of the entire group.
mplement a finalizer that deletes the entire group only when there's just one namespace associated with it.

Technical Considerations:

This solution may introduce potential race conditions as multiple operators will be working on the same group.
The operator should be designed to reconcile on failure, mitigating issues from concurrent operations.
Implement proper locking mechanisms if required or use a distributed locking service to manage concurrent access to shared resources (Optional). Interface to global distributed locking.

Acceptance Criteria:

SLO rules from multiple clusters can create a namespace in the same group in the centralized monitoring solution.
Deleting an SLO object in one cluster only removes the rules specific to the namespace which is defined in CRD.
A finalizer is implemented that deletes the entire group when only one namespace remains associated with it.
The solution is resilient to race conditions and can handle concurrent operations from multiple clusters (Out of scope). (Optional)

I can work on this if there are no concerns.


Refine CI messages, warn level, etc

This PR refines the syntactic expression of CI, status checks etc.
CI on this fork has been set up for quad double work only.
Hi @sinandredemption and @cosurgi, as we discussed offline in other chats, i added some rudimentary CI to this fork.
ALSO: Please note: At the moment, tests add, mul pass, whereas sub/div actually fail. Nonetheless, success is returned from main()for the sake of CI going green. We need to fix sub/div. Maybe I missed something simple here, as I often times do miss something simple.
Initial CI performs with the following characteristics:

CI runs on pushes to branches as well as push/PR to develop
Matrix tests GCC/clang naticve with C++11, 14, 17, 2a on Ubuntu 20.04 LTS, resulting in 8 tests
Use arithmetic tests add, sub, mul, div.
Do not use bjam and do not use Boost.Test
Pattern is command-line build with one or more tests/files like this one here
For the moment, just use 1 main() subroutine and return 0 or -1 based on all-test success/fail.


Account for defcon threshold being None when displaying server info

Fixes BOT-XK.

This works as a fix for the server command, but there's an underlying issue which is that self.threshold is set to None by default, but then becomes an empty relativedelta() upon reset. My plan was to replace the None's with a relativedelta(), as it is apparently false-y anyway, and then it's possible it's not needed to change the server command.

Yeah you could avoid it being None, but it still seemed to be an intentional choice to display the dash character rather than "0 days", so that is what I stuck to here.


OnLoad page autofocus on chrono

Hello,
I have a one page infinite scroll website and when loading the focus is setting on my react chrono.
If I put of the component, it's ok of course.
Is there a way to counter that please?
Best regards,
Loïc

could you please point me to the repo where this is happening.
thanks

You ca found it here : https://github.com/lohax/MonCvOnline
And yes, I'm using the last version 1.6
Thank you

thanks i am unable to run the repo locally. is there a gif you can attach to help me understand the problem better.
here is the error i see while running
error Problems with gatsby-source-contentful plugin options:
spaceId: undefined - "spaceId" is required
accessToken: undefined - "accessToken" is required

Yes, it's beacause of my CMS acces.
You can see it on : https://loic-chambost.dev

thanks. this needs to be fixed on the component. ill keep you posted on the patch/fix.

Thanks a lot for your work !

please update react-chrono to 1.6.1 and let me know how it works out for you.

It's ok with this new version !
Thanks a lot!


Can addObstacle improved to be more effecient?

I need to use recastnavigation in my game server,  I test the method addObstacle cost average 7ms to recast tile mesh in release O2 version, but I expected 100 battles on my server at the same time. Is there anyway to improve it? Thank You.

I guess addObstalce is not your problem, but the update -> buildNavMeshTile(..) is the slowpoke ? Did you test which function is the cause of it ?
I would say it is the rebuilding of the mesh tile(s), that is the slowest part in this process. You can try to lower the TileSize, that should speed things up.

I guess addObstalce is not your problem, but the update -> buildNavMeshTile(..) is the slowpoke ? Did you test which function is the cause of it ?
I would say it is the rebuilding of the mesh tile(s), that is the slowest part in this process. You can try to lower the TileSize, that should speed things up.

Thanks Your Reply.
Yes, I mean that it costs most in buildNavMeshTile. I test it in default tile size 48x48 and using cell size 0.17. I will try to lower the tile size. Whether it would increase cost in other method like find Path？

I don't know how the cell size relates to whatever you are doing, but if you increase the cellsize, that should also speed things up a bit.

I don't know how the cell size relates to whatever you are doing, but if you increase the cellsize, that should also speed things up a bit.
A smaller tilesize should not really affect findpath, the effect of smaller polygons in some places should be nearly zero when it comes to pathfinding.

Ok, I will try it. Thank You.


Error in opening URL when part of it is not shown

Description of the problem or steps to reproduce
Open a file containing a URL
Resize the window so that part of the URL is not visible
Ctrl+Left Click on the URL to open it
Only the visible part of the URL is opened in the browser
Occasionally, the line number of the next line may be included in the URL
Example 1:

The URL in the file is https://intranet.nyuad.nyu.edu/files/resources/driver-license.pdf but the URL opened is https://intranet.nyuad.nyu.edu/files/resources/driver-lic
Example 2:

The URL in the file is https://github.com but the URL opened is https://github.c2
Specifications
Version: 1.3.2
Commit hash: 2f587c6
OS: Solus
Terminal: GNOME Terminal 3.24.2

Opening a URL with ctrl-left click is not a feature provided by micro but rather by gnome terminal. Your terminal will look at the characters it is displaying and determine that it is a URL and will open that. The terminal doesn't have access to what the rest of the URL is so it cannot know that there are more characters.
If you turn softwrap on the terminal might figure it out but I'm not sure. This is not a bug with micro because this isn't a feature of micro -- this is just one of the annoyances of running a program from within a terminal (the terminal and the program have limited communication). Perhaps there could be a keybinding in micro that opens the url the cursor is on (like gx in vim).


Migrate to str.format() for Python 3 compat

In setup.py you use print '...' % package_dir. This is incompatible with Python 3, and the str.format() syntax is compatible with Python 2 and 3.
http://stackoverflow.com/questions/13945749/string-formatting-in-python-3

@gmr Just co confirm. You still don't have plans to port this to Python 3? Or you're welcome to all pull-requests with Python 3 support features?


Import Data.Semigroup, enabling build with GHC 8.2.2

This allows the library to be used with GHC 8.2.2

hi, thank you
could you add  in travis.yml
 - env: GHCVER=8.2.2 STACK_YAML=stack-<stackversion>.yaml
      addons:
        apt:
          sources:
          - hvr-ghc
          packages:
          - ghc-8.2.2

then we can be sure that every change in code does not Fail 8.2.2
with stack version 11.10 i had also problem with Megaparsec dependencies ? which stack version did you used?

Done!

looks good, you can merge it


ComfyUI import failed

I can't install this extension for ComfyUI, since it fails to load the extension due to the following error:
Traceback (most recent call last):
  File "S:\Library\Software\comfyui\ComfyUI\nodes.py", line 1993, in load_custom_node
    module_spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "S:\Library\Software\comfyui\ComfyUI\custom_nodes\z-tipo-extension\__init__.py", line 1, in <module>
    from .nodes.tipo import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS
  File "S:\Library\Software\comfyui\ComfyUI\custom_nodes\z-tipo-extension\nodes\tipo.py", line 38, in <module>
    import kgen.models as models
ModuleNotFoundError: No module named 'kgen'

Cannot import S:\Library\Software\comfyui\ComfyUI\custom_nodes\z-tipo-extension module for custom nodes: No module named 'kgen'

ComfyUI is up to date and I also tried with a clean install, but no dice.

the installation code failed on your env
install tipo-kgen library through pip by yourself

I'm not a python guru, but it seems to me that the extension is installing the requirements in the wrong location?
Attempting to install LLaMA-CPP-Python
Looking in indexes: https://pypi.org/simple, https://jllllll.github.io/llama-cpp-python-cuBLAS-wheels/AVX2/cu121
Requirement already satisfied: llama-cpp-python in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (0.2.26+cu121)
Requirement already satisfied: typing-extensions>=4.5.0 in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from llama-cpp-python) (4.12.2)
Requirement already satisfied: diskcache>=5.6.1 in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from llama-cpp-python) (5.6.3)
Requirement already satisfied: numpy>=1.20.0 in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from llama-cpp-python) (2.1.2)

[notice] A new release of pip is available: 23.0.1 -> 24.2
[notice] To update, run: python.exe -m pip install --upgrade pip
Requirement already satisfied: tipo-kgen>=0.1.2 in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (0.1.3)
Requirement already satisfied: huggingface-hub in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from tipo-kgen>=0.1.2) (0.25.2)
Requirement already satisfied: transformers in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from tipo-kgen>=0.1.2) (4.45.2)
Requirement already satisfied: torch in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from tipo-kgen>=0.1.2) (2.4.1)
Requirement already satisfied: requests in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from huggingface-hub->tipo-kgen>=0.1.2) (2.31.0)
Requirement already satisfied: pyyaml>=5.1 in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from huggingface-hub->tipo-kgen>=0.1.2) (6.0.2)
Requirement already satisfied: filelock in c:\users\veggie\appdata\roaming\python\python310\site-packages (from huggingface-hub->tipo-kgen>=0.1.2) (3.10.0)
Requirement already satisfied: tqdm>=4.42.1 in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from huggingface-hub->tipo-kgen>=0.1.2) (4.66.5)
Requirement already satisfied: typing-extensions>=<IP_ADDRESS> in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from huggingface-hub->tipo-kgen>=0.1.2) (4.12.2)
Requirement already satisfied: packaging>=20.9 in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from huggingface-hub->tipo-kgen>=0.1.2) (23.2)
Requirement already satisfied: fsspec>=2023.5.0 in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from huggingface-hub->tipo-kgen>=0.1.2) (2024.9.0)
Requirement already satisfied: sympy in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from torch->tipo-kgen>=0.1.2) (1.13.3)
Requirement already satisfied: jinja2 in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from torch->tipo-kgen>=0.1.2) (3.1.4)
Requirement already satisfied: networkx in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from torch->tipo-kgen>=0.1.2) (3.3)
Requirement already satisfied: tokenizers<0.21,>=0.20 in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from transformers->tipo-kgen>=0.1.2) (0.20.1)
Requirement already satisfied: numpy>=1.17 in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from transformers->tipo-kgen>=0.1.2) (2.1.2)
Requirement already satisfied: safetensors>=0.4.1 in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from transformers->tipo-kgen>=0.1.2) (0.4.5)
Requirement already satisfied: regex!=2019.12.17 in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from transformers->tipo-kgen>=0.1.2) (2024.9.11)
Requirement already satisfied: colorama in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from tqdm>=4.42.1->huggingface-hub->tipo-kgen>=0.1.2) (0.4.6)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from jinja2->torch->tipo-kgen>=0.1.2) (3.0.1)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from requests->huggingface-hub->tipo-kgen>=0.1.2) (2023.7.22)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from requests->huggingface-hub->tipo-kgen>=0.1.2) (2.0.7)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from requests->huggingface-hub->tipo-kgen>=0.1.2) (3.3.0)
Requirement already satisfied: idna<4,>=2.5 in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from requests->huggingface-hub->tipo-kgen>=0.1.2) (3.4)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in c:\users\veggie\appdata\local\programs\python\python310\lib\site-packages (from sympy->torch->tipo-kgen>=0.1.2) (1.3.0)

Yes but it is somehow expected
It is very easy to prevent any python code to install things in a specific env
So it is impossible for me to ensure all the ppl can get automatic installation
Please install required lib by yourself

I fixed the issue by using the following code instead of os.system("pip install ...")

This should be the intended approach based on other extensions that I looked into.

I fixed the issue by using the following code instead of os.system("pip install ...")  This should be the intended approach based on other extensions that I looked into.

No
This still can't ensure all the situation
But this is "better"

Should I make a pull request for it?

Should I make a pull request for it?

It's fine
If you want to do that, try to wrap the installation things into one function for possible future use


avx2: added mm256_sad_epu8, mm256_mpsadbw_epu8, mm256_maddubs_epi16

avx2: added impl for mm256_sad_epu8
avx2: added impl for mm256_mpsadbw_epu8
avx2: added impl for mm256_maddubs_epi16

I have done a straightforward implementation for mm256_maddubs_epi16 in this PR. For the parallel version, what I understood from this comment is that we have to first perform a widening multiplication after which we will get a 256 bit block. After that we have to use parallel operation. But I have few doubts, firstly I am confused about how to do the widening multiplication? Apart from this after we do the widening multiplication we need to do saturated parallel addition and return the lower 128 bit block. If we want to do it like that we need to write new deinterleave operations as existing x_mm256_deinterleave_even returns a0,a2,a4,a6,b0,b2,b4,b6,a8,a10,a12,a14,b8,b10,b12,b14 and not a0,a2,a4,a6,a8,a10,a12,a14,b0,b2,b4,b6,b8,b10,b12,b14. So in my opinion if we have to use parallel operation we will need new parallel functions and return the `'lower 128  bit block' of mm256_adds_epi16( new_mm256_deinterleaveeven(x) , new_mm256_deinterleaveodd(x) ) where x is result of widening multiplication. Am I thinking in right direction? I am a little confused about this. Please tell me what I need to do for parallel impl of maddubs or if the current one is good enough?

I think that at the time I was under the impression that hadd would do the sensible thing instead of switching back and forth between a and b, in which case it would have just been a matter of
return simde_mm256_hadd_epi16(
    simde_mm256_mul_epi16(simde_mm256_cvtepu8_epi16(a_.m128i[0]), simde_mm256_cvtepu8_epi16(b_.m128i[0])),
    simde_mm256_mul_epi16(simde_mm256_cvtepu8_epi16(a_.m128i[1]), simde_mm256_cvtepu8_epi16(b_.m128i[1])));

Unfortunately this is another example of Intel's APIs lacking any sort of consistency.  In this case, it's probably because they are trying to reuse hardware.
I'm actually thinking it might be a good idea to rename the existing simde_x_mm256_deinterleave* functions, since what they're doing isn't really deinterleaving but something rather more complicated.  They're still useful as internal APIs, but they're pretty useless as a public API.
Anyways, for now I think what you have is about as good as we're going to get.  A SIMDE_SHUFFLE_VECTOR-based implementation would be nice, but I think the fallbacks that will mainly be here is what is currently the SSSE3 fallback (I'm planning on changing it soon to be a fallback for all 128-bit ISA extensions), so it probably doesn't matter much.  I'll go ahead and merge this soon.
BTW, you keep using "parallel", I think what you mean is "pairwise" (as in operates on an adjacent pair of lanes).

BTW, you keep using "parallel", I think what you mean is "pairwise" (as in operates on an adjacent pair of lanes).

yes, my bad, I meant pairwise.


CC74600: Unnecessarily nested lists

Hello, @karann-msft,
This proposed file change comes from https://github.com/NuGet/docs.microsoft.com-nuget.es-es/pull/4 .
Could you review this contribution and help to merge if agreed?
Many thanks in advance.

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/index.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

@v-makoud I am not sure I understand the impact. What are we trying to change here? Is it this page?

Hi @karann-msft ,
Contributor claims that there were unnecessarily nested lists, making the file look bad.
As this change came from a localized PR wee need to report it but please, just take a look at the changes and let us know if you agreed.
Regards.

Hi @karann-msft ,
Contributor claims that there were unnecessarily nested lists, making the file look bad.
As this change came from a localized PR wee need to report it but please, just take a look at the changes and let us know if you agreed.
Regards.

Hi @karann-msft ,
Contributor claims that there were unnecessarily nested lists, making the file look bad.
As this change came from a localized PR wee need to report it but please, just take a look at the changes and let us know if you agreed.
Regards.

I copied the change to a new branch zivkan-test, so that it can be viewed in the docs review tool. The index page with the change looks exactly the same as without the change.  Therefore the only benefit is for people who view source, but it doesn't appear to cause any problems.

Thanks @v-makoud @v-maudel  @zivkan

In case anyone looks at this PR, I made a mistake in my zivkan-test branch and didn't actually merge the change. This PR in fact broke the layout of the index change, so it has been reverted.


Add CI pipeline to automatically format code

This PR introduces a CI pipeline to automatically format code using Prettier. In the PR there are also included the repository files modified through the CI pipeline that is introduced. For any changes required please let me know.

I faced a dilemma on whether the CI should fail and have the developers fix the styling issues or auto-format the code. I decided to choose the second approach because I thought that as long as the functional changes will be associated with the developers that wrote the code and only the formatting changes will be credited to the CI automation, then it would decrease the manual efforts required to fix the styling issues. However, it is quite simple to make the workflow fail in such cases and not auto-format the code so if you think that it is a better approach I will change this.

I faced a dilemma on whether the CI should fail and have the developers fix the styling issues or auto-format the code. I decided to choose the second approach because I thought that as long as the functional changes will be associated with the developers that wrote the code and only the formatting changes will be credited to the CI automation, then it would decrease the manual efforts required to fix the styling issues. However, it is quite simple to make the workflow fail in such cases and not auto-format the code so if you think that it is a better approach I will change this.

The problem with automatic changes is that git blame will not associate them by default with the person who made the actual change. So since you're also on the fence, let's got for a failing CI. We can provide instructions for a pre-commit action that developers can use to format the code before the commit.

I have created a pre-commit hook that runs the prettier check command and in case that there exist styling errors it suggests to the developer to run the npm run prettier-fix command (prettier-fix script runs prettier --write . command). How could I commit the pre-commit hook since it is not getting tracked ?. Would it be a good practice to have a .githooks directory that contains the pre-commit hook and then have the developers run a script that copies that hook within the .git/hooks directory ?

I don't know of a good way to do this.  In another project I documented it as part of the development processes. You can add similar instructions in a README.md section.


Error while Nordigen node client library on node version > 12.x : Must use import to load ES Module: /../node_modules/nordigen-node/lib/index.js require() of ES modules is not supported.

I am using latest stable node version i.e 16.14.2, I get below mentioned error on compiling the code.
const nordigen_node_1 = require("nordigen-node");
^
Error [ERR_REQUIRE_ESM]: require() of ES Module /home/polyscripts/polyscripts/payments-backend/node_modules/nordigen-node/lib/index.js from /home/../dist/nordigen/nordigen.controller.js not supported.
Instead change the require of index.js in /home/../dist/nordigen/nordigen.controller.js to a dynamic import() which is available in all CommonJS modules.

Hi.
You should use import NordigenClient from "nordigen-node"; instead of require
As well you will need to add "type": "module", to package.json
You can check express example app in the example directory

Hi @victory-sokolov ,
Thanks for the quick reply,
I followed the same approach as importing NordigenClient from "nordigen-node". I'am using nestJs and the code I write is in typescript. Hence the error that is being thrown is in the transpiled .js file, that I do not have any control over.
On doing so I got an error mentioned below:Must use import to load ES Module: /home/../node_modules/nordigen-node/lib/index.js
require() of ES modules is not supported.
Aslo about adding "type": "module" to package.json, on trying that I got an error as attached in the screenshot.
well I surfed on internet for further resolution, but few say index.js should be renamed to index.cjs, but I am not sure how and where that would affect the whole application.

If you want to use commonjs you can run the module through babel and it'll work.

@praji1304 Could you please share your solution? I am still stuck on it. Also using NestJS.
Thanks

@jans-y  did not find any solution that time at least, I ended up implementing the integration with their APIs. It will cost you a day but work perfectly fine.

@praji1304 Thanks for the response. I did the same. Would be nice to either have it fixed or state this into docs as known limitation so devs don't waste their time on this issue ;)

If you want to use commonjs you can run the module through babel and it'll work.

Can you explain how to do it?

Hello!
Sorry for the delay in response.
You could try to use dynamic-imports to overcome this problem.
Try something like this and let us know if this will works for you
const nordigenNode = await import('nordigen-node');
const NordigenClient = nordigenNode.default;
const client = new NordigenClient({secretId: '', secretKey: ''});


[Battle.net] Autolaunch Game

I've read over the wiki and I have a sinking feeling that I'm providing an incorrect arguement, but it doesn't appear that I can autolaunch a game. The program does launch battle.net correctly, but I still have to click "play" for Overwatch. Is this broken or not a feature? If the latter, could we get this?

I'm dumb. For anyone else with this issue: I had LauncherPath set to "Battle.net Launcher.exe" instead of "Battle.net.exe". Both open Battle.net, but only the latter will work with autolaunch options (e.g. LauncherArgs=--exec="launch Pro").


Loss became infinite or NaN

I ran it on uodd 10-shot and the loss crashed

Hi! Thanks for your question, but we tried to reproduce it by three different co-authors, and we didn't cause this error (I actually have no idea about it, may you check maybe the version of your environment.

thx, I'll try to figure it out

I solved it by changing batch size to 8, I think there is still a potential problem, but it shouldn't be related to this work

Still thanks for your feedback!


Backup of odoo database

i want to take backup of odoo database using shell command and crontab please guide step by step i am new here

@dwarkab I'm afraid you may have the wrong place. This repository has nothing to do with Odoo. I recommend contacting Odoo support through their official website, or joining in with the Odoo community on IRC/Slack/mailing lists/etc.


Last Week in Pony - January 22, 2023

Version 0.7.0 of ponylang/ponyup has been released.
See the release notes for more details.


Context question

Hi,
trying out Preact, following code is supposed to show the user name in Child component, but it shows nothing, anything missing? thanks
function Child() {
    return (

child component {this.context.user}

    )
}
class Myprovider extends Component {
    getChildContext() {
        return { user: 'Prect' }
    }
    render({ children }) {
        return children[0];
    }
}

class App extends Component {
    render(props, state) {
             return (
            <Myprovider>

 Parent

                <Child></Child>
            </Myprovider>
        )
    }
}

render(<App />, document.body)

There are actually two mistakes in your example.

Your Child doesn't extend Component and will therefore not have a context property.
Preact passes the context as the second argument to stateless components, so you can use the second argument as the context.

You are only returning the first child in your Myprovider which in this case is the
 Parent
.
So the <Child></Child> will never be rendered. You can fix this by either returning all children wrapped in e.g. a
 or by removing the
 from your App.

You example updated to return all children in Mycomponent and to use the second argument instead of this.context.
function Child(props, context) {
    return (

child component {context.user}

    )
}
class Myprovider extends Component {
    getChildContext() {
        return { user: 'Prect' }
    }
    render({ children }) {
        return
{children}
;
    }
}

class App extends Component {
    render(props, state) {
             return (
            <Myprovider>

 Parent

                <Child></Child>
            </Myprovider>
        )
    }
}

render(<App />, document.body)

Or you can view your example here within the preact repl.

Accessing context via this only works for class-based components. Functional components get it passed directly as a standard argument:
function MyComponent(props, context) {
  return
{context.whatever}

}

@Kanaye
thanks for the sample, it works! now I change the Child to extend from Component so that it will have a context, but not working in this example , what I'm missing here?

@angelochen960
Your Child2s render() is not returning anything.
Change line 13 from

child component {this.context.user}

to
return (
child component {this.context.user}
);


Ignore pod scheduling errors during deployments

We are seeing the following issue during app deployments periodically (especially when the app has many pods and the cluster is highly utilized in terms of resource requests).

Deployment is triggered (either git push or deis config:set, etc.)
Hephy modifies the K8s deployment and Pods are scaled up and terminated.
At some point hephy returns an error, similar to:

Error: Unknown Error (400): {"detail":"(app::deploy): There was a problem deploying v123. Rolling back process types to release v122.
There was a problem while deploying v123 of some-hephy-app-cmd. Additional information:
0/26 nodes are available: 26 Insufficient memory."}

In a cluster that is auto-scaled, this is a temporary condition and it should not lead to a deployment failure.

One strategy we tried to circumvent the problem is the following:

When a hephy app is created, we set the deployment strategy of the K8s deployment to type RollingUpdate, maxSurge==0, and maxUnavailable==20% (for example).
Our hope was that this would prevent K8s from creating more pods than the deployment's desired size, thus avoiding to run into any cluster resource limitations.
However that didn't work. The issue persists.
Our best guess atm as to why is: When the deployment is started (with the strategy above), a new replicaset is created with scale zero. Then the old replicaset is scaled down by e.g. -20% and the new replicaset is scaled up by +20%. There seems to be a period of time where new pods are already scheduled , but old ones still in terminating and still using the resource requests. During that time the new pods can potentially oversubscribe the cluster's resources and trigger the error.

@jfuechsl Is there a bigger problem if the scheduling failure doesn’t get fixed by autoscaling and you have DEIS_IGNORE_SCHEDULING_FAILURE set?
We have logic in our deployments to retry deployment when there are scheduling failures - I wonder if workflow could optionally do this rather than just ignoring the failures.

@dmcnaught good question. In that case I suppose the deployment would time out eventually and revert.
Retrying would not be desired in such cases as it should just wait for K8s to finish the deployment rollout and have enough time to schedule everything properly.

@jfuechsl Right - it wouldn’t really be retrying the deploy, but retrying the check to see if it was successful or still has a scheduling error. When enough time has passed to timeout (something like DEIS_IGNORE_SCHEDULING_TIMEOUT)  then it can return an accurate pass or fail.

@dmcnaught Great idea, let me update the PR.

One problem with that is, we are inspecting a pod's event stream to look for failures such as FailedScheduling. Once that event happened, it will always be included in the events. This makes a timeout based check retry impossible.
IMO, we shouldn't look at the pod events at all but only at the Pod phase and Pod conditions. They represent the point-in-time status of the Pod and can thus be handled with a timeout-based waiting logic on a per Pod basis. That would however necessitate a bigger change in a Deployment's wait_until_ready logic.

Thank you for the great discussion @jfuechsl and @dmcnaught .
Agreed that it would be nice to look at Pod phases and conditions rather than Scheduling Events on the k8s api. However, that would require a bit more work. As of right now, this little feature hack saves some headaches on autoscaling clusters and doesn't change the existing behaviour for users so it is good IMO.
@jfuechsl can you document the new ENV variable on the workflow docs. I added a note on the PR.

@Cryptophobia thanks very much. Yes, I will update the docs and submit a PR.

@Cryptophobia I submitted https://github.com/teamhephy/workflow/pull/108

All relevant PRs for this feature have been merged in a long while. Closing now. Thank you @jfuechsl .

All relevant PRs for this feature have been merged in a long while. Closing now. Thank you @jfuechsl .


feat: make logger happy

Background

Reduce logger memory usage. (Using uint8 is better)
Complete enumeration.

@bydmm  cc.

Occam's Razor

Occam's Razor


running question

do you meet the program stop without mistakes when the code running for some time ,and the GPU stops work.I changed the value of num_threads。（from 16 to 10）。i run it again.i don't know Is it because the value is too high。

@tuoniaoren I haven't met your situation before. Maybe it's the machine's problem?

ok!i'll check it out. I'll change the machine


Run stardist completely in colab

Hey,
unfortunately, my laptop is not suitable for training the model. It would take a whole day.
My question is if I can run the whole example script by colab on a better PC and get a model I can export to Fiji?
We have one in our imaging facility, but I don't think that we are allowed to install a complete python environment and so on.
Marc

Hi,

My question is if I can run the whole example script by colab on a better PC and get a model I can export to Fiji?

Yes, basically.
We have an example notebook that runs StarDist 2D on Colab. Otherwise, you can also try the StarDist notebooks provided by ZeroCostDL4Mic.

We have one in our imaging facility, but I don't think that we are allowed to install a complete python environment and so on.

I would suggest to use your own computer if you're (starting to get) serious about using deep learning in your facility.
Best,
Uwe


fix: support for macOS

macOS doesn't seem to allow port binding on <IP_ADDRESS>, to resolve that, use <IP_ADDRESS> instead

Changes

:bug: binds to <IP_ADDRESS> instead of <IP_ADDRESS> to make quickstart work on macOS

/kind bug
Fixes #423

Awesome, thanks!
/lgtm /approve

Thank you @psschwei!


net/buf: Add the possibility to manipulate externally allocated buf/data

Note that name sucks a bit, let's change to a better one if possible (function and flag name that is)

Codecov Report

Merging #8609 into master will decrease coverage by 0.01%.
The diff coverage is 12.5%.

@@            Coverage Diff             @@
##           master    #8609      +/-   ##
==========================================
- Coverage   64.17%   64.15%   -0.02%
==========================================
  Files         429      429
  Lines       41157    41170      +13
  Branches     6925     6926       +1
==========================================
+ Hits        26413    26414       +1
- Misses      11587    11598      +11
- Partials     3157     3158       +1

Impacted Files
Coverage Δ

include/net/buf.h
88.88% <ø> (ø)
:arrow_up:

subsys/net/buf.c
55.95% <12.5%> (-1.95%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4535b94...d13f44b. Read the comment docs.

@jhedberg what about the name? is "static" a good one. Well it was used already for net_buf_simple actually. I'll add a macro NET_BUF_DEFINE_STATIC() as well, it can be useful for those who don't need to select a different data array at runtime.

Instead of flag name NET_BUF_STATIC should it be instead NET_BUF_EXTERNAL?

Maybe. There's NET_BUF_EXTERNAL_DATA already, and then if both the struct and data it points to are "external" (to standard pool structures), makes sense to call the flag NET_BUF_EXTERNAL. The risk is confusion between "EXTERNAL_DATA" and "EXTERNAL" (a necessary/acceptable risk IMHO).
NET_BUF_STATIC would still work too, but indeed, there's confusion between a buffer being conceptually "static" and it and its data array being defined as static vars (that's slightly bigger confusion than the above IMHO).
Another way to deal with it would be to "decompose" settings to the most atomic properties, then define useful higher-level buffer types in terms of them. That's how lwIP does it for example. (I mention that for completeness, as a reference to "how others do that". I don't advocate doing that now, but I'm sure we'll need to define more buffer flags in the future (e.g. a flag for constant (ROM) vs volatile data). In this case, having atomic, orthogonal flags makes sense).
  PBUF_RAM = (PBUF_ALLOC_FLAG_DATA_CONTIGUOUS | PBUF_TYPE_FLAG_STRUCT_DATA_CONTIGUOUS | PBUF_TYPE_ALLOC_SRC_MASK_STD_HEAP),
  PBUF_ROM = PBUF_TYPE_ALLOC_SRC_MASK_STD_MEMP_PBUF,
  PBUF_REF = (PBUF_TYPE_FLAG_DATA_VOLATILE | PBUF_TYPE_ALLOC_SRC_MASK_STD_MEMP_PBUF),
  PBUF_POOL = (PBUF_ALLOC_FLAG_RX | PBUF_TYPE_FLAG_STRUCT_DATA_CONTIGUOUS | PBUF_TYPE_ALLOC_SRC_MASK_STD_MEMP_PBUF_POOL)

@pfalcon confusion could be a risk, though a minor one. But why not NET_BUF_EXTERNAL_FULL instead of NET_BUF_EXTERNAL then? at least it's easy to grasp the difference with NET_BUF_EXTERNAL_DATA
Yep I recall the idea of having a flag for ROM based data. But that will need first a kernel api function for checking if pointer is in ROM or not. A worthy idea non the less.

confusion could be a risk, though a minor one. But why not NET_BUF_EXTERNAL_FULL instead of NET_BUF_EXTERNAL then? at least it's easy to grasp the difference with NET_BUF_EXTERNAL_DATA

I don't have a specific preference here. NET_BUF_EXTERNAL_FULL sounds a bit "artificial", but indeed clearly distinguishable from NET_BUF_EXTERNAL_DATA.

let's try NET_BUF_ALL_EXTERNAL

FWIW, I have nothing against calling it STATIC. If I understand right, this is just a buffer that's not associated with any pool, i.e. a standalone one. Calling it "external" sounds quite unintuitive to me.  How are you planning to write safe code that will only try to reuse the buffer (manipulate the data, place it in a FIFO, etc) when no-one else uses it anymore? Since net_buf's have reference counting there might be arbitrary modules that the buffer passes through that grab a hold of the buffer with net_buf_ref, and assume they have valid access to it until they do net_buf_unref.

Usage of such net_bug should be restricted to very particular usage yes.
For instance, in my case, I want to create L2 header right before sending the packet into such net_buf. In this particular case it's totally fine since on sending won't grab any reference anywhere there.
But indeed, maybe that needs to be properly documented?

For instance, in my case, I want to create L2 header right before sending the packet into such net_buf. In this particular case it's totally fine since on sending won't grab any reference anywhere there.

@tbursztyka could you elaborate on this a bit? I remember you saying the buffer would be placed in some FIFO (which is why net_buf_simple is no good), which means it'll be processed at some later moment, presumably by some other thread. How does the code that took this buffer and passed it onward know when it can be used again?

I remember you saying the buffer would be placed in some FIFO

I'm sure @tbursztyka will have a chance to reply before he goes on vacation (which he said is today unless I missed it), but let me flex again my understanding of the matter (based on discussion with him): this functionality (standalone buffer) will be used when a packet is extracted from fifo by tx thread, which then processes it in synchronous manner. E.g. higher-level stack layers will prepare a complete IP packet and put it in fifo. TX thread will extract it, prepend L2 header using such a standalone/static net_buf, and make a call to driver to transmit a chain of buffers. When this call returns, the static buffer can be reused.
In other words, this standalone buffer is not expected to be an application-level construct, but rather an internal one.

@jhedberg it's not going to be placed in FIFO, it's going to be appended/inserted into a net_buf list. And it is not going to be processed later (or then it would be a serious problem).
Here is the idea, for instance on ethernet l2 (do not try to follow current code, it's going to change):
net_pkt being sent:

net_if: queue pkt
net_if_tx: use L2's send()  (TX thread at this point)
ethernet_send: create ethernet header in its locally allocated net_buf. Insert into net_pkt, send it to driver to send it.
driver: send the packet right away (no other thread, nothing, we are still in TX thread)
drivers returnse, either success or error: ethernet l2 removes the locally allocated net_buf from net_pkt
L2's send returns

@tbursztyka ok, thanks for the clarification. Then I would make at least a few things to avoid bugs with it:

Document this properly
Many API calls such as net_buf_ref, net_buf_unref, net_buf_put & net_buf_get fail if called for such a buffer.

Actually thinking more about this, what I suggested just opens a can of worms. E.g. until now net_buf_ref() has always succeeded and there's lots of code assuming it will. We'd need to update all those places if it can now suddenly return NULL for a buffer with the STANDALONE flag set.
Also, having the buffer in a linked list is a reference (in the sense that there's a pointer that references the buffer), and not incrementing the refcount for it is just a hack. What about the other buffers in the linked list which don't have the STANDALONE flag set - sounds like there might have been a bug there from the beginning if the linked list links don't use the refcounting mechanism.
So in summary, I'm starting to feel like the least hackish option would still be to have dedicated pool with a single buffer, and have an assert() if allocating from the pool fails using K_NO_WAIT, since that means that some code in the code path of the function that's supposed to consume the buffer hasn't actually fully consumed it.
Does @jukkar have some thoughts on this?

Make API calls such as net_buf_ref, net_buf_unref, net_buf_put & net_buf_get fail if called for such a buffer.

This is a bit problematic if we would need to check in the driver if the buf is "static" one or not before trying to unref if.

So in summary, I'm starting to feel like the least hackish option would still be to have dedicated pool with a single buffer, and have an assert() if allocating from the pool fails using K_NO_WAIT, since that means that some code in the code path of the function that's supposed to consume the buffer hasn't actually fully consumed it.

Having a pool of just one buf for this purpose sounds like a good solution. This way this "special" buffer would look exactly like any other buffer, and we could control the life cycle of it very well.

Just getting back on this.
Having a pool for a statically allocated net_buf is way too overkill. Idea of such net_buf is to use those very locally, and obviously knowing when a ref/unref is going to happen no avoid issues related to it.
I might drop that PR altogether, and instead up to - for instance - ethernet l2 to allocate statically a net_buf by itsels, put into the list, and removing it relevantly. (@jukka: there will not be any unref on driver side anymore btw). It's not going to be super clean but at least will avoid any ambiguity .

@jhedberg, @jukkar  fyi: https://github.com/tbursztyka/zephyr/commit/e6d99789d493d3ddbd2eaa4008442bf8b3a6fa7b Well, it works pretty well that way, no need to temper with net_buf code base then.


Clarify requirement instructions for App Engine

https://github.com/GoogleCloudPlatform/google-cloud-java/blob/master/APPENGINE.md
says to add appengine-api-1.0-sdk for App Engine. This should emphasize that it is only needed for local dev appserver1 (not sure about dev appserver2). The instructions mislead people to believe the dependency is required on prod too.
Also, it says

Note: the following instructions are not required for App Engine Flex and Java 8 standard environment.

However, I verified that dev appserver emulating the Java 8 runtime still needs the dependency. Also, mentioning flex here is confusing, because there is no dev appserver for flex.
(BTW, "Flex" should be "flexible environment" in docs.)

Fixed by #2948


Variable possibly unbound warning after enumerate?

Consider this code snippet-
for i, val in enumerate(range(1, 11)):
    print(val)
print(i)

I get a "i" possibly unbound: Pylance on that last print(i). But the variables in a for....in are accessible after that for....in too. As is the case here, this prints correctly. Should this be a valid warning?
More information here.

Yes, this is a valid warning. In general a for loop can execute zero (or more) times depending on the iterator provided by the expression after the keyword in. If it executes zero times, the variable i will be unbound, and the program will crash. In this particular case, you know that the iterable enumerate(range(1, 11)) will always cause the for loop to execute more than zero times, but a type checker doesn't have enough information to determine that from the type information it is provided. It must assume this is a possible bug.
If you want to use the target of a for loop (in your case, i) outside of the for loop, you should make sure it is initialized before the for loop. For example:
i = None

Hi, @erictraut, how to disable the kind of checking?

This is a potential bug in your code, so the best thing to do is to change your code to eliminate the potential bug.
If you really want to suppress the entire diagnostic rule, you can do so on a line-by-line basis with a "# type: ignore" comment, in a file-by-file basis using a "# pyright: reportUnboundVariables=false" comment, or across your entire project (or all projects) by setting the python.analysis.diagnosticSeverityOverrides setting for reportUnboundVariables to "none".

Thank you very much! It works!


Check for environment drift

If one of the putEnvironment calls fails in upload-environment.js there could be a possibility of some functions having different environments. This could be a check added to shep doctor

This has ended up becoming part of shep config sync


Dashboard

Improvements and features to the Professional Dashboard:

Expanded the contract context to support multiple contracts.
Refactored the account context for better clarity and fixed related bugs.
Organized contexts for better structure and readability.
Optimized smart contract calls for checking allowances.
Introduced a dashboard with.
Created utility functions for ERC20 token interactions.
Fixed some bugs to improve user experience.
Added functionalities to display balances for a list of tokens, along with redeem and invest buttons.
Implemented the ability to withdraw from the vault.
Enabled claiming through the UI.
Integrated event listeners to capture and display events in the front end.
Displayed a history of payments for both developers and companies.

This PR addresses the requirements of the issue "Professional Dashboard#12", enabling professionals to claim stablecoins, community tokens, and view historical payments.
Closes #12.

I understand now how semantic messages work


Packet Tracking

Router/PP (Router Utils) should periodically send packet counts to Helium Packet Router (HPR)

Note applicable anymore


Like-type change outputs

Samourai Wallet has a feature called Like-type change outputs. See:

https://samourai.kayako.com/article/53-like-type-change-outputs
https://docs.samourai.io/wallet/basic-concepts#like-type-change

Basically, the purpose is to avoid the following problem:
User sends a single UTXO from a native segwit address to a legacy address and then a native segwit change address. Basic review of the transaction will imply that the legacy address is the receiver and the native segwit address is the change address since the sending UTXO was native segwit.
With Like-type change outputs both outputs of a transaction are of the same script type. So, in the example above, both outputs would be a legacy address. This results in a little "back tracking" of the script type potentially, but offers more privacy. It also makes the wallet it bit more challenging to design, especially if the seed was already used for another wallet that used an older script type.
Therefor I'm not sure if you want to implement this, but I at least wanted to bring it up for discussion. I think native segwit and taproot addresses appear as the same format, so maybe this issue will start to go away in the future and it is not worth wasting the time on it?

Duplicate of #466 (as it is effectively required to support this). Fwiw I have moved even further away in the intervening period.

You're just saying you likely may never fix https://github.com/sparrowwallet/sparrow/issues/466

You are suggesting this is something to be "fixed" - rather it is a deliberate design decision made on the basis of understanding the tradeoffs of both options. In fact the codebase has evolved since to make this easier to do, but the same problems with UX apply, while the actual need has reduced as legacy script type usage has dropped.
For example, when a user links a BIP47 wallet, Sparrow must create and manage 3 wallets for P2PKH, P2SH-P2WSH and P2WSH script types. Luckily, since BIP47 wallets are not top level wallets, much of this complexity can be placed into areas of the interface that are less frequently visited, but if multiple script types were a top level feature for all wallets this would be impossible.