Set up a basic automated e2e smoke test

Summary
This PR sets up a basic automated e2e smoke test:

Automated means the whole testing is runnable with a single command.
e2e means it exercises the whole P4 DPDK stack including the p4c compiler (with DPDK backend), DPDK software switch, TDI runtime API, and some other libraries provided in this repo.

Background
In our current experimental usage of the P4 DPDK stack, we sometimes identify issues that could be caught early by a simple e2e smoke test. We think adding these tests to this public repo, and setting up GitHub CI to run them automatically, could be a useful addition to the existing unit tests.
Demo
Build a Docker image with the P4 DPDK stack (can take up to hours):
$ ./tools/docker_build.sh

Run the smoke test in a container:
$ ./tools/docker_run_smoke_test.sh
INFO:root:Run command:
    p4c-dpdk --arch pna --p4runtime-files e2e-test/suites/pna/basicfwd/p4c_gen/p4info.txt --bf-rt-schema e2e-test/suites/pna/basicfwd/p4c_gen/bfrt.json --tdi e2e-test/suites/pna/basicfwd/p4c_gen/tdi.json --context e2e-test/suites/pna/basicfwd/p4c_gen/context.json -o e2e-test/suites/pna/basicfwd/p4c_gen/config.spec e2e-test/suites/pna/basicfwd/main.p4

INFO:root:Using SDE: /home/sde
INFO:root:Using SDE_INSTALL: /home/sde/install
Node Pages Size Total
0    16    2Mb    32Mb
1    16    2Mb    32Mb

Hugepages mounted on /dev/hugepages
INFO:root:Run command:
    sudo -E PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LD_LIBRARY_PATH=/home/sde/install/lib:/home/sde/install/lib64:/home/sde/install/lib/x86_64-linux-gnu /home/sde/install/bin/bf_switchd --install-dir /home/sde/install --conf-file /home/work/e2e-test/suites/pna/basicfwd/conf_bf_switchd.json

INFO:root:PID: 34
INFO:root:Run command:
    /home/sde/install/bin/bfshell -f /home/work/e2e-test/suites/pna/basicfwd/cmds_bfshell.py

INFO:root:PID: 56
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
.
Sent 1 packets.
..
Sent 1 packets.
.
----------------------------------------------------------------------
Ran 2 tests in 2.277s

OK

For more details, see e2e-test/README.md.
Current status and further plans
The test added in this PR is very basic, but shall serve as a good starting point for test organization.
@lzhzero and I will keep working on adding more tests in follow-up PRs, and reporting issues if we encountered any, until we get a fully GitHub CI automated representative sets of e2e tests.

One specific question I have now is how to best set up the Docker images? I could think of several different options, and have implemented one of them. But there could be reasons to prefer other options.

(This one is implemented) A Docker image based on the existing p4lang/p4c image built using this Dockerfile in the p4c repo continuously.
We could build p4c from source within the Docker image. One benefit is the possibility to pin a specific version of p4c to test against instead of always using the latest one.
We could have separate p4c and p4-dpdk Docker images, because these 2 stacks don't depend on each other. Then in testing, we can use the p4c container to compile the P4 code, and the p4-dpdk container to run the test against compiler outputs.

Any inputs on this are appreciated!

I'm encountering a new issue that could use some help.
Short description
The add_with_xxx command fails with:
TdiTableError: Error: table_entry_add failed on table pipe.MainControl.forwarding. [Object not found]

Relevant logs shown in bf_switchd:
MatchSpec:
priority = 0
num_valid_match_bits:48
num_match_bytes:6
match_value_bits: 0x0000c0a80166
match_mask_bits: 0xffffffffffff
ActionSpec:
action_bmap:01
num_valid_action_data_bits:64
num_action_data_bytes:8
action_data: 0x0000000200000002
get_table_id failed at 693
get_table_id failed at 714 end
2023-06-02 23:54:05.700968 BF_PIPE ERROR - dpdk table id get for table forwarding failed
2023-06-02 23:54:05.701086 BF_PIPE ERROR - not able get table metadata for table forwarding
2023-06-02 23:54:05.701134 BF_PIPE ERROR - dal_table_ent_add failed

How to reproduce?
Use the latest commit in this PR. Follow the instructions in e2e-test/README.md.
For the convenience of a quick look, below I copied the bfshell commands file I used:
tdi_python

# Add ports

for port_id in range(4):
    tdi.port.port.add(
        DEV_PORT=port_id,
        PORT_TYPE="BF_DPDK_TAP",
        PORT_DIR="PM_PORT_DIR_DEFAULT",
        PORT_IN_ID=port_id,
        PORT_OUT_ID =port_id,
        PIPE_IN="pipe",
        PIPE_OUT="pipe",
        MEMPOOL="MEMPOOL0",
        PORT_NAME=f"TAP{port_id}",
        MTU=1500
    )

# Add entries

from netaddr import IPAddress

control = tdi.main.pipe.MainControl
table = control.forwarding
table.add_with_mark_and_forward(
    dst_addr=IPAddress('<IP_ADDRESS>'),
    marker=0x01,
    port=1
)
table.add_with_mark_and_forward(
    dst_addr=IPAddress('<IP_ADDRESS>'),
    marker=0x02,
    port=2
)

The error occurs on the last two table.add_with_mark_and_forward calls.

Are you able to create at least one simple test that passes, that is not blocked by the issue you have found and reported in this comment?  https://github.com/p4lang/p4-dpdk-target/pull/102#issuecomment-1574461454
If so, it might be nice to focus on creating that first simple passing test first, and file a separate issue for the bug you have found.

Thanks Andy for your suggestion! I'll try to bypass this and come to a working test first.

@satish153 @swaroopsarma @jamescchoi Can you suggest someone who can (a) review this?  and (b) do whatever approval is required for the Github "1 workflow awaiting approval" message I see on this conversation?

Thanks @fruffy for offering help! That would be great!
I just tried building the Docker image again. It takes 45 min on my computer, and the final image is 4.78 GB. I feel the building time might be too long, and the image size too large to be usable in CI.
Without a Docker container, as long as all the dependencies are installed, the testing scripts shall work. Maybe that's an easier solution on the CI side.
I still think providing a Dockerfile is useful as a reproducible environment to try things out locally, even if it's not used in the CI. I feel the best middle ground might be what's been done in the p4c repo, that the Dockerfile is simply invoking the ci-build.sh script, which is also used in CI to build required dependencies.
One challenge I faced in setting up dependencies is how to best bring in p4c. Building p4-dpdk-target itself doesn't require p4c, but in the e2e tests p4c is needed. My current solution is to use the public p4lang/p4c image as the base image, so that p4c is automatically available. This is probably not ideal, because the p4c version is not fixed, but sometimes that might matter. If we do want to fix p4c version in testing, then it seems building p4c from source is the only option?
I'd like to continue with more detailed discussions on this and some other issues, but I feel getting them all resolved in this PR might be too much. So, I'm proposing limiting the scope of this PR to a single working test that is automated locally, and expect more work to be done before it is fully automated in CI. Hopefully this could also make reviewing this PR easier.

Just rebased the pull request

@swaroopsarma Can you take another look at this to see if it looks good to you now?  If I understand correctly, with the latest commit (number 11) on this PR, there are only new log messages output in the dal_init.c code, but no other functional changes.

@jafingerhut Thanks for the ping and sorry about the premature pushes. I added 2 more commits and now they should be ready for another review.
Also thank you @swaroopsarma for your previous review. Wish to hear more from you soon.

This is probably not ideal, because the p4c version is not fixed, but sometimes that might matter. If we do want to fix p4c version in testing, then it seems building p4c from source is the only option?

HI @qobilidop This is easy to solve, you can docker pull a specific SHA of any docker image instead of using e.g. :latest or :stable, e.g. here is the version of p4c:latest on 2022-08-19. You can navigate dockerhub and look at the tag versions, see example then get the SHA from there. You can pull or run this image directly using @sha<sha code>: instead of the normal tag, for example:
docker pull p4lang/p4c@sha256:f334bfa3f3f5fc92c3935dc88dc2dbd35a17b1d332fcb98041d5a660c666b2dc

This is how you can pin a version. Just document in any scripts what the SHA corresponds to, e.g. # This is same as p4c:latest on 2023-06-23
BTW you can also use this to build your own dockerfiles using a specific base image, I do this in the SONiC-DASH project. Here's an example Dockerfile which I base on a specific p4c docker imaghe. I then make a new docker image which is smaller by throwing away unneeded backends. Then I publish this smaller custom p4c image to another container registry which a CI pipeline or user dev environment can pull down. I saved a few hundred MB this way. https://github.com/sonic-net/DASH/blob/main/dash-pipeline/dockerfiles/Dockerfile.p4c-bmv2

To avoid long docker build times, why don't you automate the docker build and publish workflow similar to many other p4lang repos, e.g. https://github.com/p4lang/p4c/blob/main/.github/workflows/ci-container-image.yml ?
Everything gets built and published to Dockerhub daily. This will save hours, and perhaps attract more users because the initial user experience can be a deal-breaker. Strive for 5 Minutes TFD (Tine for First Dopamine).

Thanks @chrispsommers! Those are great suggestions. I didn't realize it could be that easy. The example Dockerfile you linked is especially useful. I see there is also a Dockerfile.p4c-dpdk. I'll learn from those examples and improve the Dockerfile in this PR.
As to publishing Docker images using GitHub CI automatically, I completely agree with you that this would be a great thing to have. I'm just not sure if I am the right person to set it up. Because it seems this would require more permissions (both on GitHub and DockerHub) than I currently have. I'm willing to do the work, but it might be easier for someone already having those permissions to set this up in a follow-up PR. @swaroopsarma @jafingerhut @fruffy What do you think?

I'm just not sure if I am the right person to set it up. Because it seems this would require more permissions (both on GitHub and DockerHub) than I currently have. I'm willing to do the work, but it might be easier for someone already having those permissions to set this up in a follow-up PR. @swaroopsarma @jafingerhut @fruffy What do you think?

@qobilidop while developing it in your forked dev branch, you can take a few temporary measures to develop it as far as possible w/o involving others, e.g.:

comment out the final login/publish steps in the .yml CI file, this way you can debug the build process itself
change the Docker repo to a personal one on Dockerhub, I did this for DASH and we actually used it for several months in production until we migrated the docker registry to ACR.

Actually puling the trigger to publish to p4lang/p4-dpdk docker registry would probably require the p4-dpdk account admin to install the dockerhub account credentials in the repo secrets (unless they are global for all or p4lang GitHub site).

Got it. I'll try these initial steps then. Thanks again @chrispsommers!

One more piece of advice - any changes you make to this branch will trigger CI actions in the p4lang/p4-dpdk GitHub pipeline, because the PR is already opened. If you want to try out a new CI script or other potentially messy experiments without sharing with the entire world, you can make a scratch branch off of it in your workspace (git checkout -b some-name) and push to your own repo. There you can do messy experiments while you're trying CI actions, etc. in relative privacy. You must enable Git Actions in your repo, they're off by default. Then you can merge back to the PR branch and the same actions will get triggered in the PR branch. I wish I'd thought of this before a few long, embarrassing sessions. :)

Thanks @chrispsommers! Those are great suggestions. I didn't realize it could be that easy. The example Dockerfile you linked is especially useful. I see there is also a Dockerfile.p4c-dpdk. I'll learn from those examples and improve the Dockerfile in this PR.
As to publishing Docker images using GitHub CI automatically, I completely agree with you that this would be a great thing to have. I'm just not sure if I am the right person to set it up. Because it seems this would require more permissions (both on GitHub and DockerHub) than I currently have. I'm willing to do the work, but it might be easier for someone already having those permissions to set this up in a follow-up PR. @swaroopsarma @jafingerhut @fruffy What do you think?

Iiirc there is no formal process. The docker images are maintained on a voluntary basis. I believe @antoninbas has access to the p4lang Dockerhub.

@chrispsommers I've adopted the multi-stage builds technique from the examples you linked to. It helps reducing docker image size from 4.78GB to 1GB, and building time from 45min to 13min. Very impressive!
For the docker image CI setup, I still plan to do the work, but after second thoughts, I again propose to leave it to a follow-up PR, so we can leave this already-long PR more focused (and potentially easier to review). I can work on this in my fork without waiting for this PR to be merged. So that's not a blocker.
@swaroopsarma I've marked this PR as ready for review again. The main change needing your attention (based on your last review comments) is the diff in dal_init.c. I have come up with a new workaround, and would like to know if it is good enough this time.
@jafingerhut @fruffy The bmv2 ptf testing work you mentioned in another meeting sounds pretty relevant here. I'm interested to learn how it works if you can share a link, and see if I can help with integrating with it in a follow-up PR.
Thanks everyone!

@jafingerhut @fruffy The bmv2 ptf testing work you mentioned in another meeting sounds pretty relevant here. I'm interested to learn how it works if you can share a link, and see if I can help with integrating with it in a follow-up PR.

I believe Andy did a writeup here. The main advantage is the direct use of Protobuf messages. I do not think this setup requires the shell. Correct me if I am wrong, Andy.
I am also working with a student at NYU (@Hoooao) on an automated setup for p4c, which uses P4Testgen to generate PTF tests for the SoftNIC and runs them. We were thinking of using your automated workflow and then combining it with the PTF tests setup. But this is still work in progress.

I am also working with a student at NYU (@Hoooao) on an automated setup for p4c, which uses P4Testgen to generate PTF tests for the SoftNIC and runs them. We were thinking of using your automated workflow and then combining it with the PTF tests setup. But this is still work in progress.

@fruffy Interesting coincidence, I'd just messaged @jafingerhut about a proposal to include p4tools in all nightly p4c docker build/publish CI runs. It would make using p4testgen more convenient, just pull a pre-built docker. I'll file a PR shortly.

Thanks @fruffy for the information. That's great to know! This kind of test infra surely will be very useful. I look forward to you and @Hoooao's work. I'll read Andy's writeup for my own learning then.

@fruffy I've submitted PR https://github.com/p4lang/p4c/pull/4049 to incorporate p4tools (including p4testgen) into Dockerfile builds which will eliminate the need to build from scratch unless you're developing the backend itself. Consumers can just docker pull p4lang/p4c and use it. It needs a review.

I believe Andy did a writeup here. The main advantage is the direct use of Protobuf messages. I do not think this setup requires the shell. Correct me if I am wrong, Andy.

My PTF tests use a Python library called p4runtime-shell just out of personal choice: https://github.com/p4lang/p4runtime-shell
Any method of creating+sending, or receiving+parsing, the required P4Runtime Protobuf messages in Python will work.

@qobilidop  and @jafingerhut  can we merge this PR ?

I'm okay with merging it. And thanks for reviewing!

@swaroopsarma I have no objections to merging it.  Please do not count on me for a technical review of these changes -- that would be a job best requested of others who know this repo contents better than I do.
I am not sure if you were asking me to do perform the merge, but if you were, I do not have permissions on this repository to do that.


Validation of custom data directories for RKE2/K3s provisioning

https://github.com/rancher/rancher/issues/45038 introduces support for customizing data directories for provisioned RKE2/K3s clusters via spec fields like shown below:
spec:
...
  rkeConfig:
...
    dataDirectories:
      k8sDistro: /opt/rancher/rke2
      provisioning: /opt/rancher/provisioning
      systemAgent: /opt/rancher/agent
...

However, there is no (backend-side) validation for these paths making it possible for the users to misconfigure them and result in provisioning errors or unstable cluster down the line. While the exact rules may need to be further refined however below is the initial list:
• Absolute paths (start with /)
• Clean (not contain env vars, shell expressions, ., or ..)
• Not set to the same path (no duplicates are allowed between any pairs of paths out of 3 configurable)
• Not nested one within another
Note that any of these fields may be not specified or be specified as empty string in which case default value for each field is used. For the purpose of validations above these default values should be considered. For example, if provisioning K3s cluster /var/lib/rancher/k3s is the default distribution data directory so configuration where dataDirectories.provisioning is set to /var/lib/rancher/k3s should be rejected.
To Reproduce

Create an RKE2 or K3s cluster with data directory entries that match
Result
Invalid values are accepted
Expected Result

Invalid values are rejected
Screenshots

Additional context

Ticket #46381 - Test Results - ✅

Scenario
Test Case
Result

1.
Invalid entry test
PASS

Verified with HA Helm on Rancher v2.10-alpha2:
Scenario 1: Invalid entry test
Verify each of the following return scenarios return a webhook error:

All data/dirs have the same name
non-absolute paths i.e .. , ., ~, *, $envVar
1 data dir nested in another

Results: Webhook returns an error in all invalid cases


setup automod isnt a command now D:

Vortex setup automod command isnt working while running even bot has admin perms and have no role hierarchy
Image: https://cdn.discordapp.com/attachments/185037029384388608/847523317543469111/unknown.png

Correct, this feature has been removed to encourage servers to only enable features if they are knowledgeable about what the feature does and are intending to enable it. Similarly, there are no punishments that are automatically created; these must also be manually created.
The automoderation page on the wiki contains overviews of all automod features, as well as recommended values. Hopefully, this will improve the interactions that users have with automod, as fewer people will get confused by Vortex performing actions that they (indirectly) set up but didn't realize they did.


translation for pt_BR

All my considerations from #4331 were handled so I've no further points regarding this.

:+1:


Add RSSLink to SiteInfo

It would be helpful to have be able to link to the main site RSS feed from any page/template.  This commit solves that problem.

Agree.

Merged in 3abb267


Remove all inline style

This is ugly, plus it causes security issue (can not use CSP with no 'unsafe-inline'

[ ] error-center
[ ] popin
[ ] placeholder (might not be possible)
[ ] header


[go] JS error "Cannot read property 'align' of undefined"

The Go mode can get itself into a state where state.context is null/undefined:
SEVERE: (TypeError) : Cannot read property 'align' of undefined
at Unknown.token(...)
at Unknown.readToken(...)
at Unknown.processLine(...)
(File and line number data is useless due to Closure compiler stripping whitespace.)

I assume my patch fixes it. If you run into the issue again, please show me the code that triggers the problem.


Bug: (fixed in a15) ape 0.1.0a14 Exception: Not a valid ABI interface JSON file if a package.json exists in contracts folder

Environment information

ape and plugin versions: 0.1.0a14

$ ape --version
0.1.0a14

Python Version: Python 3.8.10
OS: win 10 wsl2

What went wrong?
tried to compile openzeppelin contracts folder with ape
error:
Exception: Not a valid ABI interface JSON file
How can it be fixed?
upgrade to ape 0.1.0a15
$ pip install --upgrade eth-ape

resolved by upgrading to 0.1.0a15


Module load slow

Hi,
I have a problem loading modules.
Indeed, after having analyzed my project, I remade that the last modules are more and more slow.
I am also attaching a result of my research.
One of my last modules is a support module, with a loading order (order to 50 in module.json).

And now, if I ask him to load first (order at 0 in module.json)

We see a clear difference in loading, knowing that no code has changed.
Do you have an idea about this problem?
Thank you in advance for your help
Laravel => 5.7
Laravel Module => ^4.0
Php => 7.2.6

Seems you have something that's slow in your app.
Laravel modules has been tested and used with a lot of modules (10+), and still be sub 200ms.

Hello!
This is a long shot but...
If you're using homestead with virtualbox, shared folders could be a major overhead. File synchronization and disk performance is just slow. Try using NFS if that's the case.
Regards!

Hi !
No, I'm local on a windows 10 and wamp
I would have to ask my colleague what she is using because she was the one who reported the problem to me with your much higher times
(about 7 seconds)

Other than that, blackfire.io may help you identify performance issues.
Regards!

Okay, I have found my problem !!
/**
     * Register the service provider.
     *
     * @return void
     */
    public function register()
    {
        $this->registerTranslations();
        $this->app->register(RouteServiceProvider::class);
    }

The registerTranslations function is my problem!
But my problem is that I need to add the translations because I translate my urls in the road
An idea of how I could achieve this?

Hi,
Thank you for your help
This problem is not from this package !
Sorry !

Would like to add I was seeing same issues with this package. When I was running a local linux environment with everything running locally, it was just fine (sub 70ms). I moved my project to a dockerized environment and saw 2-6 second load times... (fastest being 800ms). I went to a fresh laravel install and migrated my code for a specific part of the project to this and was down to 30-45 ms load times. Believe it has to do with dir / file scans depending on environment its used on has drastic differences.

Hi,
I have same slow down.
I create new Laravel app (ver. 9) with no code. Response time 50ms to 58ms. After that i install 35 modules with php artisan module:make ModuleNameHere and response time goes to 3.4 sec.
Your package is aweseme, but can you check why that issue occur?
Thank you in advance.

I seem to have the same thing, tracing shows a lot of time spent in this package during boot :(

I'm experiencing the same thing. It's booting all modules and taking longer time

We are working on a project in Laravel 9 with +50 modules and our Laravel app is taking time with each new module we create, do you have any advice? Currently our app takes more than 10 seconds to respond for each load.

In config/modules.php activate cache.
 'cache'      => [         'enabled'  => true,         'key'      => 'laravel-modules',         'lifetime' => config('default.app.cache.ttl_seconds'),     ],

In config/modules.php activate cache.
 'cache'=> ['enabled' => true, 'key'=> 'laravel-modules', 'lifetime' => config('default.app.cache.ttl_seconds')],

I have the cache activated
'cache' => [
'enabled' => true,
'key' => 'laravel-modules',
'lifetime' => 60,
],
The minimum time it takes to load a page is +8 seconds without counting the login page, but for example my login page loads in 6 seconds but if i change the status of all the modules to false in my modules_statuses.json file, i do a cache:clear and reload the same page, the time is reduced to 400 milliseconds.

In config/modules.php activate cache.
 'cache'=> ['enabled' => true, 'key'=> 'laravel-modules', 'lifetime' => config('default.app.cache.ttl_seconds')],

I have the cache activated
'cache' => [         'enabled' => true,         'key' => 'laravel-modules',         'lifetime' => 60,  ],
The minimum time it takes to load a page is +8 seconds without counting the login page, but for example my login page loads in 6 seconds but if i change the status of all the modules to false in my modules_statuses.json file, i do a cache:clear and reload the same page, the time is reduced to 400 milliseconds.

Then try to debug the boot method for your module.
For example if you have module "Icons" you can go to Modules\Icons\Providers\IconsServiceProvider and in boot method you can  comment each line to check the reduction of time. After that you can try to cache the static resources like translation files or migration files. In my case, translation files was the problem. It's not the best practice, but it works.

In config/modules.php activate cache.
 'cache'=> ['enabled' => true, 'key'=> 'laravel-modules', 'lifetime' => config('default.app.cache.ttl_seconds')],

It's been incredibly helpful!
Here's a brief update on my situation: With around 30 modules, the booting stage was initially taking 1.20 seconds per request, which was far from ideal.
I've managed to cut down the boot time to 700ms. It's still a bit longer than I'd like, but it's a significant improvement and manageable for now. I'll keep investigating to further optimize this. Your advice saved me a lot of time and pointed me in the right direction—much appreciated!

I have the same problem, it loads for 10 seconds with 50 modules enabled


Improvement/build

Normalize module vs type usages in fsx files
unit methods now all in Method() = form
cleaned up build.bat and its skiptests leniency, now only accepted as
the first argument
Improved usage notice in build.bat
Building now checks if dotnet.exe resolves to using the proper SDK
Moved to DotNetCli FAKE helper (#2395)
Separated Projects and Tooling out of Paths.fsx
Now have a truly quick compile target
build quick [test_filter] does an incremental build and runs the
unit tests from the bin output folder. it also skips paket.
moved global.json to the root of repos
unit tests now parallizes over all tests ignoring all clusters
rather then piggy backing on xunit which we do in 5.x (our own routine
is 10+seconds faster) or on the integration routine (which we
currently do in master)

@gmarz do you also get this after callinig build first? Had to upgrade to a later version of FAKE and test skips restore.

I did try calling build initially and was hitting the same error.  Just did a git clean -fxd and ran build again and now I'm hitting
build.bat: target="build" version= esversions= skiptests=0 apiKey="<empty>" feed="elasticsearch-net" escluster= testfilter=
The system cannot find the path specified.

Looks like this line is incorrectly setting SKIPPAKET

Sorry, it's not incorrectly setting SKIPPAKET.  It's just that we never run paket restore for the first time.  I'm not sure if SKIPPAKET is really useful, since bootstrapping is always required whether we're running "quick" or not.

Updated this PR with some clean checkout and mixing regular with quick builds on the command line resiliency changes.

LGTM!

ported to 2.x and master


Bulk profile

This is a proposal for bulk operations. Contrary to the JSON-patch approach where arbitrary operations can be sent to the API, this proposal limits itself to doing the same type of operation (eg PATCH) on the same API resource collection.
My approach for creating this profile was reading the specification for single operations and try to cover every restriction and concern raised there and adapt it to bulk operations.

Hey @kbairak thanks for creating this profile. I was wondering, how come there's no bulk fetch? Am I missing something?

Any idea on addressing asynchronous bulk operations? There is a recommendation for singular resources (https://jsonapi.org/recommendations/#asynchronous-processing), but it breaks down at the point of when the job has completed successfully, returning 303 See Other, as with bulk operations, there are n number of results.

@sebastienfilion

Hey @kbairak thanks for creating this profile. I was wondering, how come there's no bulk fetch? Am I missing something?

FETCH already works on collections. Even if none of the generic filters are good for your needs, the server could implement a filter[id]. Examples:
GET /articles?filter[id]=1&filter[id]=2&...`

or
GET /articles?filter[id][any]=1,2,...

@ckeboss

Any idea on addressing asynchronous bulk operations?

My initial approach was to support them. However, the specification would have to be complicated to support them. My wish was to copy as much of the original JSONAPI specification as possible, just operate on arrays of objects instead of objects. There was this trade-off between making the profile more capable or simpler. I went for the latter.
I suppose you can support asynchronous tasks on an application level, ie create an API resource that represents the bulk operation itself. Example (writing the payloads in yaml for simplicity):
# POST /article_uploads
data:
  - type: article_uploads
    attributes:
      title: title of article 1
  - type: article_uploads
    attributes:
      title: title of article 2

# Response: 201 Created
data:
  - type: article_uploads
    id: 1
    attributes:
      status: pending
  - type: article_uploads
    id: 2
    attributes:
      status: done
    relationships:
      article:
        data:
          type: articles
          id: 2
        links:
          related: /articles/2
included:
  - type: articles
    id: 2
    attributes:
      title: title of article 2
links: ...

In this case, the article_uploads API resources are created atomically and can be supported by this profile, even though the articles themselves are created asynchronously.

I understand why you excluded them from the profile.
I think bulk operations is one of the things that really lacks in the JSON:API spec. I believe that a robust spec for bulk operations is really needed to avoid "bikesheding" in regards to bulk operations.
In regards to the workaround you provided, it does work, but the disadvantage is that the client must know extra information.
For example, say you have a Article resource with a property of isPublished. When creating or updating that resource, changing any other property other than isPublished, the resource would respond with 200 or 204, but if isPublished is true, it must do some other, possibly longer runing modification, such as syndicate the resource to other resources, etc. So in that case, it would need to respond with 202.
Would you be interested in me proposing this further?

I understand why you excluded them from the profile.
I think bulk operations is one of the things that really lacks in the JSON:API spec. I believe that a robust spec for bulk operations is really needed to avoid "bikesheding" in regards to bulk operations.
In regards to the workaround you provided, it does work, but the disadvantage is that the client must know extra information.
For example, say you have a Article resource with a property of isPublished. When creating or updating that resource, changing any other property other than isPublished, the resource would respond with 200 or 204, but if isPublished is true, it must do some other, possibly longer runing modification, such as syndicate the resource to other resources, etc. So in that case, it would need to respond with 202.
Would you be interested in me proposing this further?

how about extend it as an extension "bulk-operations"/"bulk profile"? would love to know more about your idea whatever you feel would be better for handling bulk ops

Hey @jelhan . Thank you so much for your feedback. It's been a while since I worked on this but I will try to find some time to get caught up in how {json:api} deals with extensions/profiles and see if I can push this forwards.

I published a JSON:API bulk create extension here: https://github.com/jelhan/json-api-bulk-create-extension I consider it an early draft for now. Looking for feedback. Please use GitHub issues at that repository for feedback, questions and proposing changes.
I'm going to close this PR as I feel it is superseded by that extension. If someone wants to propose an alternative solution, please feel free to do so. I assume that we want for any extension and profile some experience from real-world usage before promoting it to an official extension or profile.


fix/standardize links to api-types pages

Signed-off-by: Steve Kriss<EMAIL_ADDRESS>Fixes some broken links and makes them consistent across all pages for ease of future copy/paste.

LGTM, and thanks!


Upgrade nightly version of hermit to nightly-2021-08-31

bors r+

bors abort

bors cancel

bors r+


iOS. Crash when exceeding some height value in vertically scrollable content of ModalBottomSheet

Describe the bug
If ModalBottomSheet has scrollable content and is shown on the screen, the following error occurs. Whether the scrolling started or not, the crash is happening on first composition of the composable. Important: ModalBottomSheet's content height must be a large number. Hard to tell the exact value.
-[MTLTextureDescriptorInternal validateWithDevice:]:1357: failed assertion `Texture Descriptor Validation
MTLTextureDescriptor has height (11700) greater than the maximum allowed size of 8192.

Affected platforms

iOS

Versions

Libraries:

Compose Multiplatform version: 1.6.10

Kotlin version: 2.0.0
OS version(s) (required for Desktop and iOS issues): 17.4
OS architecture (x86 or arm64): arm64
Device (model or simulator for iOS issues): iPhone 15 Pro iOS 17.4 Simulator (happens on other versions as well)

To Reproduce
Steps to reproduce the behavior:

Run this code snippet:

ModalBottomSheet(
    onDismissRequest = { },
    content = {
        Column(
            modifier = Modifier
                .fillMaxSize()
                .verticalScroll(rememberScrollState())
        ) {
            Box(
                modifier = Modifier
                .fillMaxWidth().height(1000.dp)
                .background(Color.Black)
            )
            Box(
                modifier = Modifier
                    .fillMaxWidth().height(1000.dp)
                    .background(Color.Red)
            )
            Box(
                modifier = Modifier
                    .fillMaxWidth().height(1000.dp)
                    .background(Color.Green)
            )
        }
    },
    sheetState = rememberModalBottomSheetState(skipPartiallyExpanded = true)
)

See error

Expected behavior
No crash should happen.

Should be fixed by https://github.com/JetBrains/compose-multiplatform-core/pull/1385

Please check the following ticket on YouTrack for follow-ups to this issue. GitHub issues will be closed in the coming weeks.


Feat/gcp kms support

Implementation of https://github.com/lukasjarosch/skipper/issues/46
Adds the GCP KMS Secrets Driver. To use it add following to your <target>.skipper.secrets.drivers target configuration:
gcpkms:
  key_name: "projects/<project>/locations/<location>/keyRings/<keyring>/cryptoKeys/<keyname>"

The key name represents the full key URI. The naming scheme might be confusing but is consistent with the SDK. Both the URI and the key name are called name. I did not (yet) implement the option to provide project, location, key ring, etc. as dedicated arguments because as opposed to the Azure SDK, the GCP SDK wants to have the URI for the request. Let me know if you think the single arguments implementation should be included as well.
Open issues
I'm still struggling with key versions. With Cloud KMS, rotating a key does not disable older versions and you can still decrypt secrets with the older version without actually needing to specify the version, even if you changed the primary (default) version. Somehow that just works until you disable the old version.
In theory you should be able to specify the key version by appending it to the URI: projects/<project>/locations/<location>/keyRings/<keyring>/cryptoKeys/<keyname>/cryptoKeyVersions/<version>. This is the URI for key versions, and supported within the SDK (as you can see here), but it the API throws me an error and I wasn't successful fixing this yet:
failed to replace secret value: failed to decrypt input: rpc error: code = InvalidArgument desc = projects/tmp-joe-skipper-test/locations/europe-west3/keyRings/test/cryptoKeys/quickstart/cryptoKeyVersions/2 is not a valid resource type for this request.

Testing instructions

Create a new key ring (equivalent to Azure Key Vault, but only for keys) and a new key:

gcloud kms keyrings create "myring" --location "europe-west3"
gcloud kms keys create "mykey" --location "europe-west3" --keyring "myring" --purpose "encryption"

Encrypt a secret:

printf "supersecret" | gcloud kms encrypt --key "mykey" --keyring "myring" --location "europe-west3" --plaintext-file=- --ciphertext-file=- | base64

Place that output as secret in your skipper inventory secret path:

data: <encryption output>
type: gcpkms

Configure the GCP KMS Secret Driver in your target as stated above.

Make sure skipper can authenticate itself against Google Cloud by either pointing to service account credentials with theGOOGLE_APPLICATION_CREDENTIALS environment variable or running gcloud auth application-default login to let skipper authenticate itself as you. As this is a test environment, just be the owner of the project to not worry about permissions.

Run skipper.

This is the URI for key versions, and supported within the SDK, but it the API throws me an error

So I figured this one out. The decrypt method does actually not support specification of a key version, as you can see here.

The resource name of the CryptoKey to use for decryption. The server will choose the appropriate version.

This means we will need to omit the key version - if specified - for the decrypt request from the key name.


MAINT Make long/short positions match gross leverage

This makes pos.get_long_short_pos return a dataframe
that matches the gross leverage of the portfolio.
I removed the gross_lev argument since the leverage is
dependent on positions dataframe.
This PR is addressing https://github.com/quantopian/pyfolio/issues/30

Am I correct in assuming that this does not fix the issue with there still being a cash allocation incorrectly inferred when there is shorting?

Otherwise looks great! Thanks for the quick turn-around.

Yes, I do think you are correct, the problem is that normalizing the data throws everything out of whack when shorting is involved. We can't use the absolute value of the short positions and still have it make sense.
Consider a portfolio with 1x leverage that is equally long and short. They sell 50% of the account value and get the cash proceeds, then buy assets with the other 50%. That means the account has 100% of the account value sitting in cash still, but some of it is collateral for the short sales. That cash earns interest and is an important component of a long/short strategy, especially in a high interest environment.
I we lose that when normalizing the long/short/cash components because they should equal 200% of the account value if using absolute values. 100% in cash, then 50/50 in long/short stocks.
Having this return the dataframe defined on line 59 before the normalizing would be more correct IMO. I don't think the area plot is the best thing for this either.
Here are a few plots of a long/short strategy without normalizing the data, the area plot is not very nice, but the density plot, box plot, and histogram capture the distribution of allocations nicely.

This area plot is not so intuitive as to what I'm looking at IMO.

A density plot captures the allocation distribution well

Histograms work too, but it's a bit more cluttered.

Box plots are not that pretty, but the information is a least meaningful.

@humdings thanks for that in depth overview. I really do like that boxplot version at the end but I agree it's a bit confusing and overkill for this visualization usecase.
I think this is a case where the most important takeaway that we are trying to convey is literally the long exposure , short exposure, and "left over cash" (cash excluding proceeds from shorting).
I agree the interest earned on cash vs cash held as collateral is important if rates are high, but I think we can exclude that nuanced level of detail here.  I totally think we should eventually have an entire "Margin profile" section in the tearsheet down the road where this additional breakdown would be awesome, especially the additional p/l earned every month for interest on cash, short interest paid of hard to borrow stocks, % of cash held as collateral, % of total margin used, etc--all as timeseries showing how they change over the lifetime of the algo's history
how would you feel with this proposed approach?
Sent from iPhone
On Sun, Sep 13, 2015 at 10:08 AM -0700, "David Edwards"<EMAIL_ADDRESS>wrote:
Yes, I do think you are correct, the problem is that normalizing the data throws everything out of whack when shorting is involved. We can't use the absolute value of the short positions and still have it make sense.
Consider a portfolio with 1x leverage that is equally long and short. They sell 50% of the account value and get the cash proceeds, then buy assets with the other 50%. That means the account has 100% of the account value sitting in cash still, but some of it is collateral for the short sales. That cash earns interest and is an important component of a long/short strategy, especially in a high interest environment.
I we lose that when normalizing the long/short/cash components because they should equal 200% of the account value if using absolute values. 100% in cash, then 50/50 in long/short stocks.
Having this return the dataframe defined on line 59 before the normalizing would be more correct IMO. I don't think the area plot is the best thing for this either.
Here are a few plots of a long/short strategy without normalizing the data, the area plot is not very nice, but the density plot, box plot, and histogram capture the distribution of allocations nicely.
This area plot is not so intuitive as to what I'm looking at IMO.
A density plot captures the allocation distribution well
Histograms work too, but it's a bit more cluttered.
Box plots are not that pretty, but the information is a least meaningful.
—
Reply to this email directly or view it on GitHub.

I doesn't feel right to me to mess with the cash level honestly. In this particular case if we want the area plot to match up with the gross leverage, we want to exclude the cash altogether. The cash can be inferred from the long/short allocations.
Starting with a positions dataframe here is a script that demos what I mean.
nocash = positions.drop('cash', axis=1)
longs = nocash[nocash > 0].sum(axis=1)
shorts = nocash[nocash < 0].abs().sum(axis=1)
cash = positions.cash

leverage = (longs + shorts) / (longs - shorts + cash)
net_liquidation = longs - shorts + cash

allocations = pd.DataFrame({'long': longs,
                            'short': shorts}) / net_liquidation

print np.allclose(allocations.sum(axis=1) - leverage, 0.0)

# Produces True

So the gross leverage matches up when the cash position is excluded. If we want to compare long vs. short allocations I'd say we don't worry about the cash component since it can be solved for if the leverage and long/short allocations are known.

@humdings Thanks for the detailed answer. Your last suggestion makes a lot of sense to me. Exclude the cash as it can be inferred from the leverage and have this analysis focus only on long vs. short exposure.
In addition, how do you feel about dropping the leverage plot which now is redundant with this one?

Also interested in what @justinlent thinks about dropping cash.

+1 @humdings and Thomas. remove cash from the exposure plot since it can be inferred, and yep we can kill off the current Leverage plot in favor of this new Gross Exposure plot
Sent from iPhone
On Mon, Sep 14, 2015 at 2:43 AM -0700, "Thomas Wiecki"<EMAIL_ADDRESS>wrote:
Also interested in what @justinlent thinks about dropping cash.
—
Reply to this email directly or view it on GitHub.

That all sounds good to me, I'll update get_long_short_pos to drop the cash component.

Running the zipline example NB I get this behavior:

Strange, I'll look into it now

That is odd because it seems to work fine for me, it looks like the long portion is not plotting for you.

Something seems to have gone wrong with the uploading of the image.

@humdings can you run the NBs and commit the updated ones?

@humdings Can you force pull? I resolved the conflicts. But I still get that issue of an empty plot. Perhaps it's a python 3 issue?

What exactly do you need me to do? Where am I pulling from and to where? I'm not sure what commands to run.

git checkout long_short_cash_investigation
git pull -f origin long_short_cash_investigation

I'm still getting conflicts, I'll just scrap the repo and clone again

I'm all up to date but the notebook is barfing now. I get this error when opening the notebook
Error loading notebook
Unreadable Notebook: /Users/dedwards/repos/pyfolio/pyfolio/examples/zipline_algo_example.ipynb NotJSONError('Notebook does not appear to be JSON: u\'{\\n "cells": [\\n {\\n "cell_type": "...',)

try git checkout master -- /Users/dedwards/repos/pyfolio/pyfolio/examples/zipline_algo_example.ipynb to restore the ipynb.

Closing in favor of https://github.com/quantopian/pyfolio/pull/147


feat: Allow to pass number of expected batches or epochs to learning rate scheduler

In this PR,

we allow the user to configure whether to step the LR scheduler every batch or epoch
introduce two constants MODYN_NUM_EPOCHS and MODYN_NUM_BATCHES which -- hopefully, guarded by assertions -- describe how many epochs and batches we will train on that can be used in the learning rate scheduler configuration instead of giving a number.

Solves #163.

haha you've reviewed before it's fully done but it should be done soon :D thank you anyways, will address your comments!

@robinholzi I just re-requested a review since now the implementation is complete and I tested it on some variations of the MNIST pipeline.  I hope it will not break in other configs with some wrong calculations :) Anyways, we only fail in case the number of batches is truly relevant (when using a LR scheduler), in other case we just warn.


Compilation error with vaadin 8.6.0

I am getting compilation error when compiling with vaadin 8.6.0. Everything works correctly with 8.5.2.
The error:
[INFO] --- vaadin-maven-plugin:8.6.0:compile (default) @ lsps-vaadin-widgets ---
[INFO] Compiling module com.whitestein.lsps.vaadin.widgets.WidgetSet
[INFO]    Tracing compile failure path for type 'com.vaadin.contextmenu.client.MyVOverlay'
[INFO]       [ERROR] Errors in 'jar:file:/C:/Users/mna/.m2/repository/com/vaadin/vaadin-context-menu/2.1.0/vaadin-context-menu-2.1.0.jar!/com/vaadin/contextmenu/client/MyVOverlay.java'
[INFO]          [ERROR] Line 22: This static method cannot hide the instance method from VOverlay
[INFO]    [ERROR] Aborting compile due to errors in some input files

Culprit is following method in MyVOverlay.java
    public static void setApplicationConnection(ApplicationConnection ac) {
        // this.ac = ac;
        ac_static = ac;
    }

Static method with the same name appeared in main framework in VOverlay with this commit: https://github.com/vaadin/framework/commit/2f34a006ed3d98acd79c48c1a2d8fc680188053a

Looks like you are using v2.1.0 of the ContextMenu add-on. Have you tried v3.0.0.beta1. There has been major update with many bug fixes there. The code has been refactored and in new version the class that is failing in your case is actually removed.

Hello. Vaadin Framework 8.6 requires 3.0 beta1 or newer from context-menu. You should use our BOM to automatically get compatible version of framework and our add-ons to it
https://github.com/vaadin/framework/blob/8.6/bom/pom.xml#L23
There should not be any changes, as stated in the release notes https://github.com/vaadin/context-menu/releases/tag/3.0.0.beta1. But you cannot use 2.1 since it is not compatible.
Closing as invalid.

Thank you for the answer. When do you plan stable release? I don't really want to use beta in production.

We will take a look at #96 which might be a regression in 3.0, and then release 3.0.0 this or next week.

Thank you, that is great. Btw, I can wait two weeks. I just needed to know whether it will be sometime soon which it is.


Hotfix/deps

Hi @TriPSs
I appreciate the initiative to make this project active, I think it is a library that has a lot of interest to exist.
I would like to participate in the development.
At the moment, I am trying to understand the structure of the project as well as possible.
In my opinion, the dependencies of each library such as query-graphql, query-typeorm, etc. should be moved to the respective package.json.
If it's right, I will continue moving the dependencies to packages/query-*/package.json
This PR is about moving dependencies of each package and upgrading them because it's not possible to install the library with the latest Nestjs packages (typeorm >= 0.3 / @nestjs/graphql >= 10)

Thanks! Not sure what's happening:

Yes, with cached modules:
➤ YN0000: Done with warnings in 2s 351ms
With clean install:
➤ YN0000: Completed in 49s 211ms
➤ YN0000: Done with warnings in 50s 735ms

Note: I refreshed yarn's lock file. Not sure if this can help.

I think that this is happening because I'm missing the "Maintainer" role in order to trigger these Github actions.
I tried to update all dependencies at once but it's harder than expected with the split of nestjs/graphql and nestjs/apollo.
I'm still on it :-)

Closing this one in favor of #26


Fix script for GA tag

What's the nature of the fix?

@fchollet Per Andrew, an incorrect script was deployed for Keras. We should not be deploying GA4 properties directly. GTM must be deployed.
This is mostly reverting to what it was after you made this change: https://github.com/keras-team/keras-io/commit/5075dc6b56702190cab54b454c1d7f5ea41ca877
I don't have context on this though, for instance I don't know why it was changed here: https://github.com/keras-team/keras-io/commit/483ce32cded4be06855404b1747fc702c4fa31c7


[$500] Chat - Offline changes made in the deleted workspace are saved when returning online

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: <IP_ADDRESS>
Reproducible in staging?: y
Reproducible in production?: y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:
Action Performed:
Precondition:

Same account is logged in on two devices.
There is a workspace in the account.

[Device A] Go offline.
[Device A] Send a few messages to the workspace chat.
[Device B] Delete the workspace.
[Device A] Go online.

Expected Result:
The offline changes made in the workspace will not be saved.
The changes made in offline will have a red dot next to them with a message below saying ""This chat room has been deleted"".
Actual Result:
The messages are still sent despite the workspace is deleted.
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

Screenshots/Videos
Add any screenshot/video evidence
https://github.com/Expensify/App/assets/93399543/fdd2a1bb-0426-4153-9c76-85c83e3afc6c
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01479c44876cd0c3db
Upwork Job ID:<PHONE_NUMBER>000124416
Last Price Increase: 2023-12-02

I think this should be fixed in the backend.
From what I can see from the frontend codebase, when a Workspace (Report) is deleted/archived, it simply changes its Status to 2. Doing this will hide the interface for sending, editing, and deleting messages (ie. create more ReportActions). But I'm guessing there's no check on the status in the backend.
Looking at the Report type, it doesn't seem to log when its status changes. If it did, then we could filter out ReportActions that was created after the Report status was changed to 2.

This looks like a backend issue. Let's get an engineer for this.
🎀 👀 🎀 C+ reviewed

This looks like a backend issue. Let's get an engineer for this. 🎀 👀 🎀 C+ reviewed

@MelvinBot : please point me to the backend repo.

Thanks for the heads up @shubham1206agra - I'll move this internal now.

On pause until this is picked up internally.

Going to un-assign myself from this as I already have other wave/vip work assigned.

Moving this to weekly since it's Low priority. Please feel free to update labels as needed when priority changes.

Moved to Medium priority but no other update at this time.

Back to Low so I'll update the title.

No update yet.

Low priority for the time being, no update.

Low priority - I'm changing this to weekly so I don't have to update it every day.

Low priority, waiting for pickup once higher priority items are finished or someone has a proposal.

Great find.  I'm reassigning this to Wave5 as this relates to workspace chats.

That PR is going to solve this already, so I'm going to take this over so we don't duplicate effort. I let @rlinoz know via DM too.

Moving to weekly since it's on hold

The backend fix was deployed, and it looks like it's working well. The error messages aren't great, but this is an edge case, so it's fine.
https://github.com/Expensify/App/assets/26260477/241d97d4-7687-416a-9c4b-02cdbc89f97a


MongoDB based EnvironmentRepository implementation.

Hi,
I've implemented a basic MongoDB based EnvironmentRepository (Issue #203). I've also added a simple test.
Configuration Properties
spring.cloud.config.server.mongodb.host - The mongodb host (optional) (default <IP_ADDRESS>)
spring.cloud.config.server.mongodb.port - The mongodb port (optional) (default 27017)
spring.cloud.config.server.mongodb.database - The mongodb database
spring.cloud.config.server.mongodb.username - The mongodb username (optional)
spring.cloud.config.server.mongodb.password - The mongodb password (optional)
Enabling MongoEnvironmentRepository
The spring.profiles.active must be set to 'mongodb' and at least the spring.cloud.config.server.mongodb.database property must be configured to enable this repository.
Due to some issues I had to close my previous pull requests. Apologies.
Please review and pull.
Thanks,
Venil

Travis is ignoring my MongoEnvironmentRepositoryIntegrationTests class. Its also showing the following output.

Warning: you are leaving 2 commits behind, not connected to any of your branches:
479c869 Merge 811a58addc1a5adf369c7fc621887742da42f0ad into 1c69c59705e4e2e7bf22d511e0b777b75f094c42
811a58a MongoDB based EnvironmentRepository implementation. #203.
If you want to keep them by creating a new branch, this may be a good time to do so with:
git branch new_branch_name 479c869

Any idea why?

Why would we need spring.cloud.* configuration for Mongo (it's already provided by Spring Boot)?

If at all one needs to use a different mongo instance for just storing configs.

Sorry, I didn't catch that. Why would you need a cloud-specific config?

I mean, if the config server is to be clubbed with a component already working on a mongo instance, then the spring.cloud.config.server.mongodb.* properties would be useful to configure a separate mongo instance for only serving configuration. (This mongo instance may as well be deployed on another server with a different IP.)
If this usecase doesn't seem to be valid then I'll remove those properties and use the spring-boot mongo instance instead.

I don't think it's a common enough use case that we'd want to support it out of the box with configuration properties. If you can just use the default autowired Mongo client that would help I think. Also, did you fill in the contributor's agreement (link in README)?

OK. I'll remove those properties. And yes, I've signed the ICLA. Thanks.

I've made the necessary changes to use the spring-boot provided MongoTemplate instance in MongoEnvironmentRepository. Could you please verify and pull?

@spencergibb done.

Hey Venil,
We make us of the MongoDBEnvironmentRepository implementation and unfortunately we've found out an issue when it retrieves a 'complex' copnfiguration  (see below the example) which is stored in MongoDB. This problem occurs when we want to obtain the configuration as a property invoking the REST API.
Below an example of this scenario:

Mongo and spring cloud configuration running locally on the default ports.
Configuration stored in mongodb.
'''
....
properties : {
"foo" : {
"bar": "fooBar"
}
}
...
'''

Then when invoking the following url: http://localhost:8888/master/sample-default.properties

And then the result is the below
properties: {foo={bar=fooBar}}
but we expected a response like the below:
proterties : foo.bar = fooBar

The reason We expected the aboveis  because when we invoke the yaml endpoint the response looks like this:
	foo:
		bar: fooBar

We have been working on fixing this issue and it would be greatful If we could contribute to this project with the fix, would you mind If we do a pull request to your repository?  If so, we would like also to make some improvements in the tests implementation by using Fongo  instead of assuming you have a mongo instance running on localhost
foo.bar: fooBar
Thanks in advance,
Cheers,

Sure @DCalata, I'll have a look at it. You can send a PR to my repository as well.

@DCalata thanks for highlighting the issue. I've fixed it in 71aee7a.

Hi @venilnoronha,
I'm a partner of @DCalata. I've been testing last changes introduced following their comments.
I found an issue due cleanSource method introduced in b7aa7cb. I make a test in a forked repository that reproduces the issue.
In order to follow the behaviour of the other profiles (native, git & subversion), I introduce this changes in the MongoEnvironmentRepository to allow the definition of any key in the name of the properties. Can you check the changes, please? What do you think about it?
If you consider appropriate, I can make a PR with these changes to your repository so you can amend it to your PR. Thanks.

Hi @javiboo,
While I see that the label and profile values are being lost, I'm still wondering what would the use case be for needing those values. And if someone needs them, would there be 2 different profile/labels values per configuration? If not, then we can update cleanSource to retain those properties, which would then be returned in the API response.
My concern is that, having 2 label/profile values might cause ambiguity to the user. Also, would it not be nice to avoid nested properties at the first level itself?
Thoughts?

Hi @venilnoronha,
I agree with you and I don't think it's a good idea to have 2 different profile/labels values per configuration.
By defining a property named label, I'm trying to demonstrate the differences between mongo-db profile and the other profiles.
Suppose we have a configuration file foo.yml stored in our fs (with native profile active) for an application with name foo with the content:
property1: value1
label: value2

so if you request your configuration through the endpoint /{name}/{profiles} (with values foo and default, respectively) you obtain the response:
{
  "name": "foo",
  "profiles": [
    "default"
  ],
  "label": null,
  "version": null,
  "propertySources": [
    {
      "name": "classpath:/foo.yml",
      "source": {
        "property1": "value1",
        "label": "value2"
      }
    }
  ]
}

With mongo-db profile, how can you address the same content without nesting properties under a field and also without restricting the use of certain properties in the configuration file (considering that here the property label is also being used as a configuration property, a part of the meaning of this field in Spring Cloud)?
What do you think about it?
Thanks!

Sure @javiboo, I get your point. We can add a source field of type LinkedHashMap<String, Object> to MongoPropertySource instead of properties (just to be consistent with the API response, see below).
"propertySources": [
  {
    "name": "classpath:/foo.yml",
    "source": {
      "property1": "value1",
      "label": "value2"
    }
  }
]

Regarding the PR, if you'd like to contribute to my repo I'm open for it. However, I'm not quite sure if you might get the attribution for it (as all of the commits of this PR would be squashed). Again, if you'd want to be sure about the attribution, it would be better if you could send in the PR to spring-cloud/spring-cloud-config directly once this PR is merged. I'm also fine with making the changes you've suggested to my repo directly, based on your fork of my repo (if you don't mind).
Thanks,
Venil

@venilnoronha I don't mind, you can make the changes suggested directly to your repo. Thanks!

@javiboo I've updated the implementation. :)

@venilnoronha thanks for the quick response. We are very interested that this PR joins the spring-cloud project, we can do something to expedite this issue?

@javiboo I'm not too sure about that. Maybe @spencergibb could tell if there are any immediate plans to merge this PR.

There are not immediate plans. We'll have a discussion about it as a team.

@spencergibb sure, thanks!

@venilnoronha what do you think about hosting this in your user org, or maybe in https://github.com/spring-cloud-incubator. That way we can see how users respond and gauge interest before we decide to move this into core.

How do we go about hosting this under spring-cloud-incubator?

@venilnoronha https://github.com/spring-cloud-incubator/spring-cloud-config-server-mongodb, invite sent.

Thanks @spencergibb. I'll commit the code there soon.

Snapshot version of spring-cloud-config-server-mongodb is now available on jfrog oss. Check out the github repository here for information on maven configuration and usage.

Closing in favor of https://github.com/spring-cloud-incubator/spring-cloud-config-server-mongodb/commit/ecedae00e22978155606cf17ca1f2c3279769aff


[css] propose ids used in other selectors

Issue Type
Bug
Description
i have issues when i try to use suggestion for css files. for classes works well but when i'm  using for id i have no suggestions try it with a file with like 500 lines of code
VS Code Info
VS Code version: Code 1.20.0 (c63189deaa8e620f650cc28792b8f5f3363f2c5b, 2018-02-07T17:09:39.780Z)
OS version: Windows_NT x64 10.0.16299

System Info

Item
Value

CPUs
Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz (4 x 2195)

Memory (System)
3.91GB (0.52GB free)

Process Argv
C:\Program Files\Microsoft VS Code\Code.exe

Screen Reader
no

VM
0%

Extensions (5)

Extension
Author (truncated)
Version

rainbow-brackets
2gu
0.0.6

vscode-eslint
dba
1.4.5

vsc-material-theme
Equ
1.3.0

prettier-vscode
esb
1.1.3

LiveServer
rit
3.2.0

Reproduces without extensions

Thanks for the issue. I update https://github.com/Microsoft/vscode/issues/28442 to also include ids.
Marking as dup.

@aeschli If I understood correctly this issue is unrelated to HTML.
It's about autocomplete of #id in CSS files which is now working fine for .class.

Oh, I assumed this is about completing id selectors in CSS. The list of suggestion would require looking at HTML files to see what ids are used there.
@Skeletoo Please correct me if I got this wrong.

@aeschli  if i have an #id created in the CSS file  and i want to use it again(in the same file) i have no suggestion at all, for .classes works well

@usernamehw in HTML files i'm not getting suggestions at all, i'm not sure if that suppose to happen or not

No, in HTML there is no suggestions by default. Probably can be done with extensions, like: https://marketplace.visualstudio.com/items?itemName=Zignd.html-css-class-completion#overview

@Skeletoo Thanks I updated the title and un-duplicated...

I too have a problem with this. Already declared, selected id's in the same CSS file is not suggesting while creating new rules. I'm expecting the same behavior as the class name suggestion which is very convenient.
The same thing happens to custom HTML element /tags.


[SPARK-23014][SS] Fully remove V1 memory sink.

What changes were proposed in this pull request?
Fully remove the V1 memory sink, now that all kinds of execution support the V2 version.
How was this patch tested?
existing unit tests

Test build #87271 has started for PR 20351 at commit efcf03d.

Test build #87271 has finished for PR 20351 at commit efcf03d.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class MemoryStreamDataReaderFactory(records: Array[UnsafeRow])

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87271/
Test FAILed.

It says "fails PySpark unit tests", but the result page says "0 failures (±0)". I don't know if there's anything I can do to investigate this.

retest this please

Test build #87406 has started for PR 20351 at commit efcf03d.

Test build #4097 has started for PR 20351 at commit efcf03d.

Test build #4098 has started for PR 20351 at commit efcf03d.

Test build #87410 has started for PR 20351 at commit 6059a7f.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87406/
Test FAILed.

Test build #4097 has finished for PR 20351 at commit efcf03d.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class MemoryStreamDataReaderFactory(records: Array[UnsafeRow])

Test build #4098 has finished for PR 20351 at commit efcf03d.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class MemoryStreamDataReaderFactory(records: Array[UnsafeRow])

Test build #87410 has finished for PR 20351 at commit 6059a7f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87410/
Test FAILed.

Can one of the admins verify this patch?


updateConsent method cause crash on iOS

I am getting this error when I call Appodeal.updateConsent(null).
stack_appodeal_flutter/SwiftAppodealFlutterPlugin.swift:87: Fatal error: Unexpectedly found nil while unwrapping an Optional value.

@mgazisalik Hello!
We have released a new version of the SDK in which we have changed update consent logic. Please, try our new version!
If you have any questions/issues I will be glad to answer.
Have a good day.


Version URLs in the bottom left menu are incorrect.

Details
Hello,  I've been creating a local install and have pretty much everything working.  ReadTheDocs really is fantastic
I've just noticed however that after creating a new tag, activating and building a tagged version of my docs that the link to it in the menu at the bottom left is incorrect.
The project documentation is now at:

http://docs.mydomain.com/docs/my_project/en/latest/  and
http://docs.mydomain.com/docs/my_project/en/<IP_ADDRESS>/

Unfortunately the ReadTheDocs manu at the bottom left lists these URLs to the available versions:

http://docs.mydomain.com/en/latest/  and
http://docs.mydomain.com/en/<IP_ADDRESS>/

in both cases it seems to be missing : docs/my_project/ in the URL so if I click on either URL I get the 'Page doe not exist yet' page.
Have I mis-configured something?
If this issue is for a specific project or build, include the following:

Build #: I pulled last week but I cannot see a build number noted anywhere.  How do I find out what build I am using?

I should probably mention that I am building with recommonmark.
Many Thanks
Chris

Thx @dmartin35, your fix works for me.
I don't know if it will work with subdomain as i don't use it.

@agjohnson Your revert on commit 602f08c858e831349eb665d3c56e3b035b0bcb89 caused this bug to come back. W/o having version.get_absolute_url here, non-subdomain project do not work, because their project path is not in the hardcoding of the versions url when using:
    ("{{ version.slug }}", "/{{ version.project.language }}/{{ version.slug}}/"),{% end

As stated in the revert, that change broke production. Feel free to submit a patch that adds this output to the API endpoint.

Is there an issue that shows the error this caused? It seems like using get_absolute_url is the better choice here, vs. making the URL anew here. It's also done this way in other parts of the codebase:
$ grep -Rl 'version.get_absolute_url' readthedocs
readthedocs/doc_builder/templates/doc_builder/conf.py.tmpl
readthedocs/projects/forms.py
readthedocs/templates/core/project_details.html
readthedocs/templates/core/project_list_detailed.html
readthedocs/templates/core/project_list_featured.html
readthedocs/templates/core/version_list.html
readthedocs/templates/projects/project_version_list.html

No issue for this. The revert has information on why this change was reverted in the first place. Our build servers don't have access to a database, so we can't rely on these methods to query.

We encounter this in our local install as well. What about checking if the build is done on a local install or on readthedocs.org like so:
{% if settings.PRODUCTION_DOMAIN == 'readthedocs.org' %}
'versions': [{% for version in versions %}
    ("{{ version.slug }}", "/{{ version.project.language }}/{{ version.slug}}/"),{% endfor %}
    ],
{% else %}
'versions': [{% for version in versions %}
    ("{{ version.slug }}", "{{ version.get_absolute_url }}"),{% endfor %}
    ],
{% endif %}

Note: I have not tested this code
It may make sense to abstract the if settings.PRODUCTION_DOMAIN == 'readthedocs.org' into some global variable such as IS_LOCAL_INSTALL.

I've put my suggested changes in PR #3023, but with a more explicit check if we're running on RTD servers or a custom install (using a config option).
Unfortunately, Travis is failing with bower unavailable (#3019) so it can't be merged yet.


net6.0-ios migration

Hello,
The existing version won't work as is with net6.0-ios due to breaking changes. These are also documented here.
Right now you get the following error - error NU1202: Package SaturdayMP.XPlugins.iOS.BEMCheckBox 1.4.3 is not compatible with net6.0-ios15.2 (.NETCoreApp,Version=v6.0). Package SaturdayMP.XPlugins.iOS.BEMCheckBox 1.4.3 supports: xamarinios10 (Xamarin.iOS,Version=v1.0)
Do you have any plans to support net6.0-ios since the GA is scheduled for Q2 2022?
The samples from https://github.com/xamarin/xamarin-macios/tree/main/tests/dotnet might be helpful.
Thank you,
Cosmin

Hello @cosminstirbu.  Sorry I currently don't have time to to work the XPlugins.iOS.BEMCheckBox but I am happy to work with you, or others, to a PR to make XPlugins.iOS.BEMCheckBox work with .NET 6.  Please feel free to submit a PR and/or ask me any questions about this project.


Android13 ”permission denied“ when saving to local storage in another user

Describe the bug
Send files or photos to Android. If the non-administrator account is currently in multi-user mode, an error will occur when saving to local storage. Users who use Admin will be saved successfully.
To reproduce

Use a Pixel mobile phone

Open settings, system, multi-user, add a new user

Switch to new user

Transfer files or photos from other devices to Pixel

Expected behavior
The file is successfully saved to local storage.
Screenshots

Manually select the save location, indicating that the location cannot be written:

Desktop (please complete the following information)
No response
Smartphone (please complete the following information)
Android 13
Pixel phone
Additional context
No response

@Tienisto already fixed: https://github.com/localsend/localsend/commit/75f1817c34bc5d5b82a84c56424ef2ad26e35377 https://github.com/localsend/localsend/pull/950


added fitness-web-app

@vivekrawat21 ,
Please add projects created using HTML+CSS/SASS+JS only.
And follow the guideline present in :  https://github.com/dev-AshishRanjan/Hacktoberfest-frontend/blob/dev/CONTRIBUTING.md
You can create new PR for new projects 👍🏽


Dock component misses the events properties

Hi @dillionverma ,
I started using the Dock component and when I wanted to attach onClick event, I found that it misses this in its properties.
##Screenshot

Current
Type '{ children: Element; className: string; onClick: () => void; }' is not assignable to type 'IntrinsicAttributes & DockIconProps'.
  Property 'onClick' does not exist on type 'IntrinsicAttributes & DockIconProps'.ts(2322)

Expected
Possibly to add events to the Dock component.

thanks!!!
just fixed in latest pr


Update AchievementHandler.cs according to data Gibbed collected

Pull request
Updated as I found out that for example Horizon Forbidden West achievements have permissions, but they are not set and also updated stats to match one that Gibbed collected in SAM I checked for some appIds that I know that work and those that do not and works as expected.

Thank you for contribution, sorry for being a slowpoke with accepting it.


Adding ICacheClient.GetOrAddAsync

Was looking at other In-Memory caching solutions and found a GetOrAdd was useful.  Ultimately I could of just made this into an extension method to use privately but figured I'd throw it in here if you thought it would be worth adding.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

My vote is to keep the interface clean and make this into an extension method. If you want to create an extension method for this and keep your unit test I'd be more than happy to accept this.

I'm going to close this, would you please add an extension method for this instead.


Node6: Remove parentheses around the body of async arrow functions

Single line arrow functions can have parenthesis wrapping their body, but multi-line arrow functions cannot.

In the browser.pages patch.


intl for localization

I was thinking of switching the translation over to the intl package as it is the recommended way for flutter apps.
I've worked with it a many times and I quite like the code generation and the safety that all invocations will work.
It also has good pluralization support that the current flutter_translate package also recently added and we maybe can't use because we sit on an older downstream branch.
Sadly I have no idea how transifex handles the arb files or if it even supports them.
In general arb files are just json files with some cool extra metadata features so we might be able to just convert it.
It'd be great if someone could verify that.

More information on the recommended flutter translation flow can be found here: https://docs.flutter.dev/development/accessibility-and-localization/internationalization

I dont find the .arb files as supported file format of transifex :S
https://help.transifex.com/en/collections/3511306-for-developers#file-formats
I dont really remember why i used the special library, but at that time it was somehow the option that pleased me the most.
We could also check to update the downstream branch of flutter_translate. It is only used because it has the fallback language. Which is not supported in the main repo for reasons i cannot understand.
(If strings are not available in given langauge they are just representet with the keys. But i want it to default to english)

how does the transifex bot work regarding new string?
Will it pick them up as soon as I merge a change to the english translation into develop?


Adds LLE plugin

This is a major rework of the EventList and associated classes. I've made a generic EventList and EventListLike to allow for the subtle difference between TTE and LLE. This is mostly involved with their handling of deadtime. The classes can be built upon as well. There is some repeat, but over all, it is very modular.
@giacomov double check my livetime calculations please

Fixing up the branch

ok, but we could have merged this as is and keep it for future improvements.
The code is good and correct, it just needs some more work to be fully general. Anyway, it's entirely up to you how you want to proceed.
thanks for doing this!

Ah! Well, let me get this one right. I'll fix it up this morning unless you want to merge now.

tomorrow is fine, it's late here and my wife would kill me if I work more ;-)


dms.Endpoint panics during preview after upgrade

What happened?
On 2nd of May all of our internal PRs were failing their build validation, our build validation consists of a couple of checks but most important a pulumi preview off course to see what changes are being made.
Our last good run was from 27th of April, all of our failing PRs were failing with the below error.
This error was persistent on all of our runs, even ones that were only adding a single comment
Diagnostics:
  aws:dms:Endpoint (target-microservicedb-datalake-v1):
    error: error reading from server: EOF

  pulumi:pulumi:Stack (datalake-v1-datalake):
    panic: value is not an object
    goroutine 532 [running]:
    github.com/hashicorp/go-cty/cty.Value.GetAttr({{{0x0?, 0x0?}}, {0x0?, 0x0?}}, {0xdc39e3c, 0xb})
    	/home/runner/go/pkg/mod/github.com/hashicorp/go-cty@v1.4.1-0.20200414143053-d3edf31b6320/cty/value_ops.go:695 +0x318
    github.com/hashicorp/terraform-provider-aws/internal/service/dms.validateSSEKMSKey({0xdc39e3c, 0xb}, 0xaf78de0?)
    	/home/runner/work/pulumi-aws/pulumi-aws/upstream/internal/service/dms/endpoint.go:1396 +0x53
    github.com/hashicorp/terraform-provider-aws/internal/service/dms.validateS3SSEKMSKeyCustomizeDiff({0xa3604d3?, 0xf163dc8?}, 0xc0053844b0?, {0x0?, 0x0?})
    	/home/runner/work/pulumi-aws/pulumi-aws/upstream/internal/service/dms/endpoint.go:1382 +0x65
    github.com/hashicorp/terraform-plugin-sdk/v2/helper/customdiff.All.func1({0xf163dc8, 0xc0053844e0}, 0xdbdaa00?, {0xdbdaa00, 0xc002c68000})
    	/home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/v2@v2.0.0-20230327102345-3fa930f86570/helper/customdiff/compose.go:50 +0xb1
    github.com/hashicorp/terraform-provider-aws/internal/provider.(*wrappedResource).CustomizeDiff.func1({0xf163d58?, 0xc0000c4080?}, 0xdc4b0c6?, {0xdbdaa00, 0xc002c68000})
    	/home/runner/work/pulumi-aws/pulumi-aws/upstream/internal/provider/intercept.go:169 +0x66
    github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.schemaMap.Diff(0xc000f9b410, {0xf163d58, 0xc0000c4080}, 0xc005490340, 0xc0052dda70, 0xc0005f56e0, {0xdbdaa00, 0xc002c68000}, 0x0)
    	/home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/v2@v2.0.0-20230327102345-3fa930f86570/helper/schema/schema.go:699 +0x4d4
    github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).SimpleDiff(0x0?, {0xf163d58?, 0xc0000c4080?}, 0xc005490340, 0x5020106?, {0xdbdaa00?, 0xc002c68000?})
    	/home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/v2@v2.0.0-20230327102345-3fa930f86570/helper/schema/resource.go:895 +0x6c
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim/sdk-v2.v2Provider.simpleDiff({0x7fac731175b8?, {0x30?, 0xc0005c0000?, 0xc0052dda70?}}, 0xc002bedba0?, 0xbe8cf40?, 0xc005490000, 0x40d907?, {{{0xf1651b0, 0xc005159338}}, ...}, ...)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.45.2/pkg/tfshim/sdk-v2/provider_diff.go:87 +0x730
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim/sdk-v2.v2Provider.Diff({0xc0004aa0c0?, {0x0?, 0x0?, 0x0?}}, {0xdc6bdc2, 0x10}, {0xf165648?, 0xc005389c70}, {0xf096fc0, 0xc0052dda70})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.45.2/pkg/tfshim/sdk-v2/provider_diff.go:63 +0x30c
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(*Provider).Diff(0xc002c3e780, {0xf163dc8?, 0xc0050a52c0?}, 0xc004d87ce0)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.45.2/pkg/tfbridge/provider.go:746 +0x571
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler.func1({0xf163dc8, 0xc0050a52c0}, {0xd578f80?, 0xc004d87ce0})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.63.0/proto/go/provider_grpc.pb.go:555 +0x78
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0xf163dc8, 0xc0050a4fc0}, {0xd578f80, 0xc004d87ce0}, 0xc004edf640, 0xc005050900)
    	/home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x3f9
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler({0xda32740?, 0xc002c3e780}, {0xf163dc8, 0xc0050a4fc0}, 0xc004d87c70, 0xc002d42560)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.63.0/proto/go/provider_grpc.pb.go:557 +0x138
    google.golang.org/grpc.(*Server).processUnaryRPC(0xc0007be000, {0xf175980, 0xc0027a8000}, 0xc0050ac360, 0xc002d39470, 0x17cbad48, 0x0)
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.53.0/server.go:1336 +0xd23
    google.golang.org/grpc.(*Server).handleStream(0xc0007be000, {0xf175980, 0xc0027a8000}, 0xc0050ac360, 0x0)
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.53.0/server.go:1704 +0xa2f
    google.golang.org/grpc.(*Server).serveStreams.func1.2()
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.53.0/server.go:965 +0x98
    created by google.golang.org/grpc.(*Server).serveStreams.func1
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.53.0/server.go:963 +0x28a

Alot of debugging around with pulumi attempting to extract any additional information(which was unsuccessfull as all the debug output did not make alot of sense to us) and attempts at changing code we finnaly found a way to stop this error from appearing on a specific version of the pulumi-aws namely anything pulumi-aws>=5.38.0
our code was setup to use the pulumi-aws>=5.0.0,<6.0.0 inside the requirements.txt so it makes sense it auto updated.
and when we lock the version to pulumi-aws==5.37.0 everything keeps on working.
We did do some slight digging into the version and saw that version 5.38.0 of this package updates the terraform-provider-aws to v4.64.0 and the changelog show that this version does have an added data source for aws_dms_endpoint
We have no idea what this might mean but it seems related to the same DMS resource that we are having issues with.
Expected Behavior
Previews should function normally even after a minor version update
Steps to reproduce
Not entirely sure if this is reproducible from another environment as it might have something to do with dms resources being created on an older version and then updated in the minor ones but in our environment using pulumi-aws>=5.0.0,<6.0.0(which currently downloads 5.40.0) does not work and pulumi-aws==5.37.0 does work.
Output of pulumi about
CLI
Version      3.64.0
Go Version   go1.20.3
Go Compiler  gc

Plugins
NAME    VERSION
aws     5.29.1
python  unknown

Host
OS       darwin
Version  13.3.1
Arch     arm64

This project is written in python: executable='/Users/[redacted]/Documents/project_buckaroo/repos/aws/datalake-v1/venv/bin/python3' version='3.10.11
'

Backend
Name           pulumi.com
URL            https://app.pulumi.com/buckaroo
User           buckaroo
Organizations  buckaroo

Additional context
Affected code that might be interesting as it was only appearing to be an issue with the DMS endpoint resource.
def create_dms_target_endpoint(
        folder_name,
        target_bucket_name: pulumi.Output,
        role: aws.iam.Role,
        partition_enabled: bool = True,
):
    return aws.dms.Endpoint(
        f"{folder_name}-{const.PROJECT_NAME}",
        endpoint_id=f"{folder_name}-{const.PROJECT_NAME}",
        endpoint_type="target",
        engine_name="s3",
        extra_connection_attributes="EnableStatistics=true;DatePartitionSequence=yyyymmddhh;DatePartitionDelimiter=dash;compressionType=NONE;",
        s3_settings=aws.dms.EndpointS3SettingsArgs(
            bucket_name=target_bucket_name,
            bucket_folder=folder_name.replace("target-", ""),
            service_access_role_arn=role.arn,
            date_partition_enabled=partition_enabled,
            data_format="parquet",
            timestamp_column_name="update_ts_dms",
        ),
        tags={const.MAP_PROGRAM_RESOURCE_TAG_KEY: const.MAP_PROGRAM_RESOURCE_TAG_VALUE},
        opts=pulumi.ResourceOptions(
            ignore_changes=["extra_connection_attributes"], depends_on=[role]
        ),
    )

Contributing
Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

This is triggered on this line https://github.com/hashicorp/terraform-provider-aws/blame/8aea8f1e15c0fdb11036c816c2abc3afea451320/internal/service/dms/endpoint.go#L1389-L1390 and appears to be an issue with support for GetRawConfig.
I think this is related to or the same as the root cause of https://github.com/pulumi/pulumi-terraform-bridge/issues/1021.

FYI seeing the same issue with our dms configs - we are attempting to pin to v5.37.0 in our go.mod and will report back.

This has been fixed together with https://github.com/pulumi/pulumi-terraform-bridge/issues/1021#issuecomment-1674887944


Corrected link text and destination

Page Views and Instances count both initial and persisted values. Occurrences does not and was incorrectly linked in the sentence.

Hi Shimone, the documentation in its current state is correct.

Page views is the number of s.t() calls where a dimension item is set or persisted
Page events is the number of s.tl() calls where a dimension item is set or persisted
Occurrences is the number of total hits where a dimension item is set or persisted
Instances are the number of total hits where a dimension item is set. One 'Instance' metric is available per dimension.

Actually, it looks like the doc page here contradicts itself. The link is correct; however, the statement 'this metric does not count persisted values' is incorrect. I'll get this fixed. Thanks for bringing it to our attention!


Add support to fulfillment events

This PR adds support to fulfillment events. Fulfillment events are simple resources that can be CRUDed under /admin/orders/:order_id/fulfillments/:fulfillment_id/events.
Please review @christianblais @csaunders

Works with me. Let's not merge before we get Tobi's approval though.

Do we have an API team anymore? Are we planning on making this public?
The code seems fine, though I'm mostly concerned about whether this is sufficient enough for what our users would need.
@jamesmacaulay might be interested in this as well.

Do we have an API team anymore? Are we planning on making this public?

Yes, the docs will be released as soon as we merge this.

The code seems fine, though I'm mostly concerned about whether this is sufficient enough for what our users would need.

For now it is, as far as I can see. Would you have any scenarios in mind not covered by these changes?

:heart: this!


torchlambda lacks non-source files in distributed package

Hi.
I have been trying hard to follow the torchlambda example, but have simply come up against a hard stop that I cannot bridge. If you can offer any help, it would be much appreciated.
My installation is as such:
I am working in a conda environment called fastai2, with torch 1.5.0 (full pip list below) and docker version 19.03.8.
I installed torchlambda with "pip3 install torchlambda". (I have also tried with "pip install --user torchlambda", which fails when executing in exactly the same later later)
(fastai2) streicher@MLPC:~$ pip3 install torchlambda
Collecting torchlambda
  Using cached torchlambda-1590904228-py3-none-any.whl (30 kB)
Requirement already satisfied: Cerberus>=1.3.2 in ./.local/lib/python3.7/site-packages (from torchlambda) (1.3.2)
Requirement already satisfied: PyYAML>=5.3 in /home/linuxbrew/.linuxbrew/lib/python3.7/site-packages (from torchlambda) (5.3.1)
Requirement already satisfied: setuptools in /home/linuxbrew/.linuxbrew/lib/python3.7/site-packages (from Cerberus>=1.3.2->torchlambda) (46.0.0)
Installing collected packages: torchlambda
Successfully installed torchlambda-1590904228
(fastai2) streicher@MLPC:~$

I then created the model as described on github:
(fastai2) streicher@MLPC:~$ python
Python 3.7.7 (default, Mar 26 2020, 15:48:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> import torchvision
>>> model = torchvision.models.resnet18()
>>> torch.jit.script(model).save("model.ptc")
>>> quit()
(fastai2) streicher@MLPC:~$ ls *.ptc
model.ptc

Here starts the problems..
(fastai2) streicher@MLPC:~$ torchlambda settings
torchlambda:: Started creating YAML settings at /home/streicher/torchlambda.yaml.
torchlambda:: Started copying YAML source code
cp: cannot stat './templates/settings/torchlambda.yaml': No such file or directory
torchlambda:: Error: Failed during copying YAML source code
(fastai2) streicher@MLPC:~$ stat ./templates/settings/torchlambda.yaml
stat: cannot stat './templates/settings/torchlambda.yaml': No such file or directory

So I create the file manually using the example
(fastai2) streicher@MLPC:~$ vi torchlambda.yaml
(fastai2) streicher@MLPC:~$ cat torchlambda.yaml
---
input:
  shape: [1, 3, width, height]
  type: byte
  cast: float
  divide: 255
normalize:
  means: [0.485, 0.456, 0.406]
  stddevs: [0.229, 0.224, 0.225]
return:
  result:
    operations: argmax
    type: int
    name: label
    item: true
(fastai2) streicher@MLPC:~$

Here comes the hard problem... I then try the template step, but it fails in a way that I cannot get around
(fastai2) streicher@MLPC:~$ torchlambda template --yaml torchlambda.yaml
torchlambda:: Started creating C++ scheme at ./torchlambda.
torchlambda:: Started reading YAML settings.
torchlambda:: Finished reading YAML settings.
torchlambda:: Started validating YAML settings.
torchlambda:: Error during YAML validation:
normalize:
- means:
  - 'means field''s shape is not broadcastable to provided input shape: [1, 3, ''width'',
    ''height'']'
  stddevs:
  - 'stddevs field''s shape is not broadcastable to provided input shape: [1, 3, ''width'',
    ''height'']'
(fastai2) streicher@MLPC:~$

Do you have any idea what I am doing wrong? I would really love to give tochlambda a try. I currently use the pytorch 1.1 ARN layer in lambda, and it takes fully 18 seconds to start up from cold...
Thanks.
Additional information:
My docker seems to be ok:
(fastai2) streicher@MLPC:~$ docker pull szymonmaszke/torchlambda:v1.5.0
v1.5.0: Pulling from szymonmaszke/torchlambda
a3f8e652bdc4: Pull complete
8f876fde9c06: Pull complete
cd0ee434189a: Pull complete
Digest: sha256:c657e856e0b3f01cd7a8c3b3f603f94b0f45541ad8851afa056c988aaf59efa5
Status: Downloaded newer image for szymonmaszke/torchlambda:v1.5.0
docker.io/szymonmaszke/torchlambda:v1.5.0

(fastai2) streicher@MLPC:~/torchlambda$ docker image list
REPOSITORY                 TAG                                   IMAGE ID            CREATED             SIZE
szymonmaszke/torchlambda   v1.5.0                                d2496183b359        20 hours ago        711MB
samcli/lambda              python3.6-355e75366d87dabec462ddf97   ac2f11a96ba6        3 weeks ago         991MB
lambci/lambda              python3.6                             46852491e8e1        3 weeks ago         882MB
ubuntu                     latest                                1d622ef86b13        5 weeks ago         73.9MB
hello-world                latest                                bf756fb1ae65        5 months ago        13.3kB

(fastai2) streicher@MLPC:~$ docker run szymonmaszke/torchlambda
torchlambda:: Building AWS Lambda .zip package.
torchlambda:: Compilation flags:
torchlambda:: Final build arguments: -DBUILD_SHARED_LIBS=OFF -DAWS_COMPONENTS=
-- The CXX compiler identification is GNU 7.3.1
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Torch: /usr/local/lib/libtorch.a
-- Found CURL: /usr/lib64/libcurl.so (found version "7.61.1")
-- Found AWS SDK for C++, Version: 1.7.344, Install Root:/usr/local, Platform Prefix:, Platform Dependent Libraries: pthread;crypto;ssl;z;curl
-- Configuring done
-- Build files have been written to: /usr/local/build
CMake Error at CMakeLists.txt:34 (add_executable):
  No SOURCES given to target: torchlambda
torchlambda:: App size:
du: cannot access '/usr/local/build/torchlambda': No such file or directory
torchlambda:: Zipped app size:
du: cannot access '/usr/local/build/torchlambda.zip': No such file or directory
torchlambda:: Deployment finished successfully.(fastai2) streicher@MLPC:~$

And the pip installation sees to be ok:
(fastai2) streicher@MLPC:~/aws-lambda$ pip3 list
Package                                    Version
------------------------------------------ ------------
adal                                       1.2.3
arrow                                      0.15.6
attrs                                      19.3.0
aws-lambda-builders                        0.8.0
aws-sam-cli                                0.48.0
aws-sam-translator                         1.22.0
azure-cognitiveservices-search-imagesearch 2.0.0
azure-common                               1.1.25
beautifulsoup4                             4.9.0
binaryornot                                0.4.4
blis                                       0.4.1
boto3                                      1.13.4
botocore                                   1.16.4
Bottleneck                                 1.3.2
bs4                                        0.0.1
catalogue                                  1.0.0
Cerberus                                   1.3.2
certifi                                    2<IP_ADDRESS>
cffi                                       1.14.0
chardet                                    3.0.4
chevron                                    0.13.1
click                                      7.1.2
cookiecutter                               1.6.0
cryptography                               2.9.2
cycler                                     0.10.0
cymem                                      2.0.3
dateparser                                 0.7.4
docker                                     4.2.0
docutils                                   0.15.2
fastcore                                   0.1.17
fastprogress                               0.2.3
Flask                                      1.0.4
future                                     0.18.2
graphviz                                   0.14
idna                                       2.9
importlib-metadata                         1.6.0
isodate                                    0.6.0
itsdangerous                               1.1.0
Jinja2                                     2.11.2
jinja2-time                                0.2.0
jmespath                                   0.9.5
joblib                                     0.14.1
jsonschema                                 3.2.0
kiwisolver                                 1.2.0
MarkupSafe                                 1.1.1
matplotlib                                 3.2.1
msrest                                     0.6.13
msrestazure                                0.6.3
murmurhash                                 1.0.2
numexpr                                    2.7.1
numpy                                      1.18.4
nvidia-ml-py3                              7.352.0
oauthlib                                   3.1.0
onnx                                       1.7.0
packaging                                  20.3
pandas                                     1.0.3
Pillow                                     7.1.2
pip                                        20.0.2
plac                                       1.1.3
poyo                                       0.5.0
preshed                                    3.0.2
protobuf                                   3.12.2
pycparser                                  2.20
PyJWT                                      1.7.1
pyparsing                                  2.4.7
pyrsistent                                 0.16.0
python-dateutil                            2.8.0
pytz                                       2020.1
PyYAML                                     5.3.1
regex                                      2020.4.4
requests                                   2.23.0
requests-oauthlib                          1.3.0
s3transfer                                 0.3.3
scikit-learn                               0.22.2.post1
scipy                                      1.4.1
serverlessrepo                             0.1.9
setuptools                                 46.0.0
six                                        1.14.0
soupsieve                                  2.0
spacy                                      2.2.4
srsly                                      1.0.2
thinc                                      7.4.0
tomlkit                                    0.5.8
torch                                      1.5.0
torchlambda<PHONE_NUMBER>
torchvision                                0.6.0
tqdm                                       4.46.0
typing-extensions                          <IP_ADDRESS>
tzlocal                                    2.0.0
urllib3                                    1.25.9
wasabi                                     0.6.0
websocket-client                           0.57.0
Werkzeug                                   1.0.1
wheel                                      0.34.2
whichcraft                                 0.6.1
zipp                                       3.1.0

Hi, thanks for the report, I've confirmed the bugs you are talking about. Will hit you up as soon as the problem is fixed.

@streicherlouw fixed, reinstall torchlambda, both settings and template subcommands should work fine now, please report whether it's fine now on your side.

Thank you for your quick response, I can confirm that the scripts work perfectly now :-)