Fix model delete failed after model undeployed

Description
When deleting a model, we need to check the model status by querying the model index: https://github.com/opensearch-project/ml-commons/blob/main/plugin/src/main/java/org/opensearch/ml/action/models/DeleteModelTransportAction.java#L108, which means when undepdeloying model response returns, we have to make sure the model status in index is updated to DELETED.
But in UndeployModelTransportAction there's no such guarantee, when updating the model index: https://github.com/opensearch-project/ml-commons/blob/main/plugin/src/main/java/org/opensearch/ml/action/undeploy/TransportUndeployModelAction.java#L192, the response returns almost at the same time when the each node's undeploy finished: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/action/support/nodes/TransportNodesAction.java#L315. Which means updating index and response to client run in parallel, so this issue happens occasionally.
So we need a mechanism to guarantee only when the update model index is completed, the response to client get run by adding a countdownlatch to block the response client action when updating index. This is safe to block since this newResponse method runs in a threadpool instead of the event loop threads. I also changed the threadpool to model deploy threadpool which is dedicated for deploy/undeploy model instead of management threadpool.
Issues Resolved
https://github.com/opensearch-project/ml-commons/issues/2419
https://github.com/opensearch-project/ml-commons/issues/2560
Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Please rebase to take changes from #2627. Thanks.

what's the root cause?

what's the root cause?

Added more details in the description.

Which flaky IT is fixed from this change?

Which flaky IT is fixed from this change?

Check this issue: https://github.com/opensearch-project/ml-commons/issues/2560

@zane-neo I remember we have add immediate refresh policy . Why that can't work?

@zane-neo I remember we have add immediate refresh policy . Why that can't work?

Please check the description, simply put, this is a race condition issue that updating document and responding to user happens at the same time, no sequence guarantee.

This issue still exists now since this PR not merged: https://github.com/opensearch-project/ml-commons/actions/runs/10245403756/job/28381967022#step:6:3426

Checked again on the issue, this fix is not fixing the root cause. When using a script like below to reproduce the issue:
reg_res = client.post('localhost:9200/_plugins/_ml/models/_register', register_model_payload)
model_id = extract_model_id(reg_res)
deploy_res = client.post('localhost:9200/_plugins/_ml/models/{model_id}/_load')
undeploy_res = client.post('localhost:9200/_plugins/_ml/models/{model_id}/_unload')
delete_res = client.post('localhost:9200/_plugins/_ml/models/{model_id}')

This issue can be reproduce occasionally, but if adding time.sleep(1) after deploy_res returned, the issue can't be reproduced anymore.
The reason is deploy model is in an async approach and when client received the response, the model status in index is not guaranteed to deployed, this doesn't have a simple fix since a lot of files are involved in deploy flow.
This issue is not able to reproduced easily on local, so will add logs in the delete model action to clearly show the model status when it's not in a deletable status, and closing this PR now.


xActtiveDirectory: MSFT_xADCommon.ps1 is renamed to MSFT_xADCommon.psm1

The helper function script file MSFT_xADCommon.ps1 was renamed to MSFT_xADCommon.psm1 to be a module script file instead. This makes it possible to report code coverage for the helper functions (issue #201).

This change is

Codecov Report

Merging #202 into dev will increase coverage by 1%.
The diff coverage is 92%.

@@         Coverage Diff          @@
##            dev   #202    +/-   ##
====================================
+ Coverage    80%    81%    +1%
====================================
  Files        14     14
  Lines      1084   1214   +130
  Branches     10     10
====================================
+ Hits        871    990   +119
- Misses      203    214    +11
  Partials     10     10

Reviewed 10 of 10 files at r1.
Review status: :shipit: complete! all files reviewed, all discussions resolved

Comments from Reviewable


Relatives buggy with mj-include

Having problems with the new changes for relative URLs in mj-inlcude tags. Compiling with the MJML commandline tool works but when using gulp-mjml we get the error:
Error: ENOENT: no such file or directory, open '../freemarker/mjml/ommon/Attributes.mjml'
at Object.fs.openSync (fs.js:646:18)
at Object.fs.readFileSync (fs.js:551:33)
at checkIncludeHash (/Users/dean/Documents/Workspace/bobe/build/gulpfile.js:447:29)
at /Users/dean/Documents/Workspace/bobe/build/gulpfile.js:393:24
at DestroyableTransform.modifyFile [as _transform] (/Users/dean/Documents/Workspace/bobe/node_modules/gulp-tap/lib/tap.js:53:11)
at DestroyableTransform.Transform._read (/Users/dean/Documents/Workspace/bobe/node_modules/readable-stream/lib/_stream_transform.js:182:10)
at DestroyableTransform.Transform._write (/Users/dean/Documents/Workspace/bobe/node_modules/readable-stream/lib/_stream_transform.js:170:83)
at doWrite (/Users/dean/Documents/Workspace/bobe/node_modules/readable-stream/lib/_stream_writable.js:406:64)
at writeOrBuffer (/Users/dean/Documents/Workspace/bobe/node_modules/readable-stream/lib/_stream_writable.js:395:5)
at DestroyableTransform.Writable.write (/Users/dean/Documents/Workspace/bobe/node_modules/readable-stream/lib/_stream_writable.js:322:11)
When we use the tag:
<mj-include path="../common/Attributes.mjml" />

Error: ENOENT: no such file or directory, open '../freemarker/mjml/ommon/Attributes.mjml'
Looks like a typo ommon
Did you tried the 2.3 version ?

We are using 2.3.0.

In 2.3 relative is now from file that does the include instead of the current pwd. Did it works on older gulp-mjml ?

Yes it did. Previously we had the URL as ../freemarker/mjml/email/common/Attributes.mjml in the include, since this was relative to the location of the gulp file. Since updating we've been updating our templates to be relative from the actual file, i.e. ../common/Attributes.mjml. This is breaking with that error I listed above.
Strange thing is it works when we change the relative to be ../../../../freemarker/mjml/email/common/Attributes.mjml. So all the way down and all the way back up.

Not sure on that one, have to condense the workspace quite a bit.
Tried something just to see and strangly adding a space fixed things:
<mj-include path="../ common/Attributes.mjml" />

I'll try a get a repo up, workspace need to be condensed quite a bit.

While trying to simplify things I've found that its actually some of our other build steps causing the issue. It wasn't updated with the change for includes being relative to the files properly.
Sorry for this.

👌 then


Restore CD automation for publishing crates, and support workspace crates

This mostly reverts 5085c5b038c46ecb9fb16329caf5479b35a963c6, and attempts to support publishing workspace crates onefetch-ascii, onefetch-image, and onefetch-manifest. This would fix https://github.com/o2sh/onefetch/issues/1363 by ensuring crates are published on a platform that fully supports symbolic links.
I have little to no experience with GitHub Workflows, and I am unable to test this PR, but it still seems like it should do the job.

I am unable to test this PR

Honestly when it comes to situations like this I think it's fine to basically test in production. In other words, just trigger the workflow when we're ready and expect it to work. If there's anything to fix we can make a new patch release to re-trigger the workflow. I've even seen big projects like Vue make new patches just to re-trigger a previously broken CD.
Though of course we would never merge in a broken CD workflow :wink:

This looks mostly good. Could you use cargo publish -p PACKAGE-NAME instead of setting the working directory? I know I suggested changing the working directories in 5085c5b, but I think using cargo publish -p PACKAGE-NAME is slightly cleaner. As an added bonus one could just copy+paste the steps and run them from the project's root to publish manually if we avoid modifying the working directory.

I’ll follow up with this change next. $ cargo publish -p onefetch-ascii --dry-run works, so it should be fine.

I think I handled all the feedback except for https://github.com/o2sh/onefetch/pull/1364#discussion_r1674351277. Let me know what you think about https://github.com/o2sh/onefetch/pull/1364#discussion_r1674414547, and I’ll be back to this in a few hours.

I think I handled all the feedback except for #1364 (comment). Let me know what you think about #1364 (comment), and I’ll be back to this in a few hours.

Great! Once you resolve https://github.com/o2sh/onefetch/pull/1364#discussion_r1674431906 this will be good to me.

I think I handled all the feedback except for #1364 (comment). Let me know what you think about #1364 (comment), and I’ll be back to this in a few hours.

Great! Once you resolve #1364 (comment) this will be good to me.

Done! Thank you for the thorough and helpful review.


use brief key/value to indicate no data at query location

Currently the API returns {"Data Status": "No data at this location"} when a query is made beyond the extent of a dataset. However, per @cstephen's review this could be reduced to something like {"nodata": bool} and implemented across all endpoints. However, returning some more nuance might be useful (e.g. "your query is beyond the spatial extent of this data layer" or "your query is within the spatial extent of the data layer, but a 'No Data' value (e.g. -9999) exists at that location."

Closed by #74


dump.after_llvm_generation is not implemented

options.setOption("dump.after_llvm_generation", true);

does not dump the generated llvm it, and appears to not be implemented at all.

True its acutually 'options.setOption("dump.llvm", true);`.
I will update the command.


[chore]: reorganize Tests crate

This PR reorganizes Tests crate because all files inside it except tests.cairo where not compiled when running scarb test.
Now everything compiles, but the scripts to generate will still need improvements

@maciejka  its all good.
I tested it locally, it returns blocks with valid syntax/organization that compile without modification.
Is just missing the comment which will be deleted each time we regenerate the blocks. Might need a bit of help on that, dont know how to do that..

tristan@Tristans-MacBook-Pro raito % ./scripts/data/get_blocks.sh

Getting block:<PHONE_NUMBER>19d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
Getting block: 00000000d1145790a8694403d4063f323d499e655c83426834d4ce2f8dd4a2ee
All blocks retrieved successfully.

@maciejka its all good.
I tested it locally, it returns blocks with valid syntax/organization that compile without modification.
Is just missing the comment which will be deleted each time we regenerate the blocks. Might need a bit of help on that, dont know how to do that..

Just add comments to the get_blocks.sh for now.

@maciejka  done


NCW2 Session: Community Groups

Copying slides from the Overall NCW Slide Deck (slides 29 to 51).
fixes: https://github.com/kubernetes/community/issues/4945
/cc @chris-short @alisondy

Heyy @nate-double-u Pritish here. This PR has been open for quite some time and I was wondering if I can help you in some way so that we can wrap up this PR 😃
Thank you!

Hi Pritish! Thanks for the offer to help, this was work done during a workshop at KubeCon with @chris-short & @alisondy. It looks like it's original issue has been closed now, and I wonder if we should decide what we want to do with this work before bringing anyone new onboard to finish it up.
Thoughts Chris & Alison?


[Travis Probablistic Failure] TestRegistrationFiltering tests

Purpose
Currently sometimes randomly generated factory registration titles coincidently match search terms in our filter tests and cause them to fail. This fixes that.
Changes

change the factory registration names to something not random.

QA Notes
No really testable.
Documentation
🐞 fix, no docs.
Side Effects
None that I know of.
Ticket
No ticket

Nice!
For future PRs like this, please send them as hotfixes so that we avoid failures on master.


Hello Microverse

Displays "Hello Microverse"

New feature branch has added index.html, style.css, .gitignore, linters.yml, .hintrc, .stylelintrc.json and test.md

Thank you for the feedback.
I have have added the necessary files and installed the linters localy

I have added a descriptive title, a license and updated the readme file

hi @levy002, I have added the new title. Is everything alright now?


Examples URL Not working

lab.hasanaydogdu.com/validetta/ is down and not working. Please try and get it up

It works for me. Are you still having problems?

Yes, not working since the last 4 days.
[image: image.png]
Warm regards,
Haresh Shah
w +91 22.25250900  m +91<PHONE_NUMBER>
[image: Description: Description: @.*
Dependable. Efficient. Flexible
On Mon, Jul 10, 2023 at 12:45 PM Hasan Aydoğdu @.***>
wrote:

It works for me. Are you still having problems?
—
Reply to this email directly, view it on GitHub
https://github.com/hsnaydd/validetta/issues/72#issuecomment-1628375276,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABJ2HNLWIYZFN7DVK3SZQ7LXPOTX3ANCNFSM6AAAAAA2EC5LZQ
.
You are receiving this because you authored the thread.Message ID:
@.***>


Keywords for stickers

How can I get keywords for stickers?

You can't do this with TDLib. They are handled internally when appropriate.


RHOAIENG-436: Add model-name to labels and generateName

Description

This PR is a simple one to include model-name as label to the pipelineruns helping in filtering the pipelineruns based on labels and also attach model-name as part of generateName to distinguish the pipelineruns name.
How Has This Been Tested?

You can create a new pipelinerun to see the change in name of the pipelinerun
You can run below command to filter the pipelineruns based on the label
oc get pipelineruns -l model-name=bike-rentals-auto-ml

Merge criteria:

[ ] The commits are squashed in a cohesive manner and have meaningful messages.
[ ] Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
[ ] The developer has manually tested the changes and verified that the changes work

As per recent discussion on the JIRA, added labels to inference container test Deployment as well.

As per recent discussion in the JIRA, added labels to inference container test Deployment as well.

I wasn't a part of the discussion but doesn't the inference container test Deployment mean this?
https://github.com/opendatahub-io/ai-edge/blob/97f36c9c35b6dbd9c8237449a2a17cc8c3931fae/pipelines/tekton/aiedge-e2e/aiedge-e2e.pipeline.yaml#L223-L273
So it would mean to add labels to this Deployment and not the apps, right?

@LaVLaS are there any conventions for labels in ODH? Like should the label key have a prefix, like opendatahub.io/model-name or something? I think a prefixed label would be a little better.

@LaVLaS are there any conventions for labels in ODH? Like should the label key have a prefix, like opendatahub.io/model-name or something? I think a prefixed label would be a little better.

That is a great suggestion for the global identifier label to separate Pipelines, Tasks, *Runs, Pods, .... that are "owned" by the ai-edge workflow.  If/When we have a component name then the format would be `app.opendatahub.io/: true'.  For model specific runtime information that is a being used as a label (model name, version, ....), that falls under a label key that is private/specific to the user so a prefix would not be required, IMHO.
I don't think we should block this PR for that discussion but we should add this label standard to the discussions for creating controller to manage this AI Edge application workflow

I don't think we should block this PR for that discussion but we should add this label standard to the discussions for creating controller to manage this AI Edge application workflow

Sure 👍

/approve
/override ci/prow/test-ai-edge
Ignoring test failures due to extended openshift-ci environment provision issues that are preventing successful test runs


Add tests

Fixes #109.
Fixes #105.

Coverage decreased (-0.06%) to 99.716% when pulling 874ac3f7d0862624a188b67a3a96e2bb4aa0d15c on add-tests into f1d2e5d52e20b6e56064e95041bb75d3e5f19965 on master.


User profile image

Pull Request Overview

Link User.profile_asset_guid to Asset (foreign key)
Use flask.json.JSONEncoder to jsonify objects
Add PATCH replace /profile_asset_guid for user

Model Updates

User.profile_asset_guid now has a foreign key constraint to Asset.guid

Database Updates

User.profile_asset_guid now has a foreign key constraint to Asset.guid

REST API Updates
This is the API for adding / modifying user profile image:
PATCH /api/v1/users/<user-guid>

[{"op": "replace",
  "path": "/profile_asset_guid",
  "value": "<asset-guid>"}]

or
[{"op": "replace",
  "path": "/profile_asset_guid",
  "value": {"transactionId": "<transaction-id>"}}]

or
[{"op": "add",
  "path": "/profile_asset_guid",
  "value": "<asset-guid>"}]

or
[{"op": "add",
  "path": "/profile_asset_guid",
  "value": {"transactionId": "<transaction-id>"}}]

To remove the user profile image:
PATCH /api/v1/users/<user-guid>

[{"op": "remove",
  "path": "/profile_asset_guid"}]

Pull Request Checklist

[x] Ensure that the PR is properly formatted

Example: All lint checks are passing

[x] Ensure that the PR is properly rebased

Example: The PR is rebased on develop (commit: <insert develop commit hash>)

[x] Ensure that the PR uses a consolidated database migration

Example: One database migration is proposed (revision <insert develop revision> -> <insert new revision>)

[x] Ensure that the PR is properly tested

Example: All automated tests are passing

[x] Ensure that the PR is properly sanitized

Example: No sensitive data or large content was added to this PR

[ ] Ensure that the PR is properly reviewed

Before continuing with this PR, we're looking into using a separate FileUpload type instead of Asset for profile image and using transaction to make sure everything is reverted if something fails.


[WIP]Feat/align vcs pull requests

This PR aligns vcs/pull-requests capability with our conventions. It also adds pagination. Missing:

[x] Use testing lib with updated one-sdk
[ ] Use CLI with updated one-sdk   [Blocked - waiting on non-beta release on OneSdk]
[x] Use updated one-sdk capable of early loop termination, fail handling

Description

Motivation and Context

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] I have read the CONTRIBUTING document.
[ ] I haven't repeated the code. (DRY)
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

Changes included in https://github.com/superfaceai/station/pull/110


Bibtex docs

Shorten the readme and make a clearer reference to the documentation. Add a bibtex file for citations.

Current coverage is 99.37% (diff: 100%)

Merging #263 into master will decrease coverage by 0.11%

@@             master       #263   diff @@
==========================================
  Files            23         23
  Lines          1758       1758
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
- Hits           1749       1747     -2
- Misses            9         11     +2
  Partials          0          0

Powered by Codecov. Last update c84bc39...ac28afd


[Unexpected Behaviour] Graph width set incorrectly if failed too early in Course Mode

What you're using
The following happens on any theme on either course mode with set multiple songs (Extended/Nonstop/Marathon & Oni/Survival):
Just fail the first song and the graph for the first song takes up the entire width. Every subsequent song has the same width and eventually goes off the screen, showing splits at positions they weren't meant to be.
The widths seems to be only calculated once a new song has been reached.
The following depicts a 5-song-course in Oni/Survival Mode.
The image displays what happens if one fails on each song (starting with the 1st).
The last snippet shows what happens if one skips a(ll) song(s).
It seems that skipping songs doesn't move the life meter towards the next segment (the GainSeconds spikes happen way before the song split marker) but are being displayed at which time they occured.

Note: Even if the screenshots were made on 4.9.8, this behavior still occurs in 4.9.9
Expected behavior
Set graph width to course's entire duration instead of length of all cleared songs.

Update: I found out what the bug is, and now i'm in the process of a fix.

The fix in testing, seems to look fine, now the widths are enclosed to its proper boundries.
This is being shown with the first reproduction case, which is failing a song early on at the beginning of the course.

But we aren't done yet!

The upper graph is being displayed correctly (I've also tested the failed case and everything was in line).
But the combo graph haven't been adjusted fully (or have been adjusted too much?).

The same could be said with "Haste".

However: I wasn't able to test the haste course normally on 4.9.10/HF4 and had to do SoundUseOldResampler=1 to get it to work. And this has it's own flaws (which might need a seperate issue?).
I could have screenshot the same thing in SoundWaves4.9.9HF4 (?) but the 3.0xmusic course didn't work initially over there but on my ITG3Encore theme. Upon reloading a bunch betweet the two, it's no longer the case... I should have turned it off and on again.

Fixed in 4.11.0


Event Engagement Analytics

Please do work for this task in a branch called issue-014.
Create a component which displays information on volunteer events visible only to organization reps and admin users. Information would include feedback, number of volunteers through site, and hours of work contributed by volunteers.

I'm going to keep this in backlog until a number of related issues are completed first. As the project currently is, this issue is not even achievable imo.


If number of pets reach 5000  and I add new one

I just inspect your code and I see you limit 5000 pet items when query table pets.
I wonder if this table reach this number and I add new one .
Am I able to see my pet ?
I think for now simply just increase limit number but for future usage if we reach 1 millions.
what's your solution ?

polling is horrible! I think we already reached the limit that we could without a nice backend to support our operations. the solution is described here: #40


process file with openaire miners

File processor with OpenAIRE classifiers/project mining
http://mining.openaire.eu/openaireplus/classifier

Closing old issue since the strategy changed. Feel free to re-open if still needed.


Feat: Adding Campaign Page

Description
Currently the website is missing the campaign page. I would like to work on it.
Changes You need to make

campaign.tsx
app.tsx

Screenshots
Before:

After:

Checklist
Before submitting your pull request, ensure that you have completed the following tasks:

[x] Gone through the rules and guidelines of the project.
[x] When you are creating PR, your PR should title follows the specified format: [Feature/Enhancement/Bug Fix]: Brief description of the change.
[x] Attached OSSI or GSSoC in the heading of the Issue.
[x] I have checked this issue is not already covered in another issue.

Expected Time to Complete
2 days

I guess there is already an issue that exists, which points towards the same problem. Find out that and continue over there.

You can go with this! As other contributors who have raised those issue, are inactive.


Re-Implement Download Feature

The downloading feature previously relied on the server. Instead we want to generate all resources locally now.
Tasks

[ ] Allow downloading of a tree and associated people as a JSON file (This should be a good first step; it works as a proof of concept to locally generating the files and will later be reused for importing)
[ ] Re-Implement the original HTML export in the browser
[ ] Consider adding an image export, if this can easily be achieved
[ ] Implement a feature to import previously exported JSON files (validate data types!)

For the image export feature, we could consider using https://github.com/canvg/canvg, if our trees render correctly using that.

For the import feature, a really nice addition would also be to allow users to upload their old Plum Tree downloads. This may require some additional work to match up the data structures more nicely, and there's no guarantee that it would work at all.
From what I've seen the data for trees and people is split into different javascript files. These could be uploaded separately with a quick check for whether the people match the references found in the tree (and showing a warning if this is not the case. I'll have to have a closer look at these scripts to determine if we want to provide such an import feature.

Something to also consider is the idea of providing cloud backups using e.g. Google Drive. This way users could log in using their Google account and trees can be stored, including a timestamp corresponding to the backup. That way we can restore that cloud backup if the user either loses their device, or even synchronize the trees across multiple devices.
Of course this feature would be optional and the main storage location would still be in the user's browser. However, since browser storage is by it's very nature not very persistent, loss of data can only be avoided using backups. Local backups as suggested in the main issue are a good first step, but may be too impractical for some users.

In regard to the original HTML output, we may want to investigate providing an alternate "database" implementation that simply offers read access to some JSON file / literal JS object. This way we could provide a version of the tree viewer component of the app with all assets bundled into one (monstrous) HTML file (assuming this doesn't cause some issues on very large trees; alternatively a small number of static files), with editing disabled.
We could then produce this file as a secondary compile target (optimally without any additional files to prevent duplication), into which we insert a placeholder for the "database" object. This object could then dynamically be added in the browser on the download page to prepare the downloaded tree.
This is still pretty vague, so some things might not work as described here and I may have missed some things. Once I have some clearer ideas I will turn this into it's own issue.


About the tool specific suggestions for improvement

This is really a great project, it helped me a lot in the real world, if I am free I am very willing to help you improve and improve this project, here I would like to make a few small suggestions for continued progress
The first point, you can continue to add more AD domain vulnerability detection in the vuln_checks module, such as HiveNightmare, Printspooler family (Nightmare and Demon) and Exchange server vulnerability detection, which is very important, because in reality Exchange is very high privilege, and easy to to attack and if the attack is successful very easy to threaten the domain controller, so I felt the need to add a check for Exchange vulnerability
(CVE-2018-8581, CVE-2020-0688, CVE-2020-16875, CVE-2021-34473, CVE-2021-26855/CVE-2021-27065, CVE-2022-41040/CVE-2022-41082)
The second point is that I think we can add an automated capture of all tour passwords in the pwd_dump module, which can be combined with LaZagne, which is a great tool, you know, in the actual infiltration of the tour passwords are likely to be a breakthrough!!! So I think it's feasible
Having said that, I hope very much that this tool will get better and better, and thank you very much for your open source spirit, keep moving!!!
Translated with www.DeepL.com/Translator (free version)

Duplicate issue


Fix lightwalletd sync tests

Motivation
zcash/lightwalletd's get_latest_block RPC returns the tip height of the full node.
Zebra's lightwalletd sync test expects for this RPC to return the height of the latest block in the lightwalletd cache to check that the full sync works.
Possible Design
Either:

Add an RPC method or field to zcash/lightwalletd that returns the sync status
Check lightwalletd logs instead of get_latest_block for cache height

Then revert #7332.

Check lightwalletd logs instead of get_latest_block for cache height

I think there are already some regexes in the tests that check the lightwalletd logs, so we could copy those.
One tricky part of this is that we might need to check the Zebra and lightwalletd logs in concurrently running threads.

Hey team! Please add your planning poker estimate with Zenhub @arya2 @oxarbitrage @teor2345 @upbqdn


Follow jsonapi spec

I see that in #3 there was some deliberation as to how to format JSON responses. The decision was made to use the jsonapi format. I would prefer having the option to return a "simple JSON" format, which is why I have opened #764, but from what I can tell the current implementation of dingo/api does not follow the jsonapi spec at all. In particular, one of the first statements in the spec is:
A document MUST contain at least one of the following top-level members:

data: the document's "primary data"
errors: an array of error objects
meta: a meta object that contains non-standard meta-information.

dingo/api's approach of keying the response by the table name from which the entities is fetched and not by data clearly flies in the face of the spec. I assume that there was a previous pre 1.0 version of the spec which allowed this, but none of the pre 1.0 versions are publicly available on jsonapi.org, also it makes no sense implementing a pre-1.0 version.

I ran into something similar, the latest version of Fractal takes care of all this I think.
But in order to get it right now you need to use dev-master for dingo/api and then specify the fractal version - "league/fractal": "0.13.*"

Thanks for the feedback Bryan! I haven't managed to get it to work quite yet with the latest version of Fractal. It looks to me as though there is an overlap between Fractal's Serializers and Dingo's Formatter, and that what one would want to do is use Fractal's JsonApiSerializer. Unfortunately it doesn't look as though this can be configured in Dingo... Can you confirm this?

Oh yea sorry I forgot about that step, this is what modifying the serializer looks like in Lumen for me -
app('Dingo\Api\Transformer\Factory')->setAdapter(function ($app) {
    $fractal = new League\Fractal\Manager;
    $serializer = new \League\Fractal\Serializer\JsonApiSerializer($_ENV['API_DOMAIN']);
    $fractal->setSerializer($serializer);

    return new Dingo\Api\Transformer\Adapter\Fractal($fractal, 'include', ',', false);
});

I think there's some mention of that on the Transformers page but I may have dug it up from somewhere else.  Hopefully that helps!

I'm really not having much luck with this. While I can set the adapter, it doesn't have any effect on the output which I'm seeing, as Dingo's formatter is the last thing to touch the json before it is returned.

So if you put this in your controller -
dd(app('Dingo\Api\Transformer\Factory')->getAdapter()->getFractal()->getSerializer());
You're getting the default DataArraySerializer back?
If that's the case something must be going awry with how the serializer is getting set in your case.  More details on your setup and the code you're using (and where it's located) might be helpful.  I have only done this in Lumen but maybe someone else has experience with issues in Laravel, though they should be very similar.

When I put that in my controller, I get back whichever serializer I have configured in my AppServiceProvider, which contains the code that you suggested I use.

If you're using transformers correctly then you'll be returning a simple array from your transformer, which when passed to the formatters will simply be formatted as JSON, nothing more.
The formatters do not follow any of the JSON API spec as it's just way to complicated to implement in a simple formatter. The methods you see there for formatting things like models and what not are merely for rapid development.


Update get_agent_data_from_coord_iter to allow no agents on the grid

Allows models on which not all cells contain an agent.
Closes #12.
Confirmed it solved the problem:

I think I accidentally broke something in this PR, because now the forest fire example:

Fixed in #16.


useSearch returns old value temporarily if loader is defined

Describe the bug
When a loader is defined, setting a search parameter will render with the new value, revert back to the old value, then render with the new value again. This occurs in specific situation where there is no loaderDeps set, or the loaderDeps does not include the changed search parameter (because the loader does not need it).
The issue only occurs once and then will behave as expected on subsequent search changes until the browser is refreshed again.
Your Example Website or App
https://stackblitz.com/edit/tanstack-router-kvlqe9?file=src/main.tsx,index.html
Steps to Reproduce the Bug or Issue

Navigate to About
Open the devtools console
Refresh the page
click "set test" button
Observe logged search params.

Expected behavior
Only the new value should be logged, or at least it should only render the old value before a render with the new value has occurred.
Screenshots or Videos

Platform

OS: macOS
Browser: Arc
Version: confirmed on 1.0.7 and 1.26.3

Additional context
In a situation where you have an effect that watches the changed search param unexpected behavior will occur due to the value reverting back after the initial change.
In our case our loaders currently just trigger prefetches for react-query, so a temporary workaround is setting staleTime: Infinity as we do not depend on the loader rerunning.
Another workaround is to define loaderDeps to include all search params, but this doesn't seem ideal for the correct caching behavior.

Retested with 1.28.9 and this looks to have been fixed.
Sandbox: https://stackblitz.com/edit/tanstack-router-rcnp7h?file=src%2Fmain.tsx


WIP: Format node bindings with clang-format

No code changes were made to kzg.cxx only format was run
Inclusions

copy .clang-format to node bindings so IDE format command works
run format on kzg.cxx
add clang-format to Makefile

@jtraglia I added the lint command to the Makefile but clang-format is not on the workflow VM.  What do you think is the best course of action for that?


[@typespec/http-specs] - Modify code to handle http-client-* packages tests

While working with China Team in https://github.com/Azure/cadl-ranch/pull/755/, I have found that a few changes are required in the specs to handle the test success in http-client-java, http-client-net and http-client-python packages. This PR handles those changes. The scenarios are also validated using the test:e2e script.
Please review and approve the PR. Thanks

No changes needing a change description found.

You can try these changes here

🛝 Playground
🌐 Website
📚 Next docs


fix donate to check delegation & unbonding & redelegation entries not…

Description
Fix vesting coin donation function to check following data not DelegatedVesting amount

Delegation
Unbonding
Relegation

Author Checklist
All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.
I have...

[ ] included the correct type prefix in the PR title
[ ] added ! to the type prefix if API or client breaking change
[ ] targeted the correct branch (see PR Targeting)
[ ] provided a link to the relevant issue or specification
[ ] followed the guidelines for building modules
[ ] included the necessary unit and integration tests
[ ] added a changelog entry to CHANGELOG.md
[ ] included comments for documenting Go code
[ ] updated the relevant documentation or specification
[ ] reviewed "Files changed" and left comments if necessary
[ ] confirmed all CI checks have passed

Reviewers Checklist
All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.
I have...

[ ] confirmed the correct type prefix in the PR title
[ ] confirmed ! in the type prefix if API or client breaking change
[ ] confirmed all author checklist items have been addressed
[ ] reviewed state machine logic
[ ] reviewed API design and naming
[ ] reviewed documentation is accurate
[ ] reviewed tests and test coverage
[ ] manually tested (if applicable)

Codecov Report

Merging #141 (100a5e0) into release/v0.45.x (0dd29f4) will decrease coverage by 0.22%.
The diff coverage is 41.65%.

@@                 Coverage Diff                 @@
##           release/v0.45.x     #141      +/-   ##
===================================================
- Coverage            60.60%   60.38%   -0.23%
===================================================
  Files                  595      599       +4
  Lines                38660    39053     +393
===================================================
+ Hits                 23429    23581     +152
- Misses               13269    13489     +220
- Partials              1962     1983      +21

Impacted Files
Coverage Δ

baseapp/msg_service_router.go
75.92% <0.00%> (-9.50%)
:arrow_down:

baseapp/options.go
62.33% <0.00%> (-2.53%)
:arrow_down:

client/grpc/tmservice/service.go
71.15% <ø> (ø)

client/keys/add.go
70.34% <ø> (ø)

client/tx/legacy.go
64.28% <ø> (ø)

codec/amino_codec.go
100.00% <ø> (ø)

codec/proto_codec.go
73.41% <ø> (ø)

codec/types/any.go
55.88% <ø> (ø)

crypto/armor.go
85.89% <ø> (ø)

crypto/hd/algo.go
0.00% <0.00%> (ø)

... and 209 more


AndroidManifest android:scheme for PayPal Integration

According to the Drop-in UI documentation on braintree to implement PayPal Integration

Edit your AndroidManifest.xml to include BraintreeBrowserSwitchActivity and set the android:scheme:

<activity android:name="com.braintreepayments.api.BraintreeBrowserSwitchActivity"     android:launchMode="singleTask">     <intent-filter>         <action android:name="android.intent.action.VIEW" />         <category android:name="android.intent.category.DEFAULT" />         <category android:name="android.intent.category.BROWSABLE" />         <data android:scheme="${applicationId}.braintree" />     </intent-filter> </activity>
https://developers.braintreepayments.com/guides/client-sdk/setup/android/v2/#browser-switch-setup
I tried using my package id for the android:scheme section  but it doesn't work...
in my case my package id is "com.Organisation.ACBCompany.Android" and used ""com.organisation.acbcompany.android" but it doesn't work. Do i have to do anything special since it isn't exactly the same as the java counterpart?

Testing on and actual device actually worked. Testing on a colleagues' simulator also works which was very odd to me. I still can't figure out why it isn't working on my simulator and its working on pretty much everything else.


Where to store metadata related to the email account and/or folder

The standard doesn't seem to make clear where to store metadata related to the email account and/or folder that may be important to understanding the contents. For example:

The history of name changes of the account
The history of mail aliases of the account
The history of mail forwarding of the account
The history of users of the account (for role-based accounts)
...

It's probably outside of the scope of the document to specify a format for this metadata, but it should acknowledge the existence of the metadata and describe where it should be stored / linked to.

I have the same question. Some examples are 1) when exporting an email account through eDiscovery in Microsoft Office 365, the export includes some csv and xml files about the export and the results. 2) when converting a pst file to eml, the TOMES pst extractor tool (https://github.com/StateArchivesOfNorthCarolina/tomes-pst-extractor) includes a tsv file mapping folder names and is essential to understanding the conversion results. While it's rightly outside the mailbag specification to lay out such required metadata and the format, is it acceptable to include such files in the format subdirectories?

Cross-linking to Google Doc comment which mentions this topic: https://docs.google.com/document/d/1X7pOHxxzZl6PyMAJWd7bIR11rE4FlKty3J7oI6ghAKo/edit?disco=AAAANEF3VwA

Thank you @stuartyeates and @jamiepb , we haven't discussed this and these use cases are extremely helpful!

With input from the Advisory Board, we are thinking of just adding a recommendation to include metadata like @stuartyeates describes in bag-info.txt. We want to be flexible and permissive with metadata, and allowing multiple accounts in 0.3 makes this more challenging to provide a generalizable recommendation. In some cases it may also be useful to manage this information outside of a mailbag per #4. I could see some example keys being useful to encourage consistency, but I'm hesitant to be more prescriptive than that. If there are any suggestions, feel free to add them here or in the doc.

Per @jamiepb 's comment we decided to describe these examples as "companion files" and make it clear in the spec that its permissible to keep them in the source format subdirectory. So if you have TSVs or similar for PSTs, they would go in mailbag_root/data/pst using the same original arrangement.
Currently, the mailbagit implementation only packages files with the matching extension. So we're planning to add an optional arg to include all files in a provided directory. Let us know it that doesn't address your concerns!


gradle artifactory plugin - resolve.ivy.mavenCompatible=true does not convert dots to paths for ivyPaths

I noticed the resolver configuration does not convert dots to paths when setting mavenCompatible = true when searching for ivy files. As a consequence the artifact is retrieved but the ivy.xml is not and thus none of the configuration dependencies are downloaded.
Example:
artifactory {

    contextUrl = "${project.getProperty('artifactory.contextUrl')}"

    resolve {
        repository {
            repoKey = 'someRepo'
            username = "someUser"
            password = "somePassword"
            maven = true
            ivy {
                mavenCompatible = true
                ivyLayout = '[organization]/[module]/[revision]/ivy.xml'
                artifactLayout = '[organization]/[module]/[revision]/[module]-[revision](-[classifier]).[ext]'
            }
        }
    }
}

Resolution for ivyLayout works fine in the case where I hardcode the organization in ivyLayout.
Publishing seems to work fine with mavenCompatible = true

Tried the following combinations:

gradle 2.4 and org.jfrog.buildinfo:build-info-extractor-gradle:3.1.1
gradle 2.4 and org.jfrog.buildinfo:build-info-extractor-gradle:3.0.3

Yes, we are facing the same issue , we have to / the org to make work !!
compile (group: 'org/company', name: 'some-name' ,version: '1.0.16', classifier: 'master', ext: 'jar')

Still seeing this!
Gradle 4.0, org.jfrog.buildinfo:build-info-extractor-gradle:4.6.2
Just cost us two man days debugging everything from top to bottom, only to find this sillyness. -.-

@hborham @doit-ceo @reitzig
We have decided to deprecate the resolve section inside the artifactory closure. We therefore recommend using:
repositories {   	ivy { 		url "${project.getProperty('artifactory.contextUrl')}/someRepo"  		credentials {  			username "someUser" 			password "somePassword" 		} 		layout "pattern", {     			ivy "[organization]/[module]/[revision]/ivy.xml"         		artifact "[organization]/[module]/[revision]/[module]-[revision](-[classifier]).[ext]"         		m2compatible = true     		}     } }
Following this deprecation, we are not planning to fix this issue - however, we are not going to remove this functionality.
We updated the Gradle Artifactory Plugin [documentation] (https://www.jfrog.com/confluence/display/RTF/Gradle+Artifactory+Plugin#GradleArtifactoryPlugin-UsingtheArtifactoryPluginDSL) accordingly.
We will soon update the gradle-examples with the new recommendations.


SQLServer unique not null -- prevents FK

Laravel Version: 5.9.0
PHP Version: 7.2.18
Database Driver & Version: sqlsrv & pdo_sqlsrv, both 5.6.1

Description:
SQL Server creates the null-excempt indexes in a different way even if the field is not nullable.
This causes FK-constraints to fail as the FK is not available
Steps To Reproduce:
run laravel/telescope migrations on an SQL server database on laravel 5.8+ with https://github.com/laravel/framework/commit/75166c484336fec7027cd0440983b13a30fe18e6 applied
Ideal solution
Expose (or use, if already exposed?) a flag indicating the field is nullable, and only add the where if true.
Alternative solution
Introduce an ignoreNull option to ->unique(), this would not apply on all DB's though as most ignore null by default

Expose (or use, if already exposed?) a flag indicating the field is nullable, and only add the where if true.

I don't think we can make this assumption. Theoretically, you could have a nullable column with a UNIQUE index and want to reference it in a foreign key. This wouldn't be possible anymore.

Fair point, I guess we'd have to add a flag to the ->unique() key to ignore nulls for sqlsrv-based implementations, as the where prevents (at least in SQL Server 2016) setting a FK on that index, but otherwise it's preventing having a unique constraint on a column while still allowing (multiple) nulls; f.e. for a GUID to an SSO
One other alternative I could think of (with the penalty of an extra index per unique()->nullable()) is to compile a non-unique index alongside the unique index to allow FK constraints to still function on those column(s). This might make the logic too complex / magic though.

I feel this is more of a feature request than an actual bug. At the very least a change in the current behavior. Feel free to open up an issue on the ideas repo to get support for your idea and discuss it.

@driesvints Please reopen, this is a bug in Laravel 5.9.

@jlsjonas Should we revert your PR until we find a good fix?

@staudenmeir this was reported as 5.8? It's a bit confusing with the title.
Which PR caused this?

Yeah, 5.8 is a typo. This was caused by #27938.

If we can't get this fixed before the release then it's probably best that we revert the PR.

Yeah, 5.8 is a typo. This was caused by #27938.

Not exactly, as I encountered the issue on 5.8 with specified commit patched into it, however, I can see how that led to confusion.
I definitely agree that we should revert the PR if we can't get it fixed before the release, as this could otherwise break existing packages.
If anyone has some thoughts on the best implementation strategy or wants to discuss possible solutions I'm available to a chat &/or contributing a(nother) PR.

I think it's best that we revert anyway for now and accept a new pr that provides a solution.

Reverted. Feel free to attempt again with the problems here addressed. Thanks.

4 years later, is there a working solution for this @staudenmeir ?
As a work-around I'm adding the index manually:
DB::statement('CREATE UNIQUE INDEX users_provider_id_unique ON users (provider_id) WHERE provider_id IS NOT NULL');


Update bookings-faq.md

Removed reference to PowerShell commands in one answer.

@JohanFreelancer9 - kindly do the copy editor review for this PR. Thanks

@kwekuako Please merge this PR. Thanks


Scale function updated with recent Dalamud font changes

Global scale and font size are now independent of each other, so scaling should factor both in.

Thank you!


pageserver: sharding followups

Problem
This PR contains pageserver changes made while shaking out and testing sharding, including shard splitting.  The API for shard splitting will come later, the related changes in this PR are enablers for that.
Recommended to review this commit-by-commit since it's a series of small independent changes
Summary of changes
Enabling changes for shard splitting:

Don't physically delete layers when removing from index, if they belong to an ancestor shard
During compaction, don't retain keys if they are present in a layer but no longer belong to the current ShardIdentity.  This avoids generating image layers with many defunct keys that inflate the size of a post-split shard.

Fixes for running multiple shards on the same node (this is permitted in general but comes up especially when splitting):

Fix a case where shard lookup could hit a secondary for a tenant and incorrectly conclude no shard found
Fix a case where page service would wait for the desired LSN on a different timeline than it subsequently read, resulting in incorrect read results.

Quality of life changes:

If a request hits the wrong shard, log more detail about it.
Make walredo code shard aware, so that its logging includes shard IDs (functionally no change)
Tweak background runtime initialization to always create at least two threads: otherwise running a pageserver pinned to a single core fails.

Checklist before requesting a review

[ ] I have performed a self-review of my code.
[ ] If it is a core feature, I have added thorough tests.
[ ] Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
[ ] If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

[ ] Do not forget to reformat commit message to not include the above checklist

test_large_schema failure was an assertion in walproposer.c, so highly unlikely to be related to this PR (https://github.com/neondatabase/neon/issues/6248)

Regarding 1ce33ad : can't spot what exactly was broken. Commit message should elaborate.

Updated it as follows:
pageserver: fix handling getpage with multiple shards on one node
Previously, we would wait for the LSN to be visible on whichever
timeline we happened to load at the start of the connection, then
proceed to look up the correct timeline for the key and do the read.
If the timeline holding the key was behind the timeline we used
for the LSN wait, then we might serve an apparently-successful read result
that actually contains data from behind the requested lsn.

The warning log of Page request routed to wrong shard: could easily flood the log should it ever happen in prod. Did you consider this? (The anyhow::anyhow! also isn't great because of CPU overhead, but, I guess that'll be solved in https://github.com/neondatabase/neon/issues/6038)

For this one, in #6038 I think we'll make this disconnect the client, so that the postgres-side backoff on disconnect will implicitly rate limit it.

Same concern for the warning added to Timeline::get get on disposable key
Regarding the get on disposable key: why can't we bail out there with a PageReconstructError?

I took another look at this path.  The warning I added there was helpful when debugging an issue looking up timelines, but I don't think we need to spend the cycles on it in prod, because it is redundant wrt. the is_key_local check we do when looking up the Timeline from page_service.  I've downgraded this to a debug_assert!().

I would have expected that splits never happen in place, but, instead after a split, each shard has its own tenant-$shard_id directory with its own layer files.

That is the case locally (shards never touch each others files).  But in remote storage, the child shards contain references to their parents' layers -- the purpose  of IndexPart containing the shard index in each layer's metadata is to enable that.

I would have expected that splits never happen in place, but, instead after a split, each shard has its own tenant-$shard_id directory with its own layer files.

That is the case locally (shards never touch each others files). But in remote storage, the child shards contain references to their parents' layers -- the purpose of IndexPart containing the shard index in each layer's metadata is to enable that.

Ok, gotcha. It's obvious to me now, wonder why it wasn't in review.
Maybe I didn't have the full context of struct Layer loaded into my head.
Or it was something in the comment, but, can't pinpoint it.
It would be nice to have a regression test for this behavior though, it's quite essential.

It would be nice to have a regression test for this behavior though, it's quite essential.

Definitely: we just can't land any sharding tests until the compute support lands (currently all the testing is happening on a branch with the compute commits in it.)


Fix tests and pre-commit

@rikroe thanks, I cannot get it working somehow for myself, will try again, but merging this for now, so we can progress. thanks a lot !


Effect of changing the reference band

Hi Cordmaur,
Thanks for sharing this outstanding work.
I am working on a project related to water body identification and would like to have data with as high resolution as possible. I have found that by modifying the reference_band = Mir to Nir or other 10m band in the ".init" file, then I can change the resolution (to 10m) and I have successfully obtained the results.
I would like to ask if the output of the model at 10m resolution (reference_band = Nir) is better than the 10m image obtained by simply interpolating the 20m result (reference_band = Mir) with NDWI+B12 as input. I am asking this question only because I am not sure why you defaulted the reference band in the code to MIR instead of the higher resolution NIR (or other 10m bands), since your article clearly states a 10m result.
Best,
Zhen Hao

Just to add: the product used is Sentinel 2 - L2A from Sen2Cor

Hi haozhen,
I'm glad to know the package is being useful.
Concerning default "reference band" it has been changed to 20m (mir) just for memory consumption reasons and overall performance. Some people were not able to run it on 16gb pc.
If you need 10m output, just leave it as Nir as you did.
For the clustering combination, we tested various configurations that are presented in the peer reviewed article. The option for a SWIR (Mir and Mir2) band in the clustering is because Nir is very sensitive to turbidity waters... so that will depend on your area of study. If you have more clear waters, you can try a pure 10m processing with NDWI and B8 in the clustering.
Hope it helps.
Mauricio
Enviado do meu iPad

Em 26 de out. de 2022, à(s) 15:53, haozhen315 @.***> escreveu:

Hi Cordmaur,
Thanks for sharing this outstanding work.
I am working on a project related to water body identification and would like to have data with as high resolution as possible. I have found that by modifying the reference_band = Mir to Nir or other 10m band in the ".init" file, then I can change the resolution (to 10m) and I have successfully obtained the results.
I would like to ask if the output of the model at 10m resolution (reference_band = Nir) is better than the 10m image obtained by simply interpolating the 20m result (reference_band = Mir) with NDWI+B12 as input. I am asking this question only because I am not sure why you defaulted the reference band in the code to MIR instead of the higher resolution NIR (or other 10m bands), since your article clearly states a 10m result.
Best,
Zhen Hao
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.

Hi Cordmaur,
Thank you very much for the reply. Yes it really helps.
Best,
Zhen Hao


Bump ks-devops chart version from 0.0.4 to 0.1.1

We can do a release after this PR and #4
fix https://github.com/kubesphere-sigs/ks-devops-helm-chart/issues/2

LGTM


Optimize response serialization with slotted dataclasses

In order to support a few different categories of features in the future, I'd like to propose a different method of serializing response data. These changes are mostly internal and not user-facing. Maybe nobody else will care about the details, but I'm writing down all my notes on this anyway for future reference.
Summary
While doing some experimenting for #244 and #116, the general problem I kept running into was how to add more features and optimizations without adding too much more code complexity. I think a good place to start is by modeling serialized objects as dataclasses; specifically, slotted dataclasses using the attrs and (optionally) cattrs libraries. These come with several benefits and a couple minor drawbacks that can be mitigated.
Any other opinions or feedback on this would be appreciated!
Examples

You can see an example of something similar (but for aiohttp rather than requests) here.
See attrs docs for more examples, specifically the examples with type hints

Short-term benefits
This would have several immediate benefits:

Performance:

Up to 25% smaller serialized storage size
Slightly less memory usage (probably only noticeable with a large number of responses)
attrs works by making changes to a class when it's defined, not at runtime, so it adds almost zero overhead

Typing & Documentation:

Having type-annotated attributes makes it much easier to see at a glance what information is being stored
This also makes it much easier to examine cached responses at runtime without referring to the docs (see Cache inspection section)
Mypy integration, which would let us easily check for some common categories of serialization-related bugs (see also #268)

Lots of "free" magic methods useful for logging, testing, and debugging

Long-term benefits
This also opens up lots of possibilities for future features to be added without adding too much complexity:

Serialization formats: attrs + cattrs.unstructure() will do the majority of the work needed to serialize with pretty much any other format (including non-binary formats like JSON: #244)
Backend-specific optimizations: for example, optionally storing requests as BSON in MongoDB (which has built-in support with cattrs 1.6.0+)
Indexing: This will make it much easier to optionally store a subset of attributes as separate indexed columns instead of in a single binary blob. This would enable faster operations (by several orders of magnitude) for some use cases like remove_expired_responses(), alternative cache eviction methods, and sorting; see #116 and https://github.com/reclosedev/requests-cache/pull/253#discussion_r621765243).
Since we'll be pickling dicts and builtin types rather than custom class instances, this allows some room for future schema changes without invalidating previously cached responses.

Drawbacks

Dependencies: This would require adding 1 new dependency and 1 optional dependency.

However, they are lightweight (pure python with no dependencies of their own), actively maintained, and thoroughly tested.
attrs is so widely used that a subset of it was adapted into the standard library as the dataclasses module in python 3.7+. There's a good chance that many or most larger applications using requests-cache already have at least one of these as an indirect dependency.

Invalidating old responses: Like with 0.6, this would invalidate previously cached responses after the next release.

Invalid responses are already handled in BaseCache, and new responses will simply be fetched instead of throwing errors.

Other considerations

Transparency: Since transparent caching is important (it's in the package description, after all), we'll need to make sure that responses returned from the cache still behave exactly the same as requests.Response objects.

Currently, the only minor exception to this is noted in #234.
The unit tests already do a fairly good job of ensuring this, and more can be added.

Future compatibility: If requests.Response, requests.PreparedRequest, or urllib3.responses.HTTPResponse removes or modifies key attributes or methods in the future, we don't want requests-cache to break.

While that's unlikely (outside of requests 3.0?), I think it would still be worthwhile to preemptively handle and test for that scenario (also see #269 and #271).

Backwards compatibility: I don't expect this to require users to make any code changes, but if so this will need to be clearly documented.

I decided to make cattrs optional for now, partly due to differences on python 3.6 and 3.7+ (see version history for releases after 1.1.0, which dropped python 3.6 support).
I'd also like to get some more testing volunteers for this before making it the default.


Brand object

Preview environment images deployed!
The preview environment has been updated:
client image tag: preview-1663-f1a31d0818134ee4ddbe774213044fa4c50d8407-20240202.1726
storybook image tag: preview-1663-f1a31d0818134ee4ddbe774213044fa4c50d8407-20240202.1726
Client URL: https://pr-1663--epp.elifesciences.org
Storybook URL: https://pr-1663--epp-storybook.elifesciences.org
Note: Updated images can take a few minutes to roll out.

Preview environment created!
The preview environment has been deployed to the cluster.
Client URL: https://pr-1663--epp.elifesciences.org
Storybook URL: https://pr-1663--epp-storybook.elifesciences.org
Note: If this is a new environment, it can take a few minutes for DNS to update.


Find orphaned documents (preferably automatically)

Nice to have: find them automatically

Some docs we've worked on (like assertions and mocking-and-stubbing) can go under "Guides"

I wonder whether we could incorporate the HTMLProofer ruby gem into the build pipeline, or use something built for Hugo?

As of 93c8dfa orphaned pages are printed at the end of make htmlproofer. The presence of orphans doesn't fail the build - it just prints them. There are currently 45 orphans. For each one of the orphan pages we should do one of the following:

Move its content to an existing (linked) file
Add a link in the nav
Remove the page altogether

Nice! I might move all the orphans to a separate folder then, so we know which documents we need to review now (as they're linked to) and which we can leave until later

Yes, completely agree with those suggestions. In the mean time I might park the ones I'm unsure about in a separate folder.

=== Orphaned pages ===
12:26:19 AM: public/wiki/cucumber-backgrounder/index.html
12:26:19 AM: public/wiki/autotest-integration/index.html
12:26:19 AM: public/wiki/a-whole-new-world/index.html
12:26:19 AM: public/highlight/index.html
12:26:19 AM: public/architecture/execution/index.html
12:26:19 AM: public/cucumber/environment-variables/index.html
12:26:19 AM: public/cucumber/cucumber.yml/index.html
12:26:19 AM: public/cucumber/checking-assertions/index.html
12:26:19 AM: public/cucumber/step-argument-transforms/index.html
12:26:19 AM: public/cucumber/wire-protocol/index.html
12:26:19 AM: public/admin/index.html
12:26:19 AM: public/tdd-introduction/index.html => merge with BDD (TODO)
12:26:19 AM: public/voting/index.html => Leave for later
12:26:19 AM: public/implementations/javascript/index.html => merge somewhere?
12:26:19 AM: public/implementations/jvm/java-di/index.html => there is a link to this from the java tutorial!
12:26:19 AM: public/implementations/jvm/index.html => merge this somewhere?
12:26:19 AM: public/implementations/cpp/index.html
12:26:19 AM: public/implementations/rust/index.html
12:26:19 AM: public/implementations/lua/index.html
12:26:19 AM: public/implementations/perl/index.html
12:26:19 AM: public/implementations/ruby/browsers-and-transactions/index.html
12:26:19 AM: public/implementations/ruby/custom-formatters/index.html
12:26:19 AM: public/implementations/ruby/sinatra/index.html
12:26:19 AM: public/implementations/ruby/ruby-on-rails/index.html
12:26:19 AM: public/implementations/ruby/console-colours/index.html
12:26:19 AM: public/implementations/ruby/ruby/index.html
12:26:19 AM: public/implementations/ruby/debugging/index.html
12:26:19 AM: public/implementations/ruby/calling-steps-from-step-definitions/index.html
12:26:19 AM: public/implementations/ruby/fixtures/index.html
12:26:19 AM: public/implementations/ruby/rake/index.html
12:26:19 AM: public/implementations/ruby/minitest/index.html
12:26:19 AM: public/implementations/cucumberish/index.html
12:26:19 AM: public/implementations/tcl/index.html
12:26:19 AM: public/implementations/godog/index.html
12:26:19 AM: public/implementations/dotnet-specflow/index.html
12:26:19 AM: public/implementations/python/index.html
12:26:19 AM: public/implementations/php/index.html

If I recall correctly, most of the implementation pages are just links to GitHub projects? So can easily be consolidated into a single page.

@plaindocs true, but then we'd need to find a place for that page
For now, we discussed to focus on the most used implementations, so imho we could park the others before go live

@aslakhellesoy I think we might need to reconfigure a little bit. Some of the docs listed are linked to from other pages, for instance:
https://cucumber.netlify.com/cucumber/state/
links to public/implementations/jvm/java-di/index.html

See also: #25

Some of the docs listed are linked to from other pages

@plaindocs and I had a chat about this, and we agreed on the principle that all internal pages should be reachable from the nav. Otherwise it might be too hard to find stuff.
To avoid an explosion of links from the front page, this means some pages will have to be quite long, but I think that's fine, as long as we have a good TOC for them.

Yeah, that sound good! I can work on it this evening.

I think we can add Implementations to the Cucumber menu. And then put all or most of that content in a single page.

@plaindocs will do. Also want to add "Checking assertions" to Guides

Working on the implementations over in #120

Merged TDD into BDD (DONE)

Updated list:
7:45:24 AM: === Orphaned pages ===
7:45:24 AM: public/implementations/php/index.html
7:45:24 AM: public/implementations/jvm/index.html
7:45:24 AM: public/implementations/ruby/sinatra/index.html
7:45:24 AM: public/implementations/ruby/debugging/index.html
7:45:24 AM: public/implementations/ruby/console-colours/index.html
7:45:24 AM: public/implementations/ruby/fixtures/index.html
7:45:24 AM: public/implementations/ruby/rake/index.html
7:45:24 AM: public/implementations/ruby/calling-steps-from-step-definitions/index.html
7:45:24 AM: public/implementations/ruby/ruby/index.html
7:45:24 AM: public/implementations/ruby/ruby-on-rails/index.html
7:45:24 AM: public/implementations/ruby/minitest/index.html
7:45:24 AM: public/implementations/ruby/custom-formatters/index.html
7:45:24 AM: public/implementations/ruby/browsers-and-transactions/index.html
7:45:24 AM: public/highlight/index.html
7:45:24 AM: public/cucumber/build-tools/index.html
7:45:24 AM: public/cucumber/cucumber.yml/index.html
7:45:24 AM: public/cucumber/wire-protocol/index.html
7:45:24 AM: public/cucumber/step-argument-transforms/index.html
7:45:24 AM: public/cucumber/environment-variables/index.html
7:45:24 AM: public/cucumber/editors/index.html
7:45:24 AM: public/architecture/execution/index.html
7:45:24 AM: public/admin/index.html
7:45:24 AM: public/wiki/autotest-integration/index.html
7:45:24 AM: public/wiki/a-whole-new-world/index.html
7:45:24 AM: public/wiki/cucumber-backgrounder/index.html
7:45:24 AM: public/voting/index.html
7:45:24 AM: public/step-reuse/index.html

@aslakhellesoy Where should build tools and editors go?
cucumber.yml and environment-variables could be merged into cucumber.md? (under configuration?) - I don't know enough about either

Since calling-steps-from-step-definitions is considered an anti-pattern (and that's the only place it's linked to), should we delete it? It also contains information on Multiline step arguments that may or may not be relevant elsewhere. Since it's a Ruby feature, I'll leave it for others to decide.

/implementations/ruby/custom-formatters/ is linked to from cucumber.md - Also a Ruby feature. Should it be merged or deleted?

wiki/autotest  is linked to from content/about/related-tools.md - Also a Ruby feature. Should it be merged or deleted?

Where should build tools and editors go?

Let’s create a tools.md page with a h1 for each category of tools (build tools, editors)

Sounds good!

@aslakhellesoy What is highlight.md for and do we still need it?

@aslakhellesoy What is your intention with step-reuse.md?

I've moved all of the unlinked orphans (i.e. docs that are not linked to from other docs) to folder _orphans; imho these docs do not need to be reviewed before GoLive.

If not needed anymore, let's delete?

I think so.

Updated list of orphans:
11:27:11 AM: === Orphaned pages ===
11:27:11 AM: public/cucumber/environment-variables/index.html
11:27:11 AM: public/cucumber/cucumber.yml/index.html
11:27:11 AM: public/wiki/autotest-integration/index.html
11:27:11 AM: public/wiki/a-whole-new-world/index.html
11:27:11 AM: public/admin/index.html
11:27:11 AM: public/_orphans/architecture/execution/index.html
11:27:11 AM: public/_orphans/implementations-php/index.html
11:27:11 AM: public/_orphans/voting/index.html
11:27:11 AM: public/_orphans/cucumber-backgrounder/index.html
11:27:11 AM: public/_orphans/wire-protocol/index.html
11:27:11 AM: public/_orphans/ruby/fixtures/index.html
11:27:12 AM: public/_orphans/ruby/minitest/index.html
11:27:12 AM: public/_orphans/ruby/console-colours/index.html
11:27:12 AM: public/_orphans/step-argument-transforms/index.html
11:27:12 AM: public/_orphans/step-reuse/index.html
11:27:12 AM: public/implementations/ruby/custom-formatters/index.html
11:27:12 AM: public/implementations/ruby/sinatra/index.html
11:27:12 AM: public/implementations/ruby/ruby-on-rails/index.html
11:27:12 AM: public/implementations/ruby/debugging/index.html
11:27:12 AM: public/implementations/ruby/calling-steps-from-step-definitions/index.html
11:27:12 AM: public/implementations/ruby/browsers-and-transactions/index.html
11:27:12 AM: public/implementations/ruby/rake/index.html
11:27:12 AM: public/highlight/index.html

Done ;)

If I understand correctly, autotest is part of zentest, which was last updated almost 2 years ago on Jun 13, 2016 - Is it still used, or can it be deleted?

Sinatra has been archived by owner, so I removed the link from related-tools as well as implementations/ruby/sinatra.md

Maybe related-tools.md could use a review/update to see if it contains other tools that might be outdated?

Closed, moved to #130


Self Signed Cert on Github Provider

Description
No option to specify TLS CA self signed cert in Provider Spec.
Details
We run a GitHub enterprise server that has a self signed CA cert attached to it. When trying to set up git commit status, the notification controller gets the below error.
failed to send notification could not list commit statuses: ... : x509: certificate signed by unknown authority

Maybe a solution similar to the ImageRepository CRD can be implemented in the provider as well. Specifically a value similar to certSecretRef?

I think the easiest way is to pass the CA in the Secret which contains the token, the issue here is that we would have to go back and implement support for all notifiers where applicable as it would be very confusing otherwise. Looking at the GitHub provider I see no issue adding an optional custom transport.
@stefanprodan any opinions?

After thinking about this for another second I realize that adding a CertSecretRef field to the provider would be better than reading it from the token secret.

@coultenholt I do not have access to a GitHub enterprise deployment, so could you have a look at the PR #184 and try it out to see if it solves your issue?

@coultenholt I do not have access to a GitHub enterprise deployment, so could you have a look at the PR #184 and try it out to see if it solves your issue?

Will test just as soon as the Docker Issue is resolved.. :) Thanks for the quick turnaround!

@coultenholt I do not have access to a GitHub enterprise deployment, so could you have a look at the PR #184 and try it out to see if it solves your issue?

Will test just as soon as the Docker Issue is resolved.. :) Thanks for the quick turnaround!

@phillebaba Your PR worked for me! Would love to get this merged in and released! Thank you!

Great to hear that! I just need to write some tests and fix some smaller issues and we should be good to go.


ExcludedHttpsHostNameRegex is broken

Adding e.g. google.com to the list stops all the traffic going to said domain name.

My bad. Seems to work.

Nope, it's broken. Google is redirecting the browser to google.fi. Adding google.fi to ExcludedHttpsHostNameRegex breaks Google.

Should be fixed in latest develop. See #204


kernel kconfig: Do not enable STATIC_INIT_GNU w the old native_posix

native_posix (unlike native_sim) or its brethren (NATIVE_APPLICATION) link together the "runner" code and the embedded code. This means that when CONFIG_STATIC_INIT_GNU is set, any host library code (like the llvm fuzzer) constructors will get postponed to the Zephyr initialization.
These libraries are unlikely to work if we do this. (the llvm fuzzer does not)
So let's instead not enable STATIC_INIT_GNU for these targets. This means possible embedded library constructors will continue to be picked during the link and be still called during the native runner initialization instead of during the Zephyr OS initialization as they were just before we introduced STATIC_INIT_GNU in
6e977ae2d54c4f5443f752ce3a88f22043dcbf07
Note that native_posix will be deprecated shortly and its users are strongly encouraged to move to native_sim.
Fixes #75407

Note the board swap seems to be working in SOF for me (https://github.com/thesofproject/sof/pull/9280), so no rush on this.  Whether it's worth merging for any other hypothetical native_posix users is not something I'll argue about. :)

Note the board swap seems to be working in SOF for me (thesofproject/sof#9280), so no rush on this. Whether it's worth merging for any other hypothetical native_posix users is not something I'll argue about. :)

Thanks @andyross I think it will be best to merge this before 3.7. We still support native_posix, and just like the fuzzer failed, other host libraries may also.
And just like SOF was still using native_posix I expect some other projects will still be using it and whatever derivatives they built over the years.
This just reverts this constructor initialization behavior to what it was only for native_posix and its old family of boards. So it should be safe enough (I run a full twister run on native_posix and this did not introduce any regression).


Implemented column comment export to DDL for SQL Server.

Because it was not output ...

Oh, thanks.
No SQLServer user exists until now...!?


ValueError: On entry to SGEMV parameter number 8 had an illegal value

Travis CI failed on examples/gym/train_ddpg_gym.py:
Traceback (most recent call last):
  File "examples/gym/train_ddpg_gym.py", line 173, in <module>
    main()
  File "examples/gym/train_ddpg_gym.py", line 170, in main
    max_episode_len=timestep_limit)
  File "/home/travis/build/pfnet/chainerrl/chainerrl/experiments/train_agent.py", line 144, in train_agent_with_evaluation
    logger=logger)
  File "/home/travis/build/pfnet/chainerrl/chainerrl/experiments/train_agent.py", line 52, in train_agent
    action = agent.act_and_train(obs, r)
  File "/home/travis/build/pfnet/chainerrl/chainerrl/agents/ddpg.py", line 314, in act_and_train
    self.replay_updater.update_if_necessary(self.t)
  File "/home/travis/build/pfnet/chainerrl/chainerrl/replay_buffer.py", line 327, in update_if_necessary
    self.update_func(transitions)
  File "/home/travis/build/pfnet/chainerrl/chainerrl/agents/ddpg.py", line 246, in update
    self.actor_optimizer.update(lambda: self.compute_actor_loss(batch))
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/chainer/optimizer.py", line 416, in update
    loss.backward()
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/chainer/variable.py", line 398, in backward
    gxs = func.backward(in_data, out_grad)
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/chainer/functions/connection/linear.py", line 59, in backward
    gW = gy.T.dot(x).astype(W.dtype, copy=False)
ValueError: On entry to SGEMV parameter number 8 had an illegal value

This may be the same issue as https://github.com/pfnet/chainer/issues/2744

I close this issue because it's fixed from Chainer v3.0.0.


Avoid Vec allocations for single-result Instructions

This shows the latest memory flamegraph of the rollup-base-private circuit.

The dominant contributors to memory usage are then the SSA instructions, their results, ACIR opcodes and the ACIR callstacks.
Most of the SSA instructions don't have any results or just a single result.
https://github.com/noir-lang/noir/blob/920bf75b9e62237b0fda94a0e56671a608eee3e2/compiler/noirc_evaluator/src/ssa/ir/dfg.rs#L343-L379
We could then save memory by:

Omitting instructions from the results map if they don't have any (We'd need to check that we're not breaking any assumptions in outside code)
Replacing the Vec<ValueId> in the map with an  enum { Single(ValueId), Multiple(Vec<ValueId>) } so we can avoid the extra Vec overhead for the vast majority of instructions which only have a single result.

This is fixed in the remove ValueIds change coming soon


updateGroup function takes a FormGroupState?

When I try to add validation to my form, i use the updateGroup function as described in the documentation, but typescript tells me it takes a FormGroupState rather than just T, so I can't set my rules.
export interface ReviewForm {
  occupation: string;
}

export const validateAndUpdateForm = updateGroup<ReviewForm>({
  occupation: validate(required)
});

Typescript tells me "Object literal may only specify known properties and occupation does not exist in FormGroupState".
I'm using version 2.3.6 and I can't quite figure out how to proceed.

Which version are you using? In pre-v3 you will have to change your code to explicitly declare the property type for the validate call, i.e. occupation: validate<string>(required). The reason you get that error is that there is an overload on updateGroup that takes the form state as the first parameter and if typescript cannot match the overload it defaults to the first one. With v3 your code should work without issues.

Thank you, that seems to work.


Update the encoding

Hi Mahyar, thank you for forking my project (;
There was an error in my encoding logic that resulted in letters R-Z to be unused and digits to be twice more likely to be used. (https://github.com/lukateras/nanoid.h/commit/f3f27a9d1d435cf1f6b0d68be34bfd38ab973abc)
This pull request updates the logic to what's now in the upstream trunk, correct and a tad bit more concise.
(What a cute pokemon on your site :3 Props for the onion service!)

Hi! Thanks for your pull request! I missed this for some reason; apologies for the delay.
Thanks for checking out my website too. 😄 The onion service is actually switched off currently, but I'll re-enable it soon when I have the time to set it up again.


Add configurable 4-bit color classes

Resolves #23
Adds in the --color-classes flag to have to-html use classes to configure span colors instead of hardcoded values as discussed in #23. This also includes the required changes in ansi-to-html to support configuring the colors through classes which did involve a significant amount of changes to the public API to replace the existing convert_escaped() and convert() functions with convert() and convert_with_opts() respectively
The switch to convert_with_opts() was done to avoid adding yet another param to convert(). The examples in the documentation are all up to date, so this example should give a good overview of its usage
use ansi_to_html::{convert_with_opts, FourBitColorType, Opts};

let opts = Opts::default()
    .skip_escape(true)
    .skip_optimize(true)
    .four_bit_color_type(FourBitColorType::Class);
let bold = "\x1b[1m";
let red = "\x1b[31m";
let reset = "\x1b[0m";
let input = format!("
  {bold}Hello {red}world!{reset}
");
let converted = convert_with_opts(&input, &opts).unwrap();

assert_eq!(
    converted,
    // The `<h1>` and `</h1>` aren't escaped, useless `<i></i>` is kept, and
    // `<span class='red'>` is used instead of `<span style='color:#a00'>`
    "
  Hello world!
",
);

Forgot that I also have to handle allowing for a custom prefix to allow for more unique class names. This PR is certainly a start though
I won't have any time for that today, so I'll have to finish up early next week. Sorry for being a bit quick to open a PR :sweat_smile:

Found some time to finish things up, so it should be good for review now
Running
to-html --prefix custom --color-classes 'echo -e "\e[100m\e[31mred with a bright black background\e[0m"'

yields

> echo -e "\e[100m\e[31mred with a bright black background\e[0m"
red with a bright black background
>

The notable parts being class='custom-bg-bright-black' and class='custom-red' for the colored spans

I had an idea for another solution: Using CSS variables to make the colors customizing while providing default values:

This also means that users don't need to have two CSS classes for each color (with and without bg-):
.term {
  --red: #f55;
  --bright-red: #faa;
  /* etc. */
}

This would be a good default because it works out of the box and is customizable. Should we still have the other options? I.e.
pub enum FourBitColorType {
    #[default]
    Variable,
    Hardcoded,
    Class { prefix: Option<String> },
}

Had some time to switch things over to using CSS variables instead of having the --color-classes flag. It seems to work great from manually testing!
Currently the prefix is passed through from to-html to ansi-to-html for prefixing the CSS variables like --{prefix}{color_name} (to-html already appends a - at the end of prefix before passing it through)
Same example as before (minus the now removed --color-classes)
to-html --prefix custom 'echo -e "\e[100m\e[31mred with a bright black background\e[0m"'

yields

> echo -e "\e[100m\e[31mred with a bright black background\e[0m"
red with a bright black background
>

The latest commit should address all the review comments


Fix typo in readme file

Problem
Typo in readme file
...
Solution
Spelling corrected
...
Notes
Use semver to bump VERSION, and remember to run the following after
merging your PR:
make push-tag

Merging this tonight! Ty for your contribution.

This is merged in PR #33 cause I needed to bump the version


fix(utils): support more platforms

Thanks for your contribution!
I think it would be better to implement RawToStr for only *const c_char instead of both i8 and u8, could you please update your PR?


Add start / end date / group logic

Given:
start date: "2017-01-01"
today's date: "2017-01-15"
NSDate *startDate, *endDate;

Expect
[ProgramDay currentWeekAndDayFromStartDate: startDate todaysDate: todaysDate];

+ (NSDictionary*) currentWeekAndDayFromStartDate:(NSDate*)startDate todaysDate:(NSDate*)todaysDate;

to return:
@{ @"weekNum": @"week3", @"dayNum": @"day1" }

Added in #23


exponential backoff for at_least_once! mode

Right now the at_least_once! mode is pretty naive in how it handles failures. If an error occurs we will just requeue the message endlessly.  If the error that occurs is not recoverable (ie. it tries to hit a service which is completely gone from the internet) then this message will get retried over and over in a tight loop.
I think a better default behavior would be to handle failures with an exponential backoff algorithm similar to sidekiq.
This is not a feature built into RabbitMQ, but there are ways to handle it client side. We could add this behavior as an additional middleware in a totally separate gem, but I think it would make sense to have this be the default.

@mmmries -- It seems like we can just create a dead-letter-exchange as the queues are created, but I'm having trouble getting the messages to dead-letter. It might just be my understanding, but I also am not fully sure how the at_least_once_spec.rb fails messages and retries them. Is there something about sleep 0.1 that causes a message to fail/retry or is this method just not entirely finished?

@camkidman good to see you! (or at least your avatar)
at_least_once! currently works by calling env.reject if an error is raised while processing a message, if no errors are raised then it calls env.acknowledge.
In the current test for at_least_once! mode it triggers this behavior by raising an exception the first two times it receives a message. This is meant to simulate something like a database failure occurring while trying to process a message.
sleep 0.1 is just giving some time for the message to get to rabbitmq and back to our subscriber which is running in a separate thread.
So that all works as-is right now, but I don't want to use that feature in production because sometimes an error can't be resolved by just retrying. For instance what if I queued up a message about a given use, but then that user got deleted from the database? When my subscriber tries to lookup that user it might fail and I don't want it to just keep retrying that job forever. So the backoff is meant to give our application several chances to process a message over the course of a good chunk of time, but eventually let the system give up on that message.

@mmmries -- Thanks for the answer! Good to see you too! Wish I could've made it to UVRB the other day! Did you have your baby yet??
So I've been looking at this and I have a suggested implementation here, but I wanted to seek your opinion before I go too far on what might be the wrong direction.
To handle the whole process, I suggest the following:

We create a single dead-letter exchange (DLX) and a single dead-letter queue when the at_least_once! mode is called. All queues using this mode will have that exchange set as their dead-letter exchange.
At the same time, we create a subscriber on a different process that binds to the dead-letter queue
When a message failed from any queue that has the "x-dead-letter-exchange" argument, it goes into the aforementioned DLX, to the dead-letter queue, and to the subscriber we created.
The subscriber handles the message appropriately (it will have to pull a message off the queue, decide whether or not it's time to re-queue it to the original queue, modify the expiration header if so, then re-queue it).

Here's a picture of a little diagram I drew to illustrate if I didn't communicate right
Some considerations that I haven't fully flushed out and was wondering if you had input on:

Ideally we'd want the dead-letter queue to be a priority queue so the subscriber for the DLQ grabs the messages in the order they have to be sent. I didn't see anything that would handle this except for this plugin, but do we want users of this gem to have to install it?
If we don't use the priority queue plugin, our subscriber would need to determine whether or not the message needed to be sent based on some criteria, probably manipulating the expiration header. But we don't want the subscriber to hold onto the message for the duration of the expiration.
Seems like, in this circumstance, the subscriber to the DLQ is the one that handles the exponential backoff.

Just wanted to see what your thoughts were before I get too far down a path that I'm not sure is right.

Cameron -
Sorry for the slow response on this. I'm having my baby today and spent last week trying to make sure I wasn't leaving any loose ends at work. I don't think we should need the priority queue for our waiting queue. I think we can let RabbitMQ handle which messages get re-published by just having a timeout set on each message when we publish it into the waiting queue.
I'll try to draw out what I'm thinking and attach it here when a have a little time.

Here's a quick summary of what I'm thinking: basically when the subscriber decides that a job has failed and needs to retry it publishes a new copy of the message to the retries exchange. The new copy of the messages looks the same as the original, except that it has a TTL header. The timeouts queue will bind to the retries exchange such that it gets a copy of every message, that way we can publish the message with the right routing key and set the dead-letter-exchange for the timeouts queue to go back to the events exchange. That way once the message times out it will get re-published and our subscriber will see the message again.

@camkidman just as a reminder, I totally don't expect you to be spending your personal time on this. I do personally believe that contributing to open source is a great way to showcase your engineering skills and communication skills in a way that can be evaluated by any potential employers, but you can feel free to ditch this project at any time and for any reason.
With that said, I had a little time the other night to spike out a rough idea of how this might all work at a code level. That branch is definitely too ugly to be considered ready to merge into this project, but it shows the general idea. Currently it passes all tests under MRI, but fails under jRuby (no idea why yet).
It works something like this:

When someone calls at_least_once! in a subscriber it sets up the retries exchange (as a topic exchange) and sets up the timeouts queue which receives a copy of any message sent to the retries exchange and always publishes its dead letters back to the events exchange.

When a message raises an unhandled error in an at_least_once! subscriber, the subscriber publishes a copy of that message to the retries queue with the same payload, headers, and routing key as the original message, but we add a TTL header to it.

When the message times out in the timeouts queue it gets sent back to the events exchange and ends up back in the subscriber code again.

I think that general flow will work for this problem, but it leaves several details unaddressed:

Can the user change the retry schedule? Currently the system waits 300ms, 900ms, 2.7sec, 8.1sec all the way up to a one day wait and then finally just chucks the message out the door.

 In theory this should almost never happen, but in practice I'm sure it will happen sometimes and it would probably be good to report those cases back to the user so that they can address them in their system.

Right now all messages go back to the "events" exchange, but the user may not be using the "events" exchange. They can pick their own exchange. So what if the message needs to go back to a different exchange? What if the user has multiple exchanges they are receiving messages from?

All of the naming in my spike branch is pretty terrible

@mmmries -- Thanks for the disclaimer! I very much would like to start contributing to open source projects like this. I don't get as much code exposure as I'd like on a day-to-day basis, so this is actually fun for me (although I'm finding myself with less time to work on this specifically, I'd still like to contribute however I can).
If I may offer some opinions on the issues:
-- Do we want to build in the ability for users to change the retry schedule? I think as a default behavior, this is better than it was. I would say "this is how it behaves by default when you call at_least_once!" And I can't imagine very many users being unhappy with that.
-- I definitely think something should be logged when a message is completely discarded. You can never have too much logging, in my experience
-- I would say this is quite the issue. We could always re-publish the message to the exchange that it originally came from, but if any link in the chain isn't there anymore that would pose an issue, wouldn't it?
-- The naming isn't that bad
I hope all is well with your newborn!!

This has been added and is the default behavior for at_least_once! subscribers


Inconsistent behavior for internal and external package definition

What version of CUE are you using (cue version)?

$ cue version
cue version (devel) linux/amd64
cue version v0.4.3 linux/amd64

Does this issue reproduce with the latest release?
yes
What did you do?
Given:
-- cue.mod/module.cue --
module: "example.com"
-- other/other.cue --
package other

#Type: {
    _short?: string
    if _short != _|_ {
        _veryLongLabel: _short
    }
    ...
}
-- root.cue --
package root

import other "example.com/other"

#Type: {
    _short?: string
    if _short != _|_ {
        veryLongLabel: _short
    }
    ...
}

Alice: #Type & {
    _short: "Alice"
}

Bob: other.#Type & {
    _short: "Bob"
}

What did you expect to see?
Alice: {
    veryLonglabel: "Alice"
}
Bob: {
    veryLonglabel: "Bob"
}

What did you see instead?
Alice: {
    veryLonglabel: "Alice"
}
Bob: {}

This issue poped up in the discussion https://github.com/cue-lang/cue/discussions/1804

This is expected, hidden fields are not accessible across package boundaries

Is there anyway to make them accessible?

You can probably use # to have the desired effect

Thanks. Closing


Edit Tab on Published or Unpublished Campaign Runs Inaccessible

Sample URL:
https://www.dosomething.org/node/5429/edit (logged in as admin)
Output from web-server logs for request:
Oct  1 10:17:37 ip-10-100-35-11 drupal: https://www.dosomething.org|1443709057|php|<IP_ADDRESS>|https://www.dosomething.org/node/5429/edit%7Chttps://www.dosomething.org/node/5356/edit%7C1193105%7C%7CRecoverable||0||Notice: Trying to get property of non-object in dosomething_user_init() (line 49 of /var/www/www.dosomething.org/releases/20150930202159/lib/modules/dosomething/dosomething_user/dosomething_user.module).
Oct  1 10:17:37 ip-10-100-35-11 drupal: https://www.dosomething.org|1443709057|access denied|<IP_ADDRESS>|https://www.dosomething.org/node/5429/edit%7Chttps://www.dosomething.org/node/5356/edit%7C1193105%7C%7CRecoverable||0||node/5429/edit|https://www.dosomething.org/node/5356/edit|1193105||Recoverable
Oct  1 10:17:37 ip-10-100-35-11 drupal: https://www.dosomething.org|1443709057|php|<IP_ADDRESS>|https://www.dosomething.org/node/5429/edit%7Chttps://www.dosomething.org/node/5356/edit%7C1193105%7C%7CRecoverable||0||Notice: Undefined index: country in dosomething_global_get_countries() (line 193 of /var/www/www.dosomething.org/releases/20150930202159/lib/modules/dosomething/dosomething_global/dosomething_global.module).

happening on thor as well, seems to be a bug

fun fact this one works... https://www.dosomething.org/node/5828/edit

A couple of additions:
-- seems that Lizzy tried to access the /edit of a campaign and saw the same error, but then shortly after was able to access the /edit tab
-- based on this list (https://www.dosomething.org/admin/content/search?title=&type=campaign_run&field_cause_tid=All&status=All) the /edit of the last three runs created can be accessed, but nothing prior

Only happening on Thor and Prod, not happening on staging or local.

@mikefantini talked with @sheyd -- can we pause this until we update the Thor DB with the latest snapshot? we think its related

@deadlybutter even though it's also happening on prod?

its not always happening on prod, if you click on the most recent campaign runs /edit works fine

@deadlybutter once you knock out your Global 1.0 issues can you please take a look at this one now that thor has production db again?
cc: @namimody @angaither


Create more uscase scripts

Would be nice to have more predefined scripts for popular P2P actions like:

wireguard
scp
rsync
ssh

Would be nice to have more predefined scripts for popular P2P actions

That's a great idea. I really hope that the community will join in its implementation. But I don't have a lot of time, so I try to spend it on core development, but I will replenish the collection as much as possible. It may be worth opening a separate submodule for this.


Getters and setters added as class attributes, not @property methods

These pymethods
#[pymethods]
impl Foo {
    #[getter]
    pub fn readwrite(slf: PyRef<'_, Self>) -> String {
        String::new()
    }

    #[setter]
    pub fn set_readwrite<'py>(mut slf: PyRefMut<'py, Self>, data: &str) {}

    #[getter]
    pub fn readonly(slf: PyRef<'_, Self>) -> String {
        String::new()
    }
}

should translate to
class Foo:
    @property
    def readwrite(self) -> str: ...

    @readwrite.setter
    def readwrite(self, data: str): ...

    @property
    def readonly(self) -> str: ...

Instead, they translate to static class attributes without getter/setter logic.

Also would mean you could generate docstrings for them, right now they don't seem to go anywhere.


Handle HMR on Blade files

Last time I tried it didn't seem to work. Apparently chokidar is configured to watch the entire project from the root (except node_modules and .git), so it might be the HMR handler that decides not to do a full reload.
Maybe adding a handler for this in the plugin is needed.
Reference:

https://github.com/vitejs/vite/blob/9af1517e47f66bb793447d998cf1a4d2fa3ef758/packages/vite/src/node/server/hmr.ts#L38

Does it also reload the page when I’m updating a blade view?

Yes, it does a full-page reload when you edit .blade.php files. For other files, it is handled by Vite, so if you edit a Vue file it does a partial reload like expected.