Help with merge_samples_to_pseudobulk rule and reproducing published results

Hello,
First of all, thank you for the very nice tool you have created; it has been very useful in my research!
I would like to ask for your assistance with a couple of issues I've encountered:
Rule merge_samples_to_pseudobulk
I'm noticing that the input for this rule is a file with the sample name repeated multiple times (corresponding to the number of batches, in my case 297). This seems incorrect, as I believe there should only be one input file when there is only one sample (name_sample.bam) and the output should be name_sample.merged.bam, which would be the same as the input in cases where there is only one sample.
This current behavior not only leads to potential discrepancies in results but also significantly slows down the pipeline. To circumvent this, I've been skipping this rule, but I realize this won't be feasible when processing multiple samples together.
Do you also have this problem when running the pipeline?
Reproducing Results from N87 Cancer Cell Line
I have run the pipeline on data from the N87 cancer cell line, as done in your published paper. However, I am not obtaining the same variants as those listed in your supplementary table, despite using the default parameters for mutation calling.
I've attached the final genotypes.txt file I obtained for this N87 sample. Could you please verify if this matches your results? If not, do you have any suggestion for potential reasons for the discrepancy?
Additionally, your paper mentions validation through clonal analysis. Could you share the clonal annotations for this dataset? Ideally, a dictionary with each of the four clones as keys and a list of corresponding cell barcodes would be extremely helpful.
If you need any further clarification to address these issues, please let me know.
Thank you in advance for your time and help,
Sara Costa
genotypes.txt

Hi Sara,
Thanks for your email.  At first look, I think you've identified some issues.  Let me take a closer look at this, and I'll get back to you.
Jeff

On Jun 28, 2024, at 6:09 AM, saracosta261299 @.***> wrote:
Hello,
First of all, thank you for the very nice tool you have created; it has been very useful in my research!
I would like to ask for your assistance with a couple of issues I've encountered:
Rule merge_samples_to_pseudobulk
I'm noticing that the input for this rule is a file with the sample name repeated multiple times (corresponding to the number of batches, in my case 297). This seems incorrect, as I believe there should only be one input file when there is only one sample (name_sample.bam) and the output should be name_sample.merged.bam, which would be the same as the input in cases where there is only one sample.
This current behavior not only leads to potential discrepancies in results but also significantly slows down the pipeline. To circumvent this, I've been skipping this rule, but I realize this won't be feasible when processing multiple samples together.
Do you also have this problem when running the pipeline?
Reproducing Results from N87 Cancer Cell Line
I have run the pipeline on data from the N87 cancer cell line, as done in your published paper. However, I am not obtaining the same variants as those listed in your supplementary table, despite using the default parameters for mutation calling.
I've attached the final genotypes.txt file I obtained for this N87 sample. Could you please verify if this matches your results? If not, do you have any suggestion for potential reasons for the discrepancy?
Additionally, your paper mentions validation through clonal analysis. Could you share the clonal annotations for this dataset? Ideally, a dictionary with each of the four clones as keys and a list of corresponding cell barcodes would be extremely helpful.
If you need any further clarification to address these issues, please let me know.
Thank you in advance for your time and help,
Sara Costa
genotypes.txt
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message ID: @.***>

Hi Jeff,
Thank you so much for your answer and for taking the time to help me.
I would like to update you on the first thing I mentioned, about the
validation of the mutations.
I realized I used in my analysis a reference genome different from the one
you used (CRCh38 instead of CRCh37).
Because of that, I couldn't find the same mutations as you did. I converted
the positions from one genome to the other and this time I was able to find
3 of the 4 positions you show in the supplementary table for the NC87 line,
when calling the mutations in pseudobulk.
However, after filtering and assessing the mutations at the single cell
level, only 1 of the 3 initial mutations remain in the final genotypes.txt
file... I used the default filtering parameters you have in the snakemake
you provide, so that is a bit odd.
Let me know what you think and if I can provide you with any other
information.
Also, if it would be easier to have a short video call and discuss these
issues, I am available for that.
Sara
On Fri, 5 Jul 2024 at 23:46, Jeffrey Chang @.***> wrote:

Hi Sara,
Thanks for your email. At first look, I think you've identified some
issues. Let me take a closer look at this, and I'll get back to you.
Jeff

On Jun 28, 2024, at 6:09 AM, saracosta261299 @.***> wrote:
Hello,
First of all, thank you for the very nice tool you have created; it has
been very useful in my research!
I would like to ask for your assistance with a couple of issues I've
encountered:
Rule merge_samples_to_pseudobulk
I'm noticing that the input for this rule is a file with the sample name
repeated multiple times (corresponding to the number of batches, in my case
297). This seems incorrect, as I believe there should only be one input
file when there is only one sample (name_sample.bam) and the output should
be name_sample.merged.bam, which would be the same as the input in cases
where there is only one sample.
This current behavior not only leads to potential discrepancies in
results but also significantly slows down the pipeline. To circumvent this,
I've been skipping this rule, but I realize this won't be feasible when
processing multiple samples together.
Do you also have this problem when running the pipeline?
Reproducing Results from N87 Cancer Cell Line
I have run the pipeline on data from the N87 cancer cell line, as done
in your published paper. However, I am not obtaining the same variants as
those listed in your supplementary table, despite using the default
parameters for mutation calling.
I've attached the final genotypes.txt file I obtained for this N87
sample. Could you please verify if this matches your results? If not, do
you have any suggestion for potential reasons for the discrepancy?
Additionally, your paper mentions validation through clonal analysis.
Could you share the clonal annotations for this dataset? Ideally, a
dictionary with each of the four clones as keys and a list of corresponding
cell barcodes would be extremely helpful.
If you need any further clarification to address these issues, please
let me know.
Thank you in advance for your time and help,
Sara Costa
genotypes.txt
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

—
Reply to this email directly, view it on GitHub
https://github.com/U54Bioinformatics/PhylinSic_Project/issues/3#issuecomment-2211445963,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BCTGXAOI7OCNBBR5LTIWOLDZK4H3LAVCNFSM6AAAAABKBU336SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJRGQ2DKOJWGM
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Thanks for the information.
I have looked into the first issue, with merging to pseudobulk.  There was a bug where samples with multiple batches were being replicated.  I have fixed this, and the version in the github should be merging correctly now.  Please update, and I apologize for the problems.
I will take a look at the N87 issue.  The scientist who did the original analysis has left the lab, so bear with me as I try to recreate this analysis.
Thanks,
Jeff

On Jul 9, 2024, at 3:18 AM, saracosta261299 @.***> wrote:
Hi Jeff,
Thank you so much for your answer and for taking the time to help me.
I would like to update you on the first thing I mentioned, about the
validation of the mutations.
I realized I used in my analysis a reference genome different from the one
you used (CRCh38 instead of CRCh37).
Because of that, I couldn't find the same mutations as you did. I converted
the positions from one genome to the other and this time I was able to find
3 of the 4 positions you show in the supplementary table for the NC87 line,
when calling the mutations in pseudobulk.
However, after filtering and assessing the mutations at the single cell
level, only 1 of the 3 initial mutations remain in the final genotypes.txt
file... I used the default filtering parameters you have in the snakemake
you provide, so that is a bit odd.
Let me know what you think and if I can provide you with any other
information.
Also, if it would be easier to have a short video call and discuss these
issues, I am available for that.
Sara
On Fri, 5 Jul 2024 at 23:46, Jeffrey Chang @.***> wrote:

Hi Sara,
Thanks for your email. At first look, I think you've identified some
issues. Let me take a closer look at this, and I'll get back to you.
Jeff

On Jun 28, 2024, at 6:09 AM, saracosta261299 @.***> wrote:
Hello,
First of all, thank you for the very nice tool you have created; it has
been very useful in my research!
I would like to ask for your assistance with a couple of issues I've
encountered:
Rule merge_samples_to_pseudobulk
I'm noticing that the input for this rule is a file with the sample name
repeated multiple times (corresponding to the number of batches, in my case
297). This seems incorrect, as I believe there should only be one input
file when there is only one sample (name_sample.bam) and the output should
be name_sample.merged.bam, which would be the same as the input in cases
where there is only one sample.
This current behavior not only leads to potential discrepancies in
results but also significantly slows down the pipeline. To circumvent this,
I've been skipping this rule, but I realize this won't be feasible when
processing multiple samples together.
Do you also have this problem when running the pipeline?
Reproducing Results from N87 Cancer Cell Line
I have run the pipeline on data from the N87 cancer cell line, as done
in your published paper. However, I am not obtaining the same variants as
those listed in your supplementary table, despite using the default
parameters for mutation calling.
I've attached the final genotypes.txt file I obtained for this N87
sample. Could you please verify if this matches your results? If not, do
you have any suggestion for potential reasons for the discrepancy?
Additionally, your paper mentions validation through clonal analysis.
Could you share the clonal annotations for this dataset? Ideally, a
dictionary with each of the four clones as keys and a list of corresponding
cell barcodes would be extremely helpful.
If you need any further clarification to address these issues, please
let me know.
Thank you in advance for your time and help,
Sara Costa
genotypes.txt
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

—
Reply to this email directly, view it on GitHub
https://github.com/U54Bioinformatics/PhylinSic_Project/issues/3#issuecomment-2211445963,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BCTGXAOI7OCNBBR5LTIWOLDZK4H3LAVCNFSM6AAAAABKBU336SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJRGQ2DKOJWGM
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: @.***>

Hi Jeff,
Thank you for the update and for fixing the bug! I appreciate your efforts
and will wait for the remaining updates as you work to recreate the
analysis. I understand it can take some time.
Please let me know if you need any additional information in the meantime.
Thanks,
Sara
On Tue, 9 Jul 2024 at 18:31, Jeffrey Chang @.***> wrote:

Thanks for the information.
I have looked into the first issue, with merging to pseudobulk. There was
a bug where samples with multiple batches were being replicated. I have
fixed this, and the version in the github should be merging correctly now.
Please update, and I apologize for the problems.
I will take a look at the N87 issue. The scientist who did the original
analysis has left the lab, so bear with me as I try to recreate this
analysis.
Thanks,
Jeff

On Jul 9, 2024, at 3:18 AM, saracosta261299 @.***> wrote:
Hi Jeff,
Thank you so much for your answer and for taking the time to help me.
I would like to update you on the first thing I mentioned, about the
validation of the mutations.
I realized I used in my analysis a reference genome different from the
one
you used (CRCh38 instead of CRCh37).
Because of that, I couldn't find the same mutations as you did. I
converted
the positions from one genome to the other and this time I was able to
find
3 of the 4 positions you show in the supplementary table for the NC87
line,
when calling the mutations in pseudobulk.
However, after filtering and assessing the mutations at the single cell
level, only 1 of the 3 initial mutations remain in the final
genotypes.txt
file... I used the default filtering parameters you have in the
snakemake
you provide, so that is a bit odd.
Let me know what you think and if I can provide you with any other
information.
Also, if it would be easier to have a short video call and discuss these
issues, I am available for that.
Sara
On Fri, 5 Jul 2024 at 23:46, Jeffrey Chang @.***> wrote:

Hi Sara,
Thanks for your email. At first look, I think you've identified some
issues. Let me take a closer look at this, and I'll get back to you.
Jeff

On Jun 28, 2024, at 6:09 AM, saracosta261299 @.***> wrote:
Hello,
First of all, thank you for the very nice tool you have created; it
has
been very useful in my research!
I would like to ask for your assistance with a couple of issues I've
encountered:
Rule merge_samples_to_pseudobulk
I'm noticing that the input for this rule is a file with the sample
name
repeated multiple times (corresponding to the number of batches, in my
case
297). This seems incorrect, as I believe there should only be one
input
file when there is only one sample (name_sample.bam) and the output
should
be name_sample.merged.bam, which would be the same as the input in
cases
where there is only one sample.
This current behavior not only leads to potential discrepancies in
results but also significantly slows down the pipeline. To circumvent
this,
I've been skipping this rule, but I realize this won't be feasible
when
processing multiple samples together.
Do you also have this problem when running the pipeline?
Reproducing Results from N87 Cancer Cell Line
I have run the pipeline on data from the N87 cancer cell line, as
done
in your published paper. However, I am not obtaining the same variants
as
those listed in your supplementary table, despite using the default
parameters for mutation calling.
I've attached the final genotypes.txt file I obtained for this N87
sample. Could you please verify if this matches your results? If not,
do
you have any suggestion for potential reasons for the discrepancy?
Additionally, your paper mentions validation through clonal
analysis.
Could you share the clonal annotations for this dataset? Ideally, a
dictionary with each of the four clones as keys and a list of
corresponding
cell barcodes would be extremely helpful.
If you need any further clarification to address these issues,
please
let me know.
Thank you in advance for your time and help,
Sara Costa
genotypes.txt
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this
thread.Message
ID: @.***>

—
Reply to this email directly, view it on GitHub
<
https://github.com/U54Bioinformatics/PhylinSic_Project/issues/3#issuecomment-2211445963>,

or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/BCTGXAOI7OCNBBR5LTIWOLDZK4H3LAVCNFSM6AAAAABKBU336SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJRGQ2DKOJWGM>

.
You are receiving this because you are subscribed to this
thread.Message
ID: @.***>

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: @.***>

—
Reply to this email directly, view it on GitHub
https://github.com/U54Bioinformatics/PhylinSic_Project/issues/3#issuecomment-2218146395,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BCTGXAIPG5YQLLPWWGMF32LZLQF65AVCNFSM6AAAAABKBU336SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJYGE2DMMZZGU
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>


About the details of datasets

Hi,
I've downloaded the all_datasets.zip from cloud you mentioned in readme, while where can I get detailed information about the dataset? For example, the traffic.csv under traffic folder seems nor attached readme file neither ref link, but every line is composed from hundreds fields.
Hope to get reply soon. Thanks a lot.
David

Hi, we include the dataset description in our previous paper: Autoformer
https://arxiv.org/pdf/2106.13008


audit complains about linking to system frameworks

Testing of the salt update PR gives a very weird audit error:
==> brew audit salt --online --git --skip-style
==> FAILED
Error: 1 problem in 1 formula detected
Missing libraries:
  unexpected (/System/Library/Frameworks/AuthenticationServices.framework/Versions/A/AuthenticationServices)
  unexpected (/System/Library/Frameworks/OSLog.framework/Versions/A/OSLog)
  unexpected (/System/Library/Frameworks/PushKit.framework/Versions/A/PushKit)
  unexpected (/System/Library/Frameworks/Speech.framework/Versions/A/Speech)
  unexpected (/System/Library/Frameworks/SystemExtensions.framework/Versions/A/SystemExtensions)
Unexpected missing library linkage detected
salt:
  * salt has broken dynamic library links:

I suspect something is wrong with the audit, because it's perfectly legitimate for a formula to link against system frameworks. If there really is something wrong here, the error message is not clear enough that I understand what the problem is.

I suspect something is wrong with the audit, because it's perfectly legitimate for a formula to link against system frameworks.

Are you sure those system frameworks exist?

Looking at the code: they do not.


Publish version 4.0.0 to npm

Could you publish the new version 4.0.0 to npm? That would be great.

Until 4.0.0 is published to npm, you can install it via GitHub:
npm i jgraph/mxgraph#v4.0.0


Switch to using GLM.GlmResp, not vectors

Changes Unknown when pulling bcd559932cbd43d6fc3d4117a36036e5d4b18b9f on GlmResp into ** on master**.

Decided not to wait on Travis OSX tests.  They seem very slow today.

Changes Unknown when pulling 78d407601297b1a453a395ac390eb2b573a38b19 on GlmResp into ** on master**.


PyTorch: pretrained models

Hi,
thanks for the releasing the TensorFlow and PyTorch code for your Transformer-XL :heart:
I would like to ask, if you plan to provide some pre-trained models for the PyTorch implementation? I was only able to find the TensorFlow checkpoints...
Thanks in advance,
Stefan

Excuse me! Have you trained transformer_xl model on Chinese corpus?

Excuse me! Have you trained transformer_xl model on Chinese corpus?

+1, I look forward to your sharing the Chinese pre-trained model!


Clarification of use of crypto?

Hi  There,
Really good idea for a protocol but I'm somewhat concerned by how your apparent useage of AEAD in the protocol.
1/ I can see a static private_key variable apparently included in both the server & client source, if it really is a private key then sharing it like this means its no longer private.
2/ When the private_key variable is used in crypto_aead_chacha20poly1305_encrypt function as parameter key it is I believe is expecting a symmetric key and not an asymmetric private key.

Thanks.
In a real application, the private key is a symmetric key known only to to the web backend (which generates the connect token), and the dedicated server instances.
At no point does the client know the private key.
Unfortunately, due to a lack of time, the example client.c also knows the private key. This is not meant as an example of usage. Normally, the client would request a connect token over REST call, and not know the private key, and pass that connect token into the connect call (and be unable to generate, modify or inspect it).
cheers


Dialog: Full screen & Action sheet support

Closes

https://github.com/github/primer/issues/2414
https://github.com/github/primer/issues/2413

Note: I've chosen not to tackle the issues mentioned here to avoid causing any disruptive changes.

Changelog

New
1⃣ action-sheet dialog
We've implemented a range slider to easily adjust the size of the dialog using only a keyboard, enhancing accessibility. We've used a similar approach for our SplitPageLayout resizer. I've also implemented reduced-motion support.
https://github.com/primer/react/assets/980622/c55fc0cb-1cea-4dab-94cb-e9239a870431
https://github.com/primer/react/assets/980622/babadea0-8794-40b1-a812-cf9b6f7617f8
2⃣  full-screen dialog
This is quite simple, as it closely resembles the existing dialog behavior, with the only difference being that it's full-screen.
https://github.com/primer/react/assets/980622/eb043543-8a06-4854-bf0b-fc6c446213d2
Changed

Storybook features always display dialogs in an open state so that snapshots can effectively capture any changes.
Removed the ability to change props in features for simplicity and consistency with other components
In the example called 'WithCustomRenderers', I've placed the header, content, and footer components inside the 'WithCustomRenderers' component. This change improves readability, especially when highlighting the code.

Removed

Rollout strategy

[ ] Patch release
[x] Minor release
[ ] Major release; if selected, include a written rollout or migration plan
[ ] None; if selected, include a brief description as to why

Testing & Reviewing

Merge checklist

[ ] Added/updated tests
[x] Added/updated documentation
[x] Added/updated previews (Storybook)
[x] Changes are SSR compatible
[x] Tested in Chrome
[x] Tested in Firefox
[x] Tested in Safari
[x] Tested in Edge
[ ] (GitHub staff only) Integration tests pass at github/github (Learn more about how to run integration tests)

Hey @maximedegreve! Sorry for the delay on this, was spending some time reading the related issues to get the context to understand the changes 👀
Just wanted to check-in with you to make sure I understood things right, the corresponding issues are talking about two things:

The ability for a dialog to go full screen (the full-screen type in this PR)
A new presentation for a dialog called an action sheet (the action-sheet type in this PR)

And it looks we're supporting using these types responsively through the type prop on Dialog to get something like:
<Dialog type={['full-screen', 'default']} />

Hope that makes sense 🤔 Let me know if I'm missing anything.
With this in mind, I had a couple of questions come up:

Are these new types restricted to any specific viewport sizes or should they be visible across viewports?
Does action sheet in this PR correspond to bottom sheets in the design guidance or are they a separate type?

@maximedegreve nice, makes sense. Would it be helpful to limit usage of these variants to only specific breakpoints (if feasible) so that these are only used in appropriate contexts? Or even just roll these into the components themselves as default behavior? (This would like require a specific component for BottomSheet but could be worth it)

@joshblack 💯 feasible. At first, I thought the same. But I have concerns about limiting it like this. Some dialogs already are playing with a full screen mode right now in desktop and bottom sheets are also very similar to side sheets. I would assume side sheets would be an addition to the future. Will our API become inconsistent and harder to read if we allow side sheets on desktop but not bottom sheets? In this case we would need type: sidesheet bottomsheetOnMobile: true or type:default bottomSheetOnMobile:true.  In all our other components we use the {narrow: value, default: value...} pattern.

I do believe that components using the dialog, like SelectPanel, might prefer these built-in defaults though.
It could be a good idea to check with the rest of team if we want to maintain flexibility at the component level here or not.

Also, we need to add tests: unit tests (if applicable) and visual regression tests.

I'm working on fixing your Jest issues, but it also looks like some styles broke.

Yeah that was intentionally because I thought you aren't swiping it away.


Where does protobuf-codegen output my files?

Hi, I'm looking to generate rust from several (>20) proto files. However I cannot figure out where they are getting generated. Please advise
Here is my build.rs:
fn main() {

    protobuf_codegen::Codegen::new()
        // Use `protoc` parser, optional.
        .protoc()
        // Use `protoc-bin-vendored` bundled protoc command, optional.
        // .protoc_path(&protobuf_gen::protobuf_gen_rust::protoc_bin_path().unwrap())
        // All inputs and imports from the inputs must reside in `includes` directories.
        .includes(&["protos"])
        // Inputs must reside in some of include paths.
        .input("protos/osmosis/gamm/v1beta1/tx.proto")
        .input("protos/osmosis/gamm/v1beta1/query.proto")
        // ... several other inputs removed
        // Specify output directory relative to Cargo output directory.
        .cargo_out_dir("outdir")
        .run_from_script();
}

All of the protos are in the protos directory, and cargo build runs without failing.
Thanks

where they are getting generated

Somewhere in target.
Can be used something like this:
include!(concat!(env!("OUT_DIR"), "/outdir/mod.rs"));

hmm, I'm still having trouble. here is my updated code
fn main() {

    protobuf_codegen::Codegen::new()
        // Use `protoc` parser, optional.
        .protoc()
        // Use `protoc-bin-vendored` bundled protoc command, optional.
        // .protoc_path(&protobuf_gen::protobuf_gen_rust::protoc_bin_path().unwrap())
        // All inputs and imports from the inputs must reside in `includes` directories.
        .includes(&["protos"])
        // Inputs must reside in some of include paths.
        .input("protos/osmosis/gamm/v1beta1/tx.proto")
        .input("protos/osmosis/gamm/v1beta1/query.proto")
        // Specify output directory relative to Cargo output directory.
        .cargo_out_dir("./target")
        .run_from_script();
}

Trust me I've searched my directory up and down, haven't found the generated files. I've changed the cargo_out_dir to a bunch of different names, sometimes the command fails if I use nested directories. Any troubleshooting advice would be greatly appreciated.

I also can't seem to find the generated files. Any updates here?

I was facing the same problem...
. cargo_out_dir("src") will create a random folder under `./target/release/build/proto-xxxx/out/src/mod.rs
instead use .out_dir("src") it will generate files under ./src/mod.rs


Vechicles don't despawn when hit by water

Describe the problem
Steam user Francis reported:

Vechicles don't despawn when hit by water

This might be unwanted side-effect of no-despawn feature?
Steps to reproduce

Plunge vehicle in to water, eg. tsunami or bulldozed hydro dam
Vehicle gets swept away by water, and should subsequently despawn - but don't

Log files
n/a
Savegame?
not provided
Screenshots?
n/a
Notes or questions?

Question is if they normally despawn while floating (I doubt, reading the code) or when they stop floating -> will be hard to guess if they floated before? I see that SimulationStep for floating only increases blockCounter(main variable used to check if should despawn) and calculates sway/velocity etc.. in general makes vehicle float. Rest of the code is ignored.
Anyways it require simple test build to check what is going on and if there is an issue, test if possible fix works (will be one line of code)

I know in vanilla they do eventually despawn, but never really paid attention to how that was working. I vaguely recollect that they floated for a while and then despawned whilst still in water, or if they somehow ended up back on dry land but not near a road (thus unable to reroute to destination).

Cannot reproduce the issue.

I've tested that in isolation (I created a road, spawned few vehicles then pushed water there).
Vehicles start floating (m_flags2 == Floating) and m_blocked counter goes to 100 then vehicle gets despawned, regardless of TM:PE despawning settings. They are not despawned when they are waiting in front of flooded segments, but that's probably ok.
Sidenote: I see in the code we support despawning when vehicle is floating:
https://github.com/CitiesSkylinesMods/TMPE/blob/3ba62f38035439670716430214e36b45e9607a12/TLM/TLM/Manager/Impl/VehicleBehaviorManager.cs#L1717-L1722
and maxBlockCounter holds value of the limit before despawn
https://github.com/CitiesSkylinesMods/TMPE/blob/3ba62f38035439670716430214e36b45e9607a12/TLM/TLM/Patch/_VehicleAI/_CarAI/SimulationStepPatch.cs#L215-L228
There is a mod on the workshop that claims it is compatible with TMPE but also patches the code for despawning. Maybe that's the problem?

Was it the heavy outside traffic mod or something like that?

My little thingy for spawning "dummy" vehicles on demand 😉 Select source lane, target lane (anywhere on the map), vehicle type, type in number of vehicles and.. click start to begin spawning the horde of brainless lemmings 😂

Sry, I meant this:

There is a mod on the workshop that claims it is compatible with TMPE but also patches the code for despawning. Maybe that's the problem?

I think it was this https://steamcommunity.com/sharedfiles/filedetails/?id=2306554537 but I haven't tested it fully to say it is or it is not compatible 🤷♂

also:

So unlikely to be that mod that's causing what the user reported (I've also asked the original reporting user to confirm what was happening).
IMO we can probably close this issue for now?

Yeah, we can close it, I have no other ideas what it could went wrong or how to break it 🤷♂

So, I just discovered that trying to update hydro dam roads (specifically, unlocking them with Network Multitool) causes castropic failure of the dam, creating an inland tidal wave that wiped out my city and port - flushing lots of vehicles in to the river/sea in the processess. They did in fact despawn as expected with the block counter, so I can confirm this was a non-issue.


Allow apex domain listeners

What
If someone adds an apex domain as a listener in the gateway, we will fail to setup DNS for that as apex domains can only have A records.
Solution
Allow the user to specify an IP for any apex domains in the DNSPolicy which we will use as the A record value. The user can then for example setup a 301 redirect to re-route traffic to a subdomain where DNS Load Balancing has been applied

https://github.com/Kuadrant/multicluster-gateway-controller/blob/main/docs/proposals/DNSRecordStructure.md#apex-domains

This issue is stale because it has been open for 60 days with no activity.

This solution allows for only 1 apex domain currently. However when we add support for a DNSPolicy to target a listener, it will then be able to used for multiple apex domans in a gateway

https://github.com/Kuadrant/multicluster-gateway-controller/issues/358

This issue is stale because it has been open for 60 days with no activity.

This issue was closed because it has been inactive for 30 days since being marked as stale.


Caravan top lvs

See commit notes.  Merging probably fails on the mgmt_protect layout and subcells.  Please cherry-pick the rest of the files for merging.

Or maybe it doesn't have issues with those files.  It only seems to be complaining about openlane/Makefile.


Update WyzeHub repository

Previous author is no longer supporting, updated code exists in repository.

PullRequest to HPM are not regularly reviewed in this DCMEGLIO repository. HPM has moved to HubitatCommunity and your PR should have been created there. Having stumbled onto this PR, I've manually merged it.


Comment out docker PackageNotInstalled preflight checks

Reopening https://github.com/apprenda/kismatic/issues/672

LGTM


Optimize the way we show tablet info

Enhancement
Considering to put tablet info into information schema table, so that we can show tablet info in a standard manner which is easier to show and filter.

fixed by #18457


fix an error

how to commit the code?

Description of PR
How was this patch tested?
For code changes:

[ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m  3s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

_ trunk Compile Tests _

+1 :green_heart:
shadedclient
31m 51s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
shadedclient
19m 38s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
asflicense
0m 47s

The patch does not generate ASF License warnings.

55m 20s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4860/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4860

Optional Tests
dupname asflicense codespell detsecrets

uname
Linux c1c6ad802392 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / b05a3322a9e9698063f8fd8c744c969324a57314

Max. process+thread count
562 (vs. ulimit of 5500)

modules
C: . U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4860/1/console

versions
git=2.25.1 maven=3.6.3

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

doesn't make sense. Please create a Jira first and then shoot a PR


[TASK] Fix admin authorization endpoints

Description
There are several admin authorization APIs that should have their paths updated to be consistent with the paths of other admin authorization APIs. Also, the admin roles and groups GET APIs should include permissions and roles and permissions, respectively, in their response bodies.
The login-related APIs on the system subrouter should be removed as well, since system admins are able to use the login APIs on the admin subrouter.
Acceptance Criteria

[ ] Update /admin/application/group/accounts APIs to /admin/application/groups/{id}/accounts and remove group_id from request bodies
[ ] Update /admin/application/role/permissions to /admin/application/roles/{id}/permissions and remove role_id from request body
[ ] Update /admin/application/account/permissions APIs to /admin/application/accounts/{id}/permissions and remove account_id from request bodies
[ ] Update /admin/application/account/roles APIs to /admin/application/accounts/{id}/roles and remove account_id from request bodies
[ ] Update GET /admin/application/roles response to include permissions associated with each role
[ ] Update GET /admin/application/groups response to include roles and permissions associated with each group
[ ] Remove /system/auth/* APIs

Waiting for #365 to be resolved by #487.


Invalid max bounds

Im trying to use the glMaxBounds directive but getting an Invalid max bounds error.
I face this error both while trying to use it as per the wiki
<mapboxgl gl-max-bounds="[[-73.9876, 40.7661], [-73.9397, 40.8002]]"></mapboxgl>
, and with the controller scope
<mapboxgl gl-max-bounds="bounds"></mapboxgl>
$scope.bounds = [[-73.9876, 40.7661], [-73.9397, 40.8002]]
When debugging it looks like maxBounds is set to undefined in mapboxGlMaxBoundsDirectiveLink().

Do you have a working example for this directive?

Off-topic: Is there a way to do what map.fitBounds() does using this project? If not, how could I get an instance of the map on my controller?

Hi @K1N5L4Y3R, I solved the problem. I forgot to insert the watcher for glMaxBounds directive.
About your off-topic, you can use mapboxglMapsData factory. You have to do something like:
<mapbox-gl id="myMap"></mapbox-gl>

angular.controller('MyController', ['mapboxglMapsData', function (mapboxglMapsData) {
   var myMap = mapboxglMapsData.getMapById("myMap");
   //....
}]);

Tell me if you need something more!
Thank you very much!

Wow, that was quick. Keep up the good work! Thanks


Add support for tuple type configurable variables

Purpose
$subject
Fixes #35024
Fixes #35305
Approach
This PR includes the following features.

Supporting tuple type configurable variables via TOML configuration.
Supporting structured type configurable variables with TOML inline tables & array of inline tables.

Samples

Tuple type support
A tuple value can be provided as a TOML array in the Config.toml file.
Eg. :
In main.bal,

type Person record {|
    string name;
|};

configurable [int, string, float, decimal, byte, boolean] simpleTuple = ?;
configurable [int[], string[][]] arrayTuple = ?;
configurable [string, map<anydata>] mapTuple = ?;
configurable [int, Person] recordTuple = ?;
configurable [string, table<map<string>>] tableTuple = ?;
configurable [string, table<Person>] tableTuple2 = ?;
configurable [string, map<string>[]] mapArrayTuple = ?;
configurable [int, string, int...] restTuple = ?;

In Config.toml
simpleTuple = [278, "string", 2.3, 4.5, 2, true]

arrayTuple = [[1, 3, 5, 7, 9], [["apple", "banana", "orange"], ["ten", "twenty", "thirty"]]]

mapTuple = [
    "Foo",
    { name = "Baz Qux", age = 22, salary = [45000.0, 150000.0], address = { number = 20, road = "Palm Grove", city = "Colombo 3" }, accounts = [{ id = "1" }, { id = "2" }] },
]

tableTuple = [
    "Bar",
    [
        { a = "a" },
        { b = "b", d = "d" },
        { c = "c" }
    ] ]

tableTuple2 = [
    "Hello",
    [
        { name = "John Doe"},
        { name = "Jane Doe"},
    ],
]

mapArrayTuple = [
    "Jane",
    [
        { name = "John Doe", age = "22", subject = "English" },
        { name = "Jane Doe", age = "27", subject = "Maths" },
    ]
]

restTuple = [1, "foo", 2, 3, 4, 5]

TOML in-line table support
TOML in-line table is supported to provide values of map, and record types.
Eg.:
In main.bal

type Person record {|
    string name;
|};

configurable map<anydata> mapVar = ?;
configurable Person recordVar = ?;
configurable table<map<int>> tableVar = ?;
configurable Person[] recordArrayVar = ?;

In Config.toml,
mapVar = { a = "a", b = 2, c = 3.4, d  = [1,2,3]}
recordVar = {name = "Hinduja"}
tableVar = [{a = 1, b = 2}, {c = 3}, {d = 4, e = 5, f = 6}]
recordArrayVar = [{name = "Tom"}, {name = "Harry"}]

Check List

[x] Read the Contributing Guide
[ ] Updated Change Log
[ ] Checked Tooling Support (#)
[x] Added necessary tests

[x] Unit Tests
[ ] Spec Conformance Tests
[x] Integration Tests
[ ] Ballerina By Example Tests

[x] Increased Test Coverage
[ ] Added necessary documentation

[ ] API documentation
[ ] Module documentation in Module.md files
[ ] Ballerina By Examples

LGTM. Can we add tests for uncovered lines?

LGTM. Can we add tests for uncovered lines?

The code coverage shows warnings for the method calls of throwTypeIncompatibleError(). This method is a refactored one we call from multiple places and it throws a ConfigException. Due to that, the lines that call this method are marked as partially covered and showing warnings.
I added a few tests for the other warnings.

LGTM. Can we add tests for uncovered lines?

The code coverage shows warnings for the method calls of throwTypeIncompatibleError(). This method is a refactored one we call from multiple places and it throws a ConfigException. Due to that, the lines that call this method are marked as partially covered and showing warnings.
I added a few tests for the other warnings.

And we already have tests for the throwTypeIncompatibleError() cases. But they still show the warnings.


Autocompletion for New function

Added InstanceClass enum to support auto completion for the New function.

Sorry for the delay! I'm catching up on my backlogs.
I don't see an associated issue for this pull request. It's required that new features are filed as issues first, because that way it's easier to get community feedback on ideas. As such, this PR will be closed, but feel free to reopen an issue - it's nothing personal, just policy.
The reason why I haven't already done this sort of thing is that it introduces a dependency between Fusion's source code and the current Roblox API. While this is nice, it also would have to be kept continuously updated. Perhaps this would be something better served by a plugin that can automatically pull the latest Roblox API and add autocomplete suggestions into the script editor using the official APIs.


[Feature request] Avoiding next mail opening on delete

Hi there,
Maybe is it already possible but I couldn't find the option to avoid the opening of the next mail when deleting one.
Have a nice day.

+1

The only workaround is to set the "Layout" to "No Split".


Module reversal revised + new example

I didn’t put - - In loop because I copy pasted, a cardinal sin yes, but I
understood the logic and reviewed it.
Rename vars okay, it’s your driver and it’s imperative the structure and
naming conventions are consistent throughout.
Yes, knowledge being my limitation, keeping it working behind the scenes
was my goal, exposing the enum to the constructor, it’s so messy and I did
say I wasn’t happy with it, I am open to suggestions 😊
“Caching” and doing so at function level is new to me, in what context? The
cast is the best I could come up with given my knowledge of what is even
possible 😊.
On Tue, 26 Feb 2019 at 23:19, Bartosz Bielawski<EMAIL_ADDRESS>wrote:

@bartoszbielawski requested changes on this pull request.
Please check my comments in the code :)
In src/LEDMatrixDriver.cpp
https://github.com/bartoszbielawski/LEDMatrixDriver/pull/25#discussion_r260532105
:

@@ -160,13 +160,32 @@ void LEDMatrixDriver::_sendCommand(uint16_t command)
SPI.endTransaction();
}

+uint8_t LEDMatrixDriver::_reverseBits(uint8_t n)
+{

uint8_t x;
for(auto i = 7; n; ) {

Why don't you put --i; in the for loop?
In src/LEDMatrixDriver.hpp
https://github.com/bartoszbielawski/LEDMatrixDriver/pull/25#discussion_r260532294
:

@@ -41,9 +41,17 @@ class LEDMatrixDriver

public:

enum class flipDirection

I'd prefer to have the type name capitalized.
In src/LEDMatrixDriver.hpp
https://github.com/bartoszbielawski/LEDMatrixDriver/pull/25#discussion_r260532478
:

  const uint8_t N;
  SPISettings spiSettings;

bool modRev = false;

uint8_t moduleFlip;

Now the type internally is the raw one... Ugh, but there's no nice way to
handle that I guess.
In src/LEDMatrixDriver.cpp
https://github.com/bartoszbielawski/LEDMatrixDriver/pull/25#discussion_r260532713
:

void LEDMatrixDriver::_displayRow(uint8_t row)
{
SPI.beginTransaction(spiSettings);
digitalWrite(ssPin, 0);
for (uint16_t d = 0; d < N; d++)
{

uint16_t cmd = ((row + 1) << 8) | frameBuffer[d + row*N];

uint8_t col = frameBuffer[d + row*N];

if ((moduleFlip & static_cast<int>(flipDirection::flipColumn)) != 0)

How about caching if it's reversed or not and not checking it every time?
You can do it once per function instead of doing it for every element.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/bartoszbielawski/LEDMatrixDriver/pull/25#pullrequestreview-208262561,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABtJb_NGuJNPl8OMBC595yhIPOokbEHuks5vRcEGgaJpZM4bQ_Ia
.

--
Sent from my iPhone 6s A1688 iOS 11.0.3

I didn’t put - - In loop because I copy pasted, a cardinal sin yes, but I understood the logic and reviewed it.

It's not about understanding only, it's about making others understand what the code does. Without this one it doesn't look like a standard for loop.

Yes, knowledge being my limitation, keeping it working behind the scenes was my goal, exposing the enum to the constructor, it’s so messy and I did say I wasn’t happy with it, I am open to suggestions 😊
“Caching” and doing so at function level is new to me, in what context? The cast is the best I could come up with given my knowledge of what is even possible 😊.

Okay, maybe I used a wrong word. You can make a const bool hReversed and const bool vReversed members that will be initialized from the enum. Then in the code you can use these constants, no reason to check every time if it's to be inverted or not.

Hi, I have just pushed a new commit to a second branch. I decided to go with a mix of your and mine code.
I have finally dropper the class enums, I found a nice way to solve it (with std::set) but I don't want to bring STL into the code. So it will stay with uint8_t flags and let's call it a day. I have also added an option to invert whole display horizontally, not only separate segments. Feel free to check the new code. If everything's fine I will make an official release.

Sounds good :)
Have you merged into master yet?  I can't see it.
Also do you still have access to my final example? The MarqueeTextClean
example is poorly derrived from the MarqueeText example.
Regards,
James Gohl
*Are you cyber streetwise?*https://www.cyberaware.gov.uk/
On Sat, 16 Mar 2019 at 17:10, Bartosz Bielawski<EMAIL_ADDRESS>wrote:

Hi, I have just pushed a new commit to a second branch. I decided to go
with a mix of your and mine code.
I have finally dropper the class enums, I found a nice way to solve it
(with std::set) but I don't want to bring STL into the code. So it will
stay with uint8_t flags and let's call it a day. I have also added an
option to invert whole display horizontally, not only separate segments.
Feel free to check the new code. If everything's fine I will make an
official release.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/bartoszbielawski/LEDMatrixDriver/pull/25#issuecomment-473566306,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABtJbypDAegsTXMdES4lsupt8lSzLjkCks5vXSWfgaJpZM4bQ_Ia
.

Please include this and delete MarqueeTextClean.ino :)
Regards,
James Gohl
*Are you cyber streetwise?*https://www.cyberaware.gov.uk/
On Sat, 16 Mar 2019 at 19:56, James Gohl<EMAIL_ADDRESS>wrote:

Sounds good :)
Have you merged into master yet?  I can't see it.
Also do you still have access to my final example? The MarqueeTextClean
example is poorly derrived from the MarqueeText example.
Regards,
James Gohl
*Are you cyber streetwise?*https://www.cyberaware.gov.uk/
On Sat, 16 Mar 2019 at 17:10, Bartosz Bielawski<EMAIL_ADDRESS>wrote:

Hi, I have just pushed a new commit to a second branch. I decided to go
with a mix of your and mine code.
I have finally dropper the class enums, I found a nice way to solve it
(with std::set) but I don't want to bring STL into the code. So it will
stay with uint8_t flags and let's call it a day. I have also added an
option to invert whole display horizontally, not only separate segments.
Feel free to check the new code. If everything's fine I will make an
official release.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/bartoszbielawski/LEDMatrixDriver/pull/25#issuecomment-473566306,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABtJbypDAegsTXMdES4lsupt8lSzLjkCks5vXSWfgaJpZM4bQ_Ia
.

Nope, wanted to have it tested by somebody else than me. It's in a separate
branch for now. So it's there but not as the master. So I'm not yet braking
all the people's projects.
BB
On Sat, 16 Mar 2019, 20:56 James Gohl<EMAIL_ADDRESS>wrote:

Sounds good :)
Have you merged into master yet? I can't see it.
Also do you still have access to my final example? The MarqueeTextClean
example is poorly derrived from the MarqueeText example.
Regards,
James Gohl
*Are you cyber streetwise?*https://www.cyberaware.gov.uk/
On Sat, 16 Mar 2019 at 17:10, Bartosz Bielawski<EMAIL_ADDRESS>wrote:

Hi, I have just pushed a new commit to a second branch. I decided to go
with a mix of your and mine code.
I have finally dropper the class enums, I found a nice way to solve it
(with std::set) but I don't want to bring STL into the code. So it will
stay with uint8_t flags and let's call it a day. I have also added an
option to invert whole display horizontally, not only separate segments.
Feel free to check the new code. If everything's fine I will make an
official release.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<
https://github.com/bartoszbielawski/LEDMatrixDriver/pull/25#issuecomment-473566306
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/ABtJbypDAegsTXMdES4lsupt8lSzLjkCks5vXSWfgaJpZM4bQ_Ia
.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/bartoszbielawski/LEDMatrixDriver/pull/25#issuecomment-473580196,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AD2sEAqO10k1Fz3kJmHp-UpfVHml8pDhks5vXUxugaJpZM4bQ_Ia
.


Please invite me to the Recode-Hive GitHub Community Organization

Name
Siraj Khatti
Discord Username (if applicable)
siraj1451
Additional Context
No response

Invitation sent to join the GitHub Organisation. Welcome to the community 🎉
Don't forget after accepting to make it public so it appears on your GitHub profile for everyone else to see. You can do this by finding your name in the GitHub organisation list and change the dropdown to public https://github.com/orgs/recodehive/people


Added additional say commands

Many point and click adventure games say something different each time you interact with a game object. Some games have randomly chosen text and others go through a list of set dialog and have a default after all options have been said.
I have added the following functions to make this easy for Popochiu users to do:

queue_say_random(say_array: PackedStringArray)
say_random(say_array: PackedStringArray)
queue_say_next(say_array: PackedStringArray)
say_next(say_array: PackedStringArray)

Below is an example of usage used while I was testing this:
var use_random_array: PackedStringArray = [
	"First line",
	"Second line",
	"third line"
]

var use_next_array: PackedStringArray = [
	"First line",
	"Second line",
	"third line",
	"Last and default line"
]

func _on_click() -> void:
	await C.player.walk_to_clicked()
	await C.player.face_clicked()
	await C.player.say_random(use_random_array)

func _on_right_click() -> void:
	await C.player.walk_to_clicked()
	await C.player.face_clicked()
	await C.player.say_next(use_next_array)

say_random() will choose a random string from the array that is passed to it and makes sure it does not say the same thing twice in a row.
say_next() will say the first element in the array and remove the first element until only 1 element remains. Then it will say that each time say_next() is used.

I updated this to support emotions as well. If the emo_array is not the same length as the dialog_array then it will append blank strings to the emo_array to make them match. By default, the user does not need to provide any emo values, the same as the say() function.
I have tested this without providing any emo values and by providing an emo_array that does not match the dialog_array size to ensure that it works correctly.
var use_bed_array: PackedStringArray = [
	"First line",
	"Second line",
	"third line"
]

var use_next_array: PackedStringArray = [
	"First line",
	"Second line",
	"third line",
	"Last and default line"
]

var use_emo_array: PackedStringArray = [
	'',
	'',
]

func _on_click() -> void:
	await C.player.walk_to_clicked()
	await C.player.face_clicked()
	await C.player.say_random(use_bed_array, use_emo_array)

func _on_right_click() -> void:
	await C.player.walk_to_clicked()
	await C.player.face_clicked()
	await C.player.say_next(use_next_array, use_emo_array)

Hi @anthonyirwin82. I was reviewing this PR of yours with @mapedorr.
First of all, thanks for that, it is a very good addition :) The only thing we think may be cumbersome is keeping emotions and strings in different arrays. It requires a bit of attention and it's error prone.
An easier solution for the dev would be to specify emotions alongside strings. Using a dictionary is a bit inconvenient, but a possible solution is this:
We can use a pipe character | as a marker to insert the emotion at the beginning of the string, like this:
angry|I don't want to pick THAT up!
And escape it if you REALLY want your character to say a pipe:
The password is "1234\|qwer"
This approach can be ported for consistency to the say functions we already have in place, maybe?
That way one has to populate just one array and each string will have its emotion attached, only where there are needed.
What do you think?

The two arrays is error prone which is why I had the code to create empty strings in the emo_array if the array sizes didn't match.
The | formatting is a nicer way to do it. I did it with the two arrays to try to match the existing say command.
I will update this.

In Godot | throws an invalid escape character error and will not let you use it. I considered using a different character as an escape character but realised we only care about the first | character that separates the emotion.
So 'emotion|dialog text that has multiple | characters in it with another | for good measure will still work'
If the user wants to use pipe characters in a dialog text they will need to define an emotion even if they don't use it.
I left the say command with the same parameters so it still works with the old format, so people with existing games will not need to change anything but can also use the new | delimited strings.
I also strip out spaces and other non-printable characters from the strings, so users can also do 'emotion | dialog text' without any issues.
Below is an example of what I used in testing:W
var use_random_array: PackedStringArray = [
	"emotion|First line",
	"emotion | Second line",
	"emotion | third line"
]

var use_next_array: PackedStringArray = [
	"emotion|First line",
	"emotion|Second line",
	"emotion|third line has multiple | characters",
	"emotion | Last and default line | has multiple | characters"
]

func _on_click() -> void:
	await C.player.walk_to_clicked()
	await C.player.face_clicked()
	await C.player.say_random(use_random_array)

func _on_double_click() -> void:
	await C.player.teleport_to_hotspot("Bed")
	_on_click()

func _on_right_click() -> void:
	await C.player.walk_to_clicked()
	await C.player.face_clicked()
	await C.player.say_next(use_next_array)

Actually you can do pipes in dialog without having an emotion by doing '|dialog text string with | pipe characters | another one for good measure'.
This will treat the first pipe as the emotion delimiter and will be a blank empty string and not used. The other pipes that are part of the dialog text will work as expected.

Note to self: review the implementation in the light of what we have in PopochiuCharacterHelper.gd from line 54.

@anthonyirwin82 I took the time to review this PR again and I think before merging it we need to rethink a bit the say functions.
Here is a quick explanation, I'll go into more details after discussing this with @mapedorr.
The E.run() short syntax for dialogues accepts stuff link:
"Guybrush: I want to be a pirate!"
"Guybrush(happy): This is the happiest day of my life!"
"Guybrush[3]: Three seconds later..."
"Guybrush(sad)[5]: Why did the ship sink?"
"Guybrush[2](angry): That's the second time today!"

with the name of the character specified as the first element, a couple of optional params for duration and emotion in random order and the line after a :  token.
My proposal to use a pipe to separate emotions was misinformed about what we already had in place. The current implementation of say() takes two params, one for the emotion, but the best option would be to make it support a similar syntax as we have for E.run(), just omitting the character name, like in these examples:
"I want to be a pirate!"
"(happy): This is the happiest day of my life!"
"[3]: Three seconds later..."
"(sad)[5]: Why did the ship sink?"
"[2](angry): That's the second time today!"

This way we can have a consistent use across all the plugin. Since we're messing with strings in a way that may collide with the devs intentions (even if it's very unlikely), we need to understand how to escape this whole thing. We also need to decide if we want to have more emotions or timings in the same say command, like:
"Guybrush: I want to be a pirate! (happy): A happy one! [2]: And a very fast talker one, too!"

This is a bit of a stretch, but once we have this nailed down I would ask you if you want to help us implement this as god intended.

@anthonyirwin82 sorry for letting this stale, I'm not ignoring it, just need some time to wrap my head over this topic and talk to @mapedorr about it.
The contribution is still valuable, we just need to make it organic with the rest of the code.
Thanks for your patience.


Imsub class v0

Added class in imagesubtraction (imsub). Allows tracking of intermediate product paths, checks if products exist, add "force" keyword to enable skipping or forcing of rerunning code.

@wmwv requesting review. I want to merge this and delete the branches so I can clean up the mess I made with the WCS issue and all the wrong branches and commits and such. But this checkpoint runs.


Add document querying

This implements a Query builder to pass to DocumentHandle.query() method to traverse the tree and return all node ID(s) that match the predicate. This also supports a FindFirst and FindAll search type. The searching of the tree terminates early on FindFirst.
I need to add some notes in the actual query.rs file explaining more of its usage, but given a Query instance, a node must match ALL conditions provided in the query (i.e., each condition is treated as an AND condition) in order to be returned.
I also want to add a simple extension DocumentHandle.query_multiple (or something along those lines) which would take a Vec<Query> list of predicates and check if a node matches at least one in order to be returned (to support OR-like conditionals.)

I believe I have everything ready now for the first iteration of document querying. Squashed all commits during development into one.


Add list cmd

Hi there,
thanks for this useful little tool. I thought it would be nice to just have a simple list command that prints a list of the addresses in a given cidr range. I am not sure if you find it useful as well but if so, please feel free to merge this pull request.
Cheers.

Hello @triole, I appreciate your effort in submitting this PR, and your kind words mean a lot. It's great to hear that you find cidr useful! I have a few comments on the pull request and would like to discuss them with you.

The intention behind cidr is to be agnostic to both IPv4 and IPv6, offering features that naturally support both, as per the existing CLI features.
While this feature is beneficial, I have concerns about its practicality for CIDR ranges larger than /24. Printing thousands of lines to the console quickly becomes overwhelming. What are your thoughts on this?
Regarding the implementation, the current approach involves building the list in-memory, which might not be the most efficient solution.

Let's discuss and brainstorm on these points.

Hi,
thanks for your ideas and comments. I am glad that we can discuss here to help each other understand a little better. Let me try to explain why I find this feature useful.
But first let me say that regarding the implementation you are absolutely right. It wasn't a good idea to store the addresses inside a list and print them later. Something that I also noticed a little later but at this point the pull request was already made. I guess sometimes it's better to sleep over things. Anyhow I updated this and hope that it is solved better now.
Concerning your other point that an address list of bigger cidr ranges can grow quite large I must admit that I do not see this as a big problem. I suppose the most use cases are just checking smaller ranges and have a look at which addresses belong to these. But of course you could also try larger ones. From my point of view this is a feature that I would not decide to keep back from users just because it might end up in a print orgy that by the way can always be aborted pressing ctrl+c.. I would leave the decision about the usefulness of the list feature to the users. If someone wants to have a look at thousands of ip addresses, why not? Furthermore there might be occasions where people just need these kind of large lists to process them further with shell tools like grep, tail, head or anything else. You could also think of piping lists into files and continue working with them from there.
So anyway the point is that I think I get what you mean, but I would not decide to not offer this feature. There might be people who have use cases for it. So let the user decide what to do with it.
I hope the above makes senses and helps a little.
Cheers.

Hi @triole,
It's great to hear that you've revisited the implementation and made updates to address the concerns regarding storing addresses in a list. Taking a step back and reconsidering the approach can indeed lead to more robust solutions.
My primary concern is around the scalability of the feature. As CIDR ranges with small prefix lengths can indeed be more manageable, the situation might become less favorable as the prefix length increases. Handling thousands or even millions of IP addresses could lead to performance issues and might not align with the intended user experience.
Considering these factors, I wonder if there could be a middle ground that allows users to explore smaller CIDR ranges more interactively while also providing warnings or limitations for larger ranges. This approach could strike a balance between user empowerment and system efficiency.
I'm not fully convinced yet of the ideal solution, and I believe it's a complex matter that requires careful consideration. It's great that you're open to feedback @triole , and I look forward to seeing how this feature develops - what are your thoughts on this @DanielRieske, @mmourick and @mvanholsteijn?

Hi :wave: @bschaatsbergen @triole
Aside from scalability concerns that I absolutely agree with, I do have some questions regarding usability.
I am wondering how useful this feature would be for larger CIDR ranges, while I won't dispute that it has its merits for smaller CIDR ranges I would look into options to make more of a summary of a given CIDR range.
Example, given CIDR range <IP_ADDRESS>/24 you can print out a summary with the following details

IP Address
Network Address
Usable Host IP Range
Total Number of Hosts

<IP_ADDRESS>/24
<IP_ADDRESS>
<IP_ADDRESS> - <IP_ADDRESS>
256

This approach can also be used to calculate IPV6 addresses, therefore we keep supporting both.
Curious what you guys think of this approach

Hi 👋 @bschaatsbergen @triole
Aside from scalability concerns that I absolutely agree with, I do have some questions regarding usability. I am wondering how useful this feature would be for larger CIDR ranges, while I won't dispute that it has its merits for smaller CIDR ranges I would look into options to make more of a summary of a given CIDR range.
Example, given CIDR range <IP_ADDRESS>/24 you can print out a summary with the following details
IP Address	Network Address	Usable Host IP Range	Total Number of Hosts
<IP_ADDRESS>/24	<IP_ADDRESS>	<IP_ADDRESS> - <IP_ADDRESS>	256
This approach can also be used to calculate IPV6 addresses, therefore we keep supporting both.
Curious what you guys think of this approach

Appreciate you sharing your view on this, @DanielRieske! Your input aligns with my current thinking, and I'm actively incorporating it into #75 as I'm typing this.

Hi @bschaatsbergen and @DanielRieske ,
I appreciate your feedback and I totally see your point regarding the scalability of the list feature. Nonetheless I do tend to leave the question of usability up to each individual who considers using the feature. Maybe a warning before printing too many addresses would be a reasonable middle ground that we all could settle with. Although of course immediately the question arises, how many actually are too many.
The explain feature definitely has its use and I'm excitedly looking forward to see it implemented. I think it'll be very handy but list is a different thing on purpose.  As said I wouldn't limit the possibilities of list but of course I can live with the fact that you see this differently.
Thanks for the constructive discussion. I really enjoy it.
Cheers.

Hi @bschaatsbergen and @DanielRieske ,
I appreciate your feedback and I totally see your point regarding the scalability of the list feature. Nonetheless I do tend to leave the question of usability up to each individual who considers using the feature. Maybe a warning before printing too many addresses would be a reasonable middle ground that we all could settle with. Although of course immediately the question arises, how many actually are too many.
The explain feature definitely has its use and I'm excitedly looking forward to see it implemented. I think it'll be very handy but list is a different thing on purpose.  As said I wouldn't limit the possibilities of list but of course I can live with the fact that you see this differently.
Thanks for the constructive discussion. I really enjoy it.
Cheers.

Hey @triole,
I've been doing some tinkering about this and please allow me to get back on this in a couple days :)

Hi, no worries. It is absolutely fine not to answer or expect an answer on the same day. I know how it is to be busy all the time and therefore being forced to postpone things over and over again. It's all fine.
But to answer your question: I like the explain command and find it quite useful to get some information about network ranges, netmask, the broadcast address and so on. As I said earlier this is a nice feature that definitely has its purpose. Nonetheless the use case is different from the list feature because if you need a stupid plain list of IP addresses to further process them somewhere else list would be the way to go.
I guess you know my standpoint. I'd like to see both features implemented. But I totally respect if you decide otherwise.
Cheers.

Thank you for your understanding, and we appreciate your input on the explain command and list feature. At this time, we're not planning to implement the list feature, but your feedback is valuable. If there's increased community support or demand for this particular feature, we'll certainly reconsider. @triole


Hhh 12484 improved error logs

Adapted identifier output accordingly.

This is already integrated as suggested in Steve's comment.


HDDS-11181. Cleanup of unnecessary try-catch block.

Cleanup of unnecessary try-catch block
In the following places, the catch block catching exception is re-throwing the same exception without performing any other action. In such cases there is no need to catch the exception.
HDDS-11181
How was this patch tested?
As this PR doesn't change anything functionally, existing tests are enough to test this patch.

Thanks @nandakumar131 for the patch.

Thanks @adoroszlai for the review and merge!


Fix the "No space left on device" issue when trying to upload large packages

While trying to upload a large package to Archivematica (~34GB), we saw an issue where the package never appeared in the dashboard, and instead we saw an error in the Archivematica logs:
OSError: [Errno 28] No space left on device

Full stack trace
File "/src/storage_service/locations/api/resources.py", line 652, in _handle_location_file_move
    return move_files_fn(data["files"], origin_location, destination_location)
File "/src/storage_service/locations/api/resources.py", line 673, in move_files
    async_task = AsyncManager.run_task(task)
Traceback (most recent call last):
File "/src/storage_service/locations/api/resources.py", line 561, in _move_files_between_locations
    raise PosixMoveUnsupportedError
locations.models.space.PosixMoveUnsupportedError
(empty)
During handling of the above exception, another exception occurred:
(empty)
Traceback (most recent call last):
File "/src/storage_service/locations/models/async_manager.py", line 140, in wrapper
    value = task_fn(*args, **kwargs)
File "/src/storage_service/locations/api/resources.py", line 669, in task
    files, origin_location, destination_location
File "/usr/local/lib/python3.6/dist-packages/q.py", line 304, in wrapper
    result = func(*args, **kwargs)
File "/src/storage_service/locations/api/resources.py", line 575, in _move_files_between_locations
    destination_space=destination_space,
File "/src/storage_service/locations/models/space.py", line 366, in move_to_storage_service
    source_path, destination_path, destination_space, *args, **kwargs
File "/usr/local/lib/python3.6/dist-packages/q.py", line 304, in wrapper
    result = func(*args, **kwargs)
File "/src/storage_service/locations/models/s3.py", line 231, in move_to_storage_service
    bucket.download_file(objectSummary.key, dest_file)
File "/usr/local/lib/python3.6/dist-packages/boto3/s3/inject.py", line 246, in bucket_download_file
    ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
File "/usr/local/lib/python3.6/dist-packages/boto3/s3/inject.py", line 172, in download_file
    extra_args=ExtraArgs, callback=Callback)
File "/usr/local/lib/python3.6/dist-packages/boto3/s3/transfer.py", line 307, in download_file
    future.result()
File "/usr/local/lib/python3.6/dist-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
File "/usr/local/lib/python3.6/dist-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
File "/usr/local/lib/python3.6/dist-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
File "/usr/local/lib/python3.6/dist-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
File "/usr/local/lib/python3.6/dist-packages/s3transfer/download.py", line 572, in _main
    fileobj.write(data)
File "/usr/local/lib/python3.6/dist-packages/s3transfer/utils.py", line 364, in write
    self._fileobj.write(data)
OSError: [Errno 28] No space left on device
ERROR     2023-04-12 10:17:23  locations.models.async_manager:async_manager:wrapper:143:  Task threw an error: [Errno 28] No space left on device
Caller's traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/gevent/baseserver.py", line 34, in _handle_and_close_when_done
    return handle(*args_tuple)
File "/usr/local/lib/python3.6/dist-packages/gunicorn/workers/ggevent.py", line 155, in handle
    super(GeventWorker, self).handle(listener, client, addr)
File "/usr/local/lib/python3.6/dist-packages/gunicorn/workers/base_async.py", line 56, in handle
    self.handle_request(listener_name, req, client, addr)
File "/usr/local/lib/python3.6/dist-packages/gunicorn/workers/ggevent.py", line 160, in handle_request
    addr)
File "/usr/local/lib/python3.6/dist-packages/gunicorn/workers/base_async.py", line 107, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/wsgi.py", line 157, in __call__
    response = self.get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/base.py", line 124, in get_response
    response = self._middleware_chain(request)
File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
File "/usr/local/lib/python3.6/dist-packages/whitenoise/middleware.py", line 25, in __call__
    response = self.get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/utils/deprecation.py", line 140, in __call__
    response = self.get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/utils/deprecation.py", line 140, in __call__
    response = self.get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/utils/deprecation.py", line 140, in __call__
    response = self.get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/utils/deprecation.py", line 140, in __call__
    response = self.get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/utils/deprecation.py", line 140, in __call__
    response = self.get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/utils/deprecation.py", line 140, in __call__
    response = self.get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/utils/deprecation.py", line 140, in __call__
    response = self.get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/utils/deprecation.py", line 140, in __call__
    response = self.get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/utils/deprecation.py", line 140, in __call__
    response = self.get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/utils/deprecation.py", line 140, in __call__
    response = self.get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python3.6/dist-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tastypie/resources.py", line 227, in wrapper
    response = callback(request, *args, **kwargs)
File "/src/storage_service/locations/api/resources.py", line 208, in wrapper
    result = func(resource, request, bundle, **kwargs)
File "/src/storage_service/locations/api/resources.py", line 687, in post_detail_async
    return self._handle_location_file_move(move_files, request, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/q.py", line 304, in wrapper
    result = func(*args, **kwargs)

Currently the Archivematica storage service isa trying to put all its content in
/src/storage-service/var/archivematica/sharedDirectory/…

which uses data on the EC2 instance's local disk, rather than the big and large shared EBS volume. This means it runs out of space pretty quickly; it should be using the shared EBS volume so it has more disk space.

To investigate this issue, I ssh'd into the Archivematica container host and ran df -h to see the available disk space:
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         16G     0   16G   0% /dev
tmpfs            16G     0   16G   0% /dev/shm
tmpfs            16G  2.1M   16G   1% /run
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/nvme0n1p1   30G  9.6G   20G  33% /
/dev/nvme1n1    2.0T   70G  1.8T   4% /ebs
tmpfs           3.1G     0  3.1G   0% /run/user/1000

When I uploaded the package, I noticed the disk used on / was increasing – but we have a large EBS volume mounted at /ebs which is meant to be where working data is being stored. What's up with that?

This does expose a couple of things I'm going to change:

Despite configuring a 2TB EBS volume for large package storage, we haven't been using it for months – all our packages have been fitting within the ~20GB of instance storage. I'm going to reduce the size of the EBS volume and save a bit of money.
While looking at that, I realise our RDS instances are probably (still) overprovisioned – newer versions of Archivematica are much more efficient with database connections, so I think we can drop another instance size. This takes us from 90 to 45 max connections; in the last month our max connections are usually less than 40, so I think we can try turning down further.

It turns out this issue was also affecting staging, so I've fixed it there too.