Add LatencyMetric class

This pull request adds a new LatencyMetric to Validate. The LatencyMetric class is a metric used to measure the latency of the model's response time. It calculates the percentage of the target time that the model takes to complete a request. This metric can be used to evaluate the performance of the model in terms of response time.

Why not just report back the the actual response time in ms?

@akamor @joeferraratonic I did it based on percentage since doing MS will really screw up the charts until we remove overall scores. So I did a percentage as it won't mess up the overall score chart. Also, in the UI we have parts of the UI which state the max value for a metric (e.g. _ out of 5). We'd need to go through the UI and fix those too. So this was just the quickest way to add the metric in since I am hesitant to make a bunch of UI changes given we might change it a bunch soon.

Ok, that makes sense. In that case, I think we should make the metric a binary metric (whether or not latency is less than target time) because capping the percentage at 1, looses information when the time the response takes is greater than the target time. Making it a binary metric also makes the target time more discoverable for the user.
We should also file an issue to add a latency metric that returns milliseconds, so we do not forget to do that after the UI changes are made.

Ok, that makes sense. In that case, I think we should make the metric a binary metric (whether or not latency is less than target time) because capping the percentage at 1, looses information when the time the response takes is greater than the target time. Making it a binary metric also makes the target time more discoverable for the user.
We should also file an issue to add a latency metric that returns milliseconds, so we do not forget to do that after the UI changes are made.

Yeah, lets go with this suggestion. Binary.


Phoebemirman/ch12069/search doesn t clear when click on markets

https://app.clubhouse.io/augur/story/12069/search-doesn-t-clear-when-click-on-markets

Coverage remained the same at 66.818% when pulling 93f85d34c1a6dac6029b1c1310e22e525ee2ede5 on phoebemirman/ch12069/search-doesn-t-clear-when-click-on-markets into b3403d8e05fe82fcec1bb06a0caa6e0deac4386b on master.


Rebrand from W4k.Either to just Either

Original name of W4k.Either may indicate it's part of another project, while Either is general-purpose library.

🤦 Either is reserved (package with that name is withdrawn but its presence prevents me to do the change)


Matt's product list spreadsheet

[x] include a product .md for each entry in the spreadsheet (add column to spreadsheet to track this?)
[ ] include link to Matt's spreadsheet on the website (need to modify sharing settings?)
[ ] create a document on steps for "how to add your MBON product"

Neil Hammerschalg specifically pointed today out that his stuff isn't on there. We need to pull those (and others?) from the spreadsheet.

@7yl4r I'm concerned about linking to the Google Spreadsheet. The sharing capabilities with NOAA accounts are limited and I don't necessarily want to be a gate keeper. It might be good to write a small python script to parse the spreadsheet, remove the contact column, and publish the data via github?
We can iterate on other potential solutions.


Info/Documentation on chunkwise training

Hi there. I want to understand how to use the RetNet to train a model with the longer context. It is not clear from available documentation how to train the model for a large context. There is no parameters for Trainer of TrainingArguments how do one actually passes 100k text with 512 chunksize into the model during the training? Should one chunk the text themselves with overlapped text fragments? Or there is a method to pass 100k text and it will be processed?
What exactly these outputs can be used for?
chunk_outputs = model(input_ids, forward_impl='chunkwise', use_cache=True, recurrent_chunk_size=4)
chunk_state = chunk_outputs.last_hidden_state
chunk_cache = chunk_outputs.past_key_values

Should one mix these outputs into the beginning of a next input during the data preparation?
Please help.

Hi there!
If you want very long ctx len, say 16k, you will have to create batch of shape [bs, ctxlen] in the dataloader and pass it to the trainer. Then, when you create the RetNetConfig, set the forward_impl="chunkwise", recurrent_chunk_size=512. You can set the chunk size to whatever you like. This will set the default forward mode of RetNet to chunkwise, and it will chunk the input ids within the forward. Take a look at chunkwise_forward.

Hello @syncdoth,
Thank you for the swift answer. I have several followup questions.

When I set "forward_impl":"chunkwise" in the config file, or after loading the config and modify it with config.forward_impl="chunkwise" printing this config does not includes the forward_impl, however the amount of VRAM required for "chunkwise" is much larger that indicates the change in the configuration.
In your example [bs, ctxlen], I assume that ctxlen is the new context len - 16k in this case. Right?
What would be the effect of changing the forward_impl for the pretrained model? Let suppose I trained the model with forward_impl="parallel", and then I want to continue training with the larger context setting the implementation to forward_impl="chunkwise". Will this work? The weights are the same it is just the pipeline of training is changed.
Can I switch it back to forward_impl="parallel" for another dataset? What change in behavior should I expect?
What with the inference. Do I need to run the model in forward_impl="chunkwise" mode for the larger context to be available?

Thanks a lot for your answers, I really appreciate your work. And thanks that you are still supporting your implementation.

Question 3,4,5 - the answer is yes. You can train with any mode then load the model for further training or inference with any mode, the output should be the same. The point of retentive network is to have equivalence of the three forward modes given the same model parameters.
Question 1 - does the chunkwise forward use more VRAM with the same seqlen? That might be a bug I should track down.
Question 2 - yes, I mean the length of the full sequence, not each chunk.

@syncdoth Thank you again, appreciate it very much.
I checked with the same content length and chunkwise forward_impl uses more VRAM
With 180 tokens sequences I can have on 16G VRAM:

per_device_train_batch_size=18  with forward_impl="chunkwise"
per_device_train_batch_size=56 with forward_impl="parallel"

Network config (150M):
{   "vocab_size": 32024,   "pad_token_id": 2,   "eos_token_id": 2,   "unk_token_id": 0,   "bos_token_id": 1,   "decoder_embed_dim": 512,   "decoder_value_embed_dim": 1024,   "decoder_ffn_embed_dim": 1024,   "decoder_layers": 32,   "decoder_retention_heads": 16,   "initializer_range": 0.02,   "layernorm_eps": 1.0e-8,   "activation_fn": "swish",   "dropout": 0.0,   "activation_dropout": 0.0,   "drop_path_rate": 0.0,   "decoder_normalize_before": true,   "layernorm_embedding": true,   "no_scale_embedding": false,   "recurrent_chunk_size": 512,   "use_lm_decay": false,   "deepnorm": false,   "subln": true,   "torch_dtype": "float32",   "output_retentions": false,   "forward_impl": "parallel",   "tie_word_embeddings": false,   "is_decoder": true,   "transformers_version": "4.36.0.dev0",   "use_cache": true,   "model_type": "retnet" }
Trainer args:
--gradient_accumulation_steps 2 --bf16 --optim adamw_torch_fused --do_train --evaluation_strategy epoch --num_train_epochs 1 --lr_scheduler_type cosine_with_restarts  --save_strategy epoch --learning_rate 3e-3 --warmup_steps 1000 --include_tokens_per_second True --bf16_full_eval True

Thanks for raising this issue. I think I might know the reason;
During parallel forward, it computes q @ k matrix, which is of size (bs, num_head, seqlen, seqlen).
During chunkwise, it must compute k * v, which is of size (bs, num_head, chunk_size, qk_dim / num_head, v_dim / num_head).

You have recurrent chunk size of 512, which is longer than 180. This means that the sequence will be padded to nearest multiple of 512 (i.e. 512) to perform chunkwise forward.

For your configuration, the seqlen should be at least 1024 for chunkwise to use the same amount of memory as parallel. In my experience, I found chunkwise to be beneficial only for training with 4k+ seqlen.


Add ability to retrieve tag names from struct

I would like to build a SQL statement without using * using an empty struct. It would be nice if I could retrieve all the "tag names", similar to how Names() works. For example:
type Person struct {
  FirstName string `structs:"first_name"`
  LastName string `structs:"last_name"`
}

p := new(Person)

names := structs.Names(p)
// names = []string{"FirstName", "LastName"}

// not currently possible
tagNames := structs.TagNames(p)
// tagNames = []string{"first_name", "last_name"}

I would argue that this is acceptable in this library since it deals with a tag that this package manages. A temporary workaround is to export the struct to a map and extract the keys from that. (but it would be great if this library could remove that boilerplate)

Hi @dominicbarnes
To get the tags you also need to define the tag key name. Such as structs, json, etc.. You can do it already with the following (for your example):
package main

import (
	"fmt"

	"github.com/fatih/structs"
)

type Person struct {
	FirstName string `structs:"first_name"`
	LastName  string `structs:"last_name"`
}

func main() {
	var tagNames []string
	for _, f := range structs.New(Person{}).Fields() {
		tagNames = append(tagNames, f.Tag("structs"))
	}

	fmt.Println(tagNames)
}

As you see it's just three lines to collect all tag names. I don't want to further expand this library as I believe it already provides the necessary API. Thanks for you feedback.

The example you have misses quite a few details, such as additional parts in the tag like omitempty or omitnested and using - or empty string to ignore a field. Plus, it's inconvenient to establish my own for loop to accomplish this.
Would you at least be open to a PR that adds this feature? (rather than needing to write it yourself?)

I'm not sure those are important. The struct pkg will take care of it already. Can you share an example what you mean?

To elaborate on my first example:
type Person struct {
  FirstName string    `structs:"first_name"`
  LastName  string    `structs:"last_name"`
  Birthdate time.Time `structs:"birthdate,omitnested"`
  CacheKey  string    `structs:"-"`
}

Using the code you have above, I'd end up with: []string{"first_name", "last_name", "birthdate,omitnested", "-"} whereas what I really want is: []string{"first_name", "last_name", "birthdate"}.

Hi @dominicbarnes
I see what you mean. I worked on a separate, improved and powerful tags package that deals with this any many other things. Let keep this open, I'll add the necessary feature once I have time (just need to replace the internal tags.go file with the package I wrote and expose it with some well defined API's)


Invoice number is not showing up correctly for the invoice

Describe the bug
When adding invoice in the portal the  invoice sequence in the Invoice number is not showing up corectly. The invoice sequence for invoice should be 000015 but it is showing 000014.

Expected behavior
Invoice sequence in  the invoice number should be generated as per sequence  correctly.

Invoice number should be genearted by incrementing the sequence number of the last invoice.


Optimizing DFFs whose initial value prevents their value from changing

I'm pretty new to the Yosys codebase so this is only a proof of concept implementation which invokes SAT solver to optimize-away DFFs in the cases that boil down:
module top(
    input  clk,
    input  a,
    output b
);

    reg b_reg;
    initial begin
        b_reg <= 0;
    end

    assign b = b_reg;
    always @(posedge clk) begin
        b_reg <= a && b_reg;
    end
endmodule

Here, the b_reg initial value of 0 is preventing the register value from ever changing. I used SAT solver to check for these situations since expressions for the DFF next value can be more complex and end up in this simple form only after minimization.
I invoked the SAT solver using string command via Pass::call, since it seemed that I would need to either copy or refactor a lot of the code from the passes/sat/sat.cc module. The generated SAT problem for the example above is:
sat -ignore_unknown_cells -prove $0\b_reg[0:0][0] 0 -set b_reg[0] 0
So for the multi-bit registers, the above check would be run for each of the individual bits. If the SAT solver proves that a portion of the register cannot change its value, that portion of the register is replaced by a constant.
Again, I'm not familiar with the codebase so I'll be willing to put in some more work into the implementation if this approach makes sense and I could be given some guidelines on how to fit it naturally into Yosys.

This breaks make test. Please investigate and fix.

Hey. Had a quick look now.

I'm pretty new to the Yosys codebase

Welcome!

so this is only a proof of concept implementation

Simply calling the sat command is an okay thing to do in a proof-of concept. However, for this to be something that can be upstreamed you should actually create an instance of SatGen that you can reuse for muliple bits, and that only contains the part of the module that is relevant to the question you are asking.

OK, the code now creates an instance of SatGen. For each register, it only includes the cells that affect the register input combinatorially (i.e. its combinatorial cone). I refactored some of the procedures for traversing the netlist into passes/opt/netlist.h file, since they might be useful in different contexts.
I'm only not sure how to reuse SatGen for different bits of the multi-bit register. Even though the bits have the same combinatorial cone, they produce different SAT assumptions (to test whether their values can change). These assumptions seem to linger in SatGen instance after running solve. Is there a way to remove only these assumptions, and reuse the combinatorial cone?

Also, what is a procedure to create an automated test that would check (given a Verilog test file) whether the registers are indeed correctly removed by this optimization?

Also, what is a procedure to create an automated test that would check (given a Verilog test file) whether the registers are indeed correctly removed by this optimization?

See for example tests/various/pmux2shiftx.{ys,v}. We use select -assert-count <N> t:<T> to make sure the results contain exactly <N> cells of type <T>.

I've now refactored this according to my comments above. See 2454ad99bf49afe752d6fd1c1567f59ee9e26736.
You should be able to pull my changes into your branch with this:
git pull https://github.com/YosysHQ/yosys.git clifford/ext1046

I've pulled the changes you made.
Should I add a sentence about what "-sat" option does in "opt_rmdff" command?
Should I make "-sat" option available via "opt" command?
I have a slightly larger example where this procedure should detect and remove FFs. It consists of 3 different modules with lots of logic that should be optimized away, which makes it more convoluted. Should I add it to the test suite?

Should I add a sentence about what "-sat" option does in "opt_rmdff" command?
Should I make "-sat" option available via "opt" command?

Yes, that would be great.

I have a slightly larger example where this procedure should detect and remove FFs. It consists of 3 different modules with lots of logic that should be optimized away, which makes it more convoluted. Should I add it to the test suite?

Yes, but don't add it to tests/ within Yosys, add it to https://github.com/YosysHQ/yosys-tests.

I created a pull request for the larger test here: https://github.com/YosysHQ/yosys-tests/pull/41
Using this test I discovered that the code had an issue when there are multiple DFF cells in the design. The test occasionally produced an exception, so I checked it with Valgrind, which in turn helped me find out that there were dangling references to the removed DFF cells left in bit2driver dict. This commit removes the issue: https://github.com/YosysHQ/yosys/pull/1046/commits/35fa7b30574244e4f99373f2a790f004b4a1dbbb
I also added the help text and possibility to pass "-sat" switch via "opt" command


Revert "feat: Added support for Node 16"

Reverts openedx/mockprock#29. The workflow file named ci.yml has code for releasing the package but it was updated by mistake due to its confusing name.

We should update the workflow name to avoid any such confusion in the future.


Allow one-word message names or change error message when parsing ambient.toml

Caused by:
    0: manifest was not valid TOML
    1: TOML parse error at line 12, column 11
          |
       12 | [messages.Move]
          |           ^^^^
       the item identifier Move is not snake_case or PascalCase

Caused by:
    0: manifest was not valid TOML
    1: TOML parse error at line 12, column 11
          |
       12 | [messages.move]
          |           ^^^^
       the item identifier move is not snake_case or PascalCase

Ah - the issue is that move isn't allowed, because move is a keyword in Rust. I'll improve the error message here and allow Move.


ARROW-5812: [Java] Refactor method name and param type in BaseIntVector

Related to ARROW-5812.
Change to void setWithPossibleTruncate(int index, long value); for better generality.

Codecov Report

Merging #4763 into master will increase coverage by 3.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4763      +/-   ##
==========================================
+ Coverage   86.44%   89.45%   +3.01%
==========================================
  Files         992      659     -333
  Lines      138020    95130   -42890
  Branches     1418        0    -1418
==========================================
- Hits       119307    85099   -34208
+ Misses      18351    10031    -8320
+ Partials      362        0     -362

Impacted Files
Coverage Δ

cpp/src/arrow/io/readahead.cc
95.91% <0%> (-1.03%)
:arrow_down:

r/src/recordbatch.cpp

r/R/Table.R

js/src/util/fn.ts

go/arrow/array/bufferbuilder.go

r/src/symbols.cpp

rust/datafusion/src/execution/projection.rs

rust/datafusion/src/execution/filter.rs

rust/arrow/src/csv/writer.rs

rust/datafusion/src/bin/main.rs

... and 324 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 91b4cbc...9bbeb3d. Read the comment docs.

+1, LGTM


Update web-development-resources.md

🛠 Fixes issue #
📇 Changes proposed
✅ Check List (Mark all the applicable boxes)

[ ] My code follows the code style of this project.
[ ] This PR does not contain plagiarized content.
[ ] This PR contains changes in the documentation
[ ] This PR contains changes in the Code of the project
[ ] The title of my pull request is a short description of the requested changes.

📄 Note to reviewers

📷 Screenshots

@anubhav1206, please link the issue here which this PR is trying to solve


Update layout.html.twig

Fix actions display on small screens.

Thanks! Is that before or after? What has changed?

Sorry I was lazy but in my defense it was late 😉
Actually this is the "new" behaviour.
While updating AdminLTE to Tabler i noticed that the actions do not go below the header once reached the mobile viewport.
You can see the "right" behaviour here: https://preview.tabler.io


[stable/envoy] Add configmap for xds

Is this a new chart
No
What this PR does / why we need it:
Add ConfigMap for Envoy-xds
Which issue this PR fixes
(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

fixes #

Special notes for your reviewer:
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[x] Variables are documented in the README.md
[x] Title of the PR starts with chart name (e.g. [stable/mychartname])

/approve
/ok-to-test

/lgtm

@renjikari It seems like I've missed including that template it in #17440. Thanks a lot for the fix ☺


Find alternative to moment.js

Description
As moment.js has nearly declared itself dead (https://momentjs.com/docs/#/-project-status/)

In most cases, you should not choose Moment for new projects.
We may choose to not fix bugs or behavioral quirks, especially if they are long-standing known issues.

We should find an alternative project to handle date/time in app-frontend.
Some alternatives may be: https://blog.logrocket.com/4-alternatives-to-moment-js-for-internationalizing-dates/
Acceptance criteria
A viable alternative is identified and has replaced moment.js where used.
Specification tasks

[ ] Identify where moment.js is used
[ ] Test design / decide test need

Development tasks

[ ] Replace with new library
[ ] Regression test

Definition of done
Verify that this issue meets DoD (Only for project members) before closing.

[ ] Documentation is updated (if relevant)

[ ] Technical documentation (docs.altinn.studio)
[ ] User documentation (altinn.github.io/docs)

[ ] QA
[ ] Manual test is complete (if relevant)
[ ] Automated test is implemented (if relevant)
[ ] All tasks in this userstory are closed (i.e. remaining tasks are moved to other user stories or marked obsolete)

Moving this to later this year as we would like to use the new date picker implementation from the design system. The date picker is currently not top priority on neither backlogs, so waiting is fine.
cc: @mrosvik

We've added a formatDateLocale function in this pr https://github.com/Altinn/app-frontend-react/pull/1610 which uses the native browser API to format dates. This would probably reduce the efforts of refactoring away from moment.

https://github.com/Altinn/app-frontend-react/issues/602

Just for info, not a strong recommendation by any means: In a previous project the team moved from moment.js to day.js, and that was very easy to do. Exactly the same API, so fixed most of it using Find and replace "momen()" with "dayjs()" (and imports).

In a previous project the team moved from moment.js to day.js, and that was very easy to do

Yeah, also according to the reccomendations from the moment team, Day.js seems like the simplest replacement. Are there any good reasons for not just doing that?

I am not sure if this is a show stopper, but the only thing I can think of at the top of my head is i18n and different date formatting for different locales. We support essentially any language in our apps, and would therefore have to preload every known locale (or default to norwegian/english for ones we decide to not support). I guess we might be able to solve this with code splitting and dynamically loading locales, but I think that might be a big complex.
If we were able to rely on the native browser Intl API instead, we could be able to support any locale, but I am not sure if we can use this API to replace our current usage of moment.


powerful idea

What feature do you want to get added on the base engine?
add mod loader from github to play songs without need to install mods just using internet
i know it's hard but not impossible
To test your sight, and reliability, please select the option of what should NOT be requested.
Useful feature

failed test
also, i won't be able to do this. i know a few mods have done it but this is too complex for me, sorry man

failed test
also, i won't be able to do this. i know a few mods have done it but this is too complex for me, sorry man

for me i can do it with chart texts scripts but not sprites or sounds

failed test
also, i won't be able to do this. i know a few mods have done it but this is too complex for me, sorry man

can you try my edited port after finish compile: https://github.com/Hiho2950/FNF-PsychEngine-Modded/actions/runs/6060584883

move sounds folder to mods folder there is a lot of errors cuz of assets folder just move the content of assets and assets/shared to mods folder

if you wanna play a normal song like week 7 just copy the content of assets/week7 to mods

and i had fix a fps drop or lag strike a little bit

https://github.com/JordanSantiagoYT/FNF-PsychEngine-NoBotplayLag/assets/61796448/5d3faa05-abad-4faa-91e4-2d676d3a0bef
it works!! and i also found this issue LOL

oh yeah i also found a potential memory leak, in freeplay if you try exiting the gameplay changer state it just goes back to that state and you can't exit

oh yeah i also found a potential memory leak, in freeplay if you try exiting the gameplay changer state it just goes back to that state and you can't exit

im i got this issue when i had try to fix gameplaychangers on pause menu substate it works on substate but i need to fix on other states

2023-09-03.05-35-14.mp4
it works!! and i also found this issue LOL

about this i didnt find anything that makes this issue


preserve selected order

Is there an option to preserve the order in wich the items are selected?
right now it keeps the selected column ordered like the available column

Unfortunately, no. I've designed multi.js to stay very close to the underlying select element. The problem with this is that a regular select has no concept of which order the elements were selected. As a result it is not possible to preserve the selected order and it would be a bit of work to implement.
I'll keep your suggestion in mind though!

This request has my vote for reconsidering :)
This could help a lot when one would like to use some kind of ranks as an input. The UI is already there.


GPS driver for Snapdragon

This brings support for the GPS driver on Snapdragon. Since QURT does not support poll and termios yet, these call needed to be wrapped.
The current state compiles and runs on NuttX and Snapdragon, before merging it should be reviewed and flight tested on NuttX.

Jenkins test this

Done, awesome, thanks!

Applied to master.


i cant use the sd card

i have inserted the sd card with inject.bin into the usb and nothing seems to happen should i flash it?
if so then how?

Don't compile the Ducky Scripts, just put them as a text file onto the MicroSD card.


Remove optional on <>voted_block_for_round vriables

Task Description
These 2 variables might not need to use optional
// Block that node saw 2t+1 soft votes for
std::optional<std::pair<blk_hash_t, std::shared_ptr>> soft_voted_block_for_round_{};
// Block that node cert voted
std::optional<std::shared_ptr> cert_voted_block_for_round_{};
Epic Parent

merged with https://github.com/Taraxa-project/taraxa-node/issues/2046


⚠ LPDM Handbook (hosted on Divio) has degraded performance

In b838a28, LPDM Handbook (hosted on Divio) (https://repohubdev.tk) experienced degraded performance:

HTTP code: 500
Response time: 5895 ms

Resolved: LPDM Handbook (hosted on Divio) performance has improved in 5c2b65b.


[FALSE-POSITIVE] dash.formaloo.com

Domains or links
Please list any domains and links listed here which you believe are a false positive.
dash.formaloo.com
formaloo.com
formaloo.co
formaloo.me
formaloo.net
More Information
How did you discover your web site or domain was listed here? Listed on Virustotal

Website was hacked
Incorrectly marked as Phishing on Phishtank or OpenPhish? Yes

Have you requested removal from other sources?
Please include all relevant links to your existing removals / whitelistings.
Additional context
Add any other context about the problem here.
Hi,
I hope this message finds you well.
This domain [dash . formaloo .com] belongs to Formaloo Platform (Formaloo Creative Inc., Reg code:<PHONE_NUMBER>). Our service (https://www.formaloo.com/en/) is a B2B SaaS that allows its users to create apps and forms.
The reported page was user-generated content (UGC) in our form builder service, similar to Google forms. We deleted all forms and pages. All accounts & IP addresses are since been suspended and banned from the service.
We also have a report abuse system in place that you can use for reports: https://formaloo.net/report
Please report any abuse to us directly, and do not flag the whole domain.
We ask you to remove the flag from our domain name as it’s hurting our reputation and brand.
Also, we do have a procedure for reporting spam/abuse content on our platform. How can we connect it so you can send the reports to us so we can take them down immediately?
Looking forward to hearing from you.
Best regards,
Mary from Formaloo
:exclamation:
We understand being listed on a Phishing Database like this can be frustrating and embarrassing for many web site owners. The first step is to remain calm. The second step is to rest assured one of our maintainers will address your issue as soon as possible. Please make sure you have provided as much information as possible to help speed up the process.
Send a Pull Request for faster removal
Users who understand github and creating Pull Requests can assist us with faster removals by sending a PR to mitchellkrogza/phishing repository, on the falsepositive.list file

https://github.com/mitchellkrogza/phishing/blob/main/falsepositive.list
Please include the same above information to help speed up the whitelisting process.

Can't see anything from those domains @farokhsh:
formaloo.com
formaloo.co
formaloo.me

However, this one is listed but not on your list:
mailservice.formaloo.co

Could you explain ?
I also see a over 78 URL with subdomains of formaloo.net, is there any reason for that Does each of your customers get its own subdomain?

Hi,
Thank you very much.
Right now we're only getting a flag on dash.formaloo.com (which is our
dashboard)
[image: Screenshot 2024-12-18 at 18.15.57.png]
These domains are used by us to create form URLs.
formaloo.coformaloo.me
Each customer gets their own subdomain on these domain. the main
domain should not get flagged as it's used by everyone but each
subdomain is specific to a user and can be suspended by our platform
when reported here: https://formaloo.formaloo.me/report
Cheers,
.
Farokh Shahabi
Chief Executive Officer
Website https://www.formaloo.com/en | Twitter
https://twitter.com/formaloo | LinkedIn
https://www.linkedin.com/in/farokhshahabi/
https://www.formaloo.com/en/reviews/
On Wed, 18 Dec 2024 at 18:10, Nissar Chababy @.***>
wrote:

Reopened #930
https://github.com/Phishing-Database/Phishing.Database/issues/930.
—
Reply to this email directly, view it on GitHub
https://github.com/Phishing-Database/Phishing.Database/issues/930#event-15709830988,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BAHTZJR6M5BK4KNIMSAU5N32GH6FNAVCNFSM6AAAAABT3GMWFWVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJVG4YDSOBTGA4TQOA
.
You are receiving this because you were mentioned.Message ID:
<Phishing-Database/Phishing.Database/issue/930/issue_event/15709830988@
github.com>


Fix introspector querying logic

It is found that the query of the introspector oracle is buggy. The current logic considers far-reach-but-low-coverage as the main oracle to be used. All other configured oracles from arguments will only invoked if and only if far-reach-but-low-coverage returns an empty result. If far-reach-but-low-coverage does return at least one function from fuzz-introspector, only the oracle easy-params-far-reach and oracle optimal-targets will be invoked and all other oracles provided are ignored. This PR fixes the logic to always invoke far-reach-but-low-coverage oracles and all other oracles provided. Then fixed the _combine_functions to consider far-reach-but-low-coverage as the main priority for introspector querying to allow more "valid" candidates from other oracles to be considered during the target generation process.

/gcbrun exp -n ascc-jvm-pr-512-1 -m vertex_ai_gemini-1-5 -b jvm-small

/gcbrun exp -n ascc-jvm-pr-512-1 -m vertex_ai_gemini-1-5 -b jvm-small

Thanks for looking into this @arthurscchan.
Previously, _combine_functions() has both b and c because I wanted to add at least one benchmark from each set so that the final set has at least one from each oracle.
I implemented the previous logic specifically for C/C++ benchmarks because they only have 3 oracles: far-reach-but-low-coverage, easy-params-far-reach, and optimal-targets. Please let me know if I was wrong.
Sorry that I did not realize we needed to update the Java benchmark frequently. Given Java needs a different orcale set, would it be easier to add a new layer to 1) check which language it is, and 2) use a different logic for Java?

I am OK with both approach, I could separate the logic for jvm on oracle querying without touching the original c/c++ logic, it maybe easier to maintain.

/gcbrun skip


Minimum formatting of completed_at

because most completed_at are not used.

@TTouka Please, fix pytest.
I've checked your PR and have 500 error on tasks page:

Sorry, I fixed my bug.
Please check again.


Microsoft Azure backed Delta tables

Contact Details
<EMAIL_ADDRESS>Is your feature request related to a problem? Please describe.
I was trying to connect sharing server with ADLS gen 2, but I am getting internal server error once I execute request.
Describe the solution you'd like.
Hi,
is supporting Microsoft Azure backed Delta tables in development? Is there some estimate when it is expected to be done, or if help is needed, I can try to help, with some minor guidelines and info from someone in charge :) ?
Thanks in advance!
Code of Conduct

[X] I agree to follow this project's Code of Conduct

I'd like to work on supporting ADLS backed Delta Lake tables. @MrPowers can you assign this issue to me?

@ognis1205 I've opened a draft PR to support ADLS backed Delta Tables. There are some open questions, so it would be great if you could take a look.


spirv-fuzz: Side effects in InstructionFromMessage

InstructionFromMessage updates module's id bound. Consider removing this functionality.

I'm going to close this as the updating of the module's id bound is deliberate and is documented in the comment for the function.  This is used by TransformationAddFunction: the instructions for the function are messages in the transformation's protobuf, and as we add the corresponding instructions to the module we want the id bound to increase.


Display ellipsis in all queries

Previously, the Query List panel on the UI was adding ellipsis only on query string length. This causes querys with multiple lines to not be truncated, and appear incomplete on the query list.
This Pull Request fix this issue https://github.com/prestosql/presto/issues/2886
Now a query with more lines than the ' maxLines ' will end on the maxLines + 1 th line, followed by a new line and ...
the code below is an example of the expected output.
select (
c.name,
c.address,
c.nationkey,
c.phone,c.name
)
...

Could you share screenshot before and after the change?

Sure!
Here are the two queries I used to test. The first have 8 lines on the command line, and the other has 11.
select (
c.name,
c.address,
c.nationkey,
c.phone,c.name
)
from customer as c

and
select (
c.name,
c.address,
c.nationkey,
c.phone,c.name
)
from customer as c
join
nation as n
on ( n.nationkey = c.nationkey)

Here's a screenshot on the master branch. The query text was only trimmed on it's length, and new lines could easily break the container:

Now a screenshot after my fixes, where the query is trimmed by whatever comes first, more than 6 lines or the already in use length limit.

I also tested it with failed queries, where it's easier to see both trimming working:

please rebase instead of using merge-branch

I'm using git rebase, but it's asking for a merge whenever I try to pull from origin, https://github.com/prestosql/presto/pull/6216/commits/17d44ce174119fac347e2d7390ca5f861fb2e3ee was pushed after a git rebase. I need to redo it and discard the merge commit?

you can:

squash all your commits
cherry pick it on top of master
force push into branch

I think I've done it now

merged, thanks!


Update whatwg-fetch.d.ts

Was doc/whatwg-fetch.d.ts added through any kind of tooling, or was a static copy included? I'm trying to get DefinitelyTyped/DefinitelyTyped/pull/8887 merged in here, but I'm not sure how to go about it.

I've create #54 assuming that doc/whatwg-fetch.d.ts is a static copy.


Black Screen when switching to gamepadui

Im running an I5-9500 (dell 7070 sff) I put in an old amd hd 8570 1gb. It got to steam bigpicture mode. I signed in and then went to terminal to switch mode. It's that point it black screens. I decided the GPU was too old so bought a slightly newer AMD 4100 2gb GPU but this did the same thing. I have even updated the firmware. I can press CTRL+ALT+F3 and it shows the cmd line for 3 seconds then goes back to _
If i am quick, I can regain control and get back to desktop mode.
Looking in the Xorg.0.log I can see it trying lots of resolutions but failing to load.
(WW) AMDGPU(0): Output DisplayPort-0: Strange aspect ratio (1193/336), consider adding a quirk
It could be my stupid monitor which often causes me headaches... Samsung Odyssey G9, 5120x1440 I am running it at half resolution though. I have also tried another monitor and now an old TV. could be unrelated.
I am primarily a windows user but happy to dig further if it will help.
Might be worth noting I had the same issue on holoiso.
Anyway, I'm not expecting a fix I just hope this helps, I saw some other people with similar issue.
This was meant to be a trial run for when I finally get a UM690 in february. Hopefully that'll be less troublesome haha.

I am afraid to disappoint you but I see the same black screen on UM690.

Sadly I suspect this would end up being the case, no worries. I will continue to troubleshoot when I get mine. Given that many people seem to have managed to get this running, it's either a new issue or an issue outside the PC hardware (ie, monitor, cable or protocol)


Support navigation during a Cupertino back gesture

Programmatically performing navigation during a Cupertino back gesture was causing buggy state. This PR attempts to fix it by canceling the gesture when any navigation happens.
Currently this is just a proof of concept. It only fixes the case of a pop happening during the gesture, but weird things also happen with a push. I tried it, and if I hold the drag gesture so that it's more than halfway back, then it gets stuck when the push happens.
Listening to isCurrent instead of using PopScope didn't work because the change came after the drag gesture had already ended.
CC @maRci002
Fixes https://github.com/flutter/flutter/issues/141268

The example app I was ultimately using to test this
import 'dart:async';

import 'package:flutter/cupertino.dart';
import 'package:flutter/gestures.dart';
import 'package:flutter/material.dart';

class AppScrollBehavior extends MaterialScrollBehavior {
  const AppScrollBehavior();

  @override
  Set<PointerDeviceKind> get dragDevices => const {...PointerDeviceKind.values};
}

void main() {
  runApp(const MainApp());
}

class MainApp extends StatelessWidget {
  const MainApp({super.key});

  @override
  Widget build(BuildContext context) {
    return const CupertinoApp(
      home: ScreenOne(),
    );
  }
}

class ScreenOne extends StatelessWidget {
  const ScreenOne({super.key});

  @override
  Widget build(BuildContext context) {
    return CupertinoPageScaffold(
      child: Center(
        child: TextButton(
          onPressed: () {
            Navigator.of(context).push(
              CupertinoPageRoute(
                title: 'Two',
                builder: (_) => const ScreenTwo(),
              ),
            );
          },
          child: const Text('Go to Screen Two'),
        ),
      ),
    );
  }
}

class ScreenTwo extends StatefulWidget {
  const ScreenTwo({super.key});

  @override
  State<ScreenTwo> createState() => _ScreenTwoState();
}

class _ScreenTwoState extends State<ScreenTwo> {
  int seconds = 5;
  late Timer timer;

  @override
  void initState() {
    super.initState();

    timer = Timer.periodic(
      const Duration(seconds: 1),
      (Timer timer) {
        final ModalRoute<dynamic>? route = ModalRoute.of(context);
        if (!mounted || route == null || !route.isCurrent) {
          timer.cancel();
          return;
        }
        setState(() => seconds--);

        if (seconds == 0) {
          timer.cancel();
          //pop();
          //maybePop();
          navigate();
        }
      },
    );
  }

  @override
  void dispose() {
    timer.cancel();
    super.dispose();
  }

  void pop() {
    Navigator.of(context).pop();
  }

  void maybePop() {
    Navigator.of(context).maybePop();
  }

  void navigate() {
    Navigator.of(context).push(
      CupertinoPageRoute(
        title: 'Three',
        builder: (_) => const ScreenThree(),
      ),
    );
  }

  @override
  Widget build(BuildContext context) {
    return PopScope(
      canPop: true,
      onPopInvoked: (didPop) => print('ScreenTwo onPopInvoked didPop? $didPop'),
      child: CupertinoPageScaffold(
        child: Center(
          child: Column(
            mainAxisSize: MainAxisSize.min,
            children: [
              Text('Auto pop in: $seconds'),
              const SizedBox(height: 8),
              TextButton(
                onPressed: pop,
                child: const Text('Pop'),
              ),
              const SizedBox(height: 8),
              TextButton(
                onPressed: maybePop,
                child: const Text('Maybe Pop'),
              ),
              const SizedBox(height: 8),
              TextButton(
                onPressed: navigate,
                child: const Text('Go to Screen Three'),
              ),
            ],
          ),
        ),
      ),
    );
  }
}

class ScreenThree extends StatelessWidget {
  const ScreenThree({super.key});

  @override
  Widget build(BuildContext context) {
    return PopScope(
      canPop: true,
      onPopInvoked: (didPop) =>
          print('ScreenThree onPopInvoked didPop? $didPop'),
      child: const CupertinoPageScaffold(
        child: Center(
          child: Text('Hello World!'),
        ),
      ),
    );
  }
}

pop
push

I think the easiest solution would be to access the current _RouteLifecycle:
https://github.com/flutter/flutter/blob/67457e669f79e9f8d13d7a68fe09775fefbb79f4/packages/flutter/lib/src/widgets/navigator.dart#L2931
The Route could have a getter similar to this::
  bool get isOk {
    if (_navigator == null) {
      return false;
    }
    final _RouteEntry? currentRouteEntry = _navigator!
        ._firstRouteEntryWhereOrNull(_RouteEntry.isRoutePredicate(this));

    if (currentRouteEntry == null) {
      return false;
    }

    return const <_RouteLifecycle>[
      _RouteLifecycle.popping,
      _RouteLifecycle.pushing
    ].contains(currentRouteEntry.currentState);
  }

@maRci002 I tried your approach but I always seemed to read _RouteLifecycle.idle at the time that the drag finished. Just checking isCurrent and isActive seems to work, though.


problem with npm install node-sass in Ubuntu: binaries not found

npm install node-sass

<EMAIL_ADDRESS>install /home/davidm/trash/prueba/node_modules/node-sass
node scripts/install.js

Can not download file from https://raw.githubusercontent.com/sass/node-sass-binaries/v2.0.1/linux-ia32-node-0.13/binding.node

<EMAIL_ADDRESS>postinstall /home/davidm/trash/prueba/node_modules/node-sass
node scripts/build.js

module.js:338
throw err;
^
Error: Cannot find module '/home/davidm/trash/prueba/node_modules/node-sass/node_modules/pangyp/bin/node-gyp'
at Function.Module._resolveFilename (module.js:336:15)
at Function.Module._load (module.js:278:25)
at Function.Module.runMain (module.js:501:10)
at startup (node.js:129:16)
at node.js:814:3
Build failed

Are you sure it is looking for node v0.13 binary? Is it released already?
For node.js, we only provide binaries for the latest stable (0.12) and the penultimate stable (0.10) version of node.js. Note that v0.11 was unstable branch.

I'm sorry. v0.13 it is unstable (current master in https://github.com/joyent/node)


Status bar issue in iOS13

If we open another view controller from side menu which is connected via push segue, then, the title bar along with back button moves upwards and gets very close to status bar. It happens in iOS 13 and onwards only. Does anyone have a solution to this problem?

I'm seeing the same issue, though in my case it's when I do a modal presentation and then dismiss the presented view controller. The main container's margins somehow gets shrunk towards the edges.

Same issue. I commented maincontainerview.transform and it worked but this transform method is providing scale animation and looks nice. I could not solve it right now but I am working on it.

same issue with me..
Have you fond the fix for it?

I am also facing the similar issue. The title and the bar button icons on the controller pushed from the SideMenu move upwards and the navigation bar reduces in height. Did anyone find any solution for it?

got stuck with same issue

got stuck with same issue


(feat):Enhanced language detection

with https://github.com/aboSamoor/pycld2
checkout https://github.com/Mimino666/langdetect/issues/51#issuecomment-538574367

pycld https://github.com/aboSamoor/pycld2/issues/60

After evaluation, this work will no longer be continued.


[3.8.0] Fuel Not Consumed While Server is Closed

General information:
Project issue is for:
Server
Expected result:
Expected amount of fuel is used up while server is closed.
Actual result:
The same amount of fuel is present no matter the time duration between when it was closed and when it opened again.
How to reproduce:

Turn on power in hideout
Exit game and close server
Wait at least an hour
Reopen server and game and check fuel amount

Log files:
Attached below.
Profile:
Attached below.

Attachments

log_2024.01.27_22-58-02_<IP_ADDRESS>.28476.zip (65 KiB)
server-2024-01-27.log (2924 KiB)
LogOutput.log (135 KiB)
65b20fb90005537e7daf9f51.zip (81 KiB)

Originally written by PhantomInTime

The logs attached in the first post are from before reopening the server. These are the logs after reopening the server a little over 24 hours later when I should've had only about 10 hours of fuel left.

Attachments

server-2024-01-28.log (25 KiB)
65b20fb90005537e7daf9f51.zip (80 KiB)
log_2024.01.29_1-56-24_<IP_ADDRESS>.28476.zip (12 KiB)
LogOutput.log (8 KiB)

Originally written by PhantomInTime

Hopefully resolved with: https://dev.sp-tarkov.com/SPT-AKI/Server/commit/ad0489a9e72e08611549bf3d6ea5538475e6afed

Originally written by chomp

Won't be able to try this tonight, so may need some others. Other concern I have is will this properly increment the Hideout Management skill also? Burning fuel is meant to increase that skill along with other things like air filters and water filters.

Originally written by PhantomInTime

It should yes, the drain amount is calculated based on the time since last update() has run, if its 12 hours, that's 12 hours worth of fuel being used which is going to give you lots of hideout skill points

Originally written by chomp

I was able to test it last night. Pulled latest before booting up the server and sure enough I had about 24 hours of fuel that had been used up and hideout management properly incremented.
Issue can be closed.

Originally written by PhantomInTime


Use keywords instead of strings in coinbase plugin

The test needs to be fixed: make the type string lowercase before checking its value.

@eprbell test should be fixed (passing locally).


Please confirm these before moving forward

[X] I have searched for my issue and not found a work-in-progress/duplicate/resolved issue.
[X] I have not been informed if the issue is resolved in a preview version of the winget client.

Category of the issue
Other
Brief description of your issue
Small nit-pick, but whenever WinGet updates I always get jumpscared by "download.exe wants to run as administrator", which looks very ominous and provides very little feedback to what needs administrator perms.
I'd like it changed to the actual filename of the download (qbittorrent_4.6.0_lt20_qt6_x64_setup.exe in my case), or alternatively something the likes of "winget-install" or "qBitTorrent.qBitTorrent" (package ID).
It does not appear to be something related to the metadata of the package, so I'm guessing the client will need to be modified.
Steps to reproduce
Run winget install qBittorrent.qBittorrent
Actual behavior
An administrator prompt pops up asking you to permit download.exe
Expected behavior
An administrator prompt popping up asking you to permit an executable the likes of qbittorrent_4.6.0_lt20_qt6_x64_setup.exe
Environment
Windows Package Manager v1.6.2771

Windows: Windows.Desktop v10.0.22621.2428
System Architecture: X64
Package: Microsoft.DesktopAppInstaller v1.21.2771.0

Admin Setting                             State
--------------------------------------------------
LocalManifestFiles                        Disabled
BypassCertificatePinningForMicrosoftStore Disabled
InstallerHashOverride                     Disabled
LocalArchiveMalwareScanOverride           Disabled

Screenshots and Logs
No response

@denelon - Seems more like a cli issue?


👷🏼♂[stdlib] Experiment with alternative ways to do grapheme breaking

This is a draft attempt at eliminating redundant lookups of Unicode properties during the grapheme breaking process. My hope is that this will (eventually) considerably speed up String's character views, but that will take a bit of work -- String indexing operations currently have no easy way to preserve information about scalars that lie on a Character boundary. (Unfortunately, most scalars tend to lie on a Character boundary...)
When finding grapheme breaks in a string such as ”Cafe\u{301}”, operations such as String.count currently execute the following independent steps:
var state = _GraphemeBreakingState()
state.shouldBreak(“C”, “a”) // 2 lookups, true
state.shouldBreak(“a”, “f”) // 2 lookups, true
state.shouldBreak(“f”, “e”) // 2 lookups, true
state.shouldBreak(“e”, “\u{301}”) // 2 lookups, false
// Total: 8 property lookups

Each shouldBreak(a, b) invocation individually looks up grapheme breaking properties for both scalar values, which includes calling nontrivial functions such as hasBreakWhenPaired and Unicode._GraphemeBreakProperty(from:).
This means that during the course of the string operation, we’re retrieving this information twice for each scalar, so we are duplicating nontrivial work. We should be able to significantly speed up grapheme breaking by caching this metadata directly in the state, and preserving it across character boundaries.
var state = _GraphemeBreakingState()
state.hasBreak(before: “C”) // 1 lookup, true
state.hasBreak(before: “a”) // 1 lookup, true
state.hasBreak(before: “f”) // 1 lookup, true
state.hasBreak(before: “e”) // 1 lookup, true
state.hasBreak(before: “\u{301}”) // 1 lookup, false
// Total: 5 property lookups

To make this work well, I think we need to change String.Index to cache the grapheme breaking property of the addressed scalar, rather than the stride of the addressed Character, as it currently does. This will somewhat slow down code that iterates over strings using indices, but I suspect the improved grapheme breaking throughput will largely make up for this. (If it doesn't, we'll "just" need to also replace the huge switch statements in the current core grapheme breaking algorithms with an artisanal, small batch lookup table.)
Additionally, having indices only store information about their addressed scalar would eliminate a large source of index invalidation surprises: currently range replacements in native Swift strings sometimes invalidate indices way below the slice that they affected, which usually comes as a shock to people.
In case this proves impractical to do in String itself, at minimum we could still use this to speed up _CharacterRecognizer.
rdar://103970243

I hope to see some improvements in CharacterRecognizer benchmarks at least. (String walks probably won't like the new indices yet, but we'll see...)
@swift-ci benchmark

@swift-ci benchmark

@swift-ci benchmark


If its not possible install on samsung a14, THEN HOW ITS POSSIBLE N INSTALL ANDROID 13 REDMI NOTE 6 PRO!

Your Question
How did you make this compatible with 2018 DEVICE? I wasn't expecting to see this question.

:(

Geode targets android 6 and newer. not specific to any manifacturer

oh

then samsung galaxy tab 10.1 (normal 4g + wifi, p4 model) with android 7 custom rom????


Build without HPC registry?

Hi! I'd like to test this out, but I don't want to have to bring in another service, aside from using Azure. Do you have an example for that? Thanks!

Hi @vsoch, thanks for your interest. I'm not with HashiCorp anymore so I'm really not maintaining these demos going forward. This one was specifically for a session on Azure + HCP Packer, but if you just want to learn how to use Packer to build Azure images, there's a bunch of good blog posts out there.
I had more recent demos in my hcp-packer-ubuntu22 repo that use more modern Azure Compute Gallery publishing, but those also use HCP. To use them without HCP, you'd just have to remove the hcp_packer_registry block from the build blocks. Then for the "downstream" builds, replace the hcp-packer-artifact data sources and instead just reference the parent image directly from the Compute Gallery following the docs.


Add more tests of ASP.NET Core receivers

Follow-on work item for #180.
Remaining work includes at least:

add functional tests involving signature verification, including unused Stripe and Trello resources

[ ] Dropbox
[ ] GitHub
[ ] Pusher
[ ] Stripe
[ ] Trello

add functional tests involving request bodies for the samples still missing them

[ ] DynamicsCRM
[ ] MailChimp
[ ] Slack

add functional tests of a few other scenarios across the samples

[ ] HTTP
[ ] incorrect signature
[ ] fill gaps e.g. GitHub isn't testing no body case
[ ] unexpected Content-Type

[ ] add unit tests of other application model providers, not just WebHookActionModelPropertyProvider
[ ] add unit tests of other core filters, not just WebHookVerifySignatureFilter
add unit tests of remaining receiver-specific filters

[ ] DropboxVerifySignatureFilter
[ ] GitHubVerifySignatureFilter
[ ] PusherVerifySignatureFilter
[ ] SalesforceVerifyOrganizationIdFilter
[ ] SlackVerifyTokenFilter
[ ] StripeTestEventRequestFilter
[ ] StripeVerifyNotificationIdFilter

add unit tests of other core classes

[ ] WebHookFilterProvider
[ ] WebHookHttpRequestUtilities
[ ] WebHookMetadataProvider
[ ] WebHookRequestReader

add unit tests of other receiver-specific classes

[ ] SalesforceNotificationsModelBinderProvider
[ ] SalesforceNotificationsModelBinder

Can you add the Vend API to this list as well please?

@Yusha813 you've already opened an issue about supporting the Vend API. Testing should be added as part of #289 or after that work is done. There's no action to take here in the meantime.

Parking in dougbu/add.functional.tests.290.1 branch

Suggest we un-assign this issue, remove the Mapped to Big Rock label, and move it to the 1.0.0-preview3 milestone

This issue was moved to aspnet/AspLabs#40


NFS export options for Filestore

Add NFS export options for Filestore that are necessary for clusters running in non-RFC1918 network ranges.

/gcbrun

Thank you for contribution!


Suggestion: short help

Hi @lelegard ,
A simple enhancement for a more user-friendly funcionality.

At time, the call to tsp --help prints the help. That's OK.
But a call to tsp (without any parameter) does nothing, no help, and needs to be closed (^C)

I feel it's preferable to print a short help when calling with no parameters.
You agree?
Regards.

Hi,
This is a standard situation. The command tsp is very similar to the standard Unix command cat: it does a lot of things depending on options but the default input is stdin and the default output is stdout. The only difference is that cat processes bytes while tsp processes TS packets.
Typing cat without parameter in a shell also does nothing until you send an EOF (^D) or interrupt (^C). This is also the case for tstables, tstabdump, tsdump and zillions of classical Unix / Linux / macOS / Solaris / etc. commands.

OK. Thank you for the response.
I close the issue! :wink:


EKS cluster is not showing logs for some apps. Turns out some of are going to this namespace called ".Orphaned"

Hello Team,
I have a customer whose EKS cluster is not showing logs for some apps. Turns out some of us are going to this namespace called “.Orphaned”. The logging from Splunk connects the agent that forwards the logs to Splunk with the node that was sending logs to the Orphaned namespace.
This is a line from the malfunctioning exporter’s log file:
2023-01-05 23:47:26 +0000 [info]: #0 stats - namespace_cache_size: 3, pod_cache_size: 3, id_cache_orphaned_record: 13648170, namespace_cache_api_nil_error: 13648171, pod_cache_api_nil_error: 13648172, id_cache_miss: 13664128
The line from another exporter:
2023-01-05 23:48:35 +0000 [info]: #0 stats - namespace_cache_size: 5, pod_cache_size: 17, namespace_cache_api_updates: 513, pod_cache_api_updates: 513, id_cache_miss: 513, pod_cache_host_updates: 17, pod_watch_gone_errors: 1, pod_watch_gone_notices: 1
Notice that the second one does not have highlighted output.
The customer has followed this link to send K8s logs into Splunk:
https://github.com/splunk/splunk-connect-for-kubernetes
Also, I have found this link for a similar issue:
https://bugzilla.redhat.com/show_bug.cgi?id=1722380
Is this a bug or something related to the configuration? Any insight into this will be much appreciated!
What happened:
Logging from certain pods (spanning multiple namespaces, listed below) on Kubernetes is being sent to .orphaned.
Namespaces, where the logs are expected to go instead of .orphaned,
my-splunk-connect-splunk-kubernetes-logging-4b42s.log
kube2iam-tdnkq.log
are: esri-timecard, sam-api, kube-system
What you expected to happen:
The logs are supposed to go to the above-mentioned namespace instead of the .orphaned for the customer.

How to reproduce it (as minimally and precisely as possible):
This issue is specific to the customer's environment-specific
Anything else we need to know?: N/A
Environment: Customer on Classic Cloud Cluster Architecture

Kubernetes version (use kubectl version): N/A
Ruby version (use ruby --version): N/A
OS (e.g: cat /etc/os-release):
NAME="Ubuntu"
VERSION="18.04.6 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.6 LTS"
VERSION_ID="18.04"
Splunk version: 9.0.2209.2
Splunk Connect for Kubernetes helm chart version: N/A
Others: N/A

Values.yml file ::
USER-SUPPLIED VALUES:
global:
kubernetes:
clusterName: eks-ist-prd
logLevel: info
metrics:
service:
enabled: true
headless: true
monitoring_agent_enabled: null
monitoring_agent_index_name: null
prometheus_enabled: null
serviceMonitor:
additionalLabels: {}
enabled: false
interval: ""
metricsPort: 24231
scrapeTimeout: 10s
splunk:
hec:
caFile: null
clientCert: null
clientKey: null
consume_chunk_on_4xx_errors: null
fullUrl: null
host: http-inputs-esri.splunkcloud.com
indexName: esri_kube_prd
indexRouting: null
insecureSSL: null
port: 443
protocol: https
token: PLACEHOLDER
splunk-kubernetes-logging:
affinity: {}
buffer:
'@type': memory
chunk_limit_records: 100000
chunk_limit_size: 20m
flush_interval: 5s
flush_thread_count: 1
overflow_action: block
retry_max_times: 10
retry_type: periodic
retry_wait: 30
total_limit_size: 600m
charEncodingUtf8: false
containers:
localTime: false
logFormat: null
logFormatType: json
path: /var/log
pathDest: /var/lib/docker/containers
refreshInterval: null
removeBlankEvents: true
customFilters: {}
customMetadata: null
customMetadataAnnotations: null
enabled: true
environmentVar: null
extraVolumeMounts: []
extraVolumes: []
fluentd:
exclude_path: null
path: /var/log/containers/*.log
image:
name: splunk/fluentd-hec
pullPolicy: IfNotPresent
pullSecretName: null
registry: docker.io
tag: 1.2.12
usePullSecret: false
indexFields: []
journalLogPath: /run/log/journal
k8sMetadata:
cache_ttl: 3600
podLabels:
- app
- k8s-app
- release
watch: true
kubernetes:
clusterName: null
securityContext: false
logLevel: null
logs:
dns-controller:
from:
pod: dns-controller
multiline:
firstline: /^\w[0-1]\d[0-3]\d/
sourcetype: kube:dns-controller
dns-sidecar:
from:
container: sidecar
pod: kube-dns
multiline:
firstline: /^\w[0-1]\d[0-3]\d/
sourcetype: kube:kubedns-sidecar
dnsmasq:
from:
pod: kube-dns
multiline:
firstline: /^\w[0-1]\d[0-3]\d/
sourcetype: kube:dnsmasq
docker:
from:
journald:
unit: docker.service
sourcetype: kube:docker
etcd:
from:
container: etcd-container
pod: etcd-server
etcd-events:
from:
container: etcd-container
pod: etcd-server-events
etcd-minikube:
from:
container: etcd
pod: etcd-minikube
kube-apiserver:
from:
pod: kube-apiserver
multiline:
firstline: /^\w[0-1]\d[0-3]\d/
sourcetype: kube:kube-apiserver
kube-audit:
from:
file:
path: /var/log/kube-apiserver-audit.log
sourcetype: kube:apiserver-audit
timestampExtraction:
format: '%Y-%m-%dT%H:%M:%SZ'
kube-controller-manager:
from:
pod: kube-controller-manager
multiline:
firstline: /^\w[0-1]\d[0-3]\d/
sourcetype: kube:kube-controller-manager
kube-dns-autoscaler:
from:
container: autoscaler
pod: kube-dns-autoscaler
multiline:
firstline: /^\w[0-1]\d[0-3]\d/
sourcetype: kube:kube-dns-autoscaler
kube-proxy:
from:
pod: kube-proxy
multiline:
firstline: /^\w[0-1]\d[0-3]\d/
sourcetype: kube:kube-proxy
kube-scheduler:
from:
pod: kube-scheduler
multiline:
firstline: /^\w[0-1]\d[0-3]\d/
sourcetype: kube:kube-scheduler
kubedns:
from:
pod: kube-dns
multiline:
firstline: /^\w[0-1]\d[0-3]\d/
sourcetype: kube:kubedns
kubelet:
from:
journald:
unit: kubelet.service
multiline:
firstline: /^\w[0-1]\d[0-3]\d/
sourcetype: kube:kubelet
nodeSelector:
beta.kubernetes.io/os: linux
podAnnotations: null
podSecurityPolicy:
apiGroup: policy
apparmor_security: true
create: false
priorityClassName: null
rbac:
create: true
openshiftPrivilegedSccBinding: false
resources:
limits:
cpu: 50m
memory: 400Mi
requests:
cpu: 45m
memory: 200Mi
secret:
create: true
name: null
sendAllMetadata: false
serviceAccount:
create: true
name: null
sourcetypePrefix: kube
splunk:
hec:
caFile: null
clientCert: null
clientKey: null
consume_chunk_on_4xx_errors: null
fullUrl: null
host: null
indexName: null
insecureSSL: null
port: null
protocol: null
token: null
ingest_api:
debugIngestAPI: null
eventsEndpoint: null
ingestAPIHost: null
ingestAuthHost: null
serviceClientIdentifier: null
serviceClientSecretKey: null
tenant: null
tokenEndpoint: null
tolerations:

effect: NoSchedule
key: node-role.kubernetes.io/master
splunk-kubernetes-metrics:
affinity: {}
aggregatorBuffer:
'@type': memory
chunk_limit_records: 10000
chunk_limit_size: 10m
flush_interval: 5s
flush_thread_count: 1
overflow_action: block
retry_max_times: 10
retry_type: periodic
retry_wait: 30
total_limit_size: 400m
aggregatorTolerations: {}
buffer:
'@type': memory
chunk_limit_records: 10000
chunk_limit_size: 10m
flush_interval: 5s
flush_thread_count: 1
overflow_action: block
retry_max_times: 10
retry_type: periodic
retry_wait: 30
total_limit_size: 400m
customFilters: {}
customFiltersAggr: {}
enabled: false
environmentVar: null
environmentVarAgg: null
image:
name: splunk/k8s-metrics
pullPolicy: IfNotPresent
pullsecretName: null
registry: docker.io
tag: 1.1.11
usePullSecret: false
imageAgg:
name: splunk/k8s-metrics-aggr
pullPolicy: IfNotPresent
pullsecretName: null
registry: docker.io
tag: 1.1.11
usePullSecret: false
kubernetes:
bearerTokenFile: null
caFile: null
clusterName: null
insecureSSL: false
kubeletAddress: null
kubeletPort: null
kubeletPortAggregator: null
secretDir: null
useRestClientSSL: true
logLevel: null
metricsInterval: 15s
nodeSelector:
beta.kubernetes.io/os: linux
podAnnotations: null
podAnnotationsAgg: null
podSecurityPolicy:
apiGroup: policy
apparmor_security: true
create: false
priorityClassName: null
priorityClassNameAgg: null
rbac:
create: true
resources:
fluent:
limits:
cpu: 200m
memory: 300Mi
requests:
cpu: 200m
memory: 300Mi
secret:
create: true
name: null
serviceAccount:
create: true
name: null
usePullSecrets: false
splunk:
hec:
caFile: null
clientCert: null
clientKey: null
consume_chunk_on_4xx_errors: null
fullUrl: null
host: null
indexName: null
insecureSSL: null
port: null
protocol: null
token: null
tolerations:
effect: NoSchedule
key: node-role.kubernetes.io/master
splunk-kubernetes-objects:
affinity: {}
buffer:
'@type': memory
chunk_limit_records: 10000
chunk_limit_size: 20m
flush_interval: 5s
flush_thread_count: 1
overflow_action: block
retry_max_times: 10
retry_type: periodic
retry_wait: 30
total_limit_size: 600m
checkpointFile:
name: kubernetes-objects.pos
volume: null
customFilters: {}
enabled: false
environmentVar: null
image:
name: splunk/kube-objects
pullPolicy: IfNotPresent
pullSecretName: null
registry: docker.io
tag: 1.1.11
usePullSecret: false
indexFields: []
kubernetes:
bearerTokenFile: null
caFile: null
clientCert: null
clientKey: null
clusterName: null
insecureSSL: false
secretDir: null
url: null
logLevel: null
nodeSelector:
beta.kubernetes.io/os: linux
objects:
core:
v1:

name: pods
name: namespaces
name: nodes
mode: watch
name: events
podAnnotations: null
podSecurityPolicy:
apiGroup: policy
apparmor_security: true
create: false
priorityClassName: null
rbac:
create: true
resources:
requests:
cpu: 100m
memory: 200Mi
secret:
create: true
name: null
serviceAccount:
create: true
name: null
usePullSecrets: false
splunk:
hec:
caFile: null
caPath: null
clientCert: null
clientKey: null
consume_chunk_on_4xx_errors: null
fullUrl: null
host: null
indexName: null
indexRouting: null
insecureSSL: null
port: null
protocol: null
token: null
tolerations: []

Raw sample data of k8s.log
Splunk log of k8s.log

Kubernetes version (use kubectl version)

1.21

Ruby version (use ruby --version)

The SCK s/w dependencies are bundled into the container image that is deployed through the Daemon set. SCK is running on an out-of-the-box EKS with the following OS/Runtime configuration.

OS (e.g: cat /etc/os-release)

OS: linux (amd64)
OS Image: Amazon Linux 2
Kernel version: 5.4.141-67.229.amzn2.x86_64

Splunk Connect for Kubernetes helm chart version

splunk-connect-for-kubernetes-1.4.15

Is the version of the SCK setup is same throughout the deployment (with version name)?

It appears to be the same. All are using the same images and are deployed from the same DaemonSet. I do not see any labels other than this - which is consistent across all the pods.( controller-revision-hash=59cfbf9f74)

Hey @aseemand, from where Splunk log of k8s.log is coming from? And what does it indicate?
It is expected behaviour from the metadat_filter plugin.
In one of the issue, they stated:
Pod info is scraped from the container log file name which provides the name of namespace but not its unique id. This means in a scenario where NS is created, deleted, recreated from the same name and logs still exist on disk, you could potentially be trying to associate meta from the original version of the NS even though its uuid is different then the latest.
The plugin has some logic to try and cache metadata based on ns, podname, uids, and creation dates. If using this combo of info it is unable to "match" a log entry to a known container, it will fall back to orphan. If you are creating a number of short lived pods you may be pushing the meta out of the LRU cache.
Source: https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter/issues/234#issuecomment-634931613

Hi @harshit-splunk ,
I got the update from the customer, can you please look into this and help me with the mentioned concerns:
The resolution for this issue is unclear in your previous email. Splunk connect is misrouting logs and this still the case – we need a way to get these logs back to the correct namespaces. Please advise – this is happening on the production cluster.
In your response, you mentioned this is expected behavior for the plugin but none of those scenarios from the comment apply.
None of the namespaces that are misrouting have been deleted or re-created.
We are not creating workloads that are short-lived – the pod that is misrouting logs has been stable for the last 42 days without any restarts.
Moreover, the kube-system namespace holds multiple pods, and only one of those pods is misrouting. (That pod also belongs to the K8s node which contains other pods that are also misrouting.) This is the same node from which I shared log snippets earlier.
The screenshot below is from a query that filters out all the “.Orphaned” namespace entries. 100% of them are coming from one node from the cluster.

Thanks!


Add feature: Rocket.chat support using API

Hi,
Just a reminder that Wefork has to support Rocket.chat.
Related to: https://github.com/RocketChat/Rocket.Chat/issues/672 and https://discuss.wekan.io/t/rocket-chat-integration/15

First we would need to have a generic API/webhook features to then
integrate to Rocket.Chat among whatever :)
On 21-12-2016 11:56, HLFH wrote:

Hi,
Just a reminder that Wefork has to support Rocket.chat.
Related to: RocketChat/Rocket.Chat#672
https://github.com/RocketChat/Rocket.Chat/issues/672 and
https://discuss.wekan.io/t/rocket-chat-integration/15
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/wefork/wekan/issues/54, or mute the thread
https://github.com/notifications/unsubscribe-auth/ABxvHZUi6HJf_SoJ8g2MwULTFzNTzceYks5rKRPggaJpZM4LS3ou.

--
I use PGP to protect our privacy, if you want to know more, you can
follow this
https://emailselfdefense.fsf.org/en/
If you have further questions, please do not hesitate to ask.
You can verify my public key here: https://keybase.io/pierreozoux

It's good to have this listed also and in mind when planning #52 that is related #43 API is part of it.

This issue was moved to wekan/wekan#803

This issue was moved to wekan/wekan#805


Fix #104

Codecov Report

Merging #105 (eece7b2) into master (f43ea87) will decrease coverage by 0.14%.
The diff coverage is 85.71%.

@@            Coverage Diff             @@
##           master     #105      +/-   ##
==========================================
- Coverage   79.79%   79.64%   -0.15%
==========================================
  Files           4        4
  Lines         960      963       +3
==========================================
+ Hits          766      767       +1
- Misses        194      196       +2

Impacted Files
Coverage Δ

TopasOpt/TopasScriptGenerator.py
66.66% <80.00%> (ø)

TopasOpt/Optimisers.py
82.36% <100.00%> (-0.18%)
:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


New source: Rezo Scans

Closes #5228
Essentially copied from luascans (luacomic) and created icons.
Checklist:

[ ] Updated extVersionCode value in build.gradle for individual extensions
[ ] Updated overrideVersionCode or baseVersionCode as needed for all multisrc extensions
[x] Referenced all related issues in the PR body (e.g. "Closes #xyz")
[ ] Added the isNsfw = true flag in build.gradle when appropriate
[ ] Have not changed source names
[ ] Have explicitly kept the id if a source's name or language were changed
[x] Have tested the modifications by compiling and running the extension through Android Studio
[x] Have removed web_hi_res_512.png when adding a new extension

No trailing slash for the baseUrls

oops, thanks!


phone-number: Change types to return Maybe.

We are considering changing the interface of the problem phone-number. The idea (#115)  is to allow a more idiomatic solution to this problem, returning a Maybe, an Either or something else, instead of signalling invalid input with the String 00000000
It could be something like this:
module Phone (areaCode, number, prettyPrint) where

areaCode    :: String -> Maybe String
number      :: String -> Maybe String
prettyPrint :: String -> Maybe String

This is just and sketch, and it can/should change. It depends only on the feedback here!
Any suggestion or comment is welcome! :smile:

a Maybe String would be perfect here. I was thinking the same when I did this exercise.

I already wrote a solution but I'm waiting for #159 to open the PR, because I changed a dependency that's already listed there. So I'm assigning it to myself to avoid duplicated work.

Oops! Didn't see your comment @rbasso. Feel free to merge or decline my pr (https://github.com/exercism/xhaskell/pull/163) at your discretion.

You already did it, so I'll merge yours. :smile:


MSL: Print early_fragment_tests specifier before fragment

The compiler in 10.14 reports an error that the attribute cannot be applied to types if the specifier is printed before the fragment specifier.
I can't see anywhere in the spec that suggests where the specifier must be inserted, but the examples place it at the beginning.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Performance issues caused by UniFile findFile implementation

Steps to reproduce
The performance of Mihon degrades linearly with the number of downloaded chapters for a specific manga. You can experience this behavior by selecting a manga with a large number of chapters and downloading about 1,000 of them. You will then observe the following behavior:

Adding additional chapters to the download queue is extremely slow. With about 1,000 chapters downloaded, a delay of about 10 seconds is incurred per chapter added to the queue. So, for example, adding 100 additional chapters to the queue takes about 15 minutes.
Navigating between chapters in the reader is extremely slow. With about 1,000 chapters downloaded, the entire interface of Mihon freezes for about 10 seconds every time a new chapter must be loaded from disk.

Expected behavior
Adding any number of chapters to the download queue should take no observable time, because this requires at most one filesystem operation to list existing downloaded chapters, no network calls, and no significant data processing.
Loading an already-downloaded chapter from disk should take no observable time unless the chapter is large, because this requires at most one filesystem operation to read the chapter images from disk.
Actual behavior
Mihon uses an unusual library called UniFile for its filesystem operations. This is described as a fork of the Android library DocumentFile, which describes itself as follows:

Representation of a document backed by either a android.provider.DocumentsProvider or a raw file on disk. This is a utility class designed to emulate the traditional File interface. It offers a simplified view of a tree of documents, but it has substantial overhead

The substantial overhead is difficult to overstate, it turns out, when dealing with directories that have a large number of files. Observe the following. When Mihon adds chapters to the download queue, it invokes the Downloader.queueChapters method, which uses the DownloadProvider.findChapterDir method to filter out chapters that have already been downloaded:
https://github.com/mihonapp/mihon/blob/f27ca3b1b2f92258c213bca6b27d8eff4c7363ad/app/src/main/java/eu/kanade/tachiyomi/data/download/Downloader.kt#L276-L277
The DownloadProvider.findChapterDir iterates through four possible filepath structures returned by the DownloadProvider.getValidChapterDirNames method (handling both a .cbz suffix and an underscore prefix), and invokes the UniFile.findFile method for each one:
https://github.com/mihonapp/mihon/blob/f27ca3b1b2f92258c213bca6b27d8eff4c7363ad/app/src/main/java/eu/kanade/tachiyomi/data/download/DownloadProvider.kt#L84-L86
The intent of this method, which is documented upstream here, is to check for the existence of a file in a directory. Normally, this operation would be near-instantaneous, as checking for file existence is O(1). However, because the DocumentFile interface acts as an abstraction over a large number of virtual document store providers, some of which do not support random access, the implementation of findFile actually lists every child of the directory individually, fetches each of their attributes, and then iterates over the returned list to see if the desired file is found:
https://github.com/tachiyomiorg/UniFile/blob/7c257e1c642caa5ec7a41c640d61cd096f7f5c2e/library/src/main/java/com/hippo/unifile/TreeDocumentFile.java#L236-L241
This has the consequence that the performance of the method degrades linearly according to how many files exist in the directory - and it is called four times per chapter. Thus, with 1,000 chapters already downloaded, and 100 requested by the user to download, only 100 (or 400, to handle backward compatibility) system calls should be needed to check each chapter. However, due to the findFile implementation, a total of 400,000 system calls are made instead, because each chapter traverses the entire download directory. In practice, this can mean a delay of 15 minutes or more, with heavy IO usage on the device which quickly drains battery life.
Since the original issue and pull request tracker for Tachiyomi has been lost, it is unclear to me why this library was introduced in the first place, or if anybody else noticed this significant shortcoming. However, the problem completely blocks me from using the download system in Mihon at all for manga that have more than a thousand chapters, so I am willing to do the necessary work to identify an appropriate fix for this performance issue, with accompanying documentation and/or tests, that maintains an acceptable level of backwards compatibility.
Any insight from the maintainers would be appreciated, especially in terms of the rationale for the existing system around UniFile.
Crash logs
No response
Mihon version
f27ca3b1b2f92258c213bca6b27d8eff4c7363ad
Android version
Android 14 (GrapheneOS)
Device
Google Pixel 6a
Other details
This issue causes https://github.com/mihonapp/mihon/issues/669.
I have verified through added logging and timing that the code path I mentioned is responsible for the performance issues around the download queue. I have not verified specifically that it is also responsible for the reader slowdown, but this seems almost certainly to be the case to me. (My testing is impaired by the fact that downloads are so slow I cannot complete them on the development version of the app.)
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[X] I have written a short but informative title.
[X] I have gone through the FAQ and troubleshooting guide.
[X] I have updated the app to version 0.16.5.
[X] I have updated all installed extensions.
[X] I will fill out all of the requested information in this form.

I see that the reason for this "document tree" abstraction may be that it is what is returned by the directory picker / permissions model that is currently in use by Mihon:
https://developer.android.com/reference/androidx/activity/result/contract/ActivityResultContracts.OpenDocumentTree
https://github.com/mihonapp/mihon/blob/f27ca3b1b2f92258c213bca6b27d8eff4c7363ad/app/src/main/java/eu/kanade/presentation/more/settings/screen/SettingsDataScreen.kt#L107-L120
For example, on my device, the returned URI is content://com.android.externalstorage.documents/tree/primary%3AScratch%2FMihonDev/document/primary%3AScratch%2FMihonDev, rather than an actual filesystem path. I also see options for Termux and Nextcloud in the document picker, although these both seem hilariously inappropriate as storage locations for local downloads of manga.
I guess the questions are:

Is it intended that storage locations other than the local filesystem can be selected? If so, what is the use case?
Is it possible to bypass the DocumentFile API and access the filesystem in a more expedient manner? If so, should that be done by DocumentContract methods as suggested in the DocumentFile reference, or are there even lower-level interfaces?
If we are using the DocumentFile or DocumentContract API, do efficient methods exist for checking file existence? Clearly the ones we are using are wrong.
If no efficient methods exist for checking file existence in the official APIs, are there workarounds? Do we need to maintain our own cache of filesystem state? (God, I hope not.) What do other apps do?

I will do some research.

Being able to support other storage locations is just a feature of SAF, which has to be used due to scoped storage.
And those file operations are IPC calls rather than system calls, which is why they are much slower.
For this case, it can be optimized by lifting listFiles outside of the lambda.

I have gotten a local development environment set up and am setting about adjusting UniFile to use more efficient DocumentContract methods to implement the commonly used filesystem operations. Given that Mihon uses UniFile methods that have O(n) performance instead of O(1) in many different places, I think fixing the issue at the library level will be the most fruitful. Pull request to follow.

I happen to disagree with you on that point, but let's keep the discussion on-topic and related to the bug report in this thread.

https://github.com/mihonapp/mihon/pull/728 and https://github.com/tachiyomiorg/UniFile/pull/2, together, fix the reported problem. There may be other areas as well where Mihon's performance suffers due to unoptimized usage of SAF, but this at least remediates a number of problem hotspots by fixing a commonly used library subroutine - for example, this also improves the performance of createDirectory since it uses findFile as a subroutine, and even just findFile is used directly in about a dozen separate places in Mihon.

https://github.com/tachiyomiorg/UniFile/pull/2 is definitely a good improvement, but do you have benchmark results of that vs lifting listFiles?
I'm a bit concerned because TreeDocumentFile.exists is also somewhat expensive.
As I said they are IPC calls and IIRC each of them will cost about 20-30 ms.
Actually, I think we could consider running them in parallel as they are rather independent.

do you have benchmark results of that vs lifting listFiles?

Sure! To perform the following tests, I used a source with 1,000 chapters already downloaded, and then initiated the download of 50 additional chapters. Based on timestamps of added logging lines, I determined the total time from initiating the add-to-queue operation to the first chapter beginning to download. Here are the results for various implementations:

No changes: 164.295s
Your UniFile change: 80.110s (2x speedup)
My UniFile change + accompanying Mihon patch: 4.364s (38x speedup)
Precomputing file list, no other changes: 4.042s (41x speedup)

The file list precomputation I did in a very silly way just to get the performance numbers - it would have to be redone more intelligently if we move forward with that option. Here is what I did:

Patch against DocumentsContractApi21.java
diff --git i/library/src/main/java/com/hippo/unifile/DocumentsContractApi21.java w/library/src/main/java/com/hippo/unifile/DocumentsContractApi21.java
index 2dddf00..52bb777 100644
--- i/library/src/main/java/com/hippo/unifile/DocumentsContractApi21.java
+++ w/library/src/main/java/com/hippo/unifile/DocumentsContractApi21.java
@@ -26,10 +26,12 @@ import android.provider.DocumentsContract;
 import android.util.Log;

 import java.util.ArrayList;
+import java.util.HashMap;
 import java.util.List;
+import java.util.Map;

 @TargetApi(Build.VERSION_CODES.LOLLIPOP)
-final class DocumentsContractApi21 {
+public final class DocumentsContractApi21 {
     private DocumentsContractApi21() {}

     private static final String TAG = DocumentsContractApi21.class.getSimpleName();
@@ -110,7 +112,15 @@ final class DocumentsContractApi21 {
         return results.toArray(new Uri[results.size()]);
     }

+    private static Map<Uri, NamedUri[]> filesCache = new HashMap<>();
+    public static boolean cachingEnabled = false;
+
     public static NamedUri[] listFilesNamed(Context context, Uri self) {
+        if (cachingEnabled && filesCache.containsKey(self)) {
+            Log.w(TAG, "Returning from cache for URI: " + self);
+            return filesCache.get(self);
+        }
+
         final ContentResolver resolver = context.getContentResolver();
         final Uri childrenUri = DocumentsContract.buildChildDocumentsUriUsingTree(self,
                 DocumentsContract.getDocumentId(self));
@@ -139,7 +149,13 @@ final class DocumentsContractApi21 {
             closeQuietly(c);
         }

-        return results.toArray(new NamedUri[results.size()]);
+        NamedUri[] rv = results.toArray(new NamedUri[results.size()]);
+        if (cachingEnabled) {
+            Log.w(TAG, "Populating in cache for URI: " + self);
+            filesCache.put(self, rv);
+        }
+
+        return rv;
     }

     public static Uri renameTo(Context context, Uri self, String displayName) {

Patch against Downloader.kt
diff --git i/app/src/main/java/eu/kanade/tachiyomi/data/download/Downloader.kt w/app/src/main/java/eu/kanade/tachiyomi/data/download/Downloader.kt
index 7c8238d5a..9dc7698ad 100644
--- i/app/src/main/java/eu/kanade/tachiyomi/data/download/Downloader.kt
+++ w/app/src/main/java/eu/kanade/tachiyomi/data/download/Downloader.kt
@@ -1,6 +1,7 @@
 package eu.kanade.tachiyomi.data.download

 import android.content.Context
+import com.hippo.unifile.DocumentsContractApi21
 import com.hippo.unifile.UniFile
 import eu.kanade.domain.chapter.model.toSChapter
 import eu.kanade.domain.manga.model.getComicInfo
@@ -270,11 +271,17 @@ class Downloader(
     fun queueChapters(manga: Manga, chapters: List<Chapter>, autoStart: Boolean) {
         if (chapters.isEmpty()) return

+        logcat { "Turning ON caching" }
+        DocumentsContractApi21.cachingEnabled = true;
+
         val source = sourceManager.get(manga.source) as? HttpSource ?: return
         val wasEmpty = queueState.value.isEmpty()
         val chaptersToQueue = chapters.asSequence()
             // Filter out those already downloaded.
-            .filter { provider.findChapterDir(it.name, it.scanlator, manga.title, source) == null }
+            .filter {
+                logcat { "Checking download directory for chapter ${it.name}" }
+                provider.findChapterDir(it.name, it.scanlator, manga.title, source) == null
+            }
             // Add chapters to queue from the start.
             .sortedByDescending { it.sourceOrder }
             // Filter out those already enqueued.
@@ -307,6 +314,9 @@ class Downloader(
                 DownloadJob.start(context)
             }
         }
+
+        logcat { "Turning OFF caching" }
+        DocumentsContractApi21.cachingEnabled = false;
     }

     /**

One more thing I've noticed while doing this testing is that the implementation and usage of findChapterDir is extremely inefficient.  Even with the actual filesystem ops stubbed out, when combined that still loses more than 30ms per call with the repeated list scanning and such.
If we fully precomputed the filesystem queries ahead of time (I didn't test this yet because it would require a bunch of changes), then that would get things even faster than my benchmarks above with the file list precomputed.
That said, even though the list precomputation produces more speedup on its own than the improved findFile implementation, the two changes can both be applied since they produce performance improvements in different areas (e.g., the better findFile will improve reader performance as well, the issue mentioned in https://github.com/mihonapp/mihon/issues/669).
So, I would vote for the following:

I need to fix tachiyomiorg/unifile#2 to fallback to slower-but-correct behavior if somebody uses a provider other than ExternalStorageProvider - should be pretty easy.
There is a minor compilation error in tachiyomiorg/unifile#1 to fix, I expect that is also a quick solve.
We can apply both tachiyomiorg/unifile#1 and tachiyomiorg/unifile#2 as well as my accompanying https://github.com/mihonapp/mihon/pull/728, that should improve things considerably.
Next step is to make another PR for Mihon that reduces the number of repetitive filesystem queries in the first place, by pre-listing the downloaded files properly (which will be faster than my hacky implementation above).

Nice! That looks promising.
I have a POC of running findChapterDir in parallel, would you mind testing #731 on top of the findFile optimization?

I will run some tests to verify that all visible performance issues relating to large numbers of downloaded chapters are now resolved, and if so, close this issue. Otherwise, will file more PRs to address the other issues.

I compiled a release build and, let me tell you, the difference is unreal. Reader performance has gone from freezing the UI for 10-15 seconds every time you flip a page, to no perceptible delay. There are further optimizations that we could make, for example https://github.com/mihonapp/mihon/pull/731 and the caching I mentioned in point 4 under https://github.com/mihonapp/mihon/issues/705#issuecomment-2087787294, but I think this is good enough to mitigate almost all of the user-visible problems in most cases.
:tada:


[FIX] compile: gcc version > 11.3, treat -Wmaybe-uninitialized as error

my gcc version
gcc (SUSE Linux) 13.2.1 20231130 [revision 741743c028dc00f27b9c8b1d5211c1f602f2fddd]
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE

compile ninja, 1 error occurs
ninja/build/_deps/googletest-src/googletest/src/gtest-death-test.cc:1301:24: error: ‘dummy’ may be used uninitialized [-Werror=maybe-uninitialized]
 1301 |   StackLowerThanAddress(&dummy, &result);
      |   ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~

just try to ignore this error

Awesome :) Can you replace the tabs in your change with 2 spaces? Then this can be merged.

Awesome :) Can you replace the tabs in your change with 2 spaces? Then this can be merged.
sorry, my bad QAQ

Awesome :) Can you replace the tabs in your change with 2 spaces? Then this can be merged.

sorry, my bad, QAQ