Port properties in from Gunrock main repo

Summary
Ports in device properties from Gunrock's cuda_properties.cuh file. Makes use of the more modern constexpr specifier to avoid using confusing macros and instead functions that are resolved at compile time.
This change is based off of the cuda-api-wrappers solution. Their implementation is more complex so if we need to evolve how device properties work for Gunrock we can first look there.
Inconsistencies with device property values
I used the compute capability tech specs in the CUDA documentation to add the Ampere architectures and noticed some of the device property values in the old cuda_properties.cuh were inconsistent with what I saw in the documentation. I was originally using the values output by the occupancy calculator excel sheet which did line up with what I saw in cuda_properties.cuh, but when I saw the docs had different values I trusted those more.
Let me know if any of the arch values should be changed.

I do think the initial PR is sufficient. Can you also add a helper utility to get architecture names based on any given cc value? So:
80 will return ampere,
75 will return turing,
...

Looks great btw, thank you! tada

I added the arch_name() function. cuda-api-wrappers bases their architecture_t around the major number, which doesn't line up exactly because of Turing and Volta both being in major == 7. So I think only having a compute_capability_t works better for us.

@neoblizz do you think it would also be good to have some additional utility for the architectures with configurable shared mem / L1 cache? I don't want the values for sm_max_smem_bytes() to be taken as absolute for the architectures that are configurable.
I think that's only on compute capability 3.X right? I read that it also has configurable shared memory bank sizes between 4B/8B (I think that means shared_memory_bank_stride() in our case).

Added an optional arg to sm_max_smem_bytes() let me know what you think.

I added the arch_name() function. cuda-api-wrappers bases their architecture_t around the major number, which doesn't line up exactly because of Turing and Volta both being in major == 7. So I think only having a compute_capability_t works better for us.

Thank you for the arch_name() function.
Actually, cuda-api-wrappers resolve on both major and minor functions, it is just a little bit hard to trace. But if you read into it, you'll find at places where it will check for the minor version. I think the initial design within the wrappers library assumed that every arch will have its unique major, but that's not the case Eyal resolves it with a hierarchical function representation.
I do agree that ours is fine with cc.

Added an optional arg to sm_max_smem_bytes() let me know what you think.

Can you add some comments to how the max_smem is calculated for configurable memory? I think it is good for now, but I might have to actually use it in a real code to see if it works as we want it to.
Also, did you check if all that can be resolved at compile time? As in if configurable memory works with constexpr?

I think that's only on compute capability 3.X right? I read that it also has configurable shared memory bank sizes between 4B/8B (I think that means shared_memory_bank_stride() in our case).

Yeah, bank size will be the stride in our case. Also, can you link me to where you read 3.X+ supports configurable shared memory? I vaguely recall it was Pascal (6.0) and above, and possibly not all GPUs either.

Added an optional arg to sm_max_smem_bytes() let me know what you think.

Can you add some comments to how the max_smem is calculated for configurable memory? I think it is good for now, but I might have to actually use it in a real code to see if it works as we want it to.
Also, did you check if all that can be resolved at compile time? As in if configurable memory works with constexpr?

The argument is an enum from the runtime API that represents the different cache/smem configurations, explained here. Since it's just an enum it should be resolved at compile time? I'm not sure how to check, but I am not getting any errors or warnings.

I think that's only on compute capability 3.X right? I read that it also has configurable shared memory bank sizes between 4B/8B (I think that means shared_memory_bank_stride() in our case).

Yeah, bank size will be the stride in our case. Also, can you link me to where you read 3.X+ supports configurable shared memory? I vaguely recall it was Pascal (6.0) and above, and possibly not all GPUs either.

I read about the configurable shared memory here. This is an old post, but it it's only on 3.x according to this (2.x as well but we don't support that).

Since it's just an enum it should be resolved at compile time? I'm not sure how to check, but I am not getting any errors or warnings.

Yeah, we'll have to test that.

I read about the configurable shared memory here. This is an old post, but it it's only on 3.x according to this (2.x as well but we don't support that).

Ok yeah, let's trust Mark's post for now. Will be worth confirming this from official docs or someone at NVIDIA. Otherwise, these changes look great!

I was able to confirm in the docs that Turing/Volta and Ampere (7.x & 8.x) have configurable shared memory and both follow a paradigm that is much more complex than just setting a constant like in Kepler. I would guess this is the standard going forward and would be nice if someone at NVIDIA can confirm.

Kepler
Turing/Volta
Ampere

In short, the newer architectures use an optionally user-provided "carveout" value as simply a suggestion and a soft boundary for the unified data cache split. The runtime driver can override the carveout suggestion if it sees room for improvement. If a carveout isn't provided, it automatically configures the memory split and "In most cases, the driver's default behavior should provide optimal performance."
Since the the shared memory size isn't hard-capped at the configured values like it is for 3.x I would be okay with ignoring the configurability and just listing the max for the newer architectures. The kernels will be able to use up to the max even if carveout is lower.

However if this new way of configuring shared data cache will be present in all the new architectures, then I'd imagine this is something we'd want to leverage by making sure the split is optimal.

Because the driver is not always aware of the full workload, it is sometimes useful for applications to provide additional hints regarding the desired shared memory configuration.

Since shared data cache configurations are done at a kernel level, this seems like something that can be tied back to the concept of a launch box. I could investigate a good way to abstract shared memory configuration for use with Gunrock's kernels, although I'd imagine the heuristics themselves would have to be individually determined. My guess is that this would make abstracting shared data cache configuration tough.

Since the the shared memory size isn't hard-capped at the configured values like it is for 3.x I would be okay with ignoring the configurability and just listing the max for the newer architectures. The kernels will be able to use up to the max even if carveout is lower.

Thanks for looking into it, if the documents  for Kepler onwards suggest a carveout, which is what I was aware of too, then I believe that should be the API going forward from the CUDA's side. And I agree with your decision of leaving it out from our device_properties API (or at least not in the sm_max_smem_bytes().

Since shared data cache configurations are done at a kernel level, this seems like something that can be tied back to the concept of a launch box.

I agree again, maybe a shared memory configuration is another parameter we'll need per-architecture for the launch box, that way, we can specify that per kernel/workload.

I could investigate a good way to abstract shared memory configuration for use with Gunrock's kernels, although I'd imagine the heuristics themselves would have to be individually determined. My guess is that this would make abstracting shared data cache configuration tough.

Maybe we won't get much out of the shared configuration, but at least having that knob in to be controlled seems really nice. Especially because future architectures may encourage more control over cache (such as the stuff I have seen with specifying cache policies, etc.)

I'm glad you agree with my thoughts on the shared memory. Also thanks for adding me as a collaborator!


Always trying to read from non-existent Spring Cloud Config Server

Because the Spring Cloud Config Client jar is on the classpath, the application is continually polling for localhost:8888 to read config data.  Is this required for the functionality of the application?

Did you add the jar yourself? Both the sample applications immediately back off when localhost:8888 is not responding.
Could you try to set the following property?
spring.cloud.config.failFast=true

@dmfrey Create a file named bootstrap.properties into your resources directory and put this into it: spring.cloud.config.enabled=false.
@andreasevers Are you planning to disable cloud config? Or are you using it optionally for something? If so are you planning to conditionally switch it off?

@jonatan-ivanov We should indeed make it conditional. Everywhere we rolled out the dashboard we use spring-cloud-config, but it shouldn't be turned on by default.
PRs for this are welcome, otherwise we'll make the change as soon as we find the time.

This will not be an issue anymore in 2.0.
Unfortunately the 1.x release had a lot of flaws and wasn't maintained.
Therefor I'm going to close this issue.
If you still have any questions, don't hesitate to reach out on Gitter or reopen this issue.


add Clear cache

public void Clear(string prefix)
{
_cache.KeyDeleteWithPrefix(prefix+"*");
}

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot

Having one PR per file is not workable. We'll discuss this change on #311


chore: all hashes in ts

This is mostly just removing a bunch of async/await stuff as we convert the final 3 functions in abis.ts to use foundation/crypto/pedersen.
There are two things that are iffy, but not sure if that should block a merge as everything passes:

vkHash, I've hardcoded a couple of values I don't fully understand (coset generator and root, which are pulled from evalutation domain in the C++).
computePublicCallStackItemHash the C++ did something funky when isExecutionRequest is true. I don't fully understand what and if it needs reproducing here, but again, everything passes.

Have left the commented C++ code in the two places above for reference.

Benchmark results
Metrics with a significant change:

circuit_simulation_time_in_ms (public-kernel-private-input): 41.3 (-93%)
circuit_simulation_time_in_ms (public-kernel-non-first-iteration): 25.9 (-94%)
l2_block_building_time_in_ms (8): 13,216 (-36%)
l2_block_building_time_in_ms (32): 52,596 (-35%)
l2_block_building_time_in_ms (128): 209,842 (-35%)
l2_block_public_tx_process_time_in_ms (8): 1,380 (-84%)
l2_block_public_tx_process_time_in_ms (32): 5,212 (-85%)
l2_block_public_tx_process_time_in_ms (128): 20,621 (-85%)

Detailed results
All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call  to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.
This benchmark source data is available in JSON format on S3 here.
Values are compared against data from master at commit 23d1e2d3 and shown if the difference exceeds 1%.
L2 block published to L1
Each column represents the number of txs on an L2 block published to L1.

Metric
8 txs
32 txs
128 txs

l1_rollup_calldata_size_in_bytes
45,444
179,588
716,132

l1_rollup_calldata_gas
222,948
868,148
3,449,408

l1_rollup_execution_gas
842,035
3,595,256
22,204,777

l2_block_processing_time_in_ms
1,941 (-1%)
7,609
29,814

note_successful_decrypting_time_in_ms
293
873
3,183 (+1%)

note_trial_decrypting_time_in_ms
99.6 (+3%)
82.6 (+2%)
134 (-15%)

l2_block_building_time_in_ms
:warning: 13,216 (-36%)
:warning: 52,596 (-35%)
:warning: 209,842 (-35%)

l2_block_rollup_simulation_time_in_ms
11,792 (-1%)
47,249
188,719 (+1%)

l2_block_public_tx_process_time_in_ms
:warning: 1,380 (-84%)
:warning: 5,212 (-85%)
:warning: 20,621 (-85%)

L2 chain processing
Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric
5 blocks
10 blocks

node_history_sync_time_in_ms
22,017
43,253 (+2%)

note_history_successful_decrypting_time_in_ms
2,047 (+1%)
3,987 (+1%)

note_history_trial_decrypting_time_in_ms
121
144

node_database_size_in_bytes
1,629,755
1,097,741

pxe_database_size_in_bytes
29,748
59,307

Circuits stats
Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit
circuit_simulation_time_in_ms
circuit_input_size_in_bytes
circuit_output_size_in_bytes

private-kernel-init
763
61,697
18,905

private-kernel-ordering
125
24,297
8,153

base-rollup
1,763
656,311
814 (-7%)

root-rollup
169 (+1%)
4,072
1,097

private-kernel-inner
787
81,568
18,905

public-kernel-private-input
:warning: 41.3 (-93%)
41,519
18,841

public-kernel-non-first-iteration
:warning: 25.9 (-94%)
41,497
18,841

merge-rollup
15.1
2,592
873

Miscellaneous
Transaction sizes based on how many contracts are deployed in the tx.

Metric
0 deployed contracts
1 deployed contracts

tx_size_in_bytes
8,787
27,547

This PR was meant to be a lot smaller, but it looks like: https://github.com/AztecProtocol/aztec-packages/pull/3345 got accidentaly merged into it early.

This PR was meant to be a lot smaller, but it looks like: #3345 got accidentaly merged into it early.

My bad!

Started getting weird conflicts so I just merged downstream https://github.com/AztecProtocol/aztec-packages/pull/3345 into this as well. So this is become a mega PR :/
You can view the two merged PR's to see those changes in isolation. To see this specific PR's changes in isolation ignoring the downstream merges will probably require a bit of manual git diff.


Increase the max_chunks limit to support large messages.

Problem description
Topic messages are limited to 20 chunks maximum by default. Each chunk is limited to 1024b (limited by HCS message size). We need a way to increase the max_chunks limit to support large messages.
Our system records MRVs with small values. Then, those MRVs aggregate to mint tokens.
In some of our use cases, It hit the limit of 20 chunks per message when more than 1000 MRVs aggregated to mint one token..
The default 20 chunks limit is set by default at:
https://vscode.dev/github/hashgraph/hedera-sdk-js/blob/develop/src/topic/TopicMessageSubmitTransaction.js#L84
But this can be set by the setMaxChunks method.
Step to reproduce
Our system records MRVs with small values. Then, those MRVs aggregate to mint tokens.
In some of our use cases, it takes over 1000 MRVs aggregated to mint one token. When this happened, we came across the following error:
{
...
severity: "ERROR"
textPayload: "2023-05-08T06:37:49.705Z [guardian-service,5e07e18b-6cab-4280-8db7-871c3691b4b2,mintDocumentBlock,block-runtime,63dc7cbd6105ab8065a4b34c]: Message with size 21306 too long for 20 chunks NR-LINKING|MzQ5NzYzMXxBUE18QVBQTElDQVRJT058MTQxMDQ2NTM0Mg|guardian-guardian-service-5ffcfddcfc-c688s|||guardian-guardian-service|"
timestamp: "2023-05-08T06:37:49.706283529Z"
}

It hit the limit of 20 chunks per message.
Expected behavior
We need a way ( maybe an environment variable) to increase this limit to support large token mint messages.

we can use the setMaxChunks function to fix this issue.  I have tested this approach and it can mint token with > 1000 dmrv

@timeout(HederaSDKHelper.MAX_TIMEOUT)
    public async submitMessage(
        topicId: string | TopicId,
        message: string,
        privateKey?: string | PrivateKey,
        transactionMemo?: string
    ): Promise<string> {
        const client = this.client;

        let messageTransaction: Transaction = new TopicMessageSubmitTransaction({
            topicId,
            message,
        }).setMaxTransactionFee(MAX_FEE)
        .setMaxChunks(50)

       ......
    }

@samuraitruong Thank you for the suggestion. We have fixed this issue by adding the maxChunks parameter to .env of worker-service.


[LOC] Invalid link in an article

There is an invalid link in an article
Repro steps:

Go to https://docs.microsoft.com/en-us/microsoftteams/setting-up-your-phone-system?bc=/skypeforbusiness/breadcrumb/toc.json&toc=/skypeforbusiness/sfbotoc/toc.json
Search for "Create a Cloud call queue"
Click on the link

Actual results:
Link does not work
Expected results:
Link should either work or be removed
GH article:
https://github.com/MicrosoftDocs/OfficeDocs-SkypeForBusiness/blob/live/Teams/setting-up-your-phone-system.md

@OfficeGlobal @ Thank you for submitting feedback and contributing to the docs. We are currently investigating this.

@officedocsbot assign @scanum

@OfficeGlobal Done, thank you.

@OfficeGlobal We appreciate any feedback that improves the content in the Microsoft docs but we haven't had any response from you so we are going to close this issue.


Document how to avoid depth camera interference when using multiple camera's

Describe the bug
When using multiple depth camera's in synchronized captures, depth camera captures should be offset from one another by 160us or more.

it is documented as part of the Hardware specification

Hi, I was going through documents about multiple devices sync. I finally ended up here because I was wondering how to set sync-delay. I think this document should also be updated to avoid confusion. To be clear, I want to check with you that the following is the correct way to do multiple devices sync:
# for master
k4arecorder.exe --device 0 --external-sync master output-1.mkv
# for subordinate-1
k4arecorder.exe --device 1 --external-sync subordinate --sync-delay 0.16 output-2.mkv
# for subordinate-2
k4arecorder.exe --device 2 --external-sync subordinate --sync-delay 0.32 output-3.mkv

Hi, I was going through documents about multiple devices sync. I finally ended up here because I was wondering how to set sync-delay. I think this document should also be updated to avoid confusion. To be clear, I want to check with you that the following is the correct way to do multiple devices sync:
for master
k4arecorder.exe --device 0 --external-sync master output-1.mkv
for subordinate-1
k4arecorder.exe --device 1 --external-sync subordinate --sync-delay 0.16 output-2.mkv
for subordinate-2
k4arecorder.exe --device 2 --external-

Hi @iammarvelous can you open a new issue please?

@wes-b Done. See ##862


[db] Support vectorized custom types in UserValue derive macro

It is not possible to provide reasonable default implementation.


URGENT: DLL on Nuget is infected with a Trojan: MSIL.Kryptik.NE@81eks7

You can see it detected by Comodo virus scanner.
Please fix it and release a new Nuget ASAP.
Bug Information
Version Number of Plugin: Latest
Device Tested On: Comodo AntiVirus
Simulator Tested On: All platforms
Version of VS: 2019
Version of Xamarin: Latest
Versions of other things you are using: Latest Comodo anti virus
Steps to reproduce the Behavior
Just scan the DLL. My app APK was detected as a Trojan, which led me to find this.
Expected Behavior
Clean and virus-free!
Actual Behavior
Dirty and viral!
Code snippet
Not applicable
Screenshotst

No idea what you are talking about.

This is a real issue James.
You probably had this virus on your system when you compiled it and it transferred itself into the DLL. You can see for yourself if you run a Comodo virus scanner on that DLL or any app APK that uses it.


Simplification doesn’t handle decomposition correctly

OptimalSimplifyingReductions carries out ReductionDecomposition + NormalizeReduction steps which introduces new local variables for the intermediate results that are higher dimensional than the original answer variable. This prevents simplification from exploiting all reuse.
An example
Take the following 4D max filter:
affine max_filter_4d [N, W] -> {: 0<W<N}
  inputs
    X : {[i,j] : 0<=i,j<=N+W}
  outputs
    Y : {[i,j] : 0<=i,j<=N+W}
  let
    Y[i,j] = reduce(max, (i,j,k,l->i,j), {: i<=k<=i+W and j<=l<=j+W} : X[k,l]);
  .

There is 2D reuse so we should be able to obtain a (4-2=2)D simplified version. OptimalSimplifyingReductions finds a 3D version using ReductionDecomposition:
// +-- Apply ReductionDecomposition to Y with [N, W] -> { [i0, i1, i2] -> [(i2), (i0)] } o [N, W] -> { [i, j, k, l] -> [(j), (k), (i)] }
// +--+-- Apply SimplifyingReduction to Y_NR with: [ 1 0 0 0 ]
affine max_filter_4d [N, W] -> {  : 0 < W < N }
  inputs
    X : {[i, j]: 0 <= i <= N + W and 0 <= j <= N + W }
  outputs
    Y : {[i, j]: 0 <= i <= N + W and 0 <= j <= N + W }
  locals
    Y_NR : {[i0, i1, i2]: 0 <= i0 <= N + W and i1 <= N + W and i2 >= 0 and -W + i1 <= i2 <= i1 }
    Y_NR_pos : {[i0, i1, i2]: i2 = i1 and 0 <= i0 <= N + W and 0 <= i1 <= N + W }
  when {  : 0 < W < N } let
    Y[i,j] = reduce(max, (i0,i1,i2->i2,i0), Y_NR);

    Y_NR[i0,i1,i2] = case  {
      {: i2 = i1 and 0 <= i0 <= N + W and 0 <= i1 <= N + W } : Y_NR_pos;
      {: 0 <= i0 <= N + W and i1 <= N + W and i2 >= 0 and -W + i1 <= i2 < i1 } : Y_NR[i0,i1,i2+1];
    };

    Y_NR_pos[i0,i1,i2] = reduce(max, (i,j,k,l->j,k,i), {: k = i and 0 <= i <= N + W and j >= 0 and j <= l <= W + j and l <= N + W } : X[k,l]);
.

However, now we have new local variables which are 3D. There is still reuse present in the residual reduction in Y_NR_pos — it’s body is 3D and the expression X[k,l] has 1D reuse. But since the variable on the LHS of the equation is 3D, the current implementation stops and says that this reduction is optimally simplified.

The variable Y_NR_pos really is 2D because of the constraint i2 = i1, and we can legally rewrite the system as:
affine max_filter_4d [N, W] -> {  : 0 < W < N }
  inputs
    X : {[i, j]: 0 <= i <= N + W and 0 <= j <= N + W }
  outputs
    Y : {[i, j]: 0 <= i <= N + W and 0 <= j <= N + W }
  locals
    Y_NR : {[i0, i1, i2]: 0 <= i0 <= N + W and i1 <= N + W and i2 >= 0 and -W + i1 <= i2 <= i1 }
    Y_NR_pos : {[i0, i1]: 0 <= i0 <= N + W and 0 <= i1 <= N + W }
  when {  : 0 < W < N } let
    Y[i,j] = reduce(max, (i0,i1,i2->i2,i0), Y_NR);

    Y_NR[i0,i1,i2] = case  {
      {: i2 = i1 and 0 <= i0 <= N + W and 0 <= i1 <= N + W } : Y_NR_pos[i0,i1];
      {: 0 <= i0 <= N + W and i1 <= N + W and i2 >= 0 and -W + i1 <= i2 < i1 } : Y_NR[i0,i1,i2+1];
    };

    Y_NR_pos[i0,i1] = reduce(max, (i,j,l->j,i), {: 0 <= i <= N + W and j >= 0 and j <= l <= W + j and l <= N + W } : X[i,l]);
.

Similarly, since Y_NR is the same value along 1D’s worth of points (i.e., the second branch of the case expression), we can legally rewrite the system as:
affine max_filter_4d [N, W] -> {  : 0 < W < N }
  inputs
    X : {[i, j]: 0 <= i <= N + W and 0 <= j <= N + W }
  outputs
    Y : {[i, j]: 0 <= i <= N + W and 0 <= j <= N + W }
  locals
    Y_NR_pos : {[i0, i1]: 0 <= i0 <= N + W and 0 <= i1 <= N + W }
  when {  : 0 < W < N } let
    Y[i,j] = reduce(max, (i0,i1,i2->i2,i0), Y_NR_pos[i0,i1]);

    Y_NR_pos[i0,i1] = reduce(max, (i,j,l->j,i), {: 0 <= i <= N + W and j >= 0 and j <= l <= W + j and l <= N + W } : X[i,l]);
.

But I’m not sure if we can always do this, since the case expression in Y_NR came from the previous application of theorem 5 from GR06.


Release/3.0.0

Release ONDEWO-S2T Angular Client 3.0.0
New Features

Track version 3.0.x of S2T-API

Release ONDEWO-S2T Angular Client 3.0.0
New Features

Track version 3.0.x of S2T-API


Losing library when '~/.config/newsboat/urls' changes

Thanks a lot for the awesome newsboat-tool - I'm using it on a daily basis.  In most cases it works flawlessly, but I have experienced sometimes a major issue which led to me lose all my library.
I'm using the following version of newsboat on my Debian Stable system:
newsboat 2.13.0
System: Linux 4.19.0-9-amd64 (x86_64)
Compiler: g++ 8.3.0
ncurses: ncurses 6.1.20181013 (compiled with 6.1)
libcurl: libcurl/7.64.0 GnuTLS/3.6.7 zlib/1.2.11 libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.5) libssh2/1.8.0 nghttp2/1.36.0 librtmp/2.3 (compiled with 7.64.0)
SQLite: 3.27.2 (compiled with 3.27.2)
libxml2: compiled with 2.9.4

My ~/.config/newsboat/urls-file is a symlink (since I'm using rcm as a dotfile managing tool which creates symlinks for config files and stores the actual files in a separate folder to be used as a vc-repository like git).  When it is available, everything works fine.  But when my  ~/.config/newsboat/urls is missing some content (e.g. because I'm commenting some parts out for testing or because I'm symlinking a different file) it will use this one to get the feeds - which is expected behaviour.  In this case I'm only able to see the parts I had written in the new urls-file.
But as soon as I use my original file again I realize that everything, which was formerly stored in the SQLite-database included in newsboat, is lost, including all my old feeds, the flags, the downloaded data, everything.
Since my library is (was) huge and I invested quite some time in tuning it to my needs, this is a major issue for me.
So my questions are as follows:

why is this happening?

is there a way to recover that data?

is there a way to prevent newsboat to delete my old data in the future?

Hi! Sorry you lost some data ._.

why is this happening?

Because Newsboat removes data that is no longer reachable. From Newsboat's point of view, you removed a feed from the urls file, which means you're no longer interested in that feed; this means you don't need the related data any more. Thus, it is removed from the database.

is there a way to recover that data?

Restoring your cache.db from a backup is the only way to get the data back.

is there a way to prevent newsboat to delete my old data in the future?

Yes, add cleanup-on-quit no to your config.
Does that answer your questions?

Thanks @Minoru for your swift reply!

add cleanup-on-quit no to your config

Awesome, that solves my issue, thanks a lot!

From Newsboat's point of view, you removed a feed from the urls file, which means you're no longer interested...

Well, that makes sense - but is that a good choice for the default behavior?  This way it's relatively easy to lose data.  If it would be the other way round (cleanup is only performed manually except if specified so on the config-file) the downside would only be a larger db - but if that bothers anyone a manual cleanup could be performed.

This way it's relatively easy to lose data.

I was going to say that this is like complaining that removing a name from the file system makes it easy to lose data, but then I realized that people already ran into that problem, and solved it with the "waste-basket" metaphor. So now I think you're right.
I can't change the default in a minor release, so that will have to wait until 3.0. But the manual cleanup can be added earlier, to the benefit of those who already disable cleanup-on-quit. I'll file separate issues about that.


Filter implementation

Greetings,
Forgive me, if same issue was opened before, or if I've misunderstood some part of API or implementation details.
I was wondering, if it's a known issue/task regarding implementation of Filters. As of right now, it seems like JsonPath's filters are not supported by JsonPathParser.
Example
Let's consider example provided in README:
val expression =<EMAIL_ADDRESS>> 1950)].title"

val path = JsonPathParser.parse(expression)

This will yield an erroneous result value
Left(Error(8, NonEmptyList(WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,',()))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,*,*)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,-,-)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,0,9)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,٠,٩)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,۰,۹)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,߀,߉)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,०,९)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,০,৯)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,੦,੯)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,૦,૯)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,୦,୯)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,௦,௯)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,౦,౯)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,೦,೯)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,൦,൯)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,෦,෯)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,๐,๙)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,໐,໙)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,༠,༩)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,၀,၉)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,႐,႙)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,០,៩)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,᠐,᠙)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,᥆,᥏)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,᧐,᧙)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,᪀,᪉)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,᪐,᪙)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,᭐,᭙)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,᮰,᮹)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,᱀,᱉)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,᱐,᱙)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,꘠,꘩)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,꣐,꣙)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,꤀,꤉)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,꧐,꧙)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,꧰,꧹)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,꩐,꩙)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,꯰,꯹)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,InRange(8,０,９)))), WithContext(expP,WithContext(selectorP,WithContext(bracketsP,WithContext(arraySliceP,InRange(8,:,:))))))))

Investigation
Upon a little bit of investigation, I've found that inside JsonPathParser, there's a defined questionMarkP:
val questionMarkP: Parser[Unit] = Parser.char('?')

Yet, as it seems, it is not being used for any of the parsers present in this class.
Also, unlike in CirceSolver, JsonPathParser is not instantiating Filter <: Exp at any point, not only syntactically, but also logically.

Added an implementation, which fixed problem for me. Though, may require additional review for completeness

Thank you @Punctuality,
We may have lost the implementation of the filter when shuffling stuff around, to generalise the implementation. I'll look in your PR, and thank you for the contribution

Released in 1.0.1


COMP: Test NumPy pre-releases (NumPy 2.0 RC) in CI

Added to the macOS Python CI builds.
Re: #4700

Maybe add it to one more, e.g. Windows or Linux?

Maybe add it to one more, e.g. Windows or Linux?

I added it to Linux, too.


Denom problem with "create-validator" command

Summery
The denom to check when running the "create-validator" command seems to be wrong.
The chain token uses uvoi, but create-validator checks with uiov.
(rawlog: 'insufficient fee: insufficient fees; got: 2000000uvoi required: 2000000uiov')
-> Validator can't be created.
Network

chain-id

iovns-galaxynet

Version

iovnscli version

$ iovnscli version --long
name: iovns
server_name: iovnsd
client_name: iovnscli
version: 0.4.5
commit: 5409c2f9e750af2df98afc57f8962cd85aeff0c7
build_tags: netgo,ledger
go: go version go1.14 linux/amd64

iovnsd version

$ iovnsd version --long
name: iovns
server_name: iovnsd
client_name: iovnscli
version: 0.4.5
commit: 5409c2f9e750af2df98afc57f8962cd85aeff0c7
build_tags: netgo,ledger
go: go version go1.14 linux/amd64

Steps to Reproduce

Check the denom in use on testnet

$ iovnscli q supply total -o json | jq
[
  {
    "denom": "uvoi",
    "amount": "1118688306555927"
  }
]

Validator creation

$ iovnscli tx staking create-validator --moniker ATEAM --amount 100000000uvoi --commission-max-change-rate 0.01 --commission-max-rate 0.2 --commission-rate 0.1 --min-self-delegation 1 --pubkey ${PUB_KEY_BECH32} --gas-prices 10.0uvoi --from ATEAM --chain-id "iovns-galaxynet"
Enter keyring passphrase:
{"chain_id":"iovns-galaxynet","account_number":"175","sequence":"0","fee":{"amount":[{"denom":"uvoi","amount":"2000000"}],"gas":"200000"},"msgs":[{"type":"cosmos-sdk/MsgCreateValidator","value":{"description":{"moniker":"ATEAM","identity":"","website":"","security_contact":"","details":""},"commission":{"rate":"0.100000000000000000","max_rate":"0.200000000000000000","max_change_rate":"0.010000000000000000"},"min_self_delegation":"1","delegator_address":"star1cdexd0ca8sddluuaydqnuj4hunql3enpl2wl3m","validator_address":"starvaloper1cdexd0ca8sddluuaydqnuj4hunql3enp6p20aj","pubkey":"starvalconspub1zcjduepqhnvffsfsu3az6ls8mfxjn07nnyh4l9j3uvn7d2ej308c33p839lszqavq8","value":{"denom":"uvoi","amount":"100000000"}}}],"memo":""}

confirm transaction before signing and broadcasting [y/N]: y
Enter keyring passphrase:
height: 0
txhash: B1080B0A3CA184D97D7CE13CD89682E4820786167FD49F6E3CFCAE7F8EE3C712
codespace: ""
code: 13
data: ""
rawlog: 'insufficient fee: insufficient fees; got: 2000000uvoi required: 2000000uiov'
logs: []
info: ""
gaswanted: 0
gasused: 0
tx: null
timestamp: ""
``

Related: https://travis-ci.com/github/iov-one/iovns/jobs/360601113.

There was a typo in https://docs.iov.one/for-validators/testnet on line 65 of the main setup script.  It's fixed now.
@wlsaud619 , please edit /opt/iovns/bin/iovnsd.sh and change uiov to uvoi and then do systemctl restart starname.service.


Could author gives a  timeline of codes？

Could author gives a  timeline of codes？

Hello, thanks for your patience and sorry for the delay! Please find the code available now!


Add to credits.md

Summary of changes
Add name to credits.md
Related Issue
(https://github.com/wwcodecolorado/scrollodex/issues/16)
How to test (for features) / reproduce (for bugs)
N/A
Screenshots
N/A
Changes include

[ ] Bugfix (non-breaking change that solves an issue)
[ ] New feature (non-breaking change that adds functionality)
[ ] Documentation (explains but does not affect functionality)
[ ] Breaking change (change that is not backwards-compatible and/or changes current functionality)

Notes

please consider including a gif that reflects your feelings about this PR

looks good @rteeter! love the GIF 🥳


Contrib dist stat

New contributivity measueament based on statistical distances between 2 distributions:

The partner-specific probability distribution of the label, wrt to the input; (estimated via maximun likelihood wrt to the partner's data)
The latent joint probability distribution of the label, wrt to the input.(estimated via maximun likelihood wrt to the joint dataset)

This difference of distributions is interpreted as a noise, which allow us to use a multiheaded adaptation of the smodel method to the multipartner case to estimate and quantify this pseudo-noise.
These contributivity metrics only need inferences to be computed, on the trained model (trained via FedSmodel)
The computational additional cost is thus neglectable
The method doesn't need a 'perfect' and global test dataset.
For now 3 distances are implemented:

KullBack- Leiber divergence
Hellinger metric
Bhattacharyya distance

These metrics are tested on the reference scenarios, see the colab notebook :
https://colab.research.google.com/drive/1DN1lLdd1b1ZmttmEiQKpx8xW5guEf_f_?usp=sharing
TODO

[ ] Add doc
[x] Investigate over s-model bug when using Advanced or Flexible splitter
[x] Handle dict contributivity score for result dataframe
[ ] Investigate std computations

Codecov Report

Merging #346 (078cbca) into master (ecc3ea8) will decrease coverage by 0.46%.
The diff coverage is 80.18%.

@@            Coverage Diff             @@
##           master     #346      +/-   ##
==========================================
- Coverage   80.68%   80.22%   -0.47%
==========================================
  Files          15       15
  Lines        3045     3084      +39
==========================================
+ Hits         2457     2474      +17
- Misses        588      610      +22

Impacted Files
Coverage Δ

mplc/multi_partner_learning/__init__.py
100.00% <ø> (ø)

mplc/multi_partner_learning/basic_mpl.py
84.85% <ø> (-0.42%)
:arrow_down:

mplc/multi_partner_learning/fast_mpl.py
60.26% <54.34%> (-1.64%)
:arrow_down:

mplc/contributivity.py
77.23% <100.00%> (+0.67%)
:arrow_up:

mplc/scenario.py
83.16% <100.00%> (+0.66%)
:arrow_up:

mplc/partner.py
92.30% <0.00%> (-1.10%)
:arrow_down:

mplc/corruption.py
79.67% <0.00%> (-0.74%)
:arrow_down:

mplc/models.py
75.32% <0.00%> (-0.63%)
:arrow_down:

... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ecc3ea8...078cbca. Read the comment docs.


🐛 Whatsapper update / installation fails

Description
I wanted to upgrade whatsapper to the latest version 2024.4.29 and get below message
De installatie van de add-on is mislukt
Can't install ghcr.io/alexbelgium/whatsapper-amd64:2024.4.29: 500 Server Error for http+docker://localhost/v1.44/images/create?tag=2024.4.29&fromImage=ghcr.io%2Falexbelgium%2Fwhatsapper-amd64&platform=linux%2Famd64: Internal Server Error ("manifest unknown")
I get this both when trying to upgrade and with a new installation (after I removed the old version).
I already resxtarted HA but after that I get the same result.
Reproduction steps
Try to install the add-on

Addon Logs
De installatie van de add-on is mislukt
Can't install ghcr.io/alexbelgium/whatsapper-amd64:2024.4.29: 500 Server Error for http+docker://localhost/v1.44/images/create?tag=2024.4.29&fromImage=ghcr.io%2Falexbelgium%2Fwhatsapper-amd64&platform=linux%2Famd64: Internal Server Error ("manifest unknown")

Architecture
amd64
OS
HAos, Virtual Machine

I'm taking care of this. ;)

see progress here
https://github.com/alexbelgium/hassio-addons/pull/1364

the issue should be solved in version 2024.4.29-1
@zandhaas could you give a try?

I think issues are still there, because of instability of the underneath library I'm using
see
https://github.com/pedroslopez/whatsapp-web.js/pull/2816

this could fix things :)
https://github.com/alexbelgium/hassio-addons/pull/1367

I confirm that the latest updates solves the issues.
I have whatsapper up and running


Expose SSIZE_MAX to enable safe/portable read and write system calls

The results of read [1] and write [2] system calls are implementation-defined if the count parameter is larger than value of SSIZE_MAX constant defined in limits.h according to POSIX.1 standard.
So in order to make it possible to use read and write in a portable manner, SSIZE_MAX should be exposed on POSIX.1-conformant systems.
[1] https://pubs.opengroup.org/onlinepubs/9699919799/functions/read.html
[2] https://pubs.opengroup.org/onlinepubs/9699919799/functions/write.html

I realize this issue is rather old, but is there any platform where SSIZE_MAX is not equal to libc::ssize::max_value()?


DatePicker Title

There should be a title when we open the datepicker dialog like "Select Date"

Thanks but that is not reflected in the gradle file. I think for that you
have to increase gradle version.
Thanks & Regards,
Krishna Meena
On Fri, Oct 30, 2015 at 2:05 AM, wdullaer<EMAIL_ADDRESS>wrote:

Closed #109
https://github.com/wdullaer/MaterialDateTimePicker/issues/109 via
8f6b1cf
https://github.com/wdullaer/MaterialDateTimePicker/commit/8f6b1cff619d1a186c29dd81f08971aeab685142
.
—
Reply to this email directly or view it on GitHub
https://github.com/wdullaer/MaterialDateTimePicker/issues/109#event-449710688
.

I fixed it in master, which is why the issue is closed.
I have yet to release a new version to maven central.
You can wait for that or check out the code in master

Ok thanks.
Thanks & Regards,
Krishna Meena
On 30-Oct-2015 6:24 PM, "wdullaer"<EMAIL_ADDRESS>wrote:

I fixed it in master, which is why the issue is closed.
I have yet to release a new version to maven central.
You can wait for that or check out the code in master
—
Reply to this email directly or view it on GitHub
https://github.com/wdullaer/MaterialDateTimePicker/issues/109#issuecomment-152515841
.


Error: bind EADDRINUSE

Trying the example on my Debian built Node v0.12.0:
drasko@Lenin:~/nw/test$ DEBUG=mdns:* node test.js
  mdns:lib:networking interface +0ms wlan0 <IP_ADDRESS>
  mdns:lib:networking creating socket for +4ms wlan0
  mdns:lib:networking creating socket for +3ms pseudo multicast
  mdns:lib:networking bindToAddress +4ms wlan0
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: bind EADDRINUSE
    at exports._errnoException (util.js:746:11)
    at dgram.js:224:28
    at dns.js:85:18
    at process._tickCallback (node.js:355:11)
    at Function.Module.runMain (module.js:503:11)
    at startup (node.js:129:16)
    at node.js:814:3
drasko@Lenin:~/nw/test$

Do you get this if you first stop the avahi daemon?
node v0.10 with avahi running, tests and examples/service.js works
node v0.12 with avahi running, tests and examples/service.js fails with EADDRINUSE
node v0.12 without avahi running, tests and examples/service.js works
More investigation needed

Ok.. as I suspected.
If I don't open the pseudo multicast "<IP_ADDRESS>:5353" it will work even with avahi running but we shouldn't do that. I don't know why v0.10 didn't fail. It should have.
Two competing mdns services is NOT a good thing. So using this to advertise services at the same time as avahi or bonjour is running will fail in one level or another.
This is probably a invalid issue but I would appreciate feedback on if it works without avahi running.

If you want to use the discovery part on a system with avahi or bonjour I have added a way of excluding the offending interface.
mdns.excludeInterface('<IP_ADDRESS>');

before ANY other activity.

I did not really try examples/service.js, but examples/simple.js. I don't need to export service, I need just to browse existing services around me.
And yes, it does not work even with avahi stopped.
How to install version with your fix? I tried:
npm install https://github.com/kmpm/node-mdns-js.git
``
but this does not work...

I don't understand - why do I have to stop avahi just to browse the existing services? Can examples/simple.js work without stopping avahi?

Ah, replacing
npm install https://github.com/kmpm/node-mdns-js.git

by
npm install git://github.com/kmpm/node-mdns-js.git

works.
And now test program works even with avahi running.

How to retrieve fullname and host?
I need to recognize the service by the name, like "myService.local". For now I am getting in name just 'http', 'smb', or 'ssh' for example:
data: { addresses: [ '<IP_ADDRESS>' ],
  query: [ '_services._dns-sd._udp.local' ],
  type:
   [ { name: 'ssh',
       protocol: 'tcp',
       subtypes: [],
       description: 'SSH Remote Terminal' },
     { name: 'smb',
       protocol: 'tcp',
       subtypes: [],
       description: 'Microsoft Windows Network' },
     { name: 'http',
       protocol: 'tcp',
       subtypes: [],
       description: 'Web Site' } ],
  interfaceIndex: 0,
  networkInterface: 'wlan0' }

I found something, on node work perfectly, but on io.js my app crash with "Error: bind EADDRINUSE <IP_ADDRESS>:5353".
the last build alpha1/2/3 of nw.js build with io.js. and mdns-js seems not compatible with io.js.

maybe help
https://github.com/iojs/io.js/issues/554

yes, reuseAddr: true seems to fix the issue.

Can you push a new version to NPM? Thanks!


Backend: Remove use of simplejson

Similar to what was done in CloudWatch https://github.com/grafana/grafana/pull/51062
Note: Marking this as a "good first issue" for anyone who would like to contribute any part of this -- the overall work may be significant, but some parts may be straightforward to re-write.

I would like to work on this

@DinethShanG sounds great, go for it! Feel free to find a way to break it up and make multiple PRs. Let us know if you want any guidance!


Medtronic basals

This will add support for:

scheduled basals
temp basals
suppressed info
suspend/resume records

Since basal schedules at a specific moment in time are required to build suppressed info and temp basals, this PR also adds support for getting the current basal schedules as well as basal schedule history.
It also refactors some common functionality out of the Animas and CareLink drivers.
For more details, see https://trello.com/c/zlAnS0iw

 [uploader] Medronic basals

Pinging @jebeck for review.

My brain is feeling a bit mushy this evening, so I skipped over the simulator and simulator tests stuff for now - will come back to that later (perhaps tomorrow?) since you said there's no rush, and I know those are the tricky bits.

@jebeck I've made more changes to this PR:

also handles nested suppressed basals
puts schedule names in suppressed scheduled basals
restarting temp basals on resume

Are you ok with reviewing the new commits, or is there a better way to review this? (e.g. by splitting changes of into a new PR). I haven't split it yet, as medtronic-device-event is also branched off here and wanted to keep the number of branches (and potential conflicts) to a minimum.

Yeah I think I can just look at the new changes, that's fine @gniezen

Note that since gniezen/medtronic is not a protected branch, you can merge without me reviewing the next updates. Go ahead and do so if it's expedient for getting a test release out or something. I was almost going to select "Approve" to give you that freedom but then realized I wouldn't have to since master is the only protected branch...

Merging this, so that only new changes are properly reflected in #333 for review.

Have looked at all changes since the last review. Just the one comment that's probably mostly a check of my understanding (and potential readability fix).


Allow explicit imports

Story
As a developer, I want to be able to explicitly import unqualified types and functions from other modules.
Motivation
Currently we can only import modules as a whole with a qualified identifier.
This leads to a lot of Result.Result which looks silly, and makes people create module local private aliases.
This is bad because:

They're adding boilerplate to work around compiler limitations
They don't get hyperlinked documentation because their type signatures point to module local private type definitions.

Demo
I want something like this to compile
import Result "Result";
// Showcases both type as well as value level imports
import { type Result, mkErr } "Result";

module {
  public fun fromOption<Ok, Err>(err: Err, opt: ?Ok) : Result<Ok, Err> {
    switch opt {
      case null { mkErr(err) };
      case (?ok) { Result.mkOk(ok) };
    };
  };
}

Design
This doesn't just work if we allow object patterns, because we can't project type components that way.
It might also be nice to have a more compact format for the "import both as qualified and explicit" pattern, as shown above.

Wouldn't this be an instance of an and pattern with the small twist that the LHS is a label pattern (appearing in the record to be destructured) and that only applies to record values?


Move query from entry point to SparkConf

Description
Move the query parameter from the entry point's main() arguments to SparkConf for batch and streaming jobs. There is no effect on interactive jobs other than removing the dummy query from the arg, which always reads from the request index.
Breaking change - together with https://github.com/opensearch-project/sql/pull/2519
Issues Resolved
https://github.com/opensearch-project/sql/issues/2376
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Thanks for the changes!
QQ regarding the dependency between SQL plugin and Spark. Is the changes on plugin side already done?

Yes, it's pending for review https://github.com/opensearch-project/sql/pull/2519


https://dndle.app/ stuck with "Loading... please wait..."

Noticed today https://dndle.app/ throws now:
Loading... please wait...

(If this doesn't go away, perhaps check is Javascript is enabled in your browser?)

This happens with every browser I have, in incognito modes as well.
Yesterday (27.3.2022) was still working fine.

Working for me. Any useful errors in your browser's debug console? (Press F12, reload page, look in "Console" tab.)

Strange. Now working, and can't reproduce this anymore on any of the same browsers (iOS Safari, Win64 Chrome & Firefox) I had this with.
If this reoccurs, I'll get the debug from console next time.

Seems to happen again, about the same time as yesterday. New monster should've appeared about ~1 h ago.
Here's the console output from Chrome (latest version, stock settings):'
dndle.js:4 Uncaught ReferenceError: Cannot access 'W' before initialization
    at x (dndle.js:4:216)
    at dndle.js:50:7
    at dndle.js:171:1041
x @ dndle.js:4
(anonymous) @ dndle.js:50
(anonymous) @ dndle.js:171

Interesting. The W in the compiled code is the instance of Reef. Sounds like a possible race condition. Taking a quick look.

I'm still unable to replicate this locally, so I can't be sure I've got a fix. Would you mind experimentally going to https://dndle.app/next.html and see if that works for you?

Yes, works fine. No errors produced in the console.
I believe the best bet to snatch this is tomorrow at around 12 EET when a new monster is updated, as this has happened now twice in a row.
I asked from a friend who plays this too, and he experienced the same "Loading... please wait..." error as I did yesterday, and today. The problem also went away for him at the same as it did for me.

It's on again, 12:15 EET.
https://dndle.app/ console output:
Uncaught ReferenceError: Cannot access 'W' before initialization
    at x (dndle.js:4:216)
    at dndle.js:50:7
    at dndle.js:171:1041

https://dndle.app/next.html console output:
dndle-next.js:547 Uncaught ReferenceError: Cannot access 'app' before initialization
    at updateCountdownTimer (dndle-next.js:547:7)
    at HTMLDocument.<anonymous> (dndle-next.js:557:3)

Thanks. Managed to replicate it locally now!

Very definitely a Daylight Savings problem. The day we're on (for the purpose of choosing a monster) is calculated entirely in UTC, which is correct, but the "time until next monster" is calculated based on the date in your local timezone and compared to the UTC date, which results in a discrepancy when daylight savings is in effect.

Experimental solution in place at https://dndle.app/next.html

Should all be fixed now. You might have to "hard refresh" (e.g. CTRL + F5 / CMD + R) on http://dndle.app/ to clear the broken JS from your cache, if you still see the problem.
Thanks for reporting this!

(There's a chance that anybody who experienced the bug today before it was fixed might not be able to do today's monster: it'll be as if they already attempted it and failed - sorry!)


Reloading like Play

I see the following related issues:

https://github.com/http4s/http4s/issues/766
https://github.com/http4s/http4s/issues/849

However from my experience with sbt-revolver that's not quite what Play offers.
In development mode, Play starts a server for you, before the sources are even compiled. Only once you hit the server via the web browser, does it compile and start up the application. Likewise, a source code change does not trigger a restart. Once sources are changed, AND you reload, that's when the recompilation happens.
I continue to use Play despite having http4s and akka-http available because I don't enjoy the reloading cycle using eager compilation. Why? I save the sources, Alt-Tab to the browser, refresh, and... bam! "Could not connect to the server". So you then have to line up with the compiler in terms of waiting times, until you can finally Refresh the page.
With Play, you save the code, Alt-Tab, Refresh, and just wait for your page to load. Lower cognitive load, much more pleasant to deal with.
So I'd like to keep this issue open, based on the understanding that the others have not answered the question exactly. Play starts a server in background and recompiles on demand.
While I am all for libraries, this one thing that is enforced by the fact that Play is a framework makes it an absolute delight to use, despite its non-uniform non-FP nature.

I'm even wondering if it's possible to embed an http4s route in Play. That'd provide an incremental adoption from Play crowd of http4s - instead of having to throw away all your Play code in one big go. Try with one endpoint, then two, and eventually all.

--- (Running the application, auto-reloading is enabled) ---

[info] p.c.s.AkkaHttpServer - Listening for HTTP on /0:0:0:0:0:0:0:0:9000

(Server started, use Enter to stop and go back to the console...)

[info] Compiling 1 Scala source to /.../target/scala-2.12/classes ...
[info] Done compiling.
[info] play.api.Play - Application started (Dev)
[info] Compiling 1 Scala source to /.../target/scala-2.12/classes ...
[info] Done compiling.

--- (RELOAD) ---

[info] play.api.Play - Application started (Dev)

I think you have a fair point. That said, this is on the sbt side of things. If someone is willing to pick up the mantle of adapting whatever plugin play is using, that would be cool.

This idea has some detractors, and we'd want it to be opt-in.  Closing for lack of progress, but if anyone is still passionate about it and intends to work on it, we can revisit.

For anyone interested, while this issue has not gone away by itself, I found a much happier solution; sbt-revolver + browser-sync.
I wrote a small http4s plug-in for browser-sync. It's not documented very much, but if anybody is interested,
libraryDependencies += "com.scalawilliam.rad4s" %% "http4s-browsersync" % "0.60.2-SNAPSHOT"

Sources @ https://github.com/ScalaWilliam/rad4s/blob/master/http4s-browsersync/src/main/scala/com/scalawilliam/rad4s/BrowserSync.scala
This would reStart the app, and then reload your browser window once it's started up.


Add resiliency and subscription CLI commands

Describe the proposal
Each type of specification should be able to be at least listed via the Dapr CLI. Currently there are CLI commands for Configurations and Components. The two other spec types are subscriptions and the newly (v1.7) introduced resiliency policies, the latter of which are global like configurations and can be applied to any applications.
Resiliency should be the same CLI as Configurations https://docs.dapr.io/reference/cli/dapr-configurations/
For example dapr resiliencies -k --name myresiliency
Subscriptions are a bit trickier if these are loaded for each app or may be global (Need to be determined)
For example dapr subscriptions -k --name mysubscription
Of course deploying these specifications is still platform dependent
Also see related issue to update the dapr dashboard to show resiliency policies and subscriptions in the UI
Release Note

RELEASE NOTE:
Add new CLI commands

Also in addition the E2E tests need to validate these configurations are there ...


Way to configure behaivior of @go.log FATAL

I am liking the new changes.  One thing I noticed was now when I use @go.log FATAL it produces my error message as well as a stack trace.
From a user point of view, if I am trapping an error (thus producing the @go.log FATAL) what  I provide to them should be sufficient... perhaps there are other cases where I wish to provide them a stack trace, but I've found most of the place I've used @go.log FATAL my message is sufficient.
For example:
If I run a script and the script expects a certain argument -c=/path/to/conf
I try to get the conf file, and if it doesn't exist
if [ ! -f "CONF_FILE" ]; then
@go.log FATAL "Please provide a valid path to configuration file with -c="
fi
The stack trace is not needed here, and if anything confuses the user. (Think "Java Errors")
Thus, I am guessing you have a way to enable or disable stack traces globally, however, maybe we should have two "FATAL"s, one for trapped errors like mine, and another when we need more debugging?
DEBUGFATAL? (lots of characters?) DBFTL? DBFATAL? FEMFATAL? (Ok that last one is pushing it).
Am I making sense or rambling?

Hey @JohnOmernik! It totally makes sense; you're not rambling. I'd wondered whether to make the trace optional. I'm thinking a global default variable that can be overridden by an invocation if needed, as other applications may want the trace and only disable it on occasion. How about _GO_LOG_FATAL_SHOW_STACK?

One of the reasons I like having two different FATAL options (one with stack and one without) is that it is easy for the developer to change as needed without changing a global variable... I guess I am worried about code clutter.

That makes sense. How about FAIL or QUIT?
And by the same token, @go.critical_section_begin can take an optional argument that can override the log level on error to FAIL or QUIT.

So FAIL would produce a stack trace, and QUIT would not? That makes sense.  I will require a bunch of changes in my code, but I can live with that :)

I was thinking of keeping FATAL as-is to produce a stack trace, and QUIT (or FAIL) to exit without one. But the idea of switching FATAL to FAIL is interesting, too.

Hey John, hope to implement @go.log QUIT by tomorrow. Currently almost finished slaying my current white whale (#49).


Use the same navbar on every website page

Use the same navbar on every page of the website so that users will not get disturbed by different navbars while using the website

We are currently redesigning the website ( Issue Number #120 ), so this improvement is not needed at the moment.


Laboratory Work #1, Gobova Anastasia - 20FPL1

its time to start implementing scrapper.py

Make sure you have green PR on Friday assignment.


docs(zh-cn): Reviewed 01_the-pipeline-function.srt

#390

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.


Vendor type person vs. organization

it is not clear from this documentation weather the 1099 functionality works the same for vendors of type person and organization. can you clarify if 1099 processing is available for vendors of type organization?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 222d0832-bb4c-31f2-a203-0b2a0826f90c
Version Independent ID: e32f8cb0-a4e5-9666-b6f7-f528360cf406
Content: Year-end 1099 reporting - Finance & Operations | Dynamics 365 | #MSDyn365FO
Content Source: articles/financials/localizations/noam-usa-year-end-1099-reporting.md
Service: dynamics-ax-applications
GitHub Login: @abruer
Microsoft Alias: abruer

Thank you for the comment. @abruer, can you please help with this question?

Bug 320705 has been logged to track this issue.

The 1099 functionality is available for vendors of type Organization and Person.


Remove limit on Filter

Motivation
While building a CLI to decode logs I reached for the limit method on Filter in order to implement some sort of pagination. To my surprise, it does not seem to have any effect. Combing through the documentation for Alchemy and Infura, the entries for eth_getLogs do not appear to include any limit option.
After pinging @gakonst in telegram, he suggested I make a PR to remove this as it doesn't appear to be part of the JSON-RPC specification.
Solution
Removed the limit option from Filter.
PR Checklist

[x] Added Tests
[x] Added Documentation
[ ] Updated the changelog

good catch


Messaging: deleteToken no longer deletes token correctly

Step 0: Are you in the right place?

For issues or feature requests related to the code in this repository
file a Github issue.

If this is a feature request please use the Feature Request template.

For general technical questions, post a question on StackOverflow
with the firebase tag.
For general (non-iOS) Firebase discussion, use the firebase-talk
google group.
For backend issues, console issues, and other non-SDK help that does not fall under one
of the above categories, reach out to
Firebase Support.
Once you've read this section and determined that your issue is appropriate for
this repository, please delete this section.

[REQUIRED] Step 1: Describe your environment

Xcode version: 12.5
Firebase SDK version: 8.5.0
Installation method: CocoaPods (select one)
Firebase Component: Messaging (Auth, Core, Database, Firestore, Messaging, Storage, etc)

[REQUIRED] Step 2: Describe the problem
FlutterFire: Issue: https://github.com/FirebaseExtended/flutterfire/issues/6766
RNFB: Issue https://github.com/invertase/react-native-firebase/issues/5570 P/R https://github.com/invertase/react-native-firebase/pull/5571
Steps to reproduce:
Call deleteTokenWithCompletion and inspect the token from tokenWithCompletion and see that it has not changed and is still valid.
From debugging, I can see that when the installations id is refreshed via  installations.deleteWithCompletion, the token will always refresh when calling messaging.deleteTokenWithCompletion
This behaviour can be replicated both in FlutterFire and RNFB. My question is what is the correct solution/method to refresh an FCM token?
I've looked through the changelog and the documentation, and it's still not 100% clear to me on how the installations id should be managed in relation to the FCM token.
Any help or guidance on this would be greatly appreciated.
Relevant Code:
By changing the following code, additional requests to deleteTokenWithCompletion will refresh the token:
From:
[[FIRMessaging messaging] deleteTokenWithCompletion:^(NSError *_Nullable error) {
    if (error) {
      [RNFBSharedUtils rejectPromiseWithNSError:reject error:error];
    } else {
      resolve([NSNull null]);
    }
  }];

To:
[[FIRInstallations installations] deleteWithCompletion:^(NSError * _Nullable error) {
    if (error) {
      [RNFBSharedUtils rejectPromiseWithNSError:reject error:error];
    } else {
      resolve([NSNull null]);
    }
  }];

More than anything, I am just very eagerly listening here for feedback.
@helenaford to be clear - do you mean that if you never delete installations token, you never get a fresh messaging token? I believe that is clear. However, do you also mean that if you delete installations token once, then you get a fresh messaging token each time you delete messaging token after even if you never delete installations token again ?
I had a hunch that this was - perhaps, maybe - related to old installations prior to the change where Messaging token moved from IID to Installations. However! Testing reports in the react-native-firebase repo indicate this happens with people that have only used react-native-firebase since after that change. Stated differently: This should be reproducible on fresh quickstarts that have never had an installation before.
Separately - I'm curious if using Installations delete vs Messaging delete will have some knock-on side-effect like resetting the in-app-messaging "run only once on this install" triggers. It seems to me that it should have that effect, but is it wanted? Stated differently: are the messaging token and the installations token effectively equal? Should deleting one delete the other? The answer to that might be an opinion vs a technical requirement?

@mikehardy yes, you have to delete the installations token once, and then it will be ok again. And, yeah it's not about old installations as I can reliably reproduce this issue by deleting the app, and reinstalling it.
I actually think it could be todo with the timing of when the Firebase App instance is initialized as I tried deleting the installations id in AppDelegate.m as the app is launching after:
    if ([FIRApp defaultApp] == nil) {
        [FIRApp configure];
    }

    // try to delete the installations token here
    [[FIRInstallations installations] deleteWithCompletion.....

with no luck. So maybe there's a race condition?  But it's very confusing because the initial FCM token that is generated works - messages can be sent to the device via firebase-admin.

Can you share with us the debugLog? We should figure out why deleteToken is broken.

@chliangGoogle for the avoidance of doubt, can you specify exactly what you mean by "debugLog" and how exactly to obtain it?
If it is what I think you mean, it is (for example) what you would see if you added -FIRDebugEnabled to as a build argument in the Xcode build scheme, then launched the app on a real device via Xcode and watched the Xcode console?
(if there was some document that clearly explained the process that would actually be helpful - I only know what I typed above from digging through different documents over the last couple years, yet it is something I need to request from people myself - as I support react-native-firebase - fairly frequently, and I have to type it out each time :sweat_smile: - plus I may be getting it wrong)

Yes. To enable debug logging set the following application argument: -FIRDebugEnabled (see http://goo.gl/RfcP7r)

@chliangGoogle thanks I can try and get this information over soon. But, not sure if it will help that much - as it can be repeated in both FlutterFire and RNFB (I don't have an iOS app I can test with), it seems the issue is within the native iOS SDK itself. If you could also take a look into it internally, it'd be much appreciated.

We are able to reproduce, I'm reaching out to our team for further investigation.

@chliangGoogle do we know if only a subset of iOS os-versions are affected? Or specific SDK versions are affected?

This is introduced with the big refactor in an effort to remove instanceID dependency in I/O release v8.0.0

IIUC, this bug is triggered under the following conditions. The condition itself seems rare in production but common during manual QA testing of an app. @chliangGoogle correct me if I'm wrong.

Using SDK v8.0.0 - v 8.6.0 (2021-04 - 2021-08)
Re-install app and launch app
race-condition triggered, authService(used by deleteToken... api) cached old checkin-credentials with some probability
deleteToken... is called.

Now if any event between 3 and 4 resets the cache in authService class then this bug goes away(on top of the probabilistic setting of the stale cache in the first place).
Nor sure about all the triggers to reset that cache but I presume the cache is deleted from the memory as the user shuffles through apps, right? Also, I expect the bug to go away if the phone restarts.
Also, I am assuming that typically there will be a large time window(days) between the reinstall and deleteToken events.

@helenaford / @mikehardy (or any other developer stumbling on this) could you share your use cases for using delete token? I'm trying to understand whether this API is serving the use-cases the developers expect it to serve, beyond the scope of just this bug, more generally. Feel free to tag me.

@spundun most use-cases of deleting a token on the client are when a user wants to log out or delete their account.

Exactly that.
If you have an app that registers the token for use in user-specific notifications, then you should as a best practice stop the flow of notifications to that token when a user logs out of the app, so that messages for the user are never delivered again
Yes on the backend the token shouldn't be used any more either so there are related processes that should occur to unsubscribe from topics and remove the token from any user correlations, but any of those may lose a race or fail to happen if the user is offline etc.
Token delete is local and should work every single time without fail, no?

Thanks @helenaford and @mikehardy . I suspected that would be the expected use case.
I'd recommend you use the "deleting the installation id" flow that @helenaford posted above for that purpose.
I'd love to know if that flow doesn't address your use case. Please feel free to reply back with any concerns or other use cases you might think of where you'd want to use deleteToken.
Personally, I'd like us to reconsider if it makes sense at all to have this deleteToken api. I think it adds a lot of confusion for the developer and seems to be adding unnecessary complexity to the API and the system.

There was some concern above that messaging and in-app-messaging would share the installation ID, and there may be a case where a user only wants to pop up a FIAM once per install, but supports multiple users and wants to keep FCM user targeted.
That was our hesitation on deleting installation token vs messaging
But perhaps they are the same token already? Insight there very appreciated

@mikehardy the token is only used by Messaging, in-app-messaging does NOT use token and only the installations ID, and even if you delete the installation ID, the new one should function as expected for in-app-messaging continue on. So you should be fine.

Ironically, the firebase-android-sdk documents their Installations.delete API with:

This call may cause Firebase Cloud Messaging, Firebase Remote Config, Firebase Predictions, or Firebase In-App Messaging to not function properly.

https://firebase.google.com/docs/reference/android/com/google/firebase/installations/FirebaseInstallations#public-taskvoid-delete
That implies something much more serious, and I guess I'm really surprised that iOS doesn't have similar side effects given that this deletes the installation on the backend completely since it is also indicated to be useful for GDPR compliance:
https://firebase.google.com/docs/reference/android/com/google/firebase/installations/FirebaseInstallations

provides a API to perform GDPR-compliant deletion of a Firebase installation.

I guess I'm just confused how we can still have all the non-messaging items behave correctly for an app installation if we delete the entire installation id/token vs just making sure that messages are no longer targeted at perhaps the wrong user :thinking:

@mikehardy I think the communication was misunderstood. What I meant is when you delete installations and get a new installation ID and new FCM token, then everything should function as expected. I was under the impression this was what you were asking. This helps you to resolve the issue and reset the state to back to normal.
If you deleted the installations and not getting a new installations ID and a FCM token, backend does a GDPR-compliant deletion and you should not expect anything is functional because the info has been wiped clean.


Exception while enabling Multiverse-Core in R5

Minor exception in v2.3-AB and R5 official build:
12:27:22 [SEVERE] Error occurred while enabling Multiverse-Core v2.3-AB (Is it up to date?)
java.lang.NoSuchMethodError: org.bukkit.plugin.PluginDescriptionFile.getAuthors()Ljava/util/ArrayList;
at com.onarandombox.MultiverseCore.MultiverseCore.getAuthors(MultiverseCore.java:525)
at com.onarandombox.MultiverseCore.MultiverseCore.onEnable(MultiverseCore.java:212)
at org.bukkit.plugin.java.JavaPlugin.setEnabled(JavaPlugin.java:215)
at org.bukkit.plugin.java.JavaPluginLoader.enablePlugin(JavaPluginLoader.java:336)
at org.bukkit.plugin.SimplePluginManager.enablePlugin(SimplePluginManager.java:363)
at org.bukkit.craftbukkit.CraftServer.loadPlugin(CraftServer.java:249)
at org.bukkit.craftbukkit.CraftServer.enablePlugins(CraftServer.java:231)
at net.minecraft.server.MinecraftServer.t(MinecraftServer.java:356)
at net.minecraft.server.MinecraftServer.a(MinecraftServer.java:343)
at net.minecraft.server.MinecraftServer.init(MinecraftServer.java:174)
at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:407)
at net.minecraft.server.ThreadServerApplication.run(SourceFile:465)
Should be easy to solve by switching from PluginDescriptionFile to newer plugin output methods :-)

I'm having this problem with the latest jenkins stable build of Multiverse. This exact problem happened with the VanishNoPacket plugin as well, and the developer confirmed it was a bug in the plugin's code calling getAuthors().

Ah I see you had already tried to fix this but it broke the Jenkins build, sorry for my redundant comment.


pom.xml dependencies scope

API 2.1.4 pom.xml includes the dependency jetty-servlet-tester, shoudn't be "test" scoped?
and javax.servlet shoudn't be provided scoped ??

All dependencies should now have the proper scope (I believe).


Fix: for doxygen

fix: looks like doxygen having issues with modern import for node modules, see if this helps

:tada: This PR is included in version 2.11.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


[GAL-4094] Sign in with wallet

Summary of Changes

Adding the wallet connect feature to the mobile apps
Added wallet connect library
Reused some authenticate function and moved it to shared packages

Demo or Before/After Pics
https://github.com/gallery-so/gallery/assets/4480258/a3185ff8-59c5-401d-9226-cde6225340f3
https://github.com/gallery-so/gallery/assets/4480258/76d388be-02c6-4dab-8507-8ec05ec25e08
Edge Cases

Sign in with existing user
Sign in with new user

Testing Steps

Changed the flag to true - https://github.com/gallery-so/gallery/blob/9e8e60af9e81e522d9b856baf5ac1705188ef53c/apps/mobile/src/screens/Login/LandingScreen.tsx#L23
You should be able to see the connect wallet button in the landing screen
For simulator testing - login with the QR scan since there is no wallet installed in the simulator

Checklist
Please make sure to review and check all of the following:

[x] I've tested the changes and all tests pass.
[x] (if mobile) I've tested the changes on both light and dark modes.

@Robinnnnn Thanks ser!

Renamed it
Added loading state to the login screen while waiting for new design for (signature stuff)

https://github.com/gallery-so/gallery/assets/4480258/62254602-a45d-44b2-9bcf-c65c5f2344fc

Update the loading state
https://github.com/gallery-so/gallery/assets/4480258/49ac287e-9371-4986-ba72-e1b2247e15ce

jakz lets gooo!!! this is sick, looking great.
left a couple of smaller comments about design and file placement, code looks great.
also noticed a few improvements we can probably make for when there's an error or user cancels the flow:
I closed the bottom sheet while it was waiting for me to sign a message, and now my app is stuck in the loading state. even if I restart the app, it automatically goes into the loading state, so I can't re-attempt the flow.
attaching a video of it automatically showing the loading spinner when I open the app.
https://github.com/gallery-so/gallery/assets/80802871/3a76406f-efe5-4047-8a2a-33e6b2a99713
I think some ways we can handle this are

if the user re-launches the app, reset the wallet connect state
if the user closes the bottom sheet during the sign in flow, reset the wallet connect state

@kaitoo1  Added the onDismiss callback to reset the wallet connection state
https://github.com/gallery-so/gallery/assets/4480258/97a83c63-5040-4c94-a3bf-54754c482758

nice jakz!!!! thanks for addressing comments, it looks great!


Performance and Profiling of the WASM plugin

What
The WASM module is in the critical data path, It is important we are able to understand how much time is spent in the WASM module and where that time is spent so that we can see bottle necks and identify areas that need optimisation and improvement
How
Export metrics that allow us to profile WASM module and analyse the code performance.
Execute Load tests and observe performance from the profiling metrics
Document the results and how to rerun these tests

https://github.com/Kuadrant/wasm-shim/issues/9   dup?


Add Readtables back

Again, apologies for the vague issue, but:

Do readtables exist?  If so where are they documented?  (Better: some simple examples?)
If not, then make this a feature request to have them...

Ah, thought I had already made an issue for this.
Indeed, they are missing and we need to add them back. We had them pre-1.0 (thanks @jlongster!) but then I rewrote the reader.

Could we use readtables to make sweetjs whitespace aware? Say for instance like coffescript or ruby?

That'd be really, really, really hard.

Readtables are more intended for a case where you want to add a new construct to an existing language with minimal changes.  An indentation-based syntax is a drastic change to the language so you really need to implement a parser for that.

I implemented indent blocks in 0.7. The table used a simple regular expression to track the flow of indentation. The syntax was:
while (true):
	while (false):
		if (1 === 2):
			return;
		y = 3
		j = 5
	x = 5
	for (i of [3]):
		i.j="k"
	i = 3

Unfortunately, outdenting two levels at once was blocked by an issue related to nested braces, IIRC.

@disnet are you looking to have the same API as the old readtables?

Not necessarily.

I'm looking to extend the utility of #. I'm thinking of looking at Racket's API for inspiration. What requirements are you targeting for readtables?

So far I've come up with a primitive api that's similar to that of syntax:
lexeme ~> = function(reader) {
  let result;
  const key = reader.key(); // ~>
  reader.read(); // eats 'skipMe'. uses default reader behavior
  // creates a reset point that is consumed when reset is called.
  // other possible names: shift, savePos, checkpoint
  reader.mark();
  const delim = reader.next().value // [
  let brackets;
  if (delim.isLeftBracket()) {
    reader.reset();
    brackets = reader.read('Delimiter');
  } else { // throw }
  return `thread { ${brackets.inner()} }`; // .inner returns another reader instance
}

foo ~> skipMe [bar, baz(1)] // foo thread { bar, baz(1) }

Implicitly, the current readtable is being extended. I do have questions about using modules at read time. Are syntactic extensions and other libraries available at read time?
import m from './m' for reader

Things I liked about the old API are mostly whitespace-related: the ability to skip whitespace without gobbling up the next token, access to the full source code, access to the current index, access to the current line number.
A read-phase equivalent of syntax isn't going to cut it for stuff like JSX and indent blocks where whitespace is significant.

I disagree. The above doesn't display the full API I have in mind. I was thinking of moving many of the 0.7 readtable utility functions to Token objects returned from read and next. There's also no reason you couldn't have the option of calling reader.read('Whitespace') or something similar.
I'm also not sure why you'd need access to the entire source string. Your coffeetable implementation doesn't seem to use anything in the source before the triggering character anyway.
I know I haven't thought things all the way through yet and I'd like to hear more useful criticism.

mhmm, I distinctly remember having a use case for access to the source code, but I'll have to get back to you on that. Also, I didn't mean to say that a syntax-like API design and whitespace awareness are mutually exclusive, just that they have to be consolidated.

My mistake. It looks like you used it to look behind one character for ellipsis. As with syntax extensions, I think we'll need limited lookbehind.

syntax is a binding form so I don't think your proposed syntax-like API quite works.
What were the extensions to # you wanted to do?

I was thinking lexeme or some other identifier would be implemented as a macro and desugar to something like:
{
  token: "~>",
  read: function(reader) {
    // ...
  }
}

But I'm guessing that doing multiple expansion passes so that readers could use syntax macros could be a lot of overhead.
As far as #, I was thinking of implementing callable, clojure-like keywords and immutable data structures:
const hm = #{ #foo 'foo', #bar 'bar' };
#foo(hm); // 'foo'

as well as some sort of lens literal:
const hm = #{ #foo 'foo', #bar #{ #baz 'baz' } };
const l = #bar/baz; // maybe #/bar/baz, #bar/baz/ or #/bar/baz/
l(hm); // 'baz'

These could potentially work on regular js objects as well
const obj = {foo: 'foo', bar: {baz: 'baz'}};
#bar/baz(obj); // 'baz'

Oh yeah, #{ and friends would be :100:.
Have you started hacking on readtables? If not I might start; I have a few other things I want to clean up in the reader anyway.

I started working from the API back (so the reader that's passed to the transform function. But I haven't started integrating it yet. I wanted to wait on answers to the details.
Can readtables make use of syntactic extensions?
How about previous readtable entries?
Are they going to take part in the module system?
How are they loaded?
I guess I could start by just wedging them in there via CLI options and not worry about the other questions for now.

Is it feasible to implement # -> syntaxQuote and syntaxQuote as reader and syntactic extensions respectively?

What were your thoughts for cleaning up the reader?

Can readtables make use of syntactic extensions?
How about previous readtable entries?

Let's say no for now, can think about this later.

Are they going to take part in the module system?
How are they loaded?

My original intention was for them to be part of the #lang pragma. One of the exports that a #lang points at would be the readtable (the other an as-yet-to-be-determined module manipulating thing). For the first pass though you can definitely just do the CLI thing.

Is it feasible to implement #... -> syntaxQuote... and syntaxQuote as reader and syntactic extensions respectively?

Yes definitely, syntax templates should definitely be implementable via a readtable.

What were your thoughts for cleaning up the reader?

I just pushed a branch with some doodling I was doing the other day. Feel free to take or ignore as you see fit.
Main things I wanted to do was pull in the tokenizer directly so we can own changes to it rather than rely on the shift parser dep. I was also thinking of changing the Reader class to composition rather than extending the Tokenizer but not sure which is better in this situation. We could also address #551.
Also, I would love if the / disambiguation logic could be in the readtable.

@disnet If you're considering removing the parser dependency anyway I've been thinking about a more radical idea. Now it'd be a bunch of monkey work but what if we implement the reader to delegate completely to a readtable?
A table entry could have a predicate (character sequence, regex or function) to match then next token paired with a tokenizer function. The function could return either a string or a token. If it's a string, the reader splices it back into the source and continues from the index before the function was called. Otherwise, the source is consumed and the token is added to the stack.
Another possibility is to have either a tokenizer or a transform function in the table entry.
Now I haven't thought out all of the implications but the idea keeps coming back to me.

Most of the work would be in pulling apart the current tokenizer and populating the base readtable with it.

but what if we implement the reader to delegate completely to a readtable?

Yes absolutely!

@disnet we previously discussed limiting the horizon of macro contexts https://github.com/sweet-js/sweet.js/issues/537#issuecomment-223113317
What are your thoughts on the same question for reader macros? My understanding is that pre 1.0 readtables allowed access to the entire source of a file.
Do we still want to have everything visible?

@gabejohnson yeah, they definitely should have access to everything. A macro needs to play nice with other macros (and binding forms) so they must be limited but a reader is low-level and get's to do whatever it wants.

@disnet  I'm currently favoring using a zipper as the reader abstraction copying some of the clojure.zip API https://clojuredocs.org/clojure.zip
This would give the user a great deal of flexibility; allowing them to transform the matched character sequence in a number of ways:
In this example the sequence is replaced with a token. The actual Reader instance will advance rdr after tokenizer returns:
{
  matcher: '|>',
  tokenizer: (rdr) => rdr.replace({
    type: TokenType.IDENTIFIER,
    value: 'PIPE_MACRO'
  })
}

Here, one substring is replaced with another and rdr is moved back one so that 'PIPE_MACRO' can be matched by the Reader:
{
  matcher: '|>',
  tokenizer: (rdr) => {
    rdr = rdr.replace('PIPE_MACRO');

    // go back one so the reader can tokenize 'PIPE_MACRO' as an identifier
    return rdr.left();
  }
}

The following two examples illustrate using a function/method to remove boilerplate:
{
  matcher: '|>',
  tokenize: (rdr) => rdr.fromString('PIPE_MACRO')
}

Although more convenient, I'm hesitant to include from* as methods on the zipper as they should be fairly easy to implement as utility functions:
{
  matcher: '|>',
  tokenize: (rdr) => ReaderUtils.fromString(rdr, 'PIPE_MACRO');
}

I'd particularly like feedback from you @elibarzilay and @jlongster as, I believe, you are both quite familiar with reader macros.

Another possibility is to have the Reader infer that if a string is returned it should be a simple replacement:
{
  matcher: '|>',
  tokenizer: (rdr) => 'PIPE_MACRO'
}

or even
{
  matcher: '|>',
  tokenizer: 'PIPE_MACRO'
}

@gabejohnson 👍  on your plans so far.
I think racket has a way of specifying modes for each mapping: terminating vs non-terminating macros, deferring a char to another mapping etc.
Does your current design have something like modes? Probably not critical for the first version but just want to make sure the API will have an obvious extension point.

@disnet I suppose modes in my scheme (no pun intended) are somewhat implicit. For instance, the last example above shows a string as the value of the table entry tokenizer property. This is akin to Racket's make-readtable with a char passed as the mode parameter.
Terminating macros would be implicit in that the author of the entry can do whatever they want once they match say {. The could treat everything that follows as a line comment, match on } and create a child in the tree or do a char/string substitution.
I honestly hadn't given any thought to non-terminating macros. Though you could write a matcher for identifiers that keeps accumulating chars until it hits one that's invalid. Then it defers to the readtable. "foo#bar" would match on foo, then if # is in the current readtable it would return a reader w/ the result of that match. Not sure what the API would look like there.
We could certainly make some convenience functions/methods in the future. In fact maybe I should start w/ matcher only accepting functions and making the string (regex?) option a convenience later.

So I've been thinking about your proposed API a bit more and have become more convinced that the API should be as low-level as possible (this of course applies to everything we put into core Sweet not just readtables). It doesn't have to be nice as long as it is possible to build nice things on top of it.
In particular I don't think you want table entries to be strings at all. A char code should be sufficient since "|>" is equivalent to having the entry be the char code | and the method attempting to consume > or defer to the parent readtable's | entry. A library can of course compile strings and regexes and other kinds of patterns down to this core API but we don't need/want to do it ourselves.
As with everything in Sweet we want to let a thousand APIs bloom.
I'm still not entirely following how the zipper API works for readtables. Could you explain how zippers work in this context? It seems to me the core things you need to be able to do inside the tokenizer method are:

peek at without consuming the raw string after the current position
consume some number of characters
read the next token using the current readtable
lookup line and column info at the current position
create a new token

need to have constructors for all token types that enforce required arguments like line info

(optional) ability to see previously read tokens?

Maybe the zipper API you have in mind does all that but I'm not following yet :)

Restricting matcher to char codes rules out implementing identifier matching in a base readtable. But I'm thinking more that we would incur a serious performance hit by implementing the entire base reader in the readtable. Only one way to find out.
As an aside, I was thinking of a scheme to use tries to match keywords and then cache identifiers in the trie was well. Just a thought.
Now let me address the zipper API:
A Focus instance represents the current position in the token tree/source string. If the focus is on a character, it contains the character/char code and location info (line,  column, index). The tokenizer function is responsible for transcribing the location info to a new token.
right moves the focus one character to the right in the case that the character hasn't already been consumed. Otherwise it moves the focus to the next token. right does NOT consume the character.
I haven't decided what happens at EOS.
left moves the focus one character to the left unless there are no unconsumed characters to the left. Otherwise it moves the focus to the next token to the left.
I haven't decided what happens if there are no more tokens to the left. Clojure would return nil. That's obviously not cool in our case.
next moves the focus one token right/down in a depth-first traversal. It consults the readtable to tokenize the following characters. If there is already a token to the right next acts like right unless the current item is a delimiter. In this case, the focus drops into the delimiter. next DOES consume characters.
prev moves the focus left/up in a depth-first traversal. It acts like left unless there are no more items to the left in which case it moves up if possible.
up moves the focus out of the current delimiter.
down moves the focus into the current delimiter (if it IS a delimiter otherwise throw?)
None of these traversal methods return nodes/characters. The all return a new, immutable focus.
That's the traversal API. I do have a slight concern that user's intuition might be violated since next is not read-only. Maybe there should be an explicit tokenize.

Now for the node access and tree modification API:
node/getNode could be a method or getter. It returns the node at the current focus.
remove removes the node at the current focus and returns a new focus moved as though prev were called.
insertLeft/insertRight insert nodes to the left and right, respectively.
Optionally other zipper methods could be implemented, but I think everything else could be written as utilities (probably overlooking something though).
node could be a getter/setter. The we wouldn't need a replace util.

Also, I forgot about path. It should be/return an object that allows you to pass it to a traverse method/function and arrive at the position you were at when you got it.
root would be a nice to have, but you can always get the path of the root node and pass it to traverse.

As for the token constructors, I've just been hand rolling them, but I see what you mean. Maybe the focus is a property on the reader and you can also call rdr.createIdentifier(string, location, ...).
I'll grant you that with only source, index, line and column you could do most of the stuff the zipper does (except for inspecting already created tokens). The reason I want to make the zipper native is so that the scanning could be implemented almost entirely in a base readtable and traversal would be a snap. Otherwise, looking behind is going O(n) and cumbersome. If zippers were implemented as a library, then reading would be O(n^2).
I'd like to easily be able to create a delimiter, jump back/forward in the tree, insert it and return to the "current" position.

If you used zippers you'd be able to do rdr.createIdentifier(startFocus, endFocus)

The reason I want to make the zipper native is so that the scanning could be implemented almost entirely in a base readtable and traversal would be a snap. Otherwise, looking behind is going O(n) and cumbersome. If zippers were implemented as a library, then reading would be O(n^2).

I don't understand this

Restricting matcher to char codes rules out implementing identifier matching in a base readtable.

This was based on an (arguably bad) assumption that matching and tokenizing would be separate concerns. Though I suppose nothing is gained by this if a null result from action means no match.
The complexity remarks were based on a complecting of tokenization and rewriting. Constructing a zipper is O(n) in the worst case and there are n readtable lookups. If there is a zipper created upon each lookup this is O(n^2). Obviously, separating them changes things.
Now I would argue that creating the token tree is already complecting tokenization and rewriting. But it is a simple transform and makes sense to do at the same time.

Are you suggesting that there could be a separate token stream rewrite phase? Or simply that it shouldn't be allowed?

Though I suppose the rewrite phase is during enforestation. There are still things that I would like to do that appear to be difficult w/o moving tokens around arbitrarily. Maybe I'm overlooking something.
Suppose I wanted to write a language that looked like the following:
#lang whatever

export {
  foo: x => ...,
  bar: (x, y) => ...,
}

local {
  baz: x => ...,
  qux: (x, y, z) => ...,
}

import {
  quux from './quux',
  corge from './corge',
}

this would translate to:
import quux from './quux';
import corge from './corge';

const baz = x => ...;
const qux = (x, y, z) => ...;

export const foo = x => ...;
export const bar = (x, y) => ...;

I can maybe define whatever to insert a whatever macro at the top of the file and then do all of my manipulation after that, but then I have to write my own matching logic.

Does it make sense to have language definitions provide a default macro that's included at the top of every macro that references the language?

This is exactly what the lang pragma is all about :)
Following Racket the lang pragma will have an API that allows you to install a readtable, set up new syntax and implicit form bindings, and in general rewrite the syntax stream.
Your proposed zipper API would probably work really nicely here but I'm pretty sure we want to keep it separate from readtables.

Sounds good! I'll start with the API you outlined above. I think I have a pretty clear idea of how to go about implementing most of it. The only mode I'm not sure of is non-terminating-macro. But I'm sure it'll come out in the wash.

@disnet for Entry.char should we be using code points instead of char codes?

Yes indeed! Good catch.

@disnet two questions:

Should action have a prefix argument (for resolving /, etc.)?

It could either be a list or a stream of Tokens. Definitely read-only.

Should undefined/null returned from and action throw an error?

At first I was thinking that returning null could cause the reader to fall-back to a previous entry for that char. Now I'm not so sure. Since we have the base readtable available to us when we make a new one, we could always grab the previous entry and call its action. I'm leaning toward being explicit, this is just a sanity check.

Should action have a prefix argument (for resolving /, etc.)?

Yes I think it should.

Should undefined/null returned from action throw an error?

Actions need a way to consume from the CharStream but not produce a token (e.g. a comment). null would work for this case right?

Actions need a way to consume from the CharStream but not produce a token (e.g. a comment). null would work for this case right?

You're right. In the fallback scenario I was going to have a special noop token, but if we're being explicit about passing down the action chain then null for noop makes more sense.

#601 and #626 Should close this. No?


Pan / Scroll Sankey Chart?

Hello
Firstly, thanks for this plugin. I've been able to quickly get some awesome charts going. Much obliged.
For most of my data, I have charts that have a few sources (say 10 or less) and hundreds of outputs. I'd like to vertically scroll the Sankey chart, if possible. Considering the number out connections I have, having a very large vertical canvas results in noticeable lag / overhead.
(ie, simply making the canvas large isn't a great solution as im drawing 4k pixels vertically... )
Id like to limit the drawing and provide some mechanism to vertically scroll / pan. Im aware Chart.js has this feature, but unless I am mistaken I can't seem to find the right way of getting It to work with this plugin.
For example, here's a zoomed out chart to provide a visual example ( had to split it over 2 images)

Is there a simple example of a working scrolling solution for Sankey charts?
Secondly, is there a way to pin some elements to a fixed / relative to the viewport  on screen position? I'd love my left side elements to remain at the top even when scrolling, so one has a consistent reference point.
Thank you again!

Nice looking chart :)
It could be possible to make this compatible with the zoom plugin, but that would not help with the pinning.
I like the idea of pinning and scrolling. Maybe one level zoom to the smallest column would work and the other columns could then be scrolled?
In your case the left column would fill the height and right column would be partially visible (and scrollable).