Virtual region managers and dynamic lookup support

This PR adds additional abstraction to halo2-base so it can more easily support dynamic (or any) lookup tables. It also addresses some previous code quality / organizational issues. halo2-ecc is unchanged aside from code updates to be compatible with the aforementioned changes.
No circuit logic was changed in this PR.
Virtual Region Managers
Previously, GateThreadBuilder was conflating many different functionalities, while being very specialized for the particular circuit configuration used in FlexGateConfig and RangeConfig, which consists of:

a basic gate in 1 column
a fixed lookup table in 1 column

However more abstractly GateThreadBuilder is an instance of "virtual cell management": we perform witness generation in a virtual region (outside of synthesize) and only at the very end map it to a "physical" region in Halo2's Plonkish arithmetization.
We formalize this into a new trait VirtualRegionManager. Any VirtualRegionManager is associated with some subset of columns (more generally, a physical Halo2 region). It can manage its own virtual region however it wants, but it must provide a deterministic way to map the virtual region to the physical region.
We have the following examples of virtual region managers:

SinglePhaseCoreManager: this is associated with FlexGateConfig in a single challenge phase.  One can think of all virtual columns as being concatenated into a single big column. Then given the target number of rows in the physical circuit, it will chunk the single virtual column appropriately into multiple physical columns.
CopyConstraintManager: this is a global manager to allow virtual cells from different regions to be referenced. Virtual cells are referred to as AssignedValue. Despite the name (which is from historical reasons), these values are not actually assigned into the physical circuit. Instead they keep track of a tag for which virtual region they belong to, and some other identifying tag that loosely maps to a CPU thread. When a virtual cell is referenced and used, a copy is performed and the CopyConstraintManager keeps track of the equality. After the virtual cells are all physically assigned, this manager will impose the equality constraints on the physical cells.

This manager also keeps track of constants that are used, deduplicates them, and assigns all constants into dedicated fixed columns. It also imposes the equality constraints between advice cells and the fixed cells.

LookupAnyManager: for any kind of lookup argument (either into a fixed table or dynamic table), we do not want to enable this lookup argument on every column of the circuit since enabling lookup is expensive. Instead, we allocate special advice columns (with no selector) where the lookup argument is always on. When we want to look up certain values, we copy them over to the special advice cells. This also means that the physical location of the cells you want to look up can be unstructured.

Dynamic lookups
I have added an example of how LookupAnyManager can be used in conjunction with SinglePhaseCoreManager for dynamic lookups. At present it still involves implementing the Circuit trait on an application-specific basis.
Diagram explaining the virtual region management philosophy:

Changelog

gates::builder has been partially moved to gates::flex_gate::threads because GateThreadBuilder was very specialized to the logic of FlexGateConfig.
As mentioned above, GateThreadBuilder was actually a conflagration of the following virtual region managers:

MultiPhaseCoreManager, which is one SinglePhaseCoreManager per challenge phase. This handles only the logic of assigning vertical advice columns that FlexGateConfig owns, and turned on the associated selector.
LookupAnyManager for the range lookup into a fixed lookup table with the entries [0, 2^lookup_bits).
CopyConstraintManager

GateThreadBuilder is no more. It corresponds most closely to MultiPhaseCoreManager. However BaseCircuitBuilder (generalization of RangeCircuitBuilder and RangeWithInstanceCircuitBuilder) now handles most of the combined functionality of the old GateThreadBuilder and RangeCircuitBuilder.

The meta-logic is that a CircuitBuilder holds the collection of VirtualRegionManagers that are needed to actually assign to a physical circuit.

The circuit builder part of gates::builder has been moved to gates::circuit::builder.
CopyConstraintManager and LookupAnyManager are derived from old code in gates::builder but now cleaned up and moved to virtual_region.
Context previously had cells_to_lookup, which was specialized for the range lookup. Since it doesn't make sense that Context would need to support arbitrary lookups, it has been moved into LookupAnyManager.
Context also previously contained the information now held in CopyConstraintManager. We have Context store an Arc (thread-safe smart pointer) to CopyConstraintManager.
GateStrategy was deleted since we've stabilized on a single strategy.

Note about multi-threading

The original goal of Context in v0.3 was that a single CPU thread should use a single Context in a deterministic way. While multi-threaded execution is non-deterministic, if we can deterministically map CPU threads to Contexts and then deterministically combine Contexts inside our virtual region, we can do parallel witness generation deterministically.
This would mean each VirtualRegionManager should have its own set of contexts for this purpose. Both for backwards compatibility and to prevent the developer from having to pass in some list of contexts each time, we have taken the approach that while Context holds the virtual columns specific to FlexGateConfig, we use context_id as a universal way to order CPU threads. This means that other region managers (for example LookupAnyManager) can store their CPU-thread local trace all together in some thread-safe struct, and as long as each CPU-thread is deterministically tagged with a context_id, we can order their physical assignments deterministically, ensuring that witness generation is still deterministic.

@nyunyunyunyu I have addressed the basic comments. The unresolved ones seem a bit more involved / complicated, so I'm inclined to save it for another PR. For now we err on the side of giving more user flexibility, at a slight cost of potential user error. We can iterate / make changes as we also use the new code in our other circuits.

@nyunyunyunyu I have addressed the basic comments. The unresolved ones seem a bit more involved / complicated, so I'm inclined to save it for another PR. For now we err on the side of giving more user flexibility, at a slight cost of potential user error. We can iterate / make changes as we also use the new code in our other circuits.

@jonathanpwang  Cool let's merge this first and leave comments open so we can address them later.


Adding support for start-after and end-before options

The regular reST include directive has support for :start-after: and :end-before: options allowing specific to text to search for for setting the start/end points of the inclusion.
This is extremely useful since it allows including, say, only specific sections of a markdown file, without always having to chase the exact line numbers.  Happy to make a PR since I see you're looking for help on this package.

That would be very useful indeed! Is there any news on this? @embray, @dmaljovec, @UncleGoogle
I think you've already proposed a solution in https://github.com/miyakogi/m2r/pull/42 @dmaljovec?


Is there any way to change download path?

I want to set it as my Downloads folder. Is there any way to do that?
And great tool, thanks for everything!!

Currently, there is no way to specify a desired folder for the images but I'll try to provide that option, as well.


ARROW-7281: [C++] Make Adaptive builders' length match expectations

The length() value should account for internally buffered data, even if not committed to the final buffer.

I've checked this doesn't affect the builder benchmarks.

Ok, hopefully it's fixed.

AppVeyor: https://ci.appveyor.com/project/pitrou/arrow/builds/30108341


Expand folder, large spacing

It's not a bug, but I don't feel good. i like boxy theme folder style.
this is boxy theme

this is one-dark-theme

This theme is not finished yet but the main idea is to make Sublime Text look as close as possible to Atom One Dark theme.


Keyboard shortcut for Switch to Inline View

I know you can toggle it permanently with "diffEditor.renderSideBySide": false
It would also be great with a keyboard shortcut to toggle between side-by-side and inline view in git diff.

@GandalfSaxe, you can assign a keybinding to toggle.diff.editorMode. You can also use the editor menu:

Ah perfect. Thanks @kieferrm! I got confused by the three different names used for this same feature:

GUI toggle: Toggle Inline View
Setting switch:  renderSidebySide
Keyboard toggle: editorMode.

You might consider to make the three names more uniform. It's hard to search for, both in settings and especially in keyboard shortcuts, since you just tend to search for whatever it's called in the GUI 🙂


Restore index command failing

I was successful to archive an index to Amazon S3. But when tried to restore the same index back to elastic, the restore script throws an exception. ie
{"error":{"root_cause":[{"type":"snapshot_restore_exception","reason":"[audit_snapshots:audit-2017.02.01] cannot restore index [audit-2017.02.01] because it's open"}],"type":"snapshot_restore_exception","reason":"[audit_snapshots:audit-2017.02.01] cannot restore index [audit-2017.02.01] because it's open"},"status":500}
I noticed that in the backup-all-indexes.sh we open the index and later delete it.
curl -sS -XPOST "${INDEX_URL}/_open"
curl -w "\n" -sS -XDELETE ${INDEX_URL}
Through the index was deleted, when I tried to check the available indices, the deleted index was still available though data was not present.
Not able to understand what is happening. Kindly help me out.

As I mentioned to you on Slack; this is happening because you set MAX_DAYS_TO_KEEP=0, so you archived an index that was being written to and which therefore got promptly recreated after you archived it.
We'll be adding a guard in the script to disallow setting MAX_DAYS_TO_KEEP to values that are too low.


Updated compatibility to Laravel 5.6

The upgrade guide  for L5.6 doesn't seem to introduce any breaking changes.

This fork/branch seems to work fine with a 5.6 project

Hi @HipsterJazzbo, any plans to accept this PR?

Hey @HipsterJazzbo , any plans to accept it?

While this PR is not merged and you want to update your Laravel version:

Change the package version:
"hipsterjazzbo/landlord": "dev-master"

Add this block to your composer.json file:

"repositories": [
    {
        "type": "vcs",
        "url":  "git@github.com:rikvdlooi/Landlord.git"
    }
]

While this PR is not merged and you want to update your Laravel version:

Change the package version:
"hipsterjazzbo/landlord": "dev-master"

Add this block to your composer.json file:

"repositories": [
    {
        "type": "vcs",
        "url":  "git@github.com:hfalucas/Landlord.git"
    }
]

@HipsterJazzbo ping 🙈

No news on accepting it? =/

Tried to reach him on Twitter some time ago (he asked for someone taking over Landlord), but did not receive any answer yet.


Where does flag come from in the example?

In the minimal example in the readme there is the line:
fn caps_flag() -> flag.FlagBuilder(Bool) {

I don't understand where flag is coming from. Tried replacing it with snag, but that also doesn't work.

Hi there! Thanks for pointing this out, the readme is currently very much in flux as I prepare to cut v1.0.0-rc2, please use the docs for either v0.18.1 or v1.0.0-rc1 with the relevant version from hex in the meantime 😁

I just have missed that when I updated it recently

Thanks. I will give that a go :)


Metadata

Would it be possible to download the lesson metadata, i.e. synopsis, etc... and embed it into the video file?

Hi, maybe its possible but i will not add it. Sorry.
I accept pull requests.

@iamfreee Can you specify the part of the script you can get the page source code to let something of the community make it happen easily?

Hi,
https://github.com/iamfreee/laracasts-downloader/blob/master/App/Http/Resolver.php#L150
This class has two methods to download the page of a serie episode and a lesson,
downloadSerieEpisode() & downloadLesson(), inside: $html = $this->getPage($path);


update deps, fix test warning, favicon

fix #6
fix #13

Coverage remained the same at 100.0% when pulling fd1f41a4345b58802e29b8a89ea2da14f6593786 on update-dep into ed1414b305660e33b7fb040c7a69b42fdc354427 on master.


Closing MessageView/MessageEditor before load complete triggers errors

... due to callbacks still registered for the view.

Errors triggered by MessageEditor not reproducable.


USB Mouse only works after unplugging and replugging it

It seems the detection and capture of USB devices is different if it happens at program launch compared to when Karabiner-Elements is already running.
Things I have noticed:

My mouse always works when Karabiner-Elements isn't running.
My mouse doesn't work when Karabiner-Elements is started with the mouse plugged in.
My mouse works fine again after unplugging and replugging it with Karabiner-Elements running.

As this means that my mouse works fine after sleep and/or whenever it is replugged this is a very minor issue for me. More of an annoyance that I only notice on reboot and whenever Karabiner-Elements is relaunched (after autoupdates).

I can confirm this happens with my Bluetooth mouse as well (MS Sculpt Confort)

I am wondering if this isn't just mice, but other USB peripherals as well. I have been having a problem with my AudioEngine 2+ USB speakers loosing connection after a period of time, even while in use. Unplugging them and replugging usually seems to work.

@nanno77 That shouldn't be an issue any longer. Under "Devices" you can select which USB devices Karabiner-Elements can modify.


Bug: Hub/Shuttle is streaming delete events but when queried hubs for latest state it says that its in merge state

What is the bug?
Basically we are receiving delete events from Shuttle but when we check for hub using the getAllVerificationMessagesByFid it says its in added state. we are assuming whenever MESSAGE_TYPE_VERIFICATION_ADD_ETH_ADDRESS appears with
merge state -> we add to our system
delete state -> remove from our system
prune state -> remove from our system
revoke state -> remove from our system
correct us if we're wrong.
for ex. FID - 284564. If you observe we're adding a a timestamp when we receive called event_timestamp. delete event is coming without any remove event.
{
	"data": {
		"type": "MESSAGE_TYPE_VERIFICATION_ADD_ETH_ADDRESS",
		"fid": 284564,
		"timestamp":<PHONE_NUMBER>000,
		"network": "FARCASTER_NETWORK_MAINNET",
		"verificationAddAddressBody": {
			"address": "Kjn9lOE5lSw5WOaRcrqRjY9TNXE=",
			"claimSignature": "jTZWmE8MJlONRBzItIqo2ahEHucShtDpSVEKKME++y5xb+a26BDFfhdInlrPBSrxqjevwlc3zFub7bpWE00+zxw=",
			"blockHash": "lGlqnrWQvyrLS6y4EUhjd3GwoqTPbRGYFrOWuc1E3Og=",
			"verificationType": 0,
			"chainId": 0,
			"protocol": "PROTOCOL_ETHEREUM"
		}
	},
	"hash": "DyGKwXxmmpWnJnJXq9MFnV64Ty8=",
	"hashScheme": "HASH_SCHEME_BLAKE3",
	"signature": "sDemPPBPGKUYAZly17eHknYCMLCyzUUVEPJN8zYCQI95v8JiUbPQs8ycw3oU8WxLjDT0uv//j/QQhgv4zjqbDw==",
	"signatureScheme": "SIGNATURE_SCHEME_ED25519",
	"signer": "9YfDCLVjOCC1TSedks0ovIhQ0QxSp9jadZq7OEtSexk=",
	"eventTimestamp":<PHONE_NUMBER>000,
	"operation": "merge"
}

{
	"data": {
		"type": "MESSAGE_TYPE_VERIFICATION_ADD_ETH_ADDRESS",
		"fid": 284564,
		"timestamp":<PHONE_NUMBER>000,
		"network": "FARCASTER_NETWORK_MAINNET",
		"verificationAddAddressBody": {
			"address": "Kjn9lOE5lSw5WOaRcrqRjY9TNXE=",
			"claimSignature": "tJnBlHSeoxYxsE29j4P/GvWXQpNOTlB5MbGdTpiHqEk3pfek71oIjhvrNa0WnAN2BRcTWZRyrvrzwE5hLY19Vhw=",
			"blockHash": "moftXqHGrPPyGWmBv523i7EA4ME7urpLZqyvJZ9INEU=",
			"verificationType": 0,
			"chainId": 0,
			"protocol": "PROTOCOL_ETHEREUM"
		}
	},
	"hash": "TsGP1MUa9rTjaNY42whm4ztP0KI=",
	"hashScheme": "HASH_SCHEME_BLAKE3",
	"signature": "AjgNy7sXqXN8lYKesgKzRM5soF0rNL1kXlIOoIA6Zhu3rEpkUSpyfmzxJfku0RRERvCHh4qKZ0T4Rrfjw6/zCQ==",
	"signatureScheme": "SIGNATURE_SCHEME_ED25519",
	"signer": "9YfDCLVjOCC1TSedks0ovIhQ0QxSp9jadZq7OEtSexk=",
	"eventTimestamp":<PHONE_NUMBER>784,
	"operation": "delete"
}

There are thousands of such FIDs which are getting affected with this issue. please try to investigate this. as its our top priority.
Feel free to reach me out if more details are needed in this issue. farcaster-dc: chetansk
We are live streaming all the events to kafka where we noticed this. you can stream using shuttle which can

This is happening for lot of FIDs where delete events are coming but seems like there's no actual deletion happened.

Hmm, so, what's seems to be happening here is that there are two verification add messages for the same address and fid.
one from May 27, with message hash "TsGP1MUa9rTjaNY42whm4ztP0KI" and
one from June 2 with message hash "DyGKwXxmmpWnJnJXq9MFnV64Ty8="
When the June 2 message is merged, it correctly overwrites and deletes the May 2 one, because hubs have a unique constraint on the address. There is no remove message. A remove is message is not required to cause a delete, because a newer message will overwrite an older one (especially for verifications and UserDataAdd).
It looks like the events have made it into your kafka stream out of order. There's not enough information here to say what caused this. If you could provide the createdat and deletedat timestamps for these two messages from shuttle's messages table for these two messages it will help narrow down whether it's happening in the hub, shuttle or at some point after.
Note that the message timestamp is different. Even though the delete event timestamp is newer, the timestamp of the message is older than the timestamp of the message in the merge event. So, a workaround would be to check for this.
To summarize,

delete events don't require a Remove message, and can happen when a newer message overwrites an older one
the events are appearing out of order in the queue, but I'm not sure what's causing this, I'm pretty sure hub and shuttle emit in the right order, but we'd need data from the messages table to confirm
check for the message timestamp can be a potential workaround while the root cause is determined for 2) above.

we have these records in our messages table
[
  {
    "id": "018f2e40-a33f-8cbc-27ca-81dab82afe3d",
    "created_at": "2024-04-30 09:05:39.124098+00",
    "updated_at": "2024-04-30 09:05:39.124098+00",
    "timestamp": "2024-04-16 23:07:19+00",
    "deleted_at": "2024-05-27 07:30:01.195+00",
    "pruned_at": null,
    "revoked_at": null,
    "fid": 284564,
    "type": 7,
    "hash_scheme": 1,
    "signature_scheme": 1,
    "hash": "20 bytes",
    "signer": "32 bytes",
    "body": "{\"address\":\"0x2a39fd94e139952c3958e69172ba918d8f533571\",\"claimSignature\":\"0x6d3460cbbc770e42913d6ae8942f47d96c9ee57e79ba62c9aaf6061c522329f906f37b1f6449730bca0567b90987d865efbc21423202304063fe3c77597248141c\",\"blockHash\":\"0x97c6ab49b01c3af5347959ae5456d8d95e7731a4f4330800b11dcd380cae1502\",\"protocol\":\"ethereum\"}",
    "raw": "267 bytes"
  },
  {
    "id": "018fb8f4-c995-709e-8540-0be48b4d8ca3",
    "created_at": "2024-05-27 07:30:01.178892+00",
    "updated_at": "2024-05-27 07:30:01.178892+00",
    "timestamp": "2024-05-27 07:29:58+00",
    "deleted_at": "2024-06-02 14:25:06.781+00",
    "pruned_at": null,
    "revoked_at": null,
    "fid": 284564,
    "type": 7,
    "hash_scheme": 1,
    "signature_scheme": 1,
    "hash": "20 bytes",
    "signer": "32 bytes",
    "body": "{\"address\":\"0x2a39fd94e139952c3958e69172ba918d8f533571\",\"claimSignature\":\"0xb499c194749ea31631b04dbd8f83ff1af59742934e4e507931b19d4e9887a84937a5f7a4ef5a088e1beb35ad169c0376051713599472aefaf3c04e612d8d7d561c\",\"blockHash\":\"0x9a87ed5ea1c6acf3f2196981bf9db78bb100e0c13bbaba4b66acaf259f483445\",\"protocol\":\"ethereum\"}",
    "raw": "267 bytes"
  },
  {
    "id": "018fd956-f8f5-4435-0ccc-53b232703349",
    "created_at": "2024-06-02 14:25:06.780531+00",
    "updated_at": "2024-06-02 14:25:06.780531+00",
    "timestamp": "2024-06-02 14:25:02+00",
    "deleted_at": null,
    "pruned_at": null,
    "revoked_at": null,
    "fid": 284564,
    "type": 7,
    "hash_scheme": 1,
    "signature_scheme": 1,
    "hash": "20 bytes",
    "signer": "32 bytes",
    "body": "{\"address\":\"0x2a39fd94e139952c3958e69172ba918d8f533571\",\"claimSignature\":\"0x8d3656984f0c26538d441cc8b48aa8d9a8441ee71286d0e949510a28c13efb2e716fe6b6e810c57e17489e5acf052af1aa37afc25737cc5b9bedba56134d3ecf1c\",\"blockHash\":\"0x94696a9eb590bf2acb4bacb81148637771b0a2a4cf6d119816b396b9cd44dce8\",\"protocol\":\"ethereum\"}",
    "raw": "267 bytes"
  }
]

Having only delete message is totally fine. but there shouldn't multiple delete/merges?
It's valid at the protocol level to keep sending messages. Newer messages will always overwrite older messages. You can keep changing your UserData.DISPLAY_NAME 50 times, and it will generate 50 merges and 49 deletes. Only the message with the latest timestamp is valid, and in the messages table there will be only one that's not marked deleted.

The above example seems to be for a different FID, but we can walk through that
018f2e40-a33f-8cbc-27ca-81dab82afe3d was created on April 30
018fb8f4-c995-709e-8540-0be48b4d8ca3 created on May 27, came in and deleted the message above (above messages was deleted_at a few ms after this one was created_at)
018fd956-f8f5-4435-0ccc-53b232703349 Came in on June 2 and in turn deleted the above message (in this case, it looks like the delete event is .5ms after the create event)
I'm not sure why the delete event is after (even if it's by less than 1 ms), I checked the code and the callbacks should be called in order (delete first, then merge):
https://github.com/farcasterxyz/hub-monorepo/blob/main/packages/shuttle/src/shuttle/hubEventProcessor.ts#L52-L62
One potential reason is the they are not part of the same event, that's unlikely, but I'll double check to make sure.
You can add some logs in your MessageHandler to confirm, but the delete callback should be happening before the merge.

Note, these are all different message hashes. The user is submitting multiple verification messages for the same eth address. The hub is doing the right thing by deleting the old message and merging the latest message and generating a event for each.
The events are for unique messages, they are not duplicates

3. checking for the message timestamp can be a potential workaround while the root cause is determined for 2) above.

The problem with relying on the timestamp is, merge and delete events of same message hash will have the same timestamp. This can cause problem in our system because if they arrive in different order we won't know which is latest.
so I think we can't rely on the timestamp. for different message hash of same address will have diff timestamp which i agree.
In the above example as you rightly said,
For the FID: 284564
first message: 018f2e40-a33f-8cbc-27ca-81dab82afe3d
created_at -> 2024-04-30 09:05:39.124098+00
deleted_at -> 2024-05-27 07:30:01.195+00
second message: 018fb8f4-c995-709e-8540-0be48b4d8ca3
created_at -> 2024-05-27 07:29:58+00
deleted_at -> 2024-06-02 14:25:06.781+00
third message -> 018fd956-f8f5-4435-0ccc-53b232703349
created_at -> 2024-06-02 14:25:02+00
we are not modifying anything in our handler, it processes in the order we receive. As you pointed out delete_at of 018fb8f4-c995-709e-8540-0be48b4d8ca3 has higher timestamp than creation of the message 018fd956-f8f5-4435-0ccc-53b232703349.
So the order events streamed from hub seems to be wrong. if we consume in this order we delete in our system instead of add. that is resulting in inconsistent data.

Hey @chetankashetti, sorry for the delay. A quick comment:

This can cause problem in our system because if they arrive in different order we won't know which is latest.
so I think we can't rely on the timestamp

In the event an add and remove message with the same timestamp arrive, the remove message trumps the add message if it was a cast, otherwise the message with the lower timestamp is discarded. This is documented in the Farcaster protocol specification. Let's take casts as an example—see the section below highlighted in bold.

3.1.3 Cast CRDT
The Cast CRDT validates and accepts CastAdd and CastRemove messages. The CRDT also ensures that the message m passes these validations:

m.signer must be a valid Signer in the add-set of the Signer CRDT for message.fid

A conflict occurs if there exists a CastAdd Message and a CastRemove message whose m.hash and m.data.body.target_hash are identical, or if there are two CastRemove messages whose m.data.body.target_hash are identical. Conflicts are resolved with the following rules:

If m.data.type is distinct, discard the CastAdd message.
If m.data.type is identical and m.data.timestamp values are distinct, discard the message with the lower timestamp.
If m.data.timestamp and m.data.type values are identical, discard the message with the lower lexicographical order.

The logic is different for reactions, links, verifications, and username proofs, in that the message with the lower timestamp is discarded (regardless of add or remove). For example, see the Reactions CRDT docs:

If m.data.type is distinct, discard the CastAdd message.

With this in mind, it should be possible to inspect the current data in your DB and make the correct decision about whether a message is a "winning" message that should be processed, or a losing message that should be ignored due to the CRDT rules.
Hope that clarifies.

One other point worth mentioning: if you look at how Shuttle is implemented (make sure you're running the latest version, as some code may have changed), you can see that Shuttle always processes the deleted messages before the merged message:
https://github.com/farcasterxyz/hub-monorepo/blob/fcb76cd9adcc2d9eff9f1a1832fc89c8bacff117/packages/shuttle/src/shuttle/hubEventProcessor.ts#L52-L63
Given this, you can safely always treat handleMessageMerge with a state parameter of deleted as safe to delete, and one with state of created as safe to create. This is much simpler than what I was previously discussing above, so I've updated my prior comment to avoid further confusion.
Just know that if you create your own handlers and you don't use the state / operation parameters, you will have to reinvent the CRDT logic locally.


docs: corrects an external link

Description
This PR corrects only an external link in security guideline.
Type of change

[X] Documentation Fix

How Has This Been Tested?
Docs site was built locally.
Checklist:

[x] My code follows the style guidelines of this project
[X] I have commented my code, particularly in hard-to-understand areas
[X] I have made corresponding changes to the documentation
[X] I have tested my code extensively
[X] I have selected the develop branch as the target branch

Thx!


Problems compiling

I am currently trying to get semantic-rs to compile. I've read that the OpenSSL versions conflict so I installed the older package openssl-1.0 to make sure to compile against the right version of SSL.
Running the following command still fails though:
OPENSSL_LIB_DIR=/usr/lib/openssl-1.0 OPENSSL_INCLUDE_DIR=/usr/include/openssl-1.0 cargo install semantic-rs

At the very end I get this error message displayed. Any idea how to make semantic-rs compile?
System information
OS: Arch Linux
OpenSSL version: OpenSSL 1.0.2q  20 Nov 2018
Rust version: rustc 1.33.0-nightly (14b96659e 2018-12-25)
Cargo version: cargo 1.33.0-nightly (2cf1f5dda 2018-12-11)

Probably just needs an update of our openssl dependency.
I won't have access to my Arch Linux machine for another week, so can't really fix & test before that.

Do you guys have tests that cover some use-cases? I can fork and update the dependency but if there are no automated tests then I'm not really sure what to look for.

Sure, we even have CI running on every push and PR: https://github.com/semantic-rs/semantic-rs/blob/master/.travis.yml
You can run tests locally: https://github.com/semantic-rs/semantic-rs#running-the-tests
(or just run make test)

For the OpenSSL thing it should mostly be a "does it compile?" test ;)


Add basic access authentication

Hi,
I use a backend based on Express.js and Passport.js for authentication.
So I found useful to add basic authentication support to fit how Passport implements that process.
It's basically based on a config var called basicAuthentication sets to false by default (in this case it uses your previous code).
If basicAuthentication is set to true, then the username and password are combined and encoded in the Authorization header using a GET request and default contentType.
Also some tests added.
Hope it fits with your requirements and thanks for your great addon.

This is great, but I'm a bit hesitant to include in this package. Seems a bit too specific and you can always just subclass authenticators, change makeRequest, and use that in your app. Let's leave this around for a little bit to see if anyone else jumps in.


Required changes based on the new dotnet-docker repo

Since with the release of dotnet 2.1, Microsoft decided to move it's docker samples to a new repo, some changes are required to reflect this new decision.

links
versions (dotnet, docker, docker images)
instructions

Summary
Describe your changes here.
Fixes #Issue_Number (if available)

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: mirsaeedi sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

Thanks, @mirsaeedi! I just approved this and will merge straightaway. You should see it live in the next few days.


add test to image uploads to use different disk allocation policy (thin or preallocated)

we need to add a test to upload_image to use different preallocation policies than the original image has :
https://github.com/oVirt/go-ovirt-client/blob/main/client_disk_uploadimage_test.go

we can see example  dealing with various image types uploads here https://github.com/oVirt/ovirt-imageio/blob/master/test/client_test.py

@eslutsky we support the sparse flag already. What's missing here?

I believe this is fixed in #131


Fix parentheses on website download page

This change fixes a problem with the markdown link on the download page.

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

R: @chamikaramj

LGTM

Thanks for the quick review!


Update Interface

This PR does several things.

Allows obj.set({ foo: 'bar' }) to work for objects
Converts meta to being a Minim object
Converts attributes to being a Minim object
Adds new getValue method which is short for obj.get('foo').toValue(). Now you can do obj.getValue('foo').

@danielgtaylor This will definitely need a look from you, as these are some big changes.

@fosrias I've addressed all comments.

@smizell Fine by me. Leave to @danielgtaylor to clarify anything.

@fosrias @smizell

Leave to danielgtaylor to clarify anything.

Just a heads up: As far as I know Daniel is for two weeks off so you may want to merge it without him (but I leave that up to you)


Collect Image Source (NGF)

As a maintainer of NGF
I want to collect the image source from all NGF/NIC installations
So that I can understand the most popular methods for installing NGF/NIC
Acceptance

The repository location of the image is NOT recorded.
A method internal to the image is used to identify if the image is custom or pulled directly.
Data is collected every 24 hours, starting 24 hours after initial deployment.
When the data is collected, the value is output in NGF logs

When we build, we may be able to inject a tag in our build pipeline via build arguments so that pre-built images are marked.


Public page for non authentificated user

It would be great to have some kind of public photos, that are accessible without user authentication. Just on the main page. And admin user should be able to "star" photos for public access.

Hi,
I find this feature would be well suited to an addon. PiGallery does not support addon yet, there are plans to implemet it.

@bpatrik Are addons available yet, or is there room to approach the implementation of this issue from a different perspective than addons? I haven't been able to find references in the code for them. I'm interested in helping develop this feature as it's one of two critical features I would need to be able to use PiGallery2 for my personal image gallery (I want to be able to easily share photos publicly).


Internal Error: @calcom/mock-payment-app: This package doesn't seem to be present in your lockfile

Looks similar to #234, but happens during docker run and not the build step.
docker build based off tag v3.7.0 with the calcom submodule up-to-date works great.
However, the docker run fails:
docker run --rm -e DATABASE_URL=postgres://postgres:password@host.docker.internal:5432/calcom -e NEXTAUTH_SECRET=secret -e CALENDSO_ENCRYPTION_KEY=[REDACTED] -e NEXT_PUBLIC_WEBAPP_URL=http://localhost:3000 -p 3000:3000 -it my/calcom:v3.7.0
+ scripts/replace-placeholder.sh https://my.calcom http://localhost:3000
Replacing all statically built instances of https://my.calcom with http://localhost:3000.
+ scripts/wait-for-it.sh -- echo database is up
Error: you need to provide a host and port to test.
Usage:
  scripts/wait-for-it.sh host:port|url [-t timeout] [-- command args]
  -q | --quiet                        Do not output any status messages
  -t TIMEOUT | --timeout=timeout      Timeout in seconds, zero for no timeout
  -- COMMAND ARGS                     Execute command with args after the test finishes
+ npx prisma migrate deploy --schema /calcom/packages/prisma/schema.prisma
Prisma schema loaded from packages/prisma/schema.prisma
Datasource "db": PostgreSQL database "calcom4", schema "public" at "host.docker.internal:5432"

234 migrations found in prisma/migrations

No pending migrations to apply.
npm notice
npm notice New minor version of npm available! 10.2.3 -> 10.3.0
npm notice Changelog: https://github.com/npm/cli/releases/tag/v10.3.0
npm notice Run npm install -g<EMAIL_ADDRESS>to update!
npm notice
+ npx ts-node --transpile-only /calcom/packages/prisma/seed-app-store.ts
📲 Updated app: 'apple-calendar'
📲 Updated app: 'caldav-calendar'
Error adding google credentials to DB: Unexpected end of JSON input
📲 Updated app: 'jitsi'
📲 Updated app: [LOTS OF DIFFERENT APPS]
📲 Updated app: 'zoom'
Skipping Routing Form - Seeding - Pro User not found
+ yarn start
Internal Error: @calcom/mock-payment-app@workspace:packages/app-store/mock-payment-app: This package doesn't seem to be present in your lockfile; run "yarn install" to update the lockfile
    at P0.getCandidates (/calcom/.yarn/releases/yarn-3.4.1.cjs:435:5145)
    at yf.getCandidates (/calcom/.yarn/releases/yarn-3.4.1.cjs:391:1264)
    at /calcom/.yarn/releases/yarn-3.4.1.cjs:439:7693
    at of (/calcom/.yarn/releases/yarn-3.4.1.cjs:390:11070)
    at ge (/calcom/.yarn/releases/yarn-3.4.1.cjs:439:7673)
    at async Promise.allSettled (index 38)
    at async $s (/calcom/.yarn/releases/yarn-3.4.1.cjs:390:10398)
    at async /calcom/.yarn/releases/yarn-3.4.1.cjs:439:8205
    at async ti.startProgressPromise (/calcom/.yarn/releases/yarn-3.4.1.cjs:390:47579)
    at async je.resolveEverything (/calcom/.yarn/releases/yarn-3.4.1.cjs:439:6233)

Thank you @zomars will try again with the updated calcom submodule: https://github.com/calcom/cal.com/commit/7b8902a769f9adb93a5909d3015730268d400ca8

It's resolved ✨


Automated dependency updates | Mon, 24 Jun 2019 12:02:06 +0000

Description
Updates js dependencies

Closing since issue will be resolved in https://github.com/transcom/mymove/pull/2337


Perhaps the static location hook should support base path by default

That could actually be a good idea. This sounds like a nice first issue for someone who would like to contribute.
To clarify this is how it is supposed to work:
<Router hook={staticLocation("/page-to-render")} base="/app">
   <Route path="/page-to-render"> {/* this will be rendered, matches static location */}
     <Link to="/another-page" />{/* this rendered link will have href="/app/another-page" *}
  </Route>
</Router>

I wish you had Deno-first code in the root so we could directly use it, even before releases :)

@molefrog I don't think there's anything wrong other than the advantages if you make a release now.

@molefrog I don't think there's anything wrong other than the advantages if you make a release now.

Fair enough. Released in v2.9.0

Thanks!


FGW get linear and quadratic term

Hello,
Given a GWOutput, am I right that out.reg_gw_cost can be split as linear + fused_penalty*quad? Is there a good way to get these linear and quadratic contributions?
I tried something but it involves instantiating out.matrix which is probably not very efficient...
Thanks a lot!
Best
GJ

yes, at this point, those are no tracked.
The small difficulty is that out.reg_gw_cost also includes entropic regularization.
I am afraid the easiest approach could be to use the same pattern as https://github.com/ott-jax/ott/blob/75cdd117871110017e50b0240f886700477e4674/src/ott/solvers/quadratic/gromov_wasserstein.py#L98 but feed the geom_xy to transport_at_geom. I would hope that if your geom_xy is a LRC (low rank cost) all show go smoothly.

Thanks a lot, I'll try that!
By the way, is out.reg_gw_cost equal to

quadratic(pi^\star) + fused_penalty*linear(pi^\star) - eps*entropy(pi^\star)
or to quadratic(pi^\star) + fused_penalty*linear(pi^\star) - eps*(1 + fused_penalty)*entropy(pi^\star) ?

Okay, that makes sense, thank you :)

Hi, me again! This made me realize that for the GW solver, epsilon is not defined in the geometries (as in the Sinkhorn solver) but in the args of WassersteinSolver, right?
IMHO this is not super clear from the docs and it could be worth setting epsilon=1.0 directly in the args of GW (as done in the LRGW solver actually)

This question has been around for a while, should epsilon be a part of Geometry or part of solver.
My thinking is that, for a linear solver, the scale of epsilon does not make sense "on its own", a solver cannot really know how to set epsilon, it should be relative to the geometry's points, and cost-function. This is why querying it from Geometry would make more sense.
On the other hand, for a quadratic solver, there is no such a thing as a "stable" Geometry: the linearization means that the cost matrix is refreshed (and quite unstable) at every iteration. Therefore, this is why one needs to move it up, at the level of the solver.
As for the discrepancy between LR and GWLR API w.r.t. epsilon, I think this is a minor bug. Both inherit from WassersteinSolver, and epsilon is there. Therefore, we should either expose it in both (in list of arguments) or hide it from both. Maybe exposing it in both would be preferable. @michalk8 do you have an opinion?


Auto corrected by following Format Ruby Code

Auto corrected by following Format Ruby Code
Click here to configure it on awesomecode.io

Coverage decreased (-0.04%) to 76.897% when pulling c44306bf5e621ed216699566eaa36808f2cc15af on awesomecode-format-ruby-code-48767 into b0e82af19faae177ac28fa721fe121ee6eb33adc on master.


Safer default Mark Deleted Tables

currently if we enable that with filtered stuff, we mark everything filtered as deleted by default. The current behavior forces us to also toggle the Mark Deleted Tables from Filter Only to keep the metadata safe. We should change this behavior around. The default marking deleted behavior should be the "safe" one, only considering filter entities, and then force users to toggle a "Mark All Deleted Tables" or something if they really want to remove all the filtered data

@pmbrull who is this going to be assigned?


Array replacement "smart heuristics" not working when injecting wrapped array into component

Version
2.3.4
Reproduction link
https://jsfiddle.net/vabrnmzg/103/
Steps to reproduce

I have an object that includes an array and some convenience methods. In this case it's used to store validation errors. (I originally encountered this issue using VeeValidate, but my reproduction is only Vue code)
This error bag object is injected by a parent into a child component that may put errors into it, or remove errors.
The convenience method "remove" of the error bag uses array.prototype.filter() to create a new array that does not include the values that need to be removed. Then it assigns this array back to this.errors inside the error bag.
The validation logic may cause "remove" to be called each time, which means that the this.errors array inside the error bag is replaced after each keystroke, when the validation runs.
Vue has "smart heuristics" that make sure the entire DOM isn't re-rendered after array replacement, but they seem to fail here. The entire DOM is re-rendered after each key stroke!
Normally you might not notice this, but the parent is also using v-model.trim on the child component. So when the child is re-rendered, it of course uses the newly trimmed value. This prevents you from typing any spaces while using v-model.trim, if the child component has a text input or textarea.
Important: this issue does not occur when injection is not used, or when the "remove" method uses array.prototype.splice() instead in order to modify the errors array in-place.

What is expected?
When injecting a (object wrapped) array, the array replacement heuristics that prevent the entire DOM from being re-rendered, should work the same as when injection is not used. Or at least any limitation should be made really clear. The in-place modification methods are working as usual, so the expectation is that array replacement works too as usual.
What is actually happening?

When injecting a (object wrapped) array, the array replacement heuristics should work the same as when injection is not used.
The code that watches the in-place modification methods such as push, splice etc seems to work fine, so a workaround is to not replace the array but use the in-place modification methods instead.
Currently, the child component re-renders its DOM when the errors array is replaced, which can cause issues when a parent component is using v-model.trim.
When using in-place modification, Vue realises that a DOM re-render is not necessary when you, for example, cleared an already-empty errors array, and it will not re-render the DOM and then everything works fine.

I discovered that injection may not be related, temporarily closing this while I investigate more.


Found CVEs in your dependencies

Your project is at risk due to the use of vulnerable dependencies. In order to avoid threats, I recommend updating to a safe version. Please note the following details.

Vulnerable Library Version: org.springframework.security.oauth : spring-security-oauth2 : 2.2.0.RELEASE
CVE ID: CVE-2019-3778, CVE-2018-1260, CVE-2019-11269, CVE-2018-15758
Import Path: spring-cloud-sleuth-core/pom.xml
Suggested Safe Versions: 2.2.5.RELEASE, 2.2.6.RELEASE, 2.3.6.RELEASE, 2.3.7.RELEASE, 2.3.8.RELEASE, 2.4.0.RELEASE
Vulnerable Library Version: com.google.guava : guava : 20.0
CVE ID: CVE-2018-10237
Import Path: tests/spring-cloud-sleuth-instrumentation-grpc-tests/pom.xml
Suggested Safe Versions: 24.1.1-android, 24.1.1-jre, 25.0-android, 25.0-jre, 25.1-android, 25.1-jre, 26.0-android, 26.0-jre, 27.0-android, 27.0-jre, 27.0.1-android, 27.0.1-jre, 27.1-android, 27.1-jre, 28.0-android, 28.0-jre, 28.1-android, 28.1-jre, 28.2-android, 28.2-jre

Thanks for trying to help. In the future, if you believe that you have found a security vulnerability and have identified a way in which it can be exploited, please report it to the Pivotal Security team.


Add Dollar sign to be replaced when creating a new URI

Is it possible to add the "$"  to the replaceAll when creating a new URI in the code below?
public URI uri() {
        String uri = newBuilder().reencodeForUri().toString();
        try {
            return new URI(uri);
        } catch (URISyntaxException e) {
            try {
                return URI.create(uri.replaceAll("[\\u0000-\\u001F\\u007F-\\u009F\\p{javaWhitespace}]", FRAGMENT_ENCODE_SET));
            } catch (Exception e2) {
                throw new RuntimeException(e);
            }
        }
    }

We have a library, that uses OkHttp crashing our android app because one of their customers is providing the following buggy URL: " https://${imp_tracker_1}/".
We have debugged the code and it's throwing an exception at the second Catch of the function above. Since $ is a reserved character for URI and does nothing I think it would be a good idea to have it treated when trying to create a new URI.

Yeah, I think we can handle this. Or you can call newBuilder() to replace it yourself.

Hey Jesse, thanks for the quick response!
Actually, I can't make any change to fix that issue on our app because is a tracking call that the library we consume does internally and is throwing the crash back at us.
We don't have any interaction with the tracking call nor with the URL that is being called. We notified them and had it disabled on our app for now but I think it may be something that could be handled by OkHTTP as well since the crash is actually happening at the code I shared above and "$" is a URL reserved sign.
Thanks!

@swankjesse Do we want to make HttpUrl stricter for the host?  Or just send through an incomplete but passing host?  What about http://$/?
  @Test public void hostContainsIllegalCharacter() throws Exception {
    assertInvalid("http://\n/", "Invalid URL host: \"\n\"");
    assertInvalid("http:// /", "Invalid URL host: \" \"");
    assertInvalid("http://%20/", "Invalid URL host: \"%20\"");
    // added here
    assertInvalid("http://$/", "Invalid URL host: \"%20\"");
  }

Or is there some sensible encoding to pass to URI constructor?
Specifically that catch block is documented as handling the fragment, which I accept for stripping.  But this case involves a $ in the host.

I don't think we'll be able to resolve a host with a $ character in it.
What do the specs say?

https://tools.ietf.org/html/rfc3986#section-3.2.2
https://tools.ietf.org/html/rfc1034#section-3.5
Pretty sure it's invalid so we should arguably fail parsing before allowing an invalid HttpUrl.

Our validation of hostnames is pretty limited. We lean on Java’s IDN class for most of it.


As a User I want to be able to find names from command line using files or pipes

Issue moved to gnames/gnfinder #90 via ZenHub


[riscv.mk] remove -g and -ffp-contract=off

In this PR:

fix the case where parameterized kernels do not use fmadd

Should this be a PR to the main repo, too?


CB-11910 Remove subnet validation logic for endpoint gateway

Puts the logic that validations endpoint gateway subnets are public behind the CDP_ENDPOINT_GATEWAY_VALIDATION
entitlement. If the entitlement is disabled, all subnets provided for the endpoint gateway will be accepted
regardless of their type. This change also removes the last of the logic first introduced in CB-11070, and
some related validation logic in AwsLoadBalancerLaunchService.
Tested with unit tests, and by providing private subnets for the endpoint gateway and verifying the environment
created successfully, and the load balancers were created with the expected subnets.

I checked the changes and looks okay, except the behaviour change of validation. That's the only open question.

@doktoric @biharitomi I've switched the logic so that it still uses the validation by default, but it can be disabled by granting the CDP_ENDPOINT_GATEWAY_SKIP_VALIDATION entitlement.


Refactor handleNativeResponse for better code reusability

This PR:

Breaks down handleNativeResponse into various blocks
Cache logic separated

Will be followed up with silent cache lookup updates in the new helper functions

Codecov Report

Merging #5757 (39210a6) into dev (81d34b4) will increase coverage by 10.55%.
The diff coverage is 100.00%.

Flag
Coverage Δ

msal-angular
96.63% <100.00%> (+0.13%)
:arrow_up:

msal-browser
?

msal-common
?

msal-core
?

msal-node
?

msal-node-extensions
?

msal-react
94.68% <ø> (ø)

node-token-validation
?

Impacted Files
Coverage Δ

lib/msal-angular/src/msal.interceptor.ts
100.00% <100.00%> (ø)

extensions/msal-node-extensions/src/index.ts

...nsions/msal-node-extensions/src/utils/Constants.ts

lib/msal-browser/src/app/ClientApplication.ts

...ib/msal-browser/src/app/PublicClientApplication.ts

...er/src/broker/nativeBroker/NativeMessageHandler.ts

.../src/interaction_client/NativeInteractionClient.ts

lib/msal-browser/src/internals.ts

...-browser/src/telemetry/BrowserPerformanceClient.ts

...ser/src/telemetry/BrowserPerformanceMeasurement.ts

... and 203 more


Auto refresh page when redline is created

As an IAO user
I want the page to automatically refresh when I create the redline
so that I can edit the redline if I need too

Assumptions & Scope
What are the assumptions for this story?
This story resolves the bug #4422 by automatically refreshing the page when a redline is created.
Unlikely that the analysts will need to edit the redline right after they create the redline as most edits will happen after the ministry comes back with requested changes.
What is IN scope?
Auto refreshing the redline layer in the redaction app when the redline is created.
What is NOT in scope?
Most of the create redline functionality is already built as part of #3437
Acceptance Criteria
Scenario 1: Click active "Redline for Sign Off"

GIVEN the "Create Records Package" dropdown is open
AND the request is in Records Review, Ministry Sign Off or Peer Review
https://jacklyn808742.invisionapp.com/console/share/8QVPHCNXSGA/981498459
WHEN the IAO analyst clicks the active "Redline for Sign Off"
THEN a modal will open with the message, "Are you sure want to create the redline PDF for ministry sign off?".
AND the modal will have the message, "When you create the redline PDF, your web browser page will automatically refresh."
AND a "Create Redline PDF" and "Cancel" button
https://jacklyn808742.invisionapp.com/console/share/8QVPHCNXSGA/981498459

Scenario 2: Click "Create Redline PDF" button

GIVEN an analyst is on the "Redline for Sign Off" modal
WHEN the IAO analyst clicks the "Create Redline PDF" button
THEN a modal will close
AND the web browser page will automatically refresh

Validation Rules? (If yes, list here)
Design
@xxx - please link the Design here
Definition of Ready

[ ] Is there a well articulated User Story?
[ ] Is there Acceptance Criteria that covers all scenarios (happy/sad paths)?
[ ] If there is a user interface, is there a design?
[ ] Does the user story need user research/validation?
[ ] Does this User Story needs stakeholder approval?
[ ] Design / Solution accepted by Product Owner
[ ] Is this user story small enough to be completed in a Sprint? Should it be split?
[ ] Are the dependencies known/ understood? (technical, business, regulatory/policy)
[ ] Has the story been estimated?

Definition of Done

[ ] Passes developer unit tests
[ ] Passes peer code review
[ ] If there's a user interface, passes UX assurance
[ ] Passes QA of Acceptance Criteria with verification in Dev and Test
[ ] Confirm Test cases built and succeeding
[ ] No regression test failures
[ ] Test coverage acceptable by Product Owner
[ ] Ticket ready to be merged to master or story branch
[ ] Developer to list Config changes/ Update documents and designs
[ ] Can be demoed in Sprint Review
[ ] Tagged as part of a Release
[ ] Feature flagged if required
[ ] Change Management activities done?

It is working as expected, so closing it


DS-3800

The booleans assigned to the metadata.hide.SCHEMA.ELEMENT.QUALIFIER configuration properties in dspace.cfg were not being used. Updated the MetadataExposureServiceImpl java class to take these booleans into account when checking if a field should be hidden.
jira ticket:
https://jira.duraspace.org/browse/DS-3800

I have updated this pull request based on the new Code Style.


mne.preprocessing.ICA - components parameters

I have a question concerning the "component parameters" in ICA, which are n_components, max_pca_components, and n_pca_components. What exactly do they do?
I understand that an ICA of say 64-channel EEG data will usually yield a W of size 64. From my current knowledge the ways how to end up with W of a lower size would be:

to perform a dimension reduction with PCA prior to ICA application
to "subsample" the sensor space, that is, drop some channels and apply only this subset to ICA

Now the three mentioned parameters to the ICA object are confusing me a bit.

max_pca_components --> this is probably corresponding to my point 1), a dimension reduction and it will select the n components of the PCA that explain the most of the variance (or all components).

If that is true, what is n_components doing?
Lastly, what is n_pca_components doing? It says that it is:

The number of PCA components used after ICA recomposition.

... but why would any PCA components be of relevance after I already back-projected my ICA (I assume that is what is meant by recomposition)? Does this parameter perhaps have something to do with the problem that is mentioned in the recent paper that warns about performing dimension reduction prior to ICA?

for @cbrnr :)

@sappelhoff I completely agree that these arguments are confusing - for an ongoing discussion please see #4856. Here's the tl;dr:

n_components and max_pca_components both perform dimensionality reduction via PCA prior to ICA. The only difference between these two arguments is the shape of the returned PCA components matrix (with n_components you get all components, whereas with max_pca_components you get only the selected components, i.e. rows, in this matrix). For the user, I'd recommend to use n_components.
You are right that n_pca_components is used in back-projecting. I still haven't figured out why exactly this argument is necessary.

We haven't finished our discussion in #4856 (feel free to participate), but at the very least, we will clarify these arguments in the docs and contrast them to equivalent EEGLAB arguments.

thanks for the reference, @cbrnr!
Now my current understanding is:

calculate PCA on data X
retain max_pca_components and project X to X_reduced
at this point, probably n_components is doing something ... but what?
Using X_reduced, we compute an ICA and have X_reduced_ica
From X_reduced_ica, we retain/reject components
We back-project X_reduced_ica using the ICA weights and have X_reduced'

( the ' indicating that some components were rejected in the ICA ... but the dimensions are the same)

X_reduced' can now be back-projected to X' using the PCA weights of dimension max_pca_components

(It would be X' instead of the original X because of two reasons: (a) some ICA components were missing (rejected) to get the same X_reduced and (b) X_reduced does not contain all pca components so we cannot recover all the data)

Here, at step 7., n_pca_components comes into play by using more than just max_pca_components to construct X' from X_reduced'

Is my intuition about the procedure correct? Given that yes, ... are there any references that suggest that step 8. (concerning max_pca_components) is sensible? And I still don't know what n_components is doing ... but for now, I will not do any dimension reduction anyway and leave the arguments at None.
However, given the discussion in #4856 and this paper that I also linked above, maybe we should include a warning that performing dimensionality reduction (even only by 1% variance) before ICA can lead to unreliable results? That information might fit well next to the warning note about filters ... ?


requestAnimationFrame not polyfilling in Android stock browser

I've been getting inconsistent results with requestAnimationFrame being detected and loading in certain versions of the Android stock browser as part of the default set.  Specifying the property with |always fixes the issue, but otherwise I get a Uncaught ReferenceError: requestAnimationFrame is not defined.
A client reported JS not working properly on two Galaxy S5 devices running versions of Android 5 and 6, but the only place I've had a chance to test the problem/fix is the Galaxy S3 Android 4.1 emulator on Browserstack.  Not sure how useful that information is, but it's all I have to go on at the moment.  I'll post an update if I get any more information - figured it was worth passing my findings along.

Thanks for reporting.  It sounds like an issue with some potential niche UA strings not being correctly identified as android.  To investigate we really need to get the raw UA strings that are being sent by the devices you believe are being wrongly classified.
And yes of course you can drop in the always flag as a workaround.

This is the UA string I'm getting from the S3 emulator on Browserstack:
Mozilla/5.0 (Linux; U; Android 4.1.2; en-us; sdk Build/MASTER) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30
Not sure how useful that is since it's an emulator, but I'm working on getting the UA strings and errors from the actual devices as well.  Thanks for looking into this, and let me know if there's any other information that would help.

That UA is detected as android/4.1.0.  It actually looks like the main problem is that we are not targeting Android:
https://github.com/Financial-Times/polyfill-service/blob/master/polyfills/requestAnimationFrame/config.json
To fix:
https://cdn.polyfill.io/v2/docs/contributing/common-scenarios#adding-browsers

It looks like that might solve the problem with the S3 emulator, but after getting some more data from the client it appears their issue was a separate one with Object.assign not being polyfilled in the S5/6.
The UA strings just about match those referenced in #509, and it sounds like #671 won't be merged in.  Happy to close this if that's the decision for now.  Appreciate all of the work on the polyfill - it's really come in handy recently!

I just updated #509 with a suggestion for a maintainable solution, so if you can contribute there you might get a fix.  Also consider adding target config for raf on Android stock browser!


[Feature Request] A big vector to support name service

In the previous discussion about large vector support, @sblackshear gave some design patterns suitable for Move programming language. I implemented DNS and another NFT example based on the sorted linked list [1]. My sample code is pushed here. This design pattern is really interesting, but I still encounter two problems, and did not find a good solution.

Each account address can only contain at most one list node. For DNS, this will result in at most one domain name for an account.
In order to reduce the overhead of adding nodes, users are required to find the "previous address" and pass it as a parameter to the add_node() function. Finding the "previous address" can be done off-chain, but it requires the VM to provide an additional interface to search the global state.

For these two problems, I still have not found a good solution, but I still believe that this is the right direction. Do you have any suggestion?
Motivation
Is your feature request related to a problem? Please describe.

Pitch
Describe the solution you'd like

Describe alternatives you've considered

Are you willing to open a pull request? (See CONTRIBUTING)
Additional context

Hi Guangyu,
Thanks for the post!

Each account address can only contain at most one list node. For DNS, this will result in at most one domain name for an account.

Good point; single name/account seems like an unfortunate limitation. One solution that came to mind is something like:
/// an identifier for a Node<T>
resource struct NodeHandle {
  /// address where the Node is stored
  addr: address,
  /// An index into the NodeVector stored under addr
  index: u64
}

resource struct Node<T> {
  /// pointer to previous and next Node's in the sorted linked list
  prev: EntryHandle, next: EntryHandle,
  data: T,
}

/// A vector of Node's stored under a single account
resource struct NodeVector<T> {
  nodes: vector<Node<T>>
}

If each account stores a vector of NodeVector instead of a single node, it can (e.g.) register as many domains as it wishes.

In order to reduce the overhead of adding nodes, users are required to find the "previous address" and pass it as a parameter to the add_node() function. Finding the "previous address" can be done off-chain, but it requires the VM to provide an additional interface to search the global state.

Ah, I was assuming that for DNS-like use-cases, someone would want to mirror the data offline and build a variety of indexes for it. For example: if tweak the insertion code to emit a InsertedAt { data, prev, next } event (or similar), you can:

Watch for InsertedAt events
Update an offline copy of the data structure each time you see an event. The offline copy of the data structure can be a map, a big vector, a DB table, or anything else that can efficiently provide the "get predecessor address" API

Also, a simpler workaround to get started quickly/test: you can expose both the "insert with previous address" API and the linear "find position and insert API". The linear API won't be realistic to call once the list gets big, but it will let you experiment without needing to implement the offline copy of the data structure.

Hi Sam,
The solution to store NodeVector for each account is pretty good. Nodes are linked with EntryHandle. Users can register as many names as needed. I'll re-write the demo with NodeVector.
Watching the "InsertedAt" event and building an offline map is also a good solution. But in Ethereum, people sometimes use "read-only" calls to view the status. Is it also an alternative solution to "find position" before insert? What do you think?
Thanks!

also noticed that in Ethereum, people sometimes use "read-only" calls to view the status. Is it an alternative solution to "find position" before insertion? What do you think?

Good question. There's no conceptual blocker for implementing an equivalent feature to read-only calls in Libra; it's just not something we've done yet. A tool that

usesstate_synchronizer to grab the latest state
spins up a local version of the VM and exposes an API for executing a script or function

shouldn't be too difficult for someone to implement. I anticipate that we'll have this eventually, but it's not currently on our roadmap.

spins up a local version of the VM and exposes an API for executing a script or function

This feature like my feature request at #4803. Spin up a local version of the VM for executing a script is easy, but MoveVM and  Scripts do not support return and obtain results in rust currently.
MoveVm's execute_script or execute_function:
pub fn execute_script(
        &mut self,
        script: Vec<u8>,
        ty_args: Vec<TypeTag>,
        args: Vec<Value>,
        sender: AccountAddress,
        cost_strategy: &mut CostStrategy,
    ) -> VMResult<()> ;

It should be:
pub fn execute_script<T>(
        &mut self,
        script: Vec<u8>,
        ty_args: Vec<TypeTag>,
        args: Vec<Value>,
        sender: AccountAddress,
        cost_strategy: &mut CostStrategy,
    ) -> VMResult<T> ;

Hello Sam
I rewrote the sorted linked list to store a NodeVector for each account (instead of a single node). Accounts can now register as many names as needed. Currently, the "find_position_and_insert()" API is used as a workaround. I will do more investigations on "read-only calls". The demo code is placed here in case someone refers to it.
Thanks again for your help! If there is no further discussion, I'll close this issue.

spins up a local version of the VM and exposes an API for executing a script or function

This feature like my feature request at #4803. Spin up a local version of the VM for executing a script is easy, but MoveVM and Scripts do not support return and obtain results in rust currently.
MoveVm's execute_script or execute_function:
pub fn execute_script(
        &mut self,
        script: Vec<u8>,
        ty_args: Vec<TypeTag>,
        args: Vec<Value>,
        sender: AccountAddress,
        cost_strategy: &mut CostStrategy,
    ) -> VMResult<()> ;

It should be:
pub fn execute_script<T>(
        &mut self,
        script: Vec<u8>,
        ty_args: Vec<TypeTag>,
        args: Vec<Value>,
        sender: AccountAddress,
        cost_strategy: &mut CostStrategy,
    ) -> VMResult<T> ;

Good point! I don't think we would want execute_script to return a result because scripts cannot have return values,  but exposing execute_function for this purpose makes sense.


[v0.14.x] - Allow an AntiAffinty tolerant to help ensure even spread of DNS pods

Changes

Adds a preferred scheduling anti-affinity to try and target nodes in different AZs if enabled.

Useful in situations where we have autoscaling enabled or are deploying DNS pods to nodes.

Codecov Report

Merging #1673 into v0.14.x will decrease coverage by <.01%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           v0.14.x   #1673      +/-   ##
==========================================
- Coverage    25.41%   25.4%   -0.01%
==========================================
  Files           98      98
  Lines         5064    5065       +1
==========================================
  Hits          1287    1287
- Misses        3634    3635       +1
  Partials       143     143

Impacted Files
Coverage Δ

pkg/api/types.go
0% <ø> (ø)
:arrow_up:

pkg/api/cluster.go
0% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 84b3370...806f4d3. Read the comment docs.


Issue: Using var Instead of let in the Project

Description
In our current project, the use of the var keyword for variable declarations is prevalent. However, var has several characteristics that can lead to bugs and maintenance challenges. It is recommended to transition to using let (or const where appropriate) for variable declarations to improve code quality and maintainability.

gonna be fixed in next commit

fixed


Fixed issue 187 - event callbacks parameters where not correct.

This is a simple workaround to resolve the issue of event handler parameters.

This is a fix for issue #187


feat: handle leading whitespace

closes #85
Adds an off by default configuration value, handle_leading_whitespace. When true, otter will track the leading whitespace of each capture, and when a command is called, properly adjust values where it needs to.
https://github.com/jmbuhr/otter.nvim/assets/56943754/951023af-aca8-4ab0-88d2-8ad8f5e8611b
The formatting bug is present in main, it appears to behave the same way in this branch as it does in main, I opened #87 to document that.

Oh wait, I didn't test rename. One sec

Officially ready for review.

I fixed #87 in https://github.com/jmbuhr/otter.nvim/pull/88. This PR also added more examples based on usecases I am currently aware off, which now also includes a norg example. Can you merge main into this branch and play around with the examples? ./tests/debug-example  <some ft or number> opens the respective example with the otter.dev_setup.
The markdown and quarto examples should just work with your setup. The ts example just needs the typescript treesitter grammar (in case you don't have that). The rust example uses some custom injections file, so don't worry too much about that for now.

This happens when formatting indented blocks. I can fix that tomorrow. Otherwise, this looks good to me.
ahh, I messed up the commit history.

@jmbuhr do you know of another language server that does range formatting? Lua LS seems to do something that doesn't make sense and it results in that last line disappearing.
Would be good to have another LS to test with.
I've tried pyright and rust analyzer.

R's language server does formatting, and it also currently works in e.g. quarto documents. So that's a good reference.

Currently on main, lua only adds one line when formatting the last chunk, so maybe it just wants a blank line after the last piece of code.

oh, wait, there is more off-by-one stuff going on. After my fix for https://github.com/jmbuhr/otter.nvim/issues/87, formatting an R chunk now refuses to format the last line of a chunk. i.e.

stays, but

becomes

Yeah this is very weird. I have something that works for lua ls right now. But I will test with R after class or when I have time.
https://github.com/jmbuhr/otter.nvim/assets/56943754/48a62d4f-205b-4eb1-86de-c2033844cbfb

I think I have an idea of whats happening. I think Lua LS is ignoring the column info and only using the line numbers.
And R LS is using the col numbers correctly.
Lua LS then, when sent 21, 0 | 22, 0 will try to format two lines in the other buffer. One will be print the other a blank line with nothing below it. And it'll respond with "format print and delete the extra blank line below it" which was removing the @ end before.
Subtracting one from line number fixes that for Lua. But now r LS is getting 21, 0 | 21,0 and it's like "you want me to format nothing? Okay here you go"

ah, indeed, I was wondering about this halfway during implementing the column-number fix and then forgot about it when it just worked for the examples I tried. Do you happen to know if there is a shortcut for "to the end of the line" instead of having to add the count? e.g. some nodes report their range formatted as
[<line>:<col> - <col in the same line>] in vim.treesitter.inspect_tree but I have not seen any range that omits the column.

I'm not familiar with a shorthand for it

Small heads up: My attempts at fixing broke too many things, so I reset main and moved those fixes back to their own branch as I should have done earlier (https://github.com/jmbuhr/otter.nvim/pull/89)

@jmbuhr I'm unable to get the R LS to range format. Is there an additional package that you have to install?

Uhhh. why does it refuse to format y = to y <-?
I've checked, the response from the LSP is actually telling us to keep y = it's not a problem with my implementation.

In other news. The different ways that R and Lua LSPs format is a good way to make sure we're doing things correctly. I wish I could spend more time on this to get it working, but it will probably have to wait another day.

@jmbuhr I have an implementation that works for pyright, R's LS, and lua_ls. But the implementation does get a little messy. There are a few different ways for the LSP to respond with a "change this text" request, and we have to account for all of them. There might be more that I'm missing I'm not sure.
There has to be a better way to do this.
One approach that would definitely be neater (although less performant) would be to just recursively search the table for range keys, b/c the spec is at least nice in that all ranges are the same, and they're all called range. But there are a lot of fields that could contain a range, and I'm currently handling all of them. This is kinda a lot (and a lot of potential maintenance overhead), and I'm not even sure that I've covered all of them.
I'm thinking that it would be nice to have access to this position data after neovim's lsp code has processed it for us (hopefully into a more uniform set of change instructions). But I don't know if that's possible, and suspect it isn't.

We currently have two ways of changing what the lsp response does. The first is by using the filter function to modify the response itself. The second is by swapping out the handler  that processes the response (after potentially being filtered). This became necessary when the nvim default handler for hovers stopped showing hovers from other buffers. Maybe the code for the builtin handler offers some clues, or can be taken and partially replaced.

Tough your solution already looks good, given the complexity of the task.

Perhaps my logic should go into a filter function then? Although, these changes apply to every request instead of only certain ones so it might make more sense to leave it where it is.
I have read through the lsp help page, but I couldn't find anything that would help us here.

The handlers have the same information available to them that we have, and no additional information as far as I could tell. So I think swapping them would work but it would look pretty similar to what we're already doing.

@jmbuhr how are you feeling about this PR?

If it is good to go from your side I will test it a bit in my own workflows and then merge it :)

Yeah, I've not found any other problems. I think it's good to go

Apologies for making you merge main so often, I just merged a change to the way we iterate over the treesitter nodes (https://github.com/jmbuhr/otter.nvim/pull/100) so we can make use of a fix in nvim (https://github.com/neovim/neovim/commit/bd5008de07d29a6457ddc7fe13f9f85c9c4619d2), which now handles offsets properly. Might even have fixed other issues along the way.

Ooh this is a nasty merge. I think b/c this branch also fixes some of the offset issues that were a part of #89
Does #100 also fix all the offset issues? b/c then I could probably just accept all the new changes and wipe my stuff out.
@jmbuhr

It doesn't address the issues in https://github.com/jmbuhr/otter.nvim/pull/89/files directly, though the upstream changes in nvim that this pulled in may have fixed some issues as a side effect. The most apparent thing fixed is where an injection would include e.g. wrapping quotes in the otter buffer, before this proper handling of offsets.
The only change in https://github.com/jmbuhr/otter.nvim/pull/100 is actually just replacing 3 occurences of
for id, node, metadata in treesitter_iterator.iter_captures(root, main_nr, query) do

with
  for pattern, match, metadata in query:iter_matches(root, main_nr, 0, -1, { all = true }) do
    for id, nodes in pairs(match) do
      local name = query.captures[id]

      -- TODO: maybe can be removed with nvim v0.10
      if type(nodes) ~= "table" then
        nodes = { nodes }
      end

      for _, node in ipairs(nodes) do

and adding the two more ends after the loop body accordingly.
It just looks more complicated in the diff because github apparently thinks aligning one end is worth shifting everything else for.
So it fixes the issues with the "offset" directives, but probably not the issues we found where we had an offset in line numbers etc.

ah that helps a lot, ty

I think you can take main and add the lines where you use text, leading_offset = trim_leading_witespace(text, main_nr, row1) pretty much in their old positions.

I forgot that I was testing with only python enabled just to make sure that setting a language list worked, and I spent like 10 minutes trying to figure out why it couldn't find the lua code cells in my document :man_facepalming:
I think this is ready now.

In the example norg document my python otter buffer ends up with
print("Hello world!")
   import numpy as np
   x = np.zeros(10)

Completion and lsp features still work, but of course the diagnostics complain.
Same for the lua block if I add a second line too it, only the first is indented.

I'll merge and if i don't experience any issues, I will turn it on by default, haven't found any downsides so far.

@benlubas I did some profiling with https://github.com/stevearc/profile.nvim and this doesn't even introduce any performance hits. Great job on the implementation!


Missing argument for parameter 'coder' in call

Hello,
I have a menu item that should do select second segment in another ViewController where ProfileGlobalVC is. I tried to do it like:
ProfileGlobalVC().setSelectedSegmentAt(2, animated: true)
Xcode gives an error:

Missing argument for parameter 'coder' in call

Then I tried to do it that way:
in my ProfileGlobalVC I've created a simple func:
func listen() {         self.setSelectedSegmentAt(2, animated: true)         }
And then tried  ProfileGlobalVC().listen()
Still getting the same error:

Missing argument for parameter 'coder' in call

My SJSegmentedViewController code is:
import UIKit
import SJSegmentedScrollView

class ProfileGlobalVC: SJSegmentedViewController, SJSegmentedViewControllerDelegate {

    var selectedSegment: SJSegmentTab?

    override func viewDidLoad() {
        self.navigationController?.navigationBar.setBackgroundImage(UIImage(), for: .default)
        self.navigationController?.navigationBar.shadowImage = UIImage()
        self.navigationController?.navigationBar.isTranslucent = false

        if let storyboard = self.storyboard {

            let headerController = storyboard
                .instantiateViewController(withIdentifier: "profile")

            let firstViewController = storyboard
                .instantiateViewController(withIdentifier: "offersTable")
            firstViewController.title = "Offers"

            let secondViewController = storyboard
                .instantiateViewController(withIdentifier: "fave")
            secondViewController.title = "Favourites"

            let thirdViewController = storyboard
                .instantiateViewController(withIdentifier: "settings")
            thirdViewController.title = "Settings"

            headerViewController = headerController
            segmentControllers = [firstViewController,
                                    secondViewController,
                                    thirdViewController]
            headerViewHeight = 180
            selectedSegmentViewHeight = 1.0
            segmentTitleColor = .gray
            selectedSegmentViewColor = .black
            segmentShadow = SJShadow.light()
            delegate = self

        }
        segmentViewHeight = 50
        title = "Segment"
        super.viewDidLoad()
}

Hi @alexeyRuin ,
Try adding the init() method in the other ViewController.

Hello,
first of all sorry for long delay.
init() didn't work for me.
I'm still having 'Missing argument' error.

@alexeyRuin,
I had the same issue when creating my instance of my SJSegmentedViewController. Initially, I instantiated it with : var leftVC = SWPProfileViewController() . What helped me was to instead instantiate it as an optional: var leftVC: SWPProfileViewController?
I hope this helps!

Any Help for this Error ? , its still ask for coder

Any Help for this Error ? , its still ask for coder