sparkctl is not working anymore

I have followed all the steps but sparkctl is not working anymore:
./sparkctl list -n spark
failed to list SparkApplications: the server could not find the requested resource (get sparkapplications.sparkoperator.k8s.io)

Here is what I followed:
dep ensure
go get -u k8s.io/code-generator/cmd/client-gen
go get -u k8s.io/code-generator/cmd/deepcopy-gen
go get -u k8s.io/code-generator/cmd/defaulter-gen
go get -u sigs.k8s.io/controller-tools/cmd/controller-gen

move to sparkctl folder and ran go build -o sparkctl.
But when I do this:
kubectl describe sparkapplication pyspark-pi

I get the response back.
Here is my YAML
apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: pyspark-pi
  namespace: spark
spec:
  type: Python
  pythonVersion: "2"
  mode: cluster
  image: "gcr.io/spark-operator/spark-py:v2.4.4"
  imagePullPolicy: Always
  mainApplicationFile: local:///opt/spark/examples/src/main/python/pi.py
  sparkVersion: "2.4.4"
  restartPolicy:
    type: OnFailure
    onFailureRetries: 3
    onFailureRetryInterval: 10
    onSubmissionFailureRetries: 5
    onSubmissionFailureRetryInterval: 20
  driver:
    cores: 1
    coreLimit: "1200m"
    memory: "512m"
    labels:
      version: 2.4.4
    serviceAccount: sparkoperator
  executor:
    cores: 1
    instances: 1
    memory: "512m"
    labels:
      version: 2.4.4

same thing here .
change sparkctl to use betav2  solved it :
func doList(crdClientset crdclientset.Interface) error {

  apps, err := crdClientset.SparkoperatorV1beta1().SparkApplications(Namespace).List(metav1.ListOptions{})

  apps, err := crdClientset.SparkoperatorV1beta2().SparkApplications(Namespace).List(metav1.ListOptions{})
  if err != nil {
          return err
  }

Fixed in #681.

Thanks @liyinan926


switch from mac runner to linux?

https://github.blog/changelog/2023-02-23-hardware-accelerated-android-virtualization-on-actions-windows-and-linux-larger-hosted-runners/ <- at least some support for android

use kvm?
use libvirt https://stackoverflow.com/questions/31828555/using-vagrant-on-cloud-ci-services/60380518#60380518

https://stackoverflow.com/questions/31828555/using-vagrant-on-cloud-ci-services/60380518#60380518

https://github.blog/changelog/2024-04-02-github-actions-hardware-accelerated-android-virtualization-now-available/

Hurray, we switched from macos-11 to ubuntu-latest for new debian-12, fedora-39 and ubuntu-2310 testing

The implementation is in APN-Pucky/gentoo-prefix-tester/actions/kvm@main


CopySymbolsFromContainer in Compile doesn't copy enough (OnPrem)

Describe the issue
I switched to using filesOnly containers for my pipelines for performance reasons. I'm entirely doing my own dependency management offline. So I'm running compile in an OnPrem filesOnly container without auth to a saas tenant. Using -CopySymbolsFromContainer I would expect to have access to all Microsoft apps but sadly that's not the case for OnPrem artifacts. Could we get similar behavior as the as the saas artifacts are using?
My proposal would be something like:

Scripts used to create container and cause the issue

Full output of scripts

Will investigate what can be done for on-premises.

It would be great to get an update on this topic as it blocks builds that for example depend on MS E-Mail Extensions.

I think the code which should fix this is:
        Invoke-ScriptInBcContainer -containerName $containerName -scriptblock { Param($appSymbolsFolder)
            if (Test-Path "C:\Extensions\*.app") {
                $paths = @(
                    "C:\Program Files\Microsoft Dynamics NAV\*\AL Development Environment\System.app"
                    "C:\Extensions\*.app"
                )
            }
            elseif (!(Test-Path "C:\Applications.*\*.app")) {
                $paths = @(
                    "C:\Program Files\Microsoft Dynamics NAV\*\AL Development Environment\System.app"
                    "C:\Applications\*\source\*.app"
                )
            }
            else {
                $paths = @(
                    "C:\Program Files\Microsoft Dynamics NAV\*\AL Development Environment\System.app"
                    "C:\Applications.*\Microsoft_Application_*.app,C:\Applications\Application\Source\Microsoft_Application.app"
                    "C:\Applications.*\Microsoft_Base Application_*.app,C:\Applications\BaseApp\Source\Microsoft_Base Application.app"
                    "C:\Applications.*\Microsoft_System Application_*.app,C:\Applications\System Application\source\Microsoft_System Application.app"
                )
            }
            $paths | ForEach-Object {
                $appFiles = $_.Split(',')
                $appFile = ""
                if (Test-Path -Path $appFiles[0]) {
                    $appFile = $appFiles[0]
                }
                elseif (Test-Path -path $appFiles[1]) {
                    $appFile = $appFiles[1]
                }
                if ($appFile) {
                    Get-ChildItem -Path $appFile -Recurse | ForEach-Object {
                        Write-Host "Copying $([System.IO.Path]::GetFileName($_.FullName)) from Container"
                        Copy-Item -Path $_.FullName -Destination $appSymbolsFolder -Force
                    }
                }
            }
        } -argumentList $containerSymbolsFolder

If you have a chance to test this, it would be great.
On a similar topic, the next containerhelper (i.e. currently in preview) introduces something called a compilerfolder, Which basically is containerless compiling (which I will use instead of filesonly in AL-Go for GitHub)
I do not have any docs yet, as this is work in progress.

I'll try to test with that code. Thank you!
On a similar topic, the next containerhelper (i.e. currently in preview) introduces something called a compilerfolder, Which basically is containerless compiling (which I will use instead of filesonly in AL-Go for GitHub)
I do not have any docs yet, as this is work in progress.
This sounds exactly like what I want for my build pipelines. Is there a preview of this feature? Will it also enable Runtime creation without an NST? This would reduce my deployment time by ~90%

The runtime creation requires the ModernDev team to add this to the compiler first - but then yes.
Will get back with some docs on how you can do this.

I just checked the repo and saw that it's all in there. I'll implement it for my CI/CD module today! This looks exciting, thanks as always.

I think the code which should fix this is:
        Invoke-ScriptInBcContainer -containerName $containerName -scriptblock { Param($appSymbolsFolder)
            if (Test-Path "C:\Extensions\*.app") {
                $paths = @(
                    "C:\Program Files\Microsoft Dynamics NAV\*\AL Development Environment\System.app"
                    "C:\Extensions\*.app"
                )
            }
            elseif (!(Test-Path "C:\Applications.*\*.app")) {
                $paths = @(
                    "C:\Program Files\Microsoft Dynamics NAV\*\AL Development Environment\System.app"
                    "C:\Applications\*\source\*.app"
                )
            }
            else {
                $paths = @(
                    "C:\Program Files\Microsoft Dynamics NAV\*\AL Development Environment\System.app"
                    "C:\Applications.*\Microsoft_Application_*.app,C:\Applications\Application\Source\Microsoft_Application.app"
                    "C:\Applications.*\Microsoft_Base Application_*.app,C:\Applications\BaseApp\Source\Microsoft_Base Application.app"
                    "C:\Applications.*\Microsoft_System Application_*.app,C:\Applications\System Application\source\Microsoft_System Application.app"
                )
            }
            $paths | ForEach-Object {
                $appFiles = $_.Split(',')
                $appFile = ""
                if (Test-Path -Path $appFiles[0]) {
                    $appFile = $appFiles[0]
                }
                elseif (Test-Path -path $appFiles[1]) {
                    $appFile = $appFiles[1]
                }
                if ($appFile) {
                    Get-ChildItem -Path $appFile -Recurse | ForEach-Object {
                        Write-Host "Copying $([System.IO.Path]::GetFileName($_.FullName)) from Container"
                        Copy-Item -Path $_.FullName -Destination $appSymbolsFolder -Force
                    }
                }
            }
        } -argumentList $containerSymbolsFolder

If you have a chance to test this, it would be great.
On a similar topic, the next containerhelper (i.e. currently in preview) introduces something called a compilerfolder, Which basically is containerless compiling (which I will use instead of filesonly in AL-Go for GitHub) I do not have any docs yet, as this is work in progress.

Did you ever have a chance to test this? - or did you switch to compilerfolder?

Hi Freddy, I did not get to test it and already switched over to use compile folders by now. (BTW loving compile folders)
From my side this topic can be closed.


Bugs: cases when whole stack of vanilla unstackables is consumed

As of this moment I have found two such bugs:

Eating from a stack of suspicious stew returns one bowl and no suspicious stew It doesn't happen with normal mushroom stew. It was dandelion / saturation stew in my case.
Using multiple books while enchanting an item consumes all books. It was stack of mending books in my case.

I did not test it extensively. There may be other such cases. Just wanted to let You know.

same problem. neat

Thanks for reporting! I'll try to get to this as soon as I can, but I've been pretty busy lately.
Unfortunately these kinds of issues are absolutely everywhere because the base game expects all these items to have a max stack size of 1 and the code makes some shortcuts with that assumption. I tried to track them all down but I'm always finding more. Please if you discover more I would really appreciate it if you'd report them!

Just fixed these two cases. Check out 1.3.1 on Modrinth!


[W6.1f][W09-B2] Charles Goh

Converted execute() method into an abstract method

Changed the title of the pull request. It should be 6.1f not 6.4f

PR not clean :-(


[Speaker Recognition] V3.0 GA swagger files

This is for submitting an updated version of Speaker Recognition swagger files from Preview (V2.0) to GA (V3.0). According to Azure REST API Stewardship guideline, I created this issue for just tracking all the work related to this PR.
PR link is here: #15685

API Stewardship Board Review / Discussion 9-Sept-2021
Minor changes between 2.0 (preview) and 3.0.

Since this is a major version revision, it would be good to move to our date based versioning. See our Azure REST API Guidelines.
There are considerations about POST being idempotent.

We should look at the retry-able request

Change the error codes from the HTTP status name to contextual things (i.e., "MissingLocale" instead of "InvalidRequest")
See the guidance on use of "top" and "nextlink"


Updated the API design to expand the message body arguments in the API functions

This PR updated the API design to expand the message body arguments in the API functions.
Previously:
local client = nakama.create_client(config)

local email =<EMAIL_ADDRESS>local password = "batsignal"
local body = nakama.create_api_account_email(email, password)
local session = nakama.authenticate_email(client, body)
pprint(session)

New:
local client = nakama.create_client(config)

local email =<EMAIL_ADDRESS>local password = "batsignal"
local session = nakama.authenticate_email(client, email, password)
pprint(session)

Hi @britzl,
Thanks for the changes you've made so far, removing the intermediate message body creation step will hugely improve the developer experience. We have a couple more requests that will help bring the Defold SDK in line with our other SDKs.
Please move client functions to the Client object rather than have them on the Nakama object.
Instead of:
local client = nakama.create_client(...)
local result = nakama.add_friends(client, ...)

It should be:
local client = nakama.create_client(...)
local result = client.add_friends(...)

Similar to above, socket functions and events should live on the Socket object.
Instead of:
nakama.on_matchpresence(socket, fn)

It should be:
socket.on_match_presence(fn)

For reference you can take a look at the IClient and ISocket definitions in the .NET SDK.
https://github.com/heroiclabs/nakama-dotnet/blob/master/Nakama/IClient.cs
https://github.com/heroiclabs/nakama-dotnet/blob/master/Nakama/ISocket.cs
Please let me know if you have any questions or issues.

Update:

Removed the semi-colon (:) notation mentioned in the previous comment

socket:send_status_update("my status") -> socket.send_status_update("my status")

Updated the example and tested against a local server
Updated the WebSocket dependency to v 3.0.0 in game.project

Note: in the docs there's three dependencies list (https://heroiclabs.com/docs/nakama/client-libraries/defold/#add-defold-plugins). It is only the extension-websocket dependency that is needed

@tomglenn could you resolve conversations that are addressed.

@britzl @tomglenn how are we tracking a version change? With the major changes we should bump a major version.

Agreed. @britzl can you also update the CHANGELOG to reflect the changes.

Done. I've retroactively added previous releases to the CHANGELOG.


dynamically add document groups

Version Info: NSwag 12.2.5
at present, this method is cumbersome. Every time i add a new version, i have to add a code once. can you dynamically group it?
https://github.com/RicoSuter/NSwag/blob/c1d0907c4e413ab2f75cc8cd0ab7018c7c688b01/src/NSwag.SwaggerGeneration.AspNetCore.Tests.Web/Startup.cs#L41-L56
another problem is that the document in the figure below is duplicated.

We could provide an extension method which registers all versions automatically, e.g.
AddVersionedSwaggerDocuments(...)
AddVersionedOpenApiDocuments(...)

@RicoSuter Thank you, when are these extension methods available?


size mismatch

my server can't load t5-v1-xl, so I am trying to run the code for t5-v1 (it could be the base model)
As said I transformed it to the model suitable for the project, however when I run the code it can't load the checkpoint with size mismatch.
Traceback (most recent call last):
  File "/home/pouramini/PPT/train.py", line 401, in <module>
    main()
  File "/home/pouramini/PPT/train.py", line 385, in main
    model, optimizer, lr_scheduler = setup_model_and_optimizer(args, tokenizer.vocab_size, ds_config, prompt_config)
  File "/home/pouramini/PPT/utils.py", line 190, in setup_model_and_optimizer
    args.iteration = load_checkpoint(model, optimizer, lr_scheduler, args, prompt_config)
  File "/home/pouramini/PPT/utils.py", line 330, in load_checkpoint
    model.module.load_state_dict(sd["module"], strict=False)
  File "/home/pouramini/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1604, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for EncDecModel:
	size mismatch for word_embeds.weight: copying a param with shape torch.Size([8032, 768]) from checkpoint, the shape in current model is torch.Size([32128, 768]).
	size mismatch for lm_head.weight: copying a param with shape torch.Size([8032, 768]) from checkpoint, the shape in current model is torch.Size([32128, 768]).
	size mismatch for encoder.word_embeds.weight: copying a param with shape torch.Size([8032, 768]) from checkpoint, the shape in current model is torch.Size([32128, 768]).
	size mismatch for encoder.blocks.0.self_attn.self_attn.project.weight: copying a param with shape torch.Size([576, 768]) from checkpoint, the shape in current model is torch.Size([2304, 768]).
	size mismatch for encoder.blocks.0.self_attn.self_attn.relative_attention_bias.weight: copying a param with shape torch.Size([32, 3]) from checkpoint, the shape in current model is torch.Size([32, 12]).
	size mismatch for encoder.blocks.0.self_attn.self_attn.dense.weight: copying a param with shape torch.Size([768, 192]) from checkpoint, the shape in current model is torch.Size([768, 768]).
	size mismatch for encoder.blocks.0.ff.dense_relu_dense.wi_0.weight: copying a param with shape torch.Size([512, 768]) from checkpoint, the shape in current model is torch.Size([2048, 768]).
``

it seems if I don't convert it, it works

Did you download the model you used from here ? You need to change the configuration to this file for the t5-v1_1-base model.
Besides, if you have only one GPU, you need to change the model parallel to 1 by setting the --mp_size in this line to 1 and MP_SIZE in this line to 1.


There's no mention in documentation of how to use TeleportHotSpots

Hi, I can't seem to find any info about how to properly implement TeleportHotSpot. How I approached it was to place an object with a collider on it and to add TeleportHotSpot component. Implementation suggests it only accepts TeleportPointers but ParabolicTeleportPointer seems to ignore my hot spots and the pointer hides when I'm over my hotspot area.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ab379974-9f28-0380-3b0a-b8ee5f83948c
Version Independent ID: f7dfa238-a18a-3019-2f1d-7b7de6964199
Content: TeleportSystemOverview
Content Source: mrtk-unity/features/teleport-system/teleport-system.md
Product: mixed-reality
GitHub Login: @RogPodge
Microsoft Alias: roliu

@RogPodge, is this a docs gaps or are there instructions we can point to?

@polar-kev yeah we're currently missing documentation (and unit testing!) on this feature. I think it was included when the teleport system was still in experimental, and when it was graduated, we never properly vetted the teleport hotspot feature.
I'm somewhat booked at the moment but I can handle graduating this particular aspect of the Teleport system in a future release!


Attribute for forcing copies of mutable types

I'm proposing an attribute that helps with tracking shallow copies vs deep copies/ownership semantics. I just spent a while tracking down a bug caused by this.
C++ surfaces errors like this because you're likely to pass in references as parameters but have to use values as member variables, to you're more likely to cause a copy than dangling reference
class Foo {
private /*magic_attribute */_bars : readonly string[]
constructor(bars : readonly string[]){
this._bars =  bars // This line now complains because I didn't call bars.slice(0)
}
}
magic_attribute named something like owning_copy? copy_mutables?
I'm working with TypesScript 3.6.4

See https://github.com/microsoft/TypeScript/issues/16148

Please use the issue template next time. Thanks!

Ah, sorry for not seeing the template (thought it was just boilerplate about trying to find dupes).
Looking at the context bug, ownership seems well litigated. How about something that tracks frozenness? That would get the immutability check I’m really trying to get at


page isn't readable with chrome

Describe the bug
ibm-garage-cloud.github.io/ibm-garage-developer-guide/starterkits/starterkittemplates is unreadable
https://ibm-garage-cloud.github.io/ibm-garage-developer-guide/starterkits/starterkittemplates/ with chrome looks like this:

Its not possible to move the left-side navigation panel, and its not possible to scroll the text, so the left 1/3 of the page can't be read.
In Firefox, its readable.

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. iOS] Linux
Browser [e.g. chrome, safari] chrome
Version [e.g. 22] current

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

works for me on chrome
https://cloudnativetoolkit.dev/codepatterns/gitrepos

Yep, works for me now, too.


Problem with bounding box in maps with google basemaps and widgets

If the user creates a map with basemap of Google, adds a widget, and then reloads the map, a "state" stored with wrong bounding box will be saved :(.
STR

Create a map.
Change basemap to a Google basemaps (of course, you have to have it enabled 😬 ).
Add a widget.
Move the map wherever you want.
Reload the map.
Everything should work.
Reload again the map.
Center is moved to [0,0][0,0].

More info: https://github.com/CartoDB/support/issues/590#issuecomment-290451205
Support ticket for context: https://github.com/CartoDB/support/issues/590.

@noguerol @jorgesancha, this is important for migrations.
cc @alonsogarciapablo

Please check it out first and let's get a sense of effort. I assume this
affects everyone with widgets and google basemaps, right?
On Thu, 30 Mar 2017 at 18:00, Javier Álvarez Medina <
<EMAIL_ADDRESS>wrote:

@noguerol https://github.com/noguerol @jorgesancha
https://github.com/jorgesancha, this is important for migrations.
cc @alonsogarciapablo https://github.com/alonsogarciapablo
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/CartoDB/cartodb/issues/11894#issuecomment-290457461,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAFQyvU0gDtNpXND4h7C3RbIQIRejLPbks5rq9GNgaJpZM4Mukb-
.

I've just seen this in CARTO.js https://github.com/CartoDB/cartodb.js/blob/v4/src/geo/gmaps/gmaps-map-view.js#L146
  getBounds: function () {
    if (this._isReady) {
      var b = this._gmapsMap.getBounds();
      var sw = b.getSouthWest();
      var ne = b.getNorthEast();
      return [
        [sw.lat(), sw.lng()],
        [ne.lat(), ne.lng()]
      ];
    }
    return [ [0, 0], [0, 0] ];
  },

Are we fixing the other issue together with this? the one for saving zoom and x,y instead of bbox?

Blocking it until we have something we agree in DI, where the source of the problem is placed.

PR about DI -> https://github.com/CartoDB/deep-insights.js/pull/533

So, there are two PR:

Builder: https://github.com/CartoDB/cartodb/pull/11901#discussion_r109371772
DI: https://github.com/CartoDB/deep-insights.js/pull/533

If we check everything is 👍 , we should:

Merge deep-insights PR into master.
Change package to point to master and update dependencies in the Builder PR.
Push it and wait for a 🍏 .

Deployed


Multipath support with non-zero length Connection IDs

This commits introduces the following features.

Leave to the application to choose the multipath extensions or not
Build on the PathEvents to let the application decide which paths to use
Provide default reasonable behavior into quiche while letting the
application provide optimised behavior

== Requesting the Multipath feature
The application can request multipath through the Config structure using a
specific API,set_multipath().
config.set_multipath(true);

The boolean value determines whether the underlying connection negotiates the
multipath extension. Once the handshake succeeded, the application can check
whether the connection has the multipath feature enabled using the
is_multipath_enabled() API on the Connection structure.
== Path management
The API requires the application to specify on which paths/4-tuples it wants
to send non-probing packets. Paths must first be validated before using them.
This is automatically done for servers, and client must use probe_path().
Once the path is validated, the application decides whether it wants active
usage through the set_active() API. It provides a single API entrypoint to
request its usage or not. For active path usage, it can use the following.
if let Some(PathEvent::Validated(local, peer)) = conn.path_event_next() {
    conn.set_active(local, peer, true).unwrap();
}

Then, the path will then be considered to use non-probing packets.
On the other hand, if for some reason the application wants to temporarily
stop sending non-probing frames on a given path, it can do the following.
conn.set_active(local, peer, false).unwrap();

Note that in such state, quiche still replies to PATH_CHALLENGEs observed on
that path.
Finally, the multipath design allows a QUIC endpoint to close/abandon a
given path along with an error code and error message, without altering the
connection's operations as long as another path is available.
conn.abandon_path(local, peer, 42, "Some error message".into()).unwrap();

-- Retrocompatibility note

The Closed variant of PathEvent is now a 4-tuple that, in addition to
the local and peer SocketAddr, also contains an u64 error code and a
Vec<u8> reason message.
There is a new variant of PathEvent: PeerPathStatus reports to the
application that the peer advertised some status for a given 4-tuple.
There are two new Error variants: UnavailablePath and
MultiPathViolation.

-- Note
Currently this API is only available when multipath feature is enabled over
the session (i.e., conn.is_multipath_enabled() returns true). If the
extension is not enabled, set_active() and abandon_path() return an
Error::InvalidState. Actually, this API might sound "double usage" along
with the migrate() API (as there is no real "connection migration" with
multipath). Should we just keep the set_active() or similarly named API
and include the migrate() functionality in set_active()? Actually, an
client application without the multipath feature could just migrate using
set_active(local, peer, true), setting the previous path in unused mode
under the hood.
== Scheduling sent packets
Similarly to the connection migration PR, there are two ways to control how
quiche schedules packets on paths.
The first consists in letting quiche handles this by itself. The
application simply uses the send() method. In the current master code,
quiche automatically handles path validation processing thanks to the
internal get_send_path_id() Connection method. The multipath patch
extends this method and iterates over all active paths following the lowest
latency path having its congestion window open heuristic (a reasonable
default in multipath protocols).
loop {
    let (write, send_info) = match conn.send(&mut out) {
            Ok(v) => v,

            Err(quiche::Error::Done) => break,

            Err(e) => {
                conn.close(false, 0x1, b"fail").ok();
                break;
            },
        };
        // write to socket bound to `send_info.from`
}

The second option is to let the application choose on which path it wants
to send the next packet. The application can iterate over the available paths
and their corresponding statistics using path_stats() and schedules packets
using send_on_path() method. This can be useful when the use case requires
some specific scheduling strategies. See apps/src/client.rs for an example
of such application-driven scheduling.

Thanks for your work on implementing MPQUIC! I played around with this code and have the suspicion that some packets are not sent on the path that they should be sent on.
My setup: I created a mininet topology to test multipath support with the quiche-server and quiche-client applications.
                Path 1
         /--- s1 --- s2 ---/
     <IP_ADDRESS>           <IP_ADDRESS>
 Client h1          Server h2
     <IP_ADDRESS>           <IP_ADDRESS>
         \--- s3 --- s4 ---/
                Path 2

I added space_id and path_id to the qlog PacketHeader that is used in the send_single and recv_single functions. From my understanding, space_id refers to the packet number space of the sent/received packet while path_id refers to the network path that the packet will be sent over / was received from. Since we have one packet number space per path, these two values should be equal.
My tests showed that, on the server, path_id always equals space_id (values are either 0 or 1). On the client, space_id and path_id are also equal in all transport:packet_sent events.
This is however not true for transport:packet_received events on the client: here, space_id != path_id in about 3% of cases (e.g., 1300 of 40000 received packets). I was not yet able to confirm if packets are actually sent on a different network path than space_id indicates in those cases (Wireshark fails to decode the QUIC packets).
The quiche logs hint that packets, that are assigned to the packet number space of one path, are sometimes sent on the other path:
[...]
[2022-11-16T14:19:49.964226504Z TRACE quiche] 14d656e811af235b787dc1ba7bc22a72524c0d80 path ID 0 now see SCID with seq num 1
[2022-11-16T14:19:49.974246915Z TRACE quiche] 14d656e811af235b787dc1ba7bc22a72524c0d80 path ID 0 now see SCID with seq num 0
[2022-11-16T14:19:49.985896148Z TRACE quiche] 14d656e811af235b787dc1ba7bc22a72524c0d80 path ID 0 now see SCID with seq num 1
[...]

Is this behavior expected or is there actually something going wrong?
Attachments:

0001-Add-path_id-and-space_id-to-qlog.patch.txt: the qlog code additions to log path_id and space_id. I used the latest commit (d6772280) from this PR.
logs.zip: qlog and quiche logs of a multipath transmission

@hendrikcech Nice to see you are experimenting with the code :)
It seems that quiche behaves correctly, but the server code does not send the packet on the right path. From the client logs,
[2022-11-16T14:19:43.721770238Z TRACE quiche] 14d656e811af235b787dc1ba7bc22a72524c0d80 rx pkt Short dcid=18f4cf5ad9df2fb7f8cb280b8ea06fafa2e45d3c key_phase=false len=1329 pn=0 src:<IP_ADDRESS>:4433 dst:<IP_ADDRESS>:8002
[2022-11-16T14:19:43.721878453Z TRACE quiche] 14d656e811af235b787dc1ba7bc22a72524c0d80 path ID 1 now see SCID with seq num 1
[2022-11-16T14:19:43.721925418Z TRACE quiche] 14d656e811af235b787dc1ba7bc22a72524c0d80 rx frm PATH_RESPONSE data=[02, e6, 51, c4, 58, 32, 39, 45]
[2022-11-16T14:19:43.721993077Z TRACE quiche] 14d656e811af235b787dc1ba7bc22a72524c0d80 rx frm PATH_CHALLENGE data=[26, 6b, ab, b8, 54, 64, 49, 0b]
[2022-11-16T14:19:43.722450331Z TRACE quiche] 14d656e811af235b787dc1ba7bc22a72524c0d80 rx frm PADDING len=1294
[2022-11-16T14:19:43.722732114Z TRACE quiche_apps::client] <IP_ADDRESS>:8002: processed 1350 bytes
[2022-11-16T14:19:43.722774926Z TRACE quiche_apps::client] <IP_ADDRESS>:8002: got 49 bytes
[2022-11-16T14:19:43.722840885Z TRACE quiche] 14d656e811af235b787dc1ba7bc22a72524c0d80 rx pkt Short dcid=14d656e811af235b787dc1ba7bc22a72524c0d80 key_phase=false len=28 pn=1 src:<IP_ADDRESS>:4433 dst:<IP_ADDRESS>:8002
[2022-11-16T14:19:43.722927964Z TRACE quiche] 14d656e811af235b787dc1ba7bc22a72524c0d80 peer reused CID 14d656e811af235b787dc1ba7bc22a72524c0d80 from path 0 on path 1
[2022-11-16T14:19:43.722971351Z TRACE quiche] 14d656e811af235b787dc1ba7bc22a72524c0d80 path ID 1 now see SCID with seq num 0
[2022-11-16T14:19:43.723004292Z TRACE quiche] 14d656e811af235b787dc1ba7bc22a72524c0d80 rx frm DATA_BLOCKED limit=10000000
[2022-11-16T14:19:43.723037715Z TRACE quiche] 14d656e811af235b787dc1ba7bc22a72524c0d80 rx frm STREAM id=7 off=0 len=1 fin=false

But at server side:
[2022-11-16T14:19:43.668765632Z INFO  quiche_server] 327765e95cde0abdb523c6de9e6b2d292c0ced18 Seen new path (<IP_ADDRESS>:4433, <IP_ADDRESS>:8002)
[2022-11-16T14:19:43.668816125Z TRACE quiche_server] recv() would block
[2022-11-16T14:19:43.669135491Z TRACE quiche] 327765e95cde0abdb523c6de9e6b2d292c0ced18 tx pkt Short dcid=18f4cf5ad9df2fb7f8cb280b8ea06fafa2e45d3c key_phase=false len=1312 pn=0 src:<IP_ADDRESS>:4433 dst:<IP_ADDRESS>:8002
[2022-11-16T14:19:43.669183980Z TRACE quiche] 327765e95cde0abdb523c6de9e6b2d292c0ced18 tx frm PATH_RESPONSE data=[02, e6, 51, c4, 58, 32, 39, 45]
[2022-11-16T14:19:43.669228104Z TRACE quiche] 327765e95cde0abdb523c6de9e6b2d292c0ced18 tx frm PATH_CHALLENGE data=[26, 6b, ab, b8, 54, 64, 49, 0b]
[2022-11-16T14:19:43.669267438Z TRACE quiche] 327765e95cde0abdb523c6de9e6b2d292c0ced18 tx frm PADDING len=1294
[2022-11-16T14:19:43.669409946Z TRACE quiche::recovery] 327765e95cde0abdb523c6de9e6b2d292c0ced18 timer=1.023455331s latest_rtt=0ns srtt=None min_rtt=0ns rttvar=166.5ms loss_time=[None, None, None] loss_probes=[0, 0, 0] cwnd=13500 ssthresh=18446744073709551615 bytes_in_flight=1350 app_limited=true congestion_recovery_start_time=None Rate { delivered: 0, delivered_time: Instant { tv_sec: 228741, tv_nsec: 941993262 }, first_sent_time: Instant { tv_sec: 228741, tv_nsec: 941993262 }, end_of_app_limited: SpacedPktNum(0, 0), last_sent_packet: SpacedPktNum(1, 0), largest_acked: SpacedPktNum(0, 0), rate_sample: RateSample { delivery_rate: 0, is_app_limited: false, interval: 0ns, delivered: 0, prior_delivered: 0, prior_time: None, send_elapsed: 0ns, ack_elapsed: 0ns, rtt: 0ns } } pacer=Pacer { enabled: true, capacity: 13500, used: 0, rate: 0, last_update: Instant { tv_sec: 228741, tv_nsec: 941993262 }, next_time: Instant { tv_sec: 228741, tv_nsec: 941993262 }, max_datagram_size: 1350, last_packet_size: None, iv: 0ns } hystart=window_end=Some(SpacedPktNum(1, 0)) last_round_min_rtt=18446744073709551615.999999999s current_round_min_rtt=18446744073709551615.999999999s css_baseline_min_rtt=18446744073709551615.999999999s rtt_sample_count=0 css_start_time=None css_round_count=0 cubic={ k=0 w_max=0 }
[2022-11-16T14:19:43.669725202Z TRACE quiche] 327765e95cde0abdb523c6de9e6b2d292c0ced18 tx pkt Short dcid=14d656e811af235b787dc1ba7bc22a72524c0d80 key_phase=false len=11 pn=1 src:<IP_ADDRESS>:4433 dst:<IP_ADDRESS>:8001
[2022-11-16T14:19:43.669773886Z TRACE quiche] 327765e95cde0abdb523c6de9e6b2d292c0ced18 tx frm DATA_BLOCKED limit=10000000
[2022-11-16T14:19:43.669803595Z TRACE quiche] 327765e95cde0abdb523c6de9e6b2d292c0ced18 tx frm STREAM id=7 off=0 len=1 fin=false
[2022-11-16T14:19:43.669936399Z TRACE quiche::recovery] 327765e95cde0abdb523c6de9e6b2d292c0ced18 timer=218.846885ms latest_rtt=104.273399ms srtt=Some(104.518737ms) min_rtt=104.273399ms rttvar=22.416993ms loss_time=[None, None, None] loss_probes=[0, 0, 0] cwnd=13500 ssthresh=18446744073709551615 bytes_in_flight=49 app_limited=true congestion_recovery_start_time=None Rate { delivered: 2194, delivered_time: Instant { tv_sec: 228741, tv_nsec: 936736141 }, first_sent_time: Instant { tv_sec: 228741, tv_nsec: 936736141 }, end_of_app_limited: SpacedPktNum(0, 1), last_sent_packet: SpacedPktNum(0, 1), largest_acked: SpacedPktNum(0, 1), rate_sample: RateSample { delivery_rate: 4958, is_app_limited: true, interval: 104.273399ms, delivered: 517, prior_delivered: 1677, prior_time: Some(Instant { tv_sec: 228741, tv_nsec: 832462742 }), send_elapsed: 0ns, ack_elapsed: 104.273399ms, rtt: 104.273399ms } } pacer=Pacer { enabled: true, capacity: 13500, used: 0, rate: 161454, last_update: Instant { tv_sec: 228741, tv_nsec: 936736141 }, next_time: Instant { tv_sec: 228741, tv_nsec: 936736141 }, max_datagram_size: 1350, last_packet_size: Some(0), iv: 0ns } hystart=window_end=Some(SpacedPktNum(0, 0)) last_round_min_rtt=18446744073709551615.999999999s current_round_min_rtt=18446744073709551615.999999999s css_baseline_min_rtt=18446744073709551615.999999999s rtt_sample_count=0 css_start_time=None css_round_count=0 cubic={ k=0 w_max=0 }
[2022-11-16T14:19:43.670211822Z TRACE quiche_server] 327765e95cde0abdb523c6de9e6b2d292c0ced18 written 1399 bytes

So quiche at the server side indeed indicates the second packet should be sent on the original path src:<IP_ADDRESS>:4433 dst:<IP_ADDRESS>:8001, but for some reason the client side notices that the "STREAM" packet is received on the same path than the PATH_CHALLENGE one.
It actually seems that quiche-server batches the conn.send() into a large buffer and calls send_to once the buffer is full (or if there is no additional packet to be sent), but forgets to check whether packets should be sent on different 4-tuples (only the 4-tuple of the first packet in the batch is considered). I can quickly look at fixing this behaviour of quiche-server in a commit.

@qdeconinck Thanks for taking a look! I can confirm that these changes to quiche-server resolve the problem.

I made a simple client/server example and tested with this multipath patch. I noticed there might be a flaw in the ACKMP sending logic.
My code is based on the one in quiche/apps. The client opens 2 paths against the server and writes data on a single QUIC stream. It turns out that the server sometimes stops sending ACKMPs which causes a client-side idle timeout and subsequent termination of the connection. This problem can be mitigated if the send() call in the server event loop is replaced with a send_on_path() call that iterates over all available paths, just like the client example (in the multipath branch) does.
The send_single() function sends an ACKMP only if there is an unacked packet receive don the same path as specified in the function argument, while the send() function just selects a "best" path and no other paths are tried even if the selected path doesn't yield a packet data.
I think the problem lies here.
I noticed the client is retransmitting STREAM packets before a timeout but it doesn't seem to help. I haven't checked on which path those retransmits are happening.

@toshiiw This is strange, as the get_send_path_id() method should address this point. Could you indicate on which commit you are based, and provide logs describing the behavior? Feel free to contact me offline if preferred.

I tested with the following version.
Attached are sqlog files with packet namespace ids.
Data is sent from a client to a server (unidirectional).
commit d67722801b043a9b82c04fa70bf1e3240492ee23 (HEAD -> multipath)
Author: Quentin De Coninck<EMAIL_ADDRESS>Date:   Mon Oct 10 17:29:15 2022 +0200

sqlog.tar.gz
I also tested with the newest code on the multipath branch but still saw premature shutdowns.

@toshiiw Oooh good catch indeed! If you configure your connection to enable datagrams on the connection, I can indeed reproduce the issue! I will push a fix with the adapted multipath test now; replacing let dgrams_to_emit = self.dgram_max_writable_len().is_some(); with let dgrams_to_emit = self.dgram_send_queue.has_pending(); indeed solves the issue.

I tested how this multpath patch performs under partial network failures.
A test client opens 2 paths against the server and continuously send data over a QUIC stream  (the same test programs as 3 weeks ago).
Then the traffic of one of the paths is disabled by issuing iptables -D INPUT -p udp -s <IP_ADDRESS> --sport 30002 -j DROP.
Then, sooner or later, the client picks up the disabled path to send STREAM resends or pings, which will never reach the server, and eventually the QUIC connection is terminated with an idle timer.
I guess multipath should be able to handle such situation.
A few observations and random ideas:

RTT based path selection might not work well with packet losses, as RTT is only updated upon successful transmit
Is it a good idea to mark a path as stale after a few packet losses?
Is there a workaround in a quiche application code? I guess counting packet losses in applications is cumbersome.

I tested how this multpath patch performs under partial network failures.
* Is there a workaround in a quiche application code? I guess counting packet losses in applications is cumbersome.

I noticed pto_count could be used as a loss counter, and the attached patch enables a multipath connection to recover from a failed path.
But the recovery is very slow (taking ~10 secs) and I'd love to see a better way to handle such a situation.
My understanding is that mark_all_inflight_as_lost must be called on a failed path to ensure that retransmits happen on a good path but closing a failed path is somewhat a slow operation.
stale-patch.txt

@masa-koz You are right on the fact that if a path experiences a PTO, it should send some packets there -- at least, to probe the path. Still, I'm not sure that the previously carried frames/data should always be retransmitted on the same path (it probably makes more sense to use an available "good" path instead).

@qdeconinck The previously carried frames/data have not been declared as lost yet. Those frames are treated as lost in Recovery::on_loss_detection_timeout() for sending them in the probe packets. So, I think that those frame should be retransmitted on the same path.
I agree with you about the fact that it makes more sense to retransmit the frames on another "good" path. But the frames should be those which were included in the actually lost packets or be those which are marked as retransmitting before lost for some reason (for example, due to the request from the application), I think.

AFAIK the multipath patch only has PTO to handle sudden 100% loss of a path, which doesn't work well because no newer packets are acked and usual loss detection never happens.
While I expect speedy recovery with multipath, there is a tradeoff with network usage etc. I'm happy to see duplicate retransmissions over available paths but other people might not.
It seems I've got this reliability issue (the former) and the availability issue (the latter) mixed up. I'll think again.

Hey @qdeconinck, thanks for your continued work on this.
We've been working on testing and integrating this into our stack, so I've now started looking into rebasing this (there's a few more conflicts due to the recent keyupdate change), as well as doing some clean-up and starting to merge some of the changes (e.g. the initial refactor commit) to make iterating over this a bit easier. If you have some time it would be useful if you could do the rebase, and then split the refactor commit into a separate PR (if not I can do the myself) so we can merge that on its own.
Regarding the refactor specifically, what was the reason for splitting the crypto fields out of PktNumSpace and into a completely separate map? Why not just leave them in PktNumSpace and use the same map for lookups?
I think we might also need to think through KeyUpdate when multi-path is used.
Also, any thoughts on the changes to the draft that were recently discussed (e.g. path IDs)? Anything we can do right now to make those changes easier later?

Hey @ghedo, sorry for the delayed answer, I was away previous week.
Regarding the rebasing and isolation of "pure refactoring changes", I can find some time this week to perform this :-)
About the splitting of PktNumSpace into two separate maps, the motivation comes from the recent optimisations trying to minimise the number of paths lookups (like a45f66fd502fa94e740db57a150a921e73f6ffb2). With this organization, it is possible to keep the crypto context and the packet number space one into local variables without having Rust compiler complaining about ownership.
Finally, regarding recent discussions, I think it would make sense to have different keys on different paths in the long run. To that end, having this notion of "internal Path ID" instead of directly using CID sequence numbers would make sense (knowing that the CID is linked to that internal path ID by the peer proposing the CID). With the current code structure, I don't think it would introduce much changes (the path_id field of ConnectionIdEntry would be a u64 instead of a Option<usize>), though I would have to try to implement such a variant to assess whether this is correct.

@ghedo Refactoring changes are isolated in #1493.

Hello everyone,
I'm new to QUIC, and I'm commencing my master's thesis entitled "Enhancing the Performance of a Single QUIC Connection with Multi-Path QUIC."
While conducting measurements, I've noticed that the Multi-Path extension is experiencing correctness issues in my setup. I'm using loop-back addresses on a single host. Below is a script that reproduces the issue. The server has one endpoint, and the client has two endpoints.
Observations:

The server validates the paths. However, it sends traffic on both paths, but the transfer ends prematurely with the client.
The client terminates with a timeout error, where the duration of the timeout equals the time of the transfer.
The occurrence of this error is non-deterministic, varying from one run to another. However, it appears to happen more frequently with larger files. I tested with 1GB and 8GB only.

I haven't delved deeply into the issue because I'm uncertain whether it's due to a misconfiguration on my part or if it's a bug in the actual source code.
Thank you in advance for your time.
#!/bin/bash

# Code partlty inspired by https://github.com/tumi8/quic-10g-paper

# Variables
QUICHE_REPO="https://github.com/qdeconinck/quiche.git"
QUICHE_COMMIT="d87332018d84fb7c429ad2ed34cbfdc6ee9477c8"
RUST_PLATFORM="x86_64-unknown-linux-gnu"
FILE_SIZE=8G
NB_RUNS=10

RED='\033[0;31m'
RESET='\033[0m'

echo_red() {
    echo -e "${RED}$1${RESET}"
}

get_unused_port(){
    local port
    port=$(shuf -i 2000-65000 -n 1)
    while netstat -atn | grep -q ":$port "; do
        port=$(shuf -i 2000-65000 -n 1)
    done
    echo "$port"
}

clone_mp_quiche() {
    if [ ! -d "./quiche" ]; then
        git clone --recursive "$QUICHE_REPO"
        cd quiche || exit
        git checkout "$QUICHE_COMMIT"
        RUSTFLAGS='-C target-cpu=native' cargo build --release
        cd ..
    fi
    if [ ! -f "./quiche-client" ]; then
        cp "quiche/target/release/quiche-client" .
    fi
    if [ ! -f "./quiche-server" ]; then
        cp "quiche/target/release/quiche-server" .
    fi
}

setup_rust() {
    # Rust
    if ! rustc --version 1>/dev/null 2>&1; then
        curl --proto '=https' --tlsv1.2 -sSf -o /tmp/rustup-init.sh https://sh.rustup.rs
        chmod +x /tmp/rustup-init.sh
        /tmp/rustup-init.sh -q -y --default-host "$RUST_PLATFORM" --default-toolchain stable --profile default
        source "$HOME/.cargo/env"
    else
        echo "Rust is already installed"
    fi
}

setup_environment() {
    mkdir -p "$(pwd)/www" "$(pwd)/responses" "$(pwd)/logs"
    fallocate -l ${FILE_SIZE} "$(pwd)/www/${FILE_SIZE}B_file"
}

iteration_loop() {
    local server_pid server_port
    for iter in $(seq 1 ${NB_RUNS}); do
        echo "Testing Multi-Path QUIC correctness - Iteration $iter"
        # Run server
        env RUST_LOG=info ./quiche-server \
            --listen <IP_ADDRESS>:19474 \
            --root "$(pwd)/www/" \
            --key "$(pwd)/quiche/apps/src/bin/cert.key" \
            --cert "$(pwd)/quiche/apps/src/bin/cert.crt" \
            --multipath \
            1>"$(pwd)/logs/server_${iter}.log" 2>&1 &
        server_pid=$!

        # Run client
        server_port=$(get_unused_port)
        client_port_1=$(get_unused_port)
        client_port_2=$(get_unused_port)
        env RUST_LOG=info ./quiche-client \
            --no-verify "https://<IP_ADDRESS>:19474/${FILE_SIZE}B_file" \
            --dump-responses "$(pwd)/responses/" \
            -A <IP_ADDRESS>:${client_port_1} \
            -A <IP_ADDRESS>:${client_port_2} \
            --multipath \
            1>"$(pwd)/logs/client_${iter}.log" 2>&1
        error_code=$?

        sleep 1

        kill -9 "$server_pid" 1>/dev/null 2>&1
        if [ $error_code -ne 0 ]; then
            echo_red "Error Client: $error_code"
            exit 1
        fi

        # Check if files are the same
        diff -q "$(pwd)/www/${FILE_SIZE}B_file" "$(pwd)/responses/${FILE_SIZE}B_file"
        if [ $? -ne 0 ]; then
            echo_red "Error: files are not the same"
            exit 1
        fi
    done
}

main() {
    # Version
    setup_rust
    [ $? -ne 0 ] && { echo_red "Error setting up rust"; exit 1; }
    clone_mp_quiche
    [ $? -ne 0 ] && { echo_red "Error cloning quiche"; exit 1; }
    setup_environment
    [ $? -ne 0 ] && { echo_red "Error setting up environment"; exit 1; }
    iteration_loop
}

main


Use package-private scope

We should consider changing some classes scope from public to package private.
This avoids exposing a lot of internal classes that aren't meant to be used outside of the API.
Some examples:

JoinType
StringUtils

The only troublesome part is that we need to move the tests accordingly to have access to the classes like .StringUtilsTest.java -> .utils.StringUtilsTest.java.

We can move JoinType, StringUtils and other enum classes from api to builder package, and make them package private. This only leaves Builder and SQLiteQueryBuilder in the original place and remove utils package completely.
No package change for unit tests.

Restructuring the test directory to match up isn't something I'm too worried about. @monxalo your suggestion works for me.

After tinkering with this, i realized that for StringUtils at least, will be hard to make package-private since it's used mostly everywhere.
If we had everything in a single package it would be straightforward but then we sacrifice organization of the classes.
For now, here's what can be made package-private:

JoinType
Contructors in every Select...Builder class.


Set up a proper CMake project structure

[x] Create an initial CMake project structure
[ ] Add a .gitignore to ignore build directories
[ ] Restructure the CMake project to build engine code as a static library and game code as an executable

Initial CMake project structure created in 88006d75e1a46a320a3ef3d61cf7ac44dabb6e02. Includes versioning support.

Adding a .gitignore is not necessary when using out of source builds.


python312Packages.solarlog-cli: 0.1.6 -> 0.2.0

Description of changes
Diff: https://github.com/dontinelli/solarlog_cli/compare/refs/tags/v0.1.6...v0.2.0
Changelog: https://github.com/dontinelli/solarlog_cli/releases/tag/v0.2.0

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 337971 run on x86_64-linux 1

15 packages built:

home-assistant-component-tests.solarlog
python311Packages.curio-compat
python311Packages.curio-compat.dist
python311Packages.pytest-aio
python311Packages.pytest-aio.dist
python311Packages.returns
python311Packages.returns.dist
python312Packages.curio-compat
python312Packages.curio-compat.dist
python312Packages.pytest-aio
python312Packages.pytest-aio.dist
python312Packages.returns
python312Packages.returns.dist
python312Packages.solarlog-cli
python312Packages.solarlog-cli.dist


Remove target="_blank" from links

Can I convince you to remove target="_blank" from all the links on the site? :)
Forcing links to open as new tabs is equivalent to pop-ups in the browser security model, so I'd have to add allow-popups to my CSP sandbox directive to use this theme. Plus target blank is just bad UX. No worries if you don't want to merge this, it would just save me the trouble of having to maintain my own fork when yours already does everything else I want my PaperMod site to do.

Agreed.


Updated main.tf.j2 as per new terraform updates.

Updated the file with the required_providers block. The version meta-argument needs to be included here post terraform version 0.13. Will also be updating the file provider.tf.j2.

Please fix also errors from failing Azure job in CI.

Please fix also errors from failing Azure job in CI.

Done.


Allow wheel events in application mode

This change allows wheel events in application mode which fixes mouse wheel
scrolling in oh-my-zsh and powershell for Linux (among others). Along with #287
a functional scroll bar will also be usable in those shells.
Fixes #151

There was quite a lot of discussion in the thread, but it seems this simple change with minimal side effects will fix it. One counter example used in #151 is that vim will call scrollDisp and cause the viewport to get broken, this is actually false since vim seems to clear the buffer.

This makes sense. Application keypad modes should only affect the cursor or numeric keypad key behavior; mouse behavior shouldn't be affected by them. This should make xterm.js behave better when running Oh My Zsh, or zsh from one of the Linux distributions which sets application keypad mode in its system config files.


Quality of favicons used for home screen shortcuts

First of all, I am very happy with the new feature to add shortcuts of websites to the home screen (#224). However, the quality of the icons used for the shortcuts is quite low for some sites. Of course not always a high quality icon is available, or it is not marked correctly on the website.
I am not sure what the process is of producing the icon for the shortcut, because the icons are fetched from https://icons.duckduckgo.com/ip3/%s.ico , but maybe you can try to fetch the best quality image available. Even if it is marked as apple-touch-icon.
For example for the website Rijnmond.nl this icon is used  when also this one is available on their website  .

Thanks for reporting this. We have logged this internally and will fix it there. Thanks again!

Internal link: https://app.asana.com/0/715106103902962/1115742071228982


Where are the .bat files

Cant seem to find them ? Can author update the readme.

LoadCapcom.bat

@echo off
copy "%~dp0\Capcom.sys" "%WINDIR%\system32\drivers\Capcom.sys"
sc create Capcom binPath= system32\drivers\Capcom.sys type= kernel
sc start Capcom
echo Driver loaded, fire up the exploit now then press a key when exploit has been done.
echo If you see any access denied, close this and relaunch the bat as Administrator.
echo DO NOT PRESS ANY KEY UNTIL YOU HAVE FINISHED LAUNCHING THE EXPLOIT
pause
PING localhost -n 2 >NUL
sc stop Capcom
sc delete Capcom
del "%WINDIR%\system32\drivers\Capcom.sys"
PING localhost -n 2 >NUL
sc stop Capcom
sc delete Capcom
del "%WINDIR%\system32\drivers\Capcom.sys"
PING localhost -n 2 >NUL
echo Cya
pause

LoadUnsignedDriver.bat

"%~dp0\drvmap.exe" "%~dp0\KsDumperDriver.sys"

check the release page https://github.com/EquiFox/KsDumper/releases/tag/v1.1 @xmenxwk


Improve typing of custom scalars

Related to https://github.com/graphql-nexus/schema/issues/360
Currently the typing of custom scalars is not great. But it is already possible to pass rootTyping to the scalarType which adds the type to the NexusGenRootTypes but then it does not use it when we actually use the type. We can use the parameter backingTypeMap, but it feels unnecessary since we already provide the root typing.
So my proposition is two fold:

Use the rootTyping when available
Accept a root typing in the method asNexusMethod

I realized that backingTypeMap also adds to the NexusGenRootTypes, but it does not use it when defining the fields.


Support remote observing

See https://github.com/Senzing/go-observing/issues/22

Pull requests:

https://github.com/Senzing/senzing-tools/pull/47


NDRS-763: Cleanup of linear_chain_sync.rs

Small clean-up of lienar_chain_sync.rs file:

Extracts ReactorEventT trait to a file,
Extracts State struct to a file.
Wraps peer-related information with a PeersState struct and extracts it to a file.
Adds peer to a DeploysNotFound result (follows the pattern of other results in the component).
Adds GetDeploysResult enum variant to follow the pattern used when downloading blocks.

Recommend reviewing commit-by-commit.

bors merge

bors merge

bors r+

bors r+


Authentication with social media

I'm using supabase as a backend and I'm so happy to use but I want to implement authentication with social media.
Can you please implement this feature?
It would be great

Hi @akram-95 !
You can actually sign users in via social account today!
You can follow the steps here to implement deeplink. Then you can use the auth.signInWithProvider() method available on supabase_flutter to sign your users in.

@dshukertjr , but how can I use this method if I'm already using Supabase-dart?

@akram-95
You could start using supabase_flutter!
supabase_flutter contains everything that supabase_dart contains, so migration should not be that hard!

Yeah but the point supabase_flutter doesn't yet support null safety

@akram-95
supabase_flutter does support null safety. You can read more about it here, but supabase_flutter not having the null-safety badge on pub.dev is just a bug on Flutter and index.db! You can safely and confidently migrate to supabase_flutter!

@dshukertjr Ok thanks


Check memory of tproc is not filled, raise informative error

I tried to run standard RB (with a non-calibrated qubit, just to see if the memory problem was solved) and it's not.
As I explained some time ago, there are two memories that can cause problems if filled: the DAC memory (that stores waveforms) and the tproc memory (that store the program).
I solved the DAC memory problem for a standard program that doesn't use too many different waveforms, but the tproc still has the same limits.
Note that the problem affects the two boards in very different ways: the RFSoC4x2 has 1024 memory limit (around 70 pulses), while the ZCU111 has a 8192 memory (around 560 pulses).
I have an idea to increase the maximum number of pulses, but in qibosoq it would probably appear as a weird patch. I am thinking about directly opening an issue and PR in qick (but maybe in the future).
In the moment, I added a check for the tproc memory so that eventually a much more informative error gets raised.
Checklist before merge:

[ ] Reviewers confirm new code works as expected.
[ ] Tests are passing.
[ ] Documentation is updated.

I have an idea to increase the maximum number of pulses, but in qibosoq it would probably appear as a weird patch. I am thinking about directly opening an issue and PR in qick (but maybe in the future).

Working directly on Qick will open a lot of options (including a bit of refactoring of their code...), but maybe this one in particular might not be a priority.
My current suggestions would be to start maintaining a list of potential improvements. Then, at some point we can decide to commit to that, and we will have a broader look, and make some planning to establish priorities (after that you can do whatever you feel more relevant by yourself... as you can do even now, but diving into someone else's code is expensive, that's why I'd recommend a bit of planning ahead of time).

Working directly on Qick will open a lot of options (including a bit of refactoring of their code...), but maybe this one in particular might not be a priority.
I think that they (or he, since there is basically a single contributor) are trying to do some refactoring

My current suggestions would be to start maintaining a list of potential improvements. Then, at some point we can decide to commit to that, and we will have a broader look, and make some planning to establish priorities (after that you can do whatever you feel more relevant by yourself... as you can do even now, but diving into someone else's code is expensive, that's why I'd recommend a bit of planning ahead of time).

Yes I agree, because of that I will not do anything for now :-)
But sooner or later, maybe when qibosoq will have all the supported features implemented, I think we should start working also on that. To speed up the development


chore: dont get plain state if not shard is there

Small optimization, if there are no history shards that means that there is not changes and we can return default values and not query the PlainStorage/PlainStorage.

Codecov Report

Merging #2959 (5874b49) into main (32772ca) will decrease coverage by 53.90%.
The diff coverage is 0.00%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
@@             Coverage Diff             @@
##             main    #2959       +/-   ##
===========================================
- Coverage   71.18%   17.28%   -53.90%
===========================================
  Files         513      520        +7
  Lines       66657    67121      +464
===========================================
- Hits        47451    11605    -35846
- Misses      19206    55516    +36310

Flag
Coverage Δ

integration-tests
17.28% <0.00%> (-0.12%)
:arrow_down:

unit-tests
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...storage/provider/src/providers/state/historical.rs
0.00% <0.00%> (-86.17%)
:arrow_down:

... and 384 files with indirect coverage changes


Integers do not undergo implicit casting in function calls

Unsure if this is by design or not, but:
def test(value : UInt64)
    pp value
end
test 0_u8

results in
no overload matches 'test' with types UInt8
Overloads are:
 - test(value : UInt64)

As this is a strict up-cast possible with any possible value, I argue that this should be implicit.
Similarly, the inverse should be true if using an integer literal that fits in the new type:
def test(value : UInt8)
    pp value
end
test 0_u64

Most of my problems with this arise from the use of an untyped constant (i.e. test 42); these constants default to Int32, which means I have to specify the type each time I want to call a function that takes a different type.

I think automatically upcasting is "fine" as it doesn't really change anything, but how would you deal with something like this?
# Somewhere above your code you get a number that is > UInt8::MAX
some_var = UInt16::MAX

def test(val : UInt8)
  pp val
  # for the current example (given it would just cast to UInt8) it would print 255
  # but you've put in a larger number, so you'd most likely wonder what happened to it.
end

test some_var

What would the desired behavior be? Simply casting to down? I think the user should be aware that he might be doing an invalid operation and should decide what to do, so either noticing "Hey I got a larger number than my function can deal with" or "I don't care if it's higher".

I agree - that shouldn't be valid behaviour. However, when the compiler can prove that the value fits within the range (i.e. an integer literal between 0 and 255), it should be implicitly casted.

Having this special case for just numbers would seem weird to me. Generalizing it would mean automatic conversions (this already is a conversion conceptually, that it can be achieved through casting is an implementation detail) and thus weak typing. And I hope we all can agree that weak typing isn't to prefer.

I've proposed earlier, and will again, that literals should be weakly typed.

@ozra so "1" + 2 == '3' should be true?

The compiler looks up method matches by first inferring the type of call arguments. Having an exception for number literals is bad, as with almost any exception. Also new errors start to appear:
def foo(x : Int64)
end

def foo(x : Int8)
end

foo 0 # Matches both, a new error message now appears

How often do you need to use UInt64 and invoke methods with literals? Also, you could do this:
def test(value : Int)
   value = value.to_u64
end

test 0 # now works

If you want to make sure that no negative values are passed, maybe add a check and raise. That makes it a runtime error instead of a compile-time one, but the only reason we have unsigned types is for C interoperability, unsigned types are discouraged from regular code. They are bug-prone and unintuitive.

My current project requires a fair amount of C interop. I believe your first example (two functions accepting ints) should be an error anyway - it's clearly ambiguous, and the behaviour will change depending on which platform you're compiling for. Your suggested workaround doesn't work for calling C methods unless they're wrapped, and it pokes holes in the type system (by throwing away all information about sizes).
This isn't a blocker by any means, but it is counter-intuitive behaviour (you'd expect 0 to be accepted by any method taking any variant of [U]IntX).

What is the C library you are interfacing with? Ruby also interfaces with C and it doesn't have unsigned  numbers. They probably use similar checks to the ones I propose, and in practice it works well.

Types are casted automatically when calling C functions, to avoid manually having to cast to the many LibC types (eg: LibC::Char, LibC::SizeT, etc.), and the following works:
lib LibX
  fun foo(x : LibC::UInt) : LibC::Int
end

LibX.foo(0_i8)   # ok
LibX.foo(0_u16)  # ok

This way you don't have to worry about (un)signed integers in Crystal code.

@jhass, I was thinking of within "SomeInt" fitting in target data-type only... Since it's a literal fitness test is compile time...

This is something that for simpler integer literals can seems to be very easy to implement with a few special cases in the compiler, but can quickly get out of hands.
For example, implicit conversion from Int8 to Int16 should probably happen. But what about Int32 to BigInt? One would probably want that too. Then we'd need to improve the language with custom conversion rules. Julia does this, and we could maybe do this in the future. On the other hand being explicit is simpler and always understandable, with implicit casts one could transform any object to any other objects, not necessarily numeric types. And with anything "implicit", it makes things harder to follow and understand.
There's also the case where maybe I want to have an array of Int32 and Int8, for some reason. Would the compiler not let me do this because Int8 is implicitly castable to Int32? What happens when a type is implicitly castable to two types inside a union? The language will quickly get out of control.
So I'll close this for now.


Unable to to customize page component tabs with Sling Resource Merger

Before #114 (up to core components 2.0.0) all page properties tabs (e.g. "basic", "advanced") were defined as part of the core components project. thus it was easy to customize the page properties dialog by hiding some tabs or properties - example:
  "cq:dialog": {
    "jcr:primaryType": "nt:unstructured",
    "content": {
      "items": {
        "tabs": {
          "items": {
            /* hide some tabs from page properties dialog */
            "sling:hideChildren": ["thumbnail","socialmedia","personalization","cloudservices"],
            /* hide some properties inside the 'basic' tab of page properties dialog */
            "basic": {
              "items": {
                "column": {
                  "items": {
                    "title": {
                      "items": {
                        "sling:hideChildren": ["tags","tagsShowOnCreate"]
                      }
                    },
                    "sling:hideChildren": ["vanityurl"]
                  }
                }
              }
            },
...

staring with release 2.0.2 because of #114 this is no longer possible:

it is still possible to hide tabs
but it is no longer possible to customize the built-in tabs (hiding fields, adding new fields to them etc.)

the reason is that since 2.0.2 the page component no longer declares the built-in tabs itself, but rather inherits them from it's supertype wcm/foundation/components/basicpage/v1/basicpage.
Unfortunately this basicpage component does not define the content of the tabs directly, but itself includes them via granite/ui/components/coral/foundation/include. This inclusion mechanism breaks the possibility to customize the tabs with Sling Resource Merger.
i know that the basicpage is part of AEM and not part of the core components - i still consider it as bug because the expected behavior was present in core components till 2.0.0 - and in my eyes this was a big benefit compared to the built-in page components that shipped with AEM in the past (e.g. wcm/foundation/components/page). now we are back to this problematic dialog tabs includes.

This was intended, as we consider the contents of each tab private and thus not customisable. We tried to document that at https://github.com/Adobe-Marketing-Cloud/aem-core-wcm-components/wiki/Versioning-Policies#dialogs.
One can either include/hide (some of) the built-in tabs - which might change in the future to support new/changed product features - or define their own tabs. I know this feels like we're taking away some of the flexibility but it was necessary to be able to evolve the product without breaking all possible customisations later on.

ah, thanks for the info - so this change was intentional.
a bit sad and more effort for application on the one side - but i understand the reasons behind it on the other side.


"GET / HTTP/1.1" 404 Not Found

I try to follow your "quick start" and launch a server, with following code:
python -m sglang.launch_server --model-path mistralai/Mistral-7B-Instruct-v0.1 \
--port 30000 --host <IP_ADDRESS>

Unfortunately I encounter some error😭
[2024-12-13 01:44:59 TP0] Load weight end. type=MistralForCausalLM, dtype=torch.bfloat16, avail mem=9.69 GB
[2024-12-13 01:44:59 TP0] Memory pool end. avail mem=2.44 GB
[2024-12-13 01:44:59 TP0] Capture cuda graph begin. This can take up to several minutes.
[2024-12-13 01:45:01 TP0] Capture cuda graph end. Time elapsed: 2.19 s
[2024-12-13 01:45:02 TP0] max_total_num_tokens=56524, max_prefill_tokens=16384, max_running_requests=2049, context_len=32768
[2024-12-13 01:45:02] INFO:     Started server process [3831493]
[2024-12-13 01:45:02] INFO:     Waiting for application startup.
[2024-12-13 01:45:02] INFO:     Application startup complete.
[2024-12-13 01:45:02] INFO:     Uvicorn running on http://<IP_ADDRESS>:30000 (Press CTRL+C to quit)
[2024-12-13 01:45:26] INFO:     <IP_ADDRESS>:40418 - "GET / HTTP/1.1" 404 Not Found
[2024-12-13 01:47:34] Initialization failed. warmup error: Traceback (most recent call last):
  File "/home/sglang/python/sglang/srt/server.py", line 621, in _wait_and_warmup
    assert res.status_code == 200, f"{res=}, {res.text=}"
AssertionError: res=<Response [502]>, res.text=''

Killed

How can I fix it? Thanks!

@LordEdison Could you change your port plz? And, provide us with your environment.

@LordEdison Hey. Have you tried this? We will close your issue in three days if you do not reply.


Installation Guide may be outdated

As I was following the installation guide found here: https://intelowlproject.github.io/docs/GreedyBear/Installation/ I came across the line
cp frontend/public/env_template.js frontend/public
I believe this file is missing from the repository

yep, you can ignore it, thanks for pointing out, we'll change the doc


Autorotating secrets do not clear the secrets file

It seems the new auto secrets mechanism can create garbage secret files. When writing a new secrets file it may need to be unlinked before. The problem is in https://metacpan.org/dist/Mojolicious-Plugin-AutoSecrets/source/lib/Mojolicious/Plugin/AutoSecrets.pm - so it needs to be fixed there.

Although I think AutoSecrets is problematic in that regard, a way to mitigate the problem is to have secrets no longer than 22 characters (default length in Session::Token). The garbage can happen when secrets are longer than 22 and they are part of the secrets file when the automatic migration happen - that way after the first full round of rotation, the extra characters are not truncated. By fixing the migration, this error won't occur anymore for new migration. The fix for already migrated instances is to manually truncate the JSON secret file.


New query builder

Two main reasons:

improve maintainability of the query builder code (code that translates FHIR Search requests into JDBC select statements)
improve the performance of select statements (see related issues)

Here is a performance comparison between 4.7.1 and current
471-latest.xlsx


Add possibility to customize docs path

Description
Adds the possibility to customize the path of documentation.
Type of change
Please delete options that are not relevant.

[x] New feature (non-breaking change which adds functionality)

How Has This Been Tested?
See the new playground file
Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] I have updated the CHANGELOG in the Unreleased section
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] Any dependent changes have been merged and published in downstream modules

Good idea, thanks for the PR :)


Add one-step presto upgrade to presto-admin

This pull request adds one-step upgrade for prestodb to presto-admin. In order to support this, it adds functionality to gather the existing presto configuration from the hosts in the cluster to a local directory, and to deploy configuration to the cluster from a directory. This ensures that the config on the cluster remains the same across an upgrade in the following situations:

A user has changed the configuration on the cluster outside of presto-admin
prestodb was installed and configured manually, and presto-admin has no config for the cluster.

The upgrade process works as follows:

Gather the cluster config to a local directory.
Uninstall presto from the cluster.
Install the new presto package on the cluster.
Deploy the previous configuration from the local directory.

@cawallin, @rschlussel please review.

Could you add product tests too?

I'll be out of the office for several weeks. The work that remains to be done before pushing is as follows:

Product tests. Per discussion with @cawallin and @rschlussel, we'd like to add a test that upgrades to a mostly-empty dummy rpm of presto-server. This is to avoid pulling another ~400 MB .rpm over the network just for the sake of a test. The dummy rpm and subsequent verification should ensure that the following scenarios are tested:

a file is added
a file is removed
a file is modified
I found the following guide to creating rpm packages, but didn't make any headway on creating the dummy package: https://fedoraproject.org/wiki/How_to_create_an_RPM_package. A forum post I read advised against creating .rpms on Ubuntu because of possible incompatibilities. I haven't gotten far enough to confirm this, but rpm -qpi presto-server-rpm... returns an error on Ubuntu, which suggests that the warning has merit.

The change to use rpm -U is currently broken because rpm -U installs the new package, then uninstalls the previous package. This would work, but the postuninstall script for the package unconditionally removes /etc/presto and /var/lib/presto. There are comments in the script explaining the rationale for the behavior. /etc/presto can be recreated before we push the saved config back out to the cluster, but we should probably look into the behavior of the postuninstall script and see if it should be updated to do something different.

@rschlussel -- can you review please?

noteworthy changes:

there is now a default temporary config storage place
connectors are saved as well
The fabric_api by rolename method was no longer necessary, so I removed it

lgtm


Improve wallet error message

Hi all 👋
This project is looking like its off to a good start 🎉!
I tried to quickly test things out by deploying a wallet. One issue I found myself stuck for a few minutes was the wallet - it wasn't immediately clear I needed to run sui genesis first, and I thought something was wrong with my installation.
I made a quick change to wallet by using anyhow::Context (which the project already uses) to add a helpful error message that could help save people some time in the future.

Good catch! We are trying to handle it more comprehensively in #1057, though. Let's keep this one open until an actual fix lands, one way or the other.

Sounds good.
Having a .sui directory in $HOME seems like a good idea. I'd also suggest maybe renaming wallet to sui_wallet, but that's a personal preference...

I would not merge it "as is" as the help message is no longer accurate after https://github.com/MystenLabs/sui/pull/1057 landed. You still have to run genesis, but you don't have to run wallet in the same directory where genesis was ran.

@no2chem - thank you for the contribution (and your kind words!) but since the original problem is mitigated I am going to close this PR.


Issue fix: onConnect callback doesn't get called if appropriate delegate with -websocketDidConnect: method not assigned.

Issue fix: onConnect callback doesn't get called if appropriate delegate with -websocketDidConnect: method not assigned.

Ah good catch, not sure how we missed that. Thanks!

My pleasure.
Thanks for a powerful library.


Add more checks for beacon client

[x] Fixes: SNO-388
[x] Fixes: SNO-391
[x] Fixes: CI breaking for tests with benchmark feature
[x] More unit tests
[x] E2E test

@claravanstaden  previously CI fails for unit test and with some fix/refactoring in this PR should be fine now.
There are still several benchmark tests for beacon light client fails as SignatureVerificationFailed, guess mock data for mainet outdated?
so I temporarily change to use minimal data for these tests and would assume using either should be fine as unit test only.
The coverage decreases a little  because some more checks added,  since we're still doing more refactoring so will ignore and back to it later

@yrong Could you rather tackle SNO-328 in a separate PR please. Makes it easier to review the changes that way.

Hey @yrong! 😄

There are still several benchmark tests for beacon light client fails as SignatureVerificationFailed, guess mock data for mainet outdated?

I quickly checked the benchmark tests for mainnet on the main branch and they look OK to me:
parachain git:(main) ✗ ./target/release/snowbridge benchmark pallet --chain /tmp/snowbridge/spec.json --execution=wasm --pallet ethereum_beacon_client --extrinsic "*" --steps 1 --repeat 1 --output pallets/ethereum-beacon-client/src/weights.rs --template templates/module-weight-template.hbs
2023-02-09 10:35:20 💫 Sync committee size is: 512
2023-02-09 10:35:20 💫 Received initial sync, starting processing.
2023-02-09 10:35:20 💫 Initial sync processing succeeded.
2023-02-09 10:35:20 assembling new collators for new session 0 at #0
2023-02-09 10:35:20 assembling new collators for new session 1 at #0
Starting benchmark: ethereum_beacon_client::sync_committee_period_update
2023-02-09 10:35:21 💫 Received initial sync, starting processing.
2023-02-09 10:35:21 💫 Initial sync processing succeeded.
2023-02-09 10:35:21 💫 Received sync committee update for period 547. Applying update
2023-02-09 10:35:21 💫 Sync committee period update for period 547 succeeded.
2023-02-09 10:35:21 💫 Received initial sync, starting processing.
2023-02-09 10:35:21 💫 Initial sync processing succeeded.
2023-02-09 10:35:21 💫 Received sync committee update for period 547. Applying update
2023-02-09 10:35:21 💫 Sync committee period update for period 547 succeeded.
2023-02-09 10:35:21 💫 Received initial sync, starting processing.
2023-02-09 10:35:21 💫 Initial sync processing succeeded.
2023-02-09 10:35:21 💫 Received sync committee update for period 547. Applying update
Starting benchmark: ethereum_beacon_client::import_finalized_header
2023-02-09 10:35:21 💫 Sync committee period update for period 547 succeeded.
2023-02-09 10:35:21 💫 Received initial sync, starting processing.
2023-02-09 10:35:21 💫 Initial sync processing succeeded.
2023-02-09 10:35:21 💫 Received finalized header for slot 4485216.
2023-02-09 10:35:21 💫 Checking weak subjectivity period. Current time is :0 Weak subjectivity period check:<PHONE_NUMBER>.
2023-02-09 10:35:21 💫 Stored finalized beacon header at slot 4485216.
2023-02-09 10:35:21 💫 Received initial sync, starting processing.
2023-02-09 10:35:21 💫 Initial sync processing succeeded.
2023-02-09 10:35:21 💫 Received finalized header for slot 4485216.
2023-02-09 10:35:21 💫 Checking weak subjectivity period. Current time is :0 Weak subjectivity period check:<PHONE_NUMBER>.
2023-02-09 10:35:21 💫 Stored finalized beacon header at slot 4485216.
2023-02-09 10:35:21 💫 Received initial sync, starting processing.
2023-02-09 10:35:21 💫 Initial sync processing succeeded.
2023-02-09 10:35:21 💫 Received finalized header for slot 4485216.
2023-02-09 10:35:21 💫 Checking weak subjectivity period. Current time is :0 Weak subjectivity period check:<PHONE_NUMBER>.
Starting benchmark: ethereum_beacon_client::import_execution_header
2023-02-09 10:35:22 💫 Stored finalized beacon header at slot 4485216.
2023-02-09 10:35:22 💫 Received initial sync, starting processing.
2023-02-09 10:35:22 💫 Initial sync processing succeeded.
2023-02-09 10:35:22 💫 Received header update for slot 4485185.
2023-02-09 10:35:22 💫 Stored execution header 0x2802…cbbe at beacon slot 4485185.
2023-02-09 10:35:22 💫 Received initial sync, starting processing.
2023-02-09 10:35:22 💫 Initial sync processing succeeded.
2023-02-09 10:35:22 💫 Received header update for slot 4485185.
2023-02-09 10:35:22 💫 Stored execution header 0x2802…cbbe at beacon slot 4485185.
2023-02-09 10:35:22 💫 Received initial sync, starting processing.
2023-02-09 10:35:22 💫 Initial sync processing succeeded.
2023-02-09 10:35:22 💫 Received header update for slot 4485185.
Pallet: "ethereum_beacon_client", Extrinsic: "sync_committee_period_update", Lowest values: [], Highest values: [], Steps: 1, Repeat: 1
Raw Storage Info
========
Storage: EthereumBeaconClient Blocked (r:1 w:0)
Storage: EthereumBeaconClient SyncCommittees (r:1 w:1)
Storage: EthereumBeaconClient ValidatorsRoot (r:1 w:0)
Storage: EthereumBeaconClient LatestSyncCommitteePeriod (r:1 w:0)
Storage: EthereumBeaconClient LatestFinalizedHeaderState (r:1 w:0)
Storage: EthereumBeaconClient FinalizedBeaconHeaders (r:0 w:1)

Can you let me know if you get a different result? You must just make sure that the snowblink or snowbridge runtime is being used for the mainnet benchmarks, as the snowbase runtime won't work (smaller sync committee). I'd prefer if we keep using the mainnet data and if there is a problem with the data we should rather fix it.
I also tested the beacon client unit tests for minimal and mainnet using the following commands:
cd parachain/pallets/ethereum-beacon-client
Minimal: cargo test --features="minimal"
Mainnet: cargo test"

I quickly checked the benchmark tests for mainnet on the main branch and they look OK to me
Can you let me know if you get a different result?

Ah, seems my comment above is a little confusing.  The benchmark tests here I mean actually are unit tests with runtime-benchmarks features which same as what defined in CI, and I run all test command including this feature as following:
cd parachain/pallets/ethereum-beacon-client
cargo test --features="runtime-benchmarks"
cargo test --features="runtime-benchmarks" --features="minimal"

this PR just make it work

@yrong OK cool, thanks! I'll have a look why they are failing. 🤔 I'd rather we fix them than changing the mainnet tests to use the minimal data spec, preferably.

I'd rather we fix them than changing the mainnet tests to use the minimal data spec, preferably.

Yes,Agree. After some furthur tests mainnet data just work fine so I won't change it either. just seperate benchmark_test and data spec to make it a bit cleaner here
https://github.com/Snowfork/snowbridge/blob/7affb7738d78cde5cd49972601db965159d1a7a5/parachain/pallets/ethereum-beacon-client/src/benchmarking/mod.rs#L102-L114

@yrong awesome, thank you! I will do a final review today.

with e2e setup, seems the sync flow works as expected

@yrong awesome! I'll do a final check tomorrow, then we can merge. 😄

Hey @yrong! I ran the start-services.sh script to do a final test on this PR, but ran into the following error:
Address: {89b4ab1ef20763630df9743acf155865600daff2}
Waiting for geth API to be ready
Starting beacon node
Starting lodestar local node
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1523  100  1435  100    88   216k  13580 --:--:-- --:--:-- --:--:--  743k
Deploying contracts
~/IdeaProjects/snowbridge/core/packages/contracts ~/IdeaProjects/snowbridge/core/packages/test
Error:
Failed to resolve file: "/Users/claravanstaden/IdeaProjects/snowbridge/core/packages/contracts/lib/forge-std/src/Script.sol": No such file or directory (os error 2).
 Check configured remappings..
    --> "/Users/claravanstaden/IdeaProjects/snowbridge/core/packages/contracts/contracts/deploy/foundry/Deploy.sol"
        "forge-std/Script.sol"
Killing chains

I'm not sure if this issue is in any case obsolete after the Nix changes? 🤷♀ Perhaps we should rebase from main on the branch and then I'll test with Nix.

Error:
Failed to resolve file: "/Users/claravanstaden/IdeaProjects/snowbridge/core/packages/contracts/lib/forge-std/src/Script.sol": No such file or directory (os error 2).

I'm not sure if this issue is in any case obsolete after the Nix changes? 🤷♀ Perhaps we should rebase from main on the branch and then I'll test with Nix.

@claravanstaden I've synced with main branch and resolve the conflicts and I test the nix setup should be fine. The issue you met is some forge library missing and could be resolved with running git submodule update --init in root directory.


End User Scale Service error

When the user has no role(default granted user)  select options scale or modify parameters of a deployed service, the UI client will call the API /xpanse/service_templates/{serviceTemplateId} which need users have role ISV and return 403 error, this cause the page error.
Modify parameters of the service error:

Scale the service error:

This error must be fixed. There are two solutions as below:

The UI Client change to call the API /xpanse/catalog/services/{serviceTemplateId} which is for the end users in all pages for the end users.
In the back-end, add role User into the needed roles of the API /xpanse/service_templates/{serviceTemplateId}.

@baixinsui we must use the API which is specifically for end user. This must be fixed on the UI.

@swaroopar I tried to replace the API in the UI client but there are other errors. I could not find out the caused. Can you fix it in the UI Client or assign it to other partners.

@Alice1319 could you please check this bug? We are using a wrong API to fetch service template details.

Sure,I'll check this


Create a beginner's tutorial for the Dash UI

Create a beginner's tutorial for the Dash UI that is shown upon loading in hub:
Walk the user through the initial process of creating a repo, uploading their data, creating a pipeline, reiterating, processing the results.

@JoeyZwicker @svekars: re-prioritizing this based off conversations today. The short of it is that we think we'll need this in order for dash to be digestible for hub users, some of whom will have no prior experience using pachyderm.

I think that this is pretty doable for end of Sept or early Oct (before we demo/present hub publicly). It might not be full "beginner tutorial in Hub" but we can certainly scope out some specifics to make sure first-time pachyderm users accessing things through Hub can have a good first experience.
We can write the whole doc with current screenshots and then just replace all of those as the UI develops. There's a little bit of chicken-and-egg here, but it's manageable and something the customer team can probably handle managing to make sure Hub team doesn't have to spend extra work on this. @svekars does that seem ok to you or any major issues that I'm not thinking of?

Yes, @JoeZwicker, looks good.


Convolution updateWeights function needs to be modified.

open func updateWeights(device: MTLDevice) {
        guard let network = network else {
            return
        }

        if #available(iOS 11.0, *) {
            if let weightsPointer = weightsPointer {
                dataSource = ConvolutionDataSource(cnnDescriptor: cnnDescriptor,
                                                   weights: UnsafeMutableRawPointer(mutating: weightsPointer.pointer()),
                                                   bias: UnsafeMutablePointer(mutating: biasPointer?.pointer() as UnsafePointer<Float>?))
            } else {
                dataSource = ConvolutionDataSource(cnnDescriptor: cnnDescriptor, parameterLoader: network.parameterLoader,
                                                   layerId: id, weightCount: getWeightsSize(), biasCount:  convSize.outputChannels)
            }
            makeConv(device: device, weights: nil, bias: nil)
        } else {
            let weights = weightsPointer?.pointer() ?? network.parameterLoader.loadWeights(for: id,
                                                                                modifier: Convolution.weightModifier,
                                                                                size: getWeightsSize())

            var bias: UnsafePointer<Float>? = nil
            if useBias {
                bias = biasPointer?.pointer() ?? network.parameterLoader.loadWeights(for: id,
                                                                                modifier: Convolution.biasModifier,
                                                                                size: convSize.outputChannels)
            }
            makeConv(device: device, weights: weights, bias: bias)
        }
    }

i think that "ConvolutionDataSource" code will be modified.
This code does not consider whether to use bias or not.
So I suggest the following code.
else {
                dataSource = ConvolutionDataSource(cnnDescriptor: cnnDescriptor, parameterLoader: network.parameterLoader,
                                                   layerId: id, weightCount: getWeightsSize(), biasCount: useBias ? convSize.outputChannels : 0)
            }

Thank you.

It seems you are right. @mats-claassen can you take a look when you are back?


Feature Request - Native custom check scheduler

This is a feature requests for the implementation of a feature to allow scheduling the execution of custom metric checks via the splunk-otel-collector.
For example, with the Datadog agent (which in many ways is the Datadog equivalent of the splunk-otel-collector), it comes with a built in cron-like ability to execute custom checks on a schedule. So for example, you create a python scripts to retrieve/generate a metric value, and the agent sends a JSON payload containing metric name and value to the REST API based on the frequency you decide.
With the splunk-otel-collector, at the moment, we are executing our custom metric python script via the hosts native cron. Cron trigger the script, the script generates the metric and builds a payload, sends the payload to the splunk-otel-collector http exporter (http://<IP_ADDRESS>:4318/v1/metrics), which in turn send it to Splunk Observability. This works ok, but it would be nicer to be able to schedule this via the splunk-otel-collector, rather than relying on the systems cron.

Thank you for entering this issue.
Would this exclusively call out to a REST API? How often are you looking to poll?

I am currently only looking to call a REST API, however I was thinking of a feature which would be more generic, allowing users to execute any checks/scripts that generate metrics in the correct format, so not necessarily only for polling endpoints.
Looking to poll every minute, but finer checks would also be good.

I have filed https://ideas.splunk.com/ideas/SFXIMMID-I-486 to follow up. Let's continue the conversation in Splunk Ideas if you don't mind.


Closed-ended classifiers: Inconsistency in handling blank strings

Blank strings/ strings that contain none of the feature words are currently handled differently by the three closed-ended classifiers.
For form and target, such strings are predicted to belong to the most common class in the training set (rally/ demonstration and domestic government, respectively). For issue, they are classified as none, which is not the most common class in the training set.
See page 19 of Alex's thesis chapter 2, and the following example code
import pandas as pd
from mpeds.classify_protest import MPEDS

test_classifier = MPEDS()
test_data = pd.Series(['', 'avocados and grapefruits'])

test_classifier.getIssue(test_data)
test_classifier.getForm(test_data)
test_classifier.getTarget(test_data)

Oh, that seems bizarre. We should probably return a Nonetype if this is the case and throw a warning that says something like "No words found in vectorizer."


Terraform Cloud Provisioner Tutorial

Summary
New tutorial on "Terraform Cloud Provisioning"
What Type of PR is This?

[x] Doc/Tutorial

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app

This PR has been automatically closed due to inactivity.


ble device disconnected for long duration: App goes to sleep

Summary

I created a stackoverflow question for this issue (http://stackoverflow.com/questions/40903276/rxandroidble-ble-device-disconnected-for-long-duration-app-goes-to-sleep),
but I'm not sure if it belongs there or in this issue queue,  so I thought I'd put it here too:
I  am using rxandroidble using autoconnect = true to continuously monitor data from a sensor. The app continuously scans for sensors it has previously connected to.
The data monitoring and scanning for sensors should continue all night, even if the phone is not connected to a power source.
If a sensor is connected in the evening, the app will stay connected all night, even if it disconnects momentarily.
However, if a sensor is disconnected for 6 hours during the night (because I pulled the sensor's battery), and then I reconnect the sensor battery in the morning, the phone does not appear to reconnect to the sensor.
I am constantly scanning every 8 seconds for new sensors in an Android service, but its not a WakefulService, nor is it started by a WakefulIntent. Should it be?
Any idea what could be going on? Is rxandroidble designed to keep scanning for sensors in this situation (ble device connects, goes out of range for 8 hours, then comes back into range)? Or do i need to manually call reconnect after the disconnect, and then rxandroid will continually attempt to reconnect.
See stack overflow link for code.
http://stackoverflow.com/questions/40903276/rxandroidble-ble-device-disconnected-for-long-duration-app-goes-to-sleep
Preconditions
 autoConnect=true always
Asus Zenfone 3 Ultra
Steps to reproduce actual result

Open app
Scanning will pick up ble sensor and connect to it
move ble sensor out of range  (or disconnect sensor battery)
Wait 8 hours - Phone is not connected to power, is immobile
bring ble sensor back into range (or reconnect sensor battery)
App should automatically reconnect, and then play a sound and send reconnect event data to server, however, none of this occurs

Actual result
 App does not reconnect to sensor
Expected result
 App should play a sound and send reconnect event data to server upon reconnect
Let me know what else i need to provide....
Many thanks!

Hello @swilsonian
I have posted an answer on the Stackoverflow.
Long story short - the library works as intended. After the connection being established and closed it won't be reopened without a call to RxBleDevice.establishConnection().
Best Regards


chore(main): release 1.2.0-alpha

:robot: I have created a release beep boop
1.2.0-alpha (2023-03-11)
Features

test releasing (53e09f8)

Bug Fixes

build: Use token for auto releasing (eb58d2f)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/evantill/plantuml-cheerpj/releases/tag/v1.2.0-alpha :sunflower:


WLError getter for "stack" breaks bluebird

https://github.com/balderdashy/waterline/blob/master/lib/waterline/error/WLError.js#L146-L148
see https://github.com/petkaantonov/bluebird/issues/552
Bluebird needs to be able to set the .stack property on WLError but is unable to do so

Looks like using __defineGetter__ is just generally a bad idea anyway: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/defineGetter

This shouldn't break bluebird as of 2.9.21

Thanks for chiming in, @petkaantonov !  We should update the WL dependency to reflect that, and also make a change here to avoid using __defineGetter__.

I already started work on this, which includes a needed refactor anyway.
Thanks for your input @petkaantonov


[BUG]Uploading large files (>= 200 MB) seems to hang when using upload(InputStream) API in DataLakeFileAsyncClient

When uploading a large file using upload(InputStream) API, the subscriber to the Mono object returned from the API never receives onComplete() or onError() calls
A clear and concise description of what the bug is.
Exception or Stack Trace
Add the exception log and stack trace if available
To Reproduce
Please refer to the example in https://github.com/masokan/test-azure-upload
Code Snippet
Code is available in the above github repository
Expected behavior
The upload should complete in a short time
Screenshots
If applicable, add screenshots to help explain your problem.
Setup (please complete the following information):

OS: Linux
IDE : [e.g. IntelliJ]
12.5.0-beta.3

Additional context
Add any other context about the problem here.
Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

[X] Bug Description Added
[X] Repro Steps Added
[X] Setup information Added

@gapra-msft PTAL

Hi @masokan
Thank you for reporting this issue, we will take a look and get back to you.

Hi @masokan
Just to update you, we have found the source of the issue. I am currently working on a fix and validating it.

@gapra-msft Thanks for your quick response on this issue.


Choose other example for linked dev on GitHub

Re. screenshot of example issue: choose/create issue with mention, link to other issue and maybe link to a commit.

With https://github.com/hbz/swib16-workshop/commit/0c3f2758d30b5e1ef5e3d10d451fb4ce9a8c3187 an example was added for linked development on GitHub that might not be the best example for the job.
I am thinking of using an issue from this repo using the full potential for self-reference. Maybe #3 or – even better – this one? What do you think @fsteeg?

I'm fine with using this issue here, though I don't see the problem with the original sample.