Delegates and generic types

If I have a delegate sub like this:
Public Delegate Sub ExecuteBusinessServiceDelegate(Of TRequest As {BsRequest, New}, TResponse As {BsResponse, New})(pBusinessService As BusinessService(Of TRequest, TResponse), pExecutionContext As BsExecutionContext)

How to do the declaration using this delegate?
Private myExecuteBusinessServiceDelegate As ExecuteBusinessServiceDelegate(……

Edit by @billwagner to add code formatting.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 1388f63d-1c05-2eb6-e505-67140b165d0c
Version Independent ID: 41ccef34-e68c-dba3-2129-9ed720197257
Content: Anonymous Functions (C# Programming Guide)
Content Source: docs/csharp/programming-guide/statements-expressions-operators/anonymous-functions.md
Product: dotnet-csharp
GitHub Login: @BillWagner
Microsoft Alias: wiwagn

@jnad72
What kind of declaration do you want to make? If you want a field, then it could be something like:
Private myExecuteBusinessServiceDelegate As ExecuteBusinessServiceDelegate(Of MyBsRequest, MyBsResponse)

Also, I'm not sure why are you looking for information about VB syntax in the C# Guide and not in the VB Guide.

@jnad72
Did the above comment answer your question? Can you tell me if the Visual Basic article on Delegates helps? If so, I'll close this issue.

Yes it did. You can close this issue. Thanks
From: Bill Wagner<EMAIL_ADDRESS>Sent: den 13 september 2018 03:12
To: dotnet/docs<EMAIL_ADDRESS>Cc: Johan Andersson (External - Capgemini Sverige AB<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [dotnet/docs] Delegates and generic types (#7187)
@jnad72https://github.com/jnad72
Did the above comment answer your question? Can you tell me if the Visual Basic article on Delegateshttps://docs.microsoft.com/dotnet/visual-basic/programming-guide/language-features/delegates/index helps? If so, I'll close this issue.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/dotnet/docs/issues/7187#issuecomment-420849252, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Aoonz0ToxLI62E3FI-hc9ELKOipWBoTMks5uabD2gaJpZM4WHgVe.

Thanks. Closing as resolved.


Support struct Data assignment operation

I have a problem using Pinocchio as a collision detection model for path planning. The path planning library I used is OMPL. According to OMPL's document, I need to implement my StateValidityChecker.
class myValidityChecker : public ompl::base::StateValidityChecker
{
public:
    myValidityChecker (const ompl::base::SpaceInformationPtr &si) :  ompl::base::StateValidityChecker(si)
    {
        pinocchio::urdf::buildModel("urdf_path", robot_model);
        pinocchio::urdf::buildGeom(robot_model, "urdf_path", pinocchio::COLLISION, robot_geom_model, "model_path");
    }

    bool isValid(const ompl::base::State *state) const override
    {
        pinocchio::Data data(robot_model);
        pinocchio::GeometryData gemo_data(robot_geom_model);
        // do collision check
        return true;
    }

    pinocchio::Model robot_model;
    pinocchio::GeometryModel robot_geom_model;
};

Because of the interface of isValid function, I can not pass pinocchio::Data. Everytime isValid called, I need to create Data  and GeometryData before collision check, which I think is time-consuming.
I also find that struct Data  has no assignment operation.
Is it a good choice for Data assignment operation:
class myValidityChecker : public ompl::base::StateValidityChecker
{
public:
    myValidityChecker (const ompl::base::SpaceInformationPtr &si) :  ompl::base::StateValidityChecker(si)
    {
        pinocchio::urdf::buildModel("urdf_path", robot_model);
        pinocchio::urdf::buildGeom(robot_model, "urdf_path", pinocchio::COLLISION, robot_geom_model, "model_path");
        //
        data = pinocchio::Data(robot_model);
        gemo_data = pinocchio::GeometryData(robot_geom_model);
    }

    bool isValid(const ompl::base::State *state) const override
    {
        // do collision check
        return true;
    }

    pinocchio::Model robot_model;
    pinocchio::GeometryModel robot_geom_model;
    pinocchio::Data data;
    pinocchio::GeometryData gemo_data;
};

Or any good suggestions to solve the problem?

Cannot you use pinocchio::Data* data or mutable pinocchio::Data data instead ?
I don't see any clean solution.

1、Using pinocchio::Data* data works.
2、But using mutable pinocchio::Data data causes an error:
no matching function for call to 'pinocchio::GeometryData::GeometryData()'
I would use the first solution.
Thanks for your reply!

Yes, it isn't possible to create a Data without a Model.
Thanks for the feedback.

You can be able to allocate the data with an emtpy model and then proceed to the copy ;)

@jcarpent I try to take your advice:
   mutable pinocchio::Data data(pinocchio::Model());

but this cause an error:
function 'data' cannot be declared 'mutable'
The version of pinocchio I used is 2.2.2 installed from robotpkg.  Is this a problem with the version I used or any version would cause an error?

From https://en.cppreference.com/w/cpp/language/cv:

mutable specifier

mutable - permits modification of the class member declared mutable even if the containing object is declared const.

mutable can only appear in class members.

@jmirabel  I should have described it more accurately. Here is the code I used:
class myValidityChecker : public ompl::base::StateValidityChecker
{
public:
    myValidityChecker (const ompl::base::SpaceInformationPtr &si) :  ompl::base::StateValidityChecker(si)
    {
    }

    mutable pinocchio::Model robot_model;    // it works
    mutable pinocchio::GeometryModel robot_geom_model;  // it works
    mutable pinocchio::Data (pinocchio::Model()); // causes an error
    mutable pinocchio::GeometryData gemo_data(pinocchio::GeometryModel()); // causes an error
};

The difference between Model and Data is that Model has a default constructor but Data not.
So why Data struct does not include a default constructor?

The error is coming from the fact you should init as follows:
class myValidityChecker : public ompl::base::StateValidityChecker
{
public:
    myValidityChecker (const ompl::base::SpaceInformationPtr &si) :   ompl::base::StateValidityChecker(si)
, data(pinocchio::Model())
, gemo_data(pinocchio::GeometryModel())
    {
    }

    mutable pinocchio::Model robot_model;    // it works
    mutable pinocchio::GeometryModel robot_geom_model;  // it works
    mutable pinocchio::Data data; // shoudl work
    mutable pinocchio::GeometryData gemo_data(pinocchio::GeometryModel()); // should work
};

The idea of not having a default constructor with no argument is to remind the user that a Data should be allocated after a Model has been created. Otherwise, you may have some inconsistencies between both, if, for instance, you add a Joint to a model, just after the creation of a Data.

The code you provided is correct.
Thank you for your patience and helpful solution!

You are welcome.


Segmentation error when running without torchpack and distributed mode

I noticed that the following function call
https://github.com/mit-han-lab/bevfusion/blob/db75150717a9462cb60241e36ba28d65f6908607/mmdet3d/ops/voxel/voxelize.py#L55-L66
throws a segmentation error if I am not running with torchpack.
(base) root@abf60d2d48ce:/home/bevfusion# python tools/visualize.py /home/bevfusion/configs/nuscenes/seg/lidar-centerpoint-bev128.yaml --mode pred --checkpoint pretrained/lidar-only-seg.pth --split val
load checkpoint from local path: pretrained/lidar-only-seg.pth
0it [00:00, ?it/s]Segmentation fault (core dumped)

I narrowed down the bug to the hard_voxelise call above but could not get a trace
PS: I made some modifications in the visualize.py code to put the distributed code (like the init and MMDistributedDataParallel) inside an argument check.

Hi,
Did you figure out how to resolve this?

I noticed that when using torchpack the tensors are placed on the GPU memory, but without it (also commenting out dist.init() and MMDistributedDataParallel stuff) they are placed on the CPU memory, and there seems to be a bug in the hard_voxelize() CPU code. What ended up working for me was manually placing the inputs on the GPU memory, i.e. doing this in the bevfusion.py file:
@auto_fp16(apply_to=("img", "points"))
    def forward(
        self,
        img,
        points,
        camera2ego,
        lidar2ego,
        lidar2camera,
        lidar2image,
        camera_intrinsics,
        camera2lidar,
        img_aug_matrix,
        lidar_aug_matrix,
        metas,
        gt_masks_bev=None,
        gt_bboxes_3d=None,
        gt_labels_3d=None,
        **kwargs,
    ):
        if isinstance(img, list):
            raise NotImplementedError
        else:
            outputs = self.forward_single(
                img.cuda(),
                points.cuda(),
                camera2ego.cuda(),
                lidar2ego.cuda(),
                lidar2camera.cuda(),
                lidar2image.cuda(),
                camera_intrinsics.cuda(),
                camera2lidar.cuda(),
                img_aug_matrix.cuda(),
                lidar_aug_matrix.cuda(),
                metas,
                gt_masks_bev,
                gt_bboxes_3d,
                gt_labels_3d,
                **kwargs,
            )
            return outputs

Hope this helps.


Explore NeoFS use cases

https://neospcc.medium.com/neofs-n3-testnet-release-and-first-episode-of-neofs-beginners-guide-54d618e1c86b


Alignment of cards in explorer

this alignment was solved in the past ?

Originally posted by @yangwao in https://github.com/kodadot/nft-gallery/issues/7697#issuecomment-1766308189

hm interesting, commented on this in the pr for stacked cards


Improve folder logic popup

Replaces this popup

with this

Makes it a bit less error prone.

there's also MIXED ;-)

Good catch. Never used that one. Fixed.


patch: Fix etcher link in instructions

Only a minor point, but I just noticed that some entries say "Write the OS file you downloaded to an SD card..." whereas others say "Write the OS file you downloaded to your SD card...".

@balena-ci rebase

Will address the consistency in another PR

@balena-ci rebase


datetime_millis util function could return wrong value

found this when executing test_venue_configuration from a different timezone
the pytest fail at assertion of

invitation = pc_client.get_invitation('ICLR.cc/2025/Conference/Submission1/-/Confidential_Comment')

the super confidential comment invitation is created but process function didn't create paper specific confidential comment invitations because the super invitation is not active yet.
the cdate set in pytest is correct which is

now - datetime.timedelta(minutes=1)

but when it's converted to mili seconds by tools.datetime_millis, the result was wrong which caused the super invitation to be inactive.
this is because

utcfromtimestamp(0)

was treated as local time
and it looks like utcfrormtimestamp function will be deprecated soon

This method is deprecated and we should update the code with the new method. I'm aware of that.

do you want to make the change in the repo?


[ARTEMIS-1171] Improve log readability around clustering

Issue: https://issues.apache.org/jira/browse/ARTEMIS-1171

I'm a bit concerned about breaking up log statements across multiple lines as it reduces the effectiveness of tools like grep to search for particular bits of logging along with their context.  My recommendation would be to weed out the details in the logging statement that really aren't necessary.

Although I agree, knowing what is necessary or not make this quite challenging since what is needed for one is not needed for someone else.
My main concern was the cascade of toString() calls that can pile up quite quickly as seen here.

@iweiss this has been reverted as this broke management.
it would need some work if you want it back.

@clebertsuconic Understood. I will rework it in the coming days. Sorry for the trouble.


Coverage binaries prevent usage of shacles

When trying to run shaclex from command line I get errors about coverage files not present in my path. Just cloning a master and running it we get:
$ sbt run
[info] Loading settings for project global-plugins from plugins.sbt ...
[info] Loading global plugins from /Users/cebriant/.sbt/1.0/plugins
[info] Loading settings for project shaclex-build from plugins.sbt ...
[info] Loading project definition from /Users/cebriant/GIT/shaclex/project
[info] Loading settings for project shaclex from version.sbt,build.sbt ...
[info] Set current project to shaclex (in build file:/Users/cebriant/GIT/shaclex/)
[info] running (fork) es.weso.shaclex.Main
[error] Exception in thread "main" java.lang.ExceptionInInitializerError
[error]         at es.weso.schema.ShaclexSchema$.empty(ShaclexSchema.scala:180)
[error]         at es.weso.schema.Schemas$.shaclex$lzycompute(Schemas.scala:15)
[error]         at es.weso.schema.Schemas$.shaclex(Schemas.scala:15)
[error]         at es.weso.schema.Schemas$.<init>(Schemas.scala:18)
[error]         at es.weso.schema.Schemas$.<clinit>(Schemas.scala)
[error]         at es.weso.shaclex.MainOpts.schemaFormats$lzycompute(MainOpts.scala:16)
[error]         at es.weso.shaclex.MainOpts.schemaFormats(MainOpts.scala:16)
[error]         at es.weso.shaclex.MainOpts.<init>(MainOpts.scala:43)
[error]         at es.weso.shaclex.Main$.run(Main.scala:67)
[error]         at cats.effect.IOApp.$anonfun$main$3(IOApp.scala:67)
[error]         at cats.effect.internals.IOAppPlatform$.mainFiber(IOAppPlatform.scala:37)
[error]         at cats.effect.internals.IOAppPlatform$.main(IOAppPlatform.scala:24)
[error]         at cats.effect.IOApp.main(IOApp.scala:67)
[error]         at cats.effect.IOApp.main$(IOApp.scala:66)
[error]         at es.weso.shaclex.Main$.main(Main.scala:25)
[error]         at es.weso.shaclex.Main.main(Main.scala)
[error] Caused by: java.io.FileNotFoundException: /home/labra/src/shapes/shacl-s/modules/shacl/target/scala-2.12/scoverage-data/scoverage.measurements.1 (No such file or directory)
[error]         at java.base/java.io.FileOutputStream.open0(Native Method)
[error]         at java.base/java.io.FileOutputStream.open(FileOutputStream.java:292)
[error]         at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:235)
[error]         at java.base/java.io.FileWriter.<init>(FileWriter.java:113)
[error]         at scoverage.Invoker$.$anonfun$invoked$1(Invoker.scala:55)
[error]         at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:86)
[error]         at scoverage.Invoker$.invoked(Invoker.scala:55)
[error]         at es.weso.shacl.Schema$.<init>(Schema.scala:116)
[error]         at es.weso.shacl.Schema$.<clinit>(Schema.scala)
[error]         ... 16 more
[error] Nonzero exit code returned from runner: 1
[error] (Compile / run) Nonzero exit code returned from runner: 1
[error] Total time: 2 s, completed Dec 9, 2019, 9:17:52 AM

It's strange...I have just cloned it into a new folder and sbt run works.
When did you clone it? because I had detected that issue about the coverage code and I had tried to repair it...

I've just cloned it right now and still see the problem:
$ rm -rf shaclex/
✔ cebriant@C02YM3Z3LVCG:~/GIT
$ git clone https://github.com/weso/shaclex
Cloning into 'shaclex'...
remote: Enumerating objects: 114, done.
remote: Counting objects: 100% (114/114), done.
remote: Compressing objects: 100% (67/67), done.
remote: Total 27756 (delta 31), reused 81 (delta 18), pack-reused 27642
Receiving objects: 100% (27756/27756), 8.94 MiB | 1.92 MiB/s, done.
Resolving deltas: 100% (13307/13307), done.
✔ cebriant@C02YM3Z3LVCG:~/GIT
$ cd shaclex/
✔ cebriant@C02YM3Z3LVCG:~/GIT/shaclex [master]
$ sbt run
[info] Loading settings for project global-plugins from plugins.sbt ...
[info] Loading global plugins from /Users/cebriant/.sbt/1.0/plugins
[info] Loading settings for project shaclex-build from plugins.sbt ...
[info] Loading project definition from /Users/cebriant/GIT/shaclex/project
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[info] Loading settings for project shaclex from version.sbt,build.sbt ...
[info] Set current project to shaclex (in build file:/Users/cebriant/GIT/shaclex/)
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[info] Compiling 14 Scala sources to /Users/cebriant/GIT/shaclex/modules/slang/target/scala-2.12/classes ...
[info] Compiling 5 Scala sources to /Users/cebriant/GIT/shaclex/modules/sgraph/target/scala-2.12/classes ...
[info] Compiling 5 Scala sources to /Users/cebriant/GIT/shaclex/modules/converter/target/scala-2.12/classes ...
[info] Compiling 10 Scala sources to /Users/cebriant/GIT/shaclex/modules/schema/target/scala-2.12/classes ...
[info] Compiling 12 Scala sources to /Users/cebriant/GIT/shaclex/modules/schemaInfer/target/scala-2.12/classes ...
[info] Compiling 3 Scala sources to /Users/cebriant/GIT/shaclex/target/scala-2.12/classes ...
[info] running (fork) es.weso.shaclex.Main
[error] Exception in thread "main" java.lang.ExceptionInInitializerError
[error]         at es.weso.schema.ShaclexSchema$.empty(ShaclexSchema.scala:180)
[error]         at es.weso.schema.Schemas$.shaclex$lzycompute(Schemas.scala:15)
[error]         at es.weso.schema.Schemas$.shaclex(Schemas.scala:15)
[error]         at es.weso.schema.Schemas$.<init>(Schemas.scala:18)
[error]         at es.weso.schema.Schemas$.<clinit>(Schemas.scala)
[error]         at es.weso.shaclex.MainOpts.schemaFormats$lzycompute(MainOpts.scala:16)
[error]         at es.weso.shaclex.MainOpts.schemaFormats(MainOpts.scala:16)
[error]         at es.weso.shaclex.MainOpts.<init>(MainOpts.scala:43)
[error]         at es.weso.shaclex.Main$.run(Main.scala:67)
[error]         at cats.effect.IOApp.$anonfun$main$3(IOApp.scala:67)
[error]         at cats.effect.internals.IOAppPlatform$.mainFiber(IOAppPlatform.scala:37)
[error]         at cats.effect.internals.IOAppPlatform$.main(IOAppPlatform.scala:24)
[error]         at cats.effect.IOApp.main(IOApp.scala:67)
[error]         at cats.effect.IOApp.main$(IOApp.scala:66)
[error]         at es.weso.shaclex.Main$.main(Main.scala:25)
[error]         at es.weso.shaclex.Main.main(Main.scala)
[error] Caused by: java.io.FileNotFoundException: /home/labra/src/shapes/shacl-s/modules/shacl/target/scala-2.12/scoverage-data/scoverage.measurements.1 (No such file or directory)
[error]         at java.base/java.io.FileOutputStream.open0(Native Method)
[error]         at java.base/java.io.FileOutputStream.open(FileOutputStream.java:292)
[error]         at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:235)
[error]         at java.base/java.io.FileWriter.<init>(FileWriter.java:113)
[error]         at scoverage.Invoker$.$anonfun$invoked$1(Invoker.scala:55)
[error]         at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:86)
[error]         at scoverage.Invoker$.invoked(Invoker.scala:55)
[error]         at es.weso.shacl.Schema$.<init>(Schema.scala:116)
[error]         at es.weso.shacl.Schema$.<clinit>(Schema.scala)
[error]         ... 16 more
[error] Nonzero exit code returned from runner: 1
[error] (Compile / run) Nonzero exit code returned from runner: 1
[error] Total time: 16 s, completed Dec 9, 2019, 10:22:48 AM
✘ cebriant@C02YM3Z3LVCG:~/GIT/shaclex [master]
$

Could it be that it works in your case because you effectively have the file in /home/labra/src/shapes/shacl-s/modules/shacl/target/scala-2.12/scoverage-data/scoverage.measurements.1 or you're able to write there?

You were right!, I think I have solved it in the new commit updating the shacl-s version.
Thanks for spotting the problem, and the solution!

Ummm...I stil think it is failing. I tried to change .travis.yml to use also scala 2.12.10 and now it fails.
My guess is that the coverage code has been included in the scala 2.12 binaries for some reason. But I don't know why.
In order to publish, the command we run is:
$ sbt clean coverageOff +publish

Yep, it doesn't work locally neither. I guess one of your published dependencies still has that coverage. I you only need to do the coverageOff when doing an interactive session. Starting sbt from the console and doing sbt clean +publish should be enough.
I think you best option is to publish artefacts only from Travis and bump versions in all libraries.

Also make sure that your repo forbids overwriting artefacts, the moment something is published remains forever. I say so because I'm using the exact same version of shex in my own project and it works, so maybe on this library (or other) the binary got overwritten.

Is this solved now? I still get the exception from above.

We thought it was solved...but maybe it isn't. We will try to check which of the published libraries is generating the problem.
If you know a way to detect the library, it would be great...the problem is that the only way we found was to go one by one library re-publishing it without the coverage code.

Sorry, I have no idea how to identify the respective library.


setting felix healthport does not work

I would like to change the felix healthport from the default 9099. By looking at felixconfigurations.crd.projectcalico.org, I can see the parameter there:
          healthPort:
            type: integer

However, after setting that parameter, I still see felix using 9099.
I guess the problem is that the operator has that value hardcoded as 9099:
https://github.com/tigera/operator/blob/master/pkg/render/node.go#L1586
Expected Behavior

I would like to have a way to set healthports, for example felix or typha, in case I have some other pods using that port too
Current Behavior

Not possible to change those ports when using the operator
Possible Solution

Expose the healthport numbers in the operator CRD
Steps to Reproduce (for bugs)

Deploy calico via the operator
Change the felix healthport in felixconfigurations.crd.projectcalico.org
Reboot the calico-node daemonset with the new value

Context

Deploy with other apps using the default port
Your Environment

Calico version: v3.21.2
Orchestrator version (e.g. kubernetes, mesos, rkt): kubernetes
Operating System and version: Ubuntu 20
Link to your project (optional):

Yep agree this is a bug. Operator should not be hard-coding that value.

We could, in theory, just have the operator monitor the value in FelixConfiguration and use that. The awkward part is that changes to FelixConfiguration take effect immediately, where as changing the probes on the pod requires a rolling update and so will take some amount of time.
The alternative is to deprecate the felix configuration version of this field and switch to an operator-only based configuration.
CC @tmjd @fasaxc for input.

Whatever we decide, felix HealthHost probably needs similar treatment.

If we have the operator set an env var then Felix would not change until the pod was rolled. So we can force the value to only change with the pod.

If we have the operator set an env var then Felix would not change until the pod was rolled. So we can force the value to only change with the pod.

Correct - that's my second option above, but it would mean that the FelixConfiguration is essentially dead and deprecated (setting it would have no effect on an operator-managed cluster)

Thanks for investing time in this! We will have the same problem with typha healthport, should I open a new issue?

@manuelbuil nope we can track both with this issue.

I think it'd be good to solve the problem of operator defaulting felixconfig values so that the user can just write to felixconfig and operator will realise "user has overriden my value of this non-critical field".

If we have the operator set an env var then Felix would not change until the pod was rolled. So we can force the value to only change with the pod.

Correct - that's my second option above, but it would mean that the FelixConfiguration is essentially dead and deprecated (setting it would have no effect on an operator-managed cluster)

I'm not suggesting that FelixConfiguration should be dead. I'm suggesting the operator can monitor FelixConfiguration for the value to use but take that value and set it as an Environment Variable. That will keep all the running pods from switching until the pod itself is updated.

I think it'd be good to solve the problem of operator defaulting felixconfig values so that the user can just write to felixconfig and operator will realise "user has overriden my value of this non-critical field". ManagedFields is supposed to do that?

I'm not sure if we'd need to be concerned with ManagedFields for this, I don't think the operator will need to care if it previously set the field or if the user set the field. The operator only needs to take what is set and apply the value read from configuration.
I think the only cases where we need to be concerned with ManagedFields is if the operator needs to know if it previously set a field or if the user has modified it. Maybe an edge case we need to consider is a disabling case for the metrics.
Unfortunately all this discussion ^ is not applicable to Typha because we don't have a current config option for Typha. My hacky (avoiding coming up with a new API) thought is to just make Typha's port be Node-1 (since it is set to 9098 right now).

I'm suggesting the operator can monitor FelixConfiguration for the value to use but take that value and set it as an Environment Variable. That will keep all the running pods from switching until the pod itself is updated.

Ah, I originally thought this wouldn't work because all of the felixes in the cluster will have already spotted the FelixConfiguration change and restarted, but I suppose if it's always set via the env var then the config update will be ignored by the felixes and we'll get a nice rolling update. That is actually good enough I think.

I'm not sure if we'd need to be concerned with ManagedFields for this, I don't think the operator will need to care if it previously set the field or if the user set the field

Yep, we should look at managed fields but that's for a different problem IIUC - namely the case where the operator and a user might want to set a field within FelixConfiguration. In this case, the operator's use of the field is read-only.

I had some code lying around from before so stuck it in a PR: https://github.com/tigera/operator/pull/1846
Need to test it out and review it, but it might do the trick for felix at least.

Could we please ensure that Felix healthport and typha healthport are both exposed via the operator, to avoid conflicts with existing services during installation.

After discussing internally we think a better solution might be to move off of using HTTP probes for readiness / liveness altogether and switch to something like a unix domain socket + exec based probe that way there will be no TCP ports opened on the host.

After discussing internally we think a better solution might be to move off of using HTTP probes for readiness / liveness altogether and switch to something like a unix domain socket + exec based probe that way there will be no TCP ports opened on the host.

Sounds more robust and less likely to clash with existing host services.

@caseydavenport looks like https://github.com/tigera/operator/pull/1846 is going to deal with the Felix healthport, can we also add a similar fix to manage the Typha healthport? Both sit on a couple of contiguous IANA un-claimed ports by default and likelihood of clashing with other applications is pretty high.

I think @howels you are aware that PR will adjust Typha also but if not or for any others that come across this issue. That PR also sets the Typha healthport, it sets it to FelixHealth-1. So if Felix HealthPort is set to 9399, typha will use 9398.

I don't think there is anything left to do for this issue so I'm going to close it.
If I'm wrong please comment and we can re-open this issue. Thank you. :wave:


cleanup(driver/bpf): split bpf_val_to_ring into many int-pushing helpers

What type of PR is this?
/kind cleanup
Any specific area of the project related to this PR?
/area driver-bpf
Does this PR require a change in the driver versions?
/version driver-API-version-patch
What this PR does / why we need it:
This PR splits up all the integer-related cases from bpf_val_to_ring.
We still need to:

figure out how to port sys_autofill
figure out how to port sys_single -> we only have 3 syscalls using it nowadays; therefore it should be pretty simple to drop it completely.
figure out what to do with PT_DYN params

Moreover, the PR also uses CHECK_RES macro in all fillers.
I think the TODO can be addressed in a subsequent PR since this is already pretty huge and will lead to multiple conflicts everytime a bpf touching PR gets merged in master.
Which issue(s) this PR fixes:

#940
Special notes for your reviewer:
This PR already cuts down total number of bpf programs instructions by ~25%:

Total number of bpf instructions in this branch: insns: 158593
Total number of bpf instructions on master:  insns: 212830

See comment.
Does this PR introduce a user-facing change?:

cleanup(driver/bpf): split up bpf_val_to_ring into many helpers

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

driver/OWNERS [FedeDP]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

This is :heart: and a great first step towards https://github.com/falcosecurity/libs/issues/940

Total number of bpf instructions in this branch: insns: 158593
Total number of bpf instructions on master:  insns: 212830

This is a ~25% reduction!
Patch used (thanks @Andreagit97 !): libs.txt

/milestone next-driver
Tagging @Molter73 and @incertum that were interested in this.
My personal opinion is to merge this as soon as possible, because it is huge and will lead to lots of conflicts.
Moreover, as previously shared, it already gives us great results.
We can then subsequently fix up remaining things (if we want to!), like:

dropping sys_single
better managing PT_DYN params

Still, sys_autofill is used for ~50 syscalls, therefore i am not sure we will ever get rid of it (and this means bpf_val_to_ring will still have to manage integer cases).

As a side note, @Andreagit97 promised that he will be testing this one on multiple clang/kernel releases :) :crossed_fingers:

Yay, and yeah we should have multiple folks test it!

Yay, and yeah we should have multiple folks test it!

I don't know why but i read `we should have multiple forks of it and i was like "uh, ok?" :rofl:

Tested on:

5.19.0-38-generic 22.04.1-Ubuntu x86 :heavy_check_mark:
4.14.309-231.529.amzn2.x86_64 Amazon linux 2 x86 :heavy_check_mark:
5.19.0-1022-aws 22.04.1-Ubuntu aarch64 :heavy_check_mark:
6.1.21-1.45.amzn2023.x86_64 Amazon linux 2 x86 :heavy_check_mark:
5.15.0-56-generic Ubuntu s390x :heavy_check_mark:
4.18.0-425.13.1.el8_7.x86_64 almalinux8 x86 :heavy_check_mark:
5.14.0-162.6.1.el9_1.x86_64 almalinux9 x86 :heavy_check_mark:

This PR looks good to me :heavy_check_mark: :heavy_check_mark: :heavy_check_mark: !

Total number of bpf instructions in this branch: insns: 158593
Total number of bpf instructions on master: insns: 212830

This is a ~25% reduction!

This is really valuable, we could improve perf for real here! if we are also able to fix recent verifier issues we won!
Next possible steps after this:

remove sys_single filler
remove counters at runtime and fully rely on the test framework
Extract from val-to-ring PT_CHARBUF PT_BYTEBUF types

The next steps could also be done in the following months, no hurry IMHO! We should just get rid of all the late verifier issues!
But yes we should always strive for improvements :)

As promised @Andreagit97 and @FedeDP compared this PR with current master leveraging the driver_sanity VM tests.
Good news, we have more data driven and systematic evidence that these changes are an improvement and we are heading in the right direction.
Re the issues re compiling w/ different compiler versions, don't mind them, you see in the driver_sanity PR https://github.com/falcosecurity/libs/pull/524 that I use certain ubuntu containers w/ GLIBC version issues, will explore if some of the build containers can be adjusted.
Even though we observe good improvements, however we can't deny that it is a bit like playing Tetris, few cases of failure I now observed in 4.16 are again familiar error messages:
326: (0f) r8 += r5
math between map_value pointer and register with unbounded min value is not allowed

-- END PROG LOAD LOG --
libscap: bpf_load_program() event=tracepoint/filler/sys_open_by_handle_at_x: Operation not permitted (1)
FAILED unit test for ../../build/driver/clang-15/4.16.18-041618-generic

Current master baseline [commit 543755587a9d21686b52ca2a0995babb074d9b90]

This PR [commit f374596e19cf0d0748bc7fe84ac7b4aa0aa30b57]

This is a super great news! :rocket: Thank you very much!
We may want to fix 4.16 issues in a follow up PR, imho. It would be great if we could reproduce it ourselves, @Andreagit97 any idea?

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97, FedeDP
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

driver/OWNERS [Andreagit97,FedeDP]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97, FedeDP, incertum
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

driver/OWNERS [Andreagit97,FedeDP,incertum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Coco - OS compatability

Can we know the list of OS that COCO will be compatible with? I am at presenting working on Hyperledger sawtooth - that can be installed in Ubuntu or AWS and not on Azure. If COCO has to work with sawtooth, then COCO has to be installed on Ubuntu or AWS? Will that be possible?

Ubuntu and other Linux distributions can be installed on Azure. Does that help?
As long as machines can talk to each other by IP (for example on different networks - such as a Coco machine on Azure and a Sawtooth machine at Rackspace) then in principle you should be able to write code to integrate different frameworks.

Coco has been released as CCF now: https://github.com/Microsoft/CCF
At the moment, it primarily targets Ubuntu 18.04.
Thank you for your patience!


Change to use tab indentation (jsbeautifier.org)

Change to correct beginning comment block to contain version
Removal of trailing whitespace from entire file

Is this PR worthwhile?

sorry, I'm not a fan of hard tabs. thx anyway

It was more that there was a mixture of tabs and spaces...

already cleaned it up. thx!


Mismatch API Doc and Markdown Doc

Description
The documentation on the fields for a trusted reference upload does not match the API doc (swagger)
https://worldhealthorganization.github.io/ddcc-gateway/#/Trusted Reference/uploadTrustedReference
https://github.com/WorldHealthOrganization/ddcc-gateway/blob/main/docs/Architecture.md#trusted-references
Details

URL is in MD, not in swagger
referenceVersion is in swagger, not in MD
domain is in swagger, not in MD
"Content-Type" in MD, "contentType" in swagger

Also

upper/lower case mismatch (Country/country, UUID, uuid)
uuid seems to be set by the gateway, should not be part of the upload request then

URL is in MD, not in swagger
** [ ] fixed with #43
referenceVersion is in swagger, not in MD
** renaming was required because version attribute is already reserved for federation feature.
domain is in swagger, not in MD
** Optional Attribute required for federation
** [ ] Should be marked as optional in Upload DTO
"Content-Type" in MD, "contentType" in swagger, upper/lower case mismatch (Country/country, UUID, uuid)
** adapted to lowercase for common naming
uuid seems to be set by the gateway, should not be part of the upload request then
** [ ] UUID will be removed from Upload Request


Regridding masks, robustness of dataset_utils functions and bug fixes

Pull Request Checklist:

[x] This PR addresses an already opened issue (for bug fixes / features)

This PR fixes issue #342
but addresses also other (unlisted) issues and includes new features as well

[x] Tests for the changes have been added (for bug fixes / features)
[ ] Documentation has been added / updated (for bug fixes / features)
[ ] HISTORY.rst has been updated (with summary of main changes)
[ ] I have added my relevant user information to AUTHORS.md

What kind of change does this PR introduce?:
Updates to clisops.utils.dataset_utils

Adapted roocs_utils.xarray_utils.xarray_utils for clisops:
(they were added to clisops.utils.dataset_utils)

get_coord_by_type
Returns now the coordinate name instead of the coordinate as
xarray.DataArray (and optionally a list of further
matches with the coordinate type)

Added function determine_lon_lat_range to determine the min. and max. lat and lon values
Added function fix_unmasked_missing_values_lon_lat to identify and mask yet unmasked
missing values in lat and lon arrays.
Fixed issue in cf_convert_between_lon_frames causing the lon frame to not be adjusted
in case of NaNs in the longitude array, also added force option
Addressed issues in generate_bounds_curvilinear:

latitudes are now clipped above 90 or below -90 degrees north
longitudes are converted to longitude frame -180, 180
longitude bounds are adjusted at the Greenwich meridian or anti meridian to avoid grid cells wrapping once or more times around the globe
Bounds are generated significantly faster due to making use of index slicing and numpy.vectorize

Adapted code for changes to get_coord_by_type and other roocs_utils functions in

clisops.core.average
clisops.core.subset
clisops.ops.base_operation
clisops.utils.output_utils
clisops.utils.file_namers

Updates to clisops.ops.regrid

if input and output grid are alike, the original dataset is passed through
and a warning issued, rather than an exception raised
Added option to request a land/sea mask for the output grid

Updates to clisops.core.regrid.Grid

Added Grid() attributes xinc, yinc, res, smash_mask, degen_mask,
contains_smashed_cells, contains_degenerate_cells
Added method _verify_bounds to
drop the lat/lon bounds if an integrity check fails
Added method _grid_detect_smashed_cells adapted from ESMF to
detect smashed cells
mask DataArray will be added to Grid().ds during initialization,
masking degenerate cells (collapsed and smashed cells)
Updated repr, added

grid approx. resolution
information whether grid contains degenerate cells
information regarding a potential mask

Add method _apply_lsm to apply land or ocean mask, if present in the file
Updated detect_extent to mask unmasked missing lon/lat values and potentially apply cf_convert_between_lon_frames
Updated detect_coordinate to remove attributes from (coordinate) variables that
would result in false positives when calling ds.cf[coord_type]
Fixed _grid_detect_collapsed_cells to not only detect cells
collapsing to points, but also cells collapsing to lines
Cleaned up duplicate methods between clisops.utils.dataset_utils
and clisops.core.regrid.Grid

Tests

Added reference to multiple test datasets newly added to mini-esgf
Tests previously skipped (as relying on data on HPC's data pools)
now using mini-esgf datasets
Tests previously skipped ("bug no longer exists") were adapted to the code and activated
Adapted multiple tests from roocs_utils for the adapted functions
Adapted multiple tests after changes to above functions
Added multiple tests for the newly added functions

Does this PR introduce a breaking change?:
The behaviour of the regrid operator changes:

if source and target grid are evaluated equal, before, an Exception was raised, now, a warning is issued and the original dataset is passed through

Some functions of roocs_utils.xarray_utils.xarray_utils have been adapted (and modified) into clisops.utils.dataset_utils:

get_coord_by_type now returns the name of the coordinate variable and not the coordinate variable
get_coord_by_type optionally returns an additional list with further matches for the coordinate variable and does not raise an Exception when more than one coordinate variable matches the requested type
some functions in clisops.utils.dataset_utils and clisops.core.regrid behave a little differently in terms of raised Expections / Errors:

clisops.utils.dataset_utils

get_coord_by_type: raises ValueError instead of an Exception, if coord_type is not known
detect_coordinate: raises KeyError instead of AttributeError if no coordinate could be detected
detect_gridtype: raises ValueError if grid type is not supported (before InvalidParameterValue and Exceptions were raised inconsistently)

clisops.core.regrid

Grid.detect_coordinate: raises KeyError instead of AttributeError if no coordinate could be detected

clisops.ops.regrid

Regrid._calculate: issues warning instead of letting clisops.core.Weights.__init__ raise an Exception when input and output grid are alike

Other information:

@sol1105 @Zeitsperre could we move all methods of xarray_utils from roocs_utils to clisops?
https://github.com/roocs/roocs-utils/blob/master/roocs_utils/xarray_utils/xarray_utils.py
... and either remove them in roocs-utils or mark them as deprecated.

@cehbrecht I think that would be fine. Maybe just add a new file to the utils module here?
What's the timeline on replacing roocs-utils? Beyond that migration, what else needs to be done to fully deprecate that library?

What's the timeline on replacing roocs-utils? Beyond that migration, what else needs to be done to fully deprecate that library?

@Zeitsperre we also need to move the parameters utils to clisops. Moving the xarray_utils  to clisops as a first step might be a good thing. We just need to make sure that we use the right roocs_utils library as long we need it ... keep the upper version limit:
https://github.com/roocs/clisops/blob/002941a890fd8e0878c9c76ad6457ad0d86dcdc6/pyproject.toml#L69
daops and rook can always follow with the changes.

@sol1105 I'll manage the merging of the main branch here. Lots of changs, but it should not be too bad, perhaps it'll even run the tests faster? Will move onto this tomorrow or next week.

@sol1105
This is getting close to compliant with the new testing data structure.
The KeyError: -1 issues are coming from the testing helper function that extracts the lon values from a Dataset. I don't think this will be too hard to address.
For the test_roll failures, these are definitely due to changes to get_coord_by_type and how they interact with the changes to calculate_offset, another easy fix.
Will see if I can make another pass at this next week. Please let me know if you have questions.

@Zeitsperre Thanks a lot for merging in your changes. I will have a look asap.

@Zeitsperre The failing tests are because of the known problem https://github.com/pydata/xarray/issues/7794
I do not think there is any problem on top of that, at least locally all tests passed with xarray==2023.2.0 and the few tests fail only for newer versions of xarray.
Should binder/environment.yml be a link to the repo's environment.yml ?

@Zeitsperre The failing tests are because of the known problem pydata/xarray#7794 I do not think there is any problem on top of that, at least locally all tests passed with xarray==2023.2.0 and the few tests fail only for newer versions of xarray.

We should maybe set the pytest.mark.xfail for these tests with the xarray version as the allowed condition. I can add that later today.

Should binder/environment.yml be a link to the repo's environment.yml ?

I think that's a good idea. Fewer files that need to be updated.

Trying to re-run tests without multiple workers. I think something in the test may be locking processes, hence why we're seeing hanging. Odd that it only happened on conda though...

@sol1105 I was able to use pytest-timeout to fail out the slowest tests and have marked them as such. Most tests should pass now.
I think the regridding tests are way too heavy to reliably use on CI. If we want to be able to use the test builds to validate that our code is working, the build shouldn't take longer than 7 or 10 minutes at most. Is there a way that we can simplify some of these tests while maintaining code coverage?

Pull Request Test Coverage Report for Build<PHONE_NUMBER>3
Details

437 of 520 (84.04%)  changed or added relevant lines in 10 files are covered.
36 unchanged lines in 4 files lost coverage.
Overall coverage increased (+0.2%) to 73.415%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

clisops/core/subset.py
7
8
87.5%

clisops/core/regrid.py
147
164
89.63%

clisops/ops/regrid.py
1
21
4.76%

clisops/utils/dataset_utils.py
267
312
85.58%

Files with Coverage Reduction
New Missed Lines
%

clisops/core/subset.py
1
76.84%

clisops/ops/base_operation.py
5
83.04%

clisops/utils/dataset_utils.py
9
78.4%

clisops/core/regrid.py
21
61.52%

Totals

Change from base Build<PHONE_NUMBER>2:
0.2%

Covered Lines:
2107

Relevant Lines:
2870

💛  - Coveralls

@sol1105 @Zeitsperre could we move all methods of xarray_utils from roocs_utils to clisops? https://github.com/roocs/roocs-utils/blob/master/roocs_utils/xarray_utils/xarray_utils.py
... and either remove them in roocs-utils or mark them as deprecated.

I'm thinking we can talk about this tomorrow, but I've started a set of branches (all named modernize) here and also in roocs-utils and daops. I'm thinking we can work within those to move the utilities around as needed.

@cehbrecht
Thanks again for the chat today!
Tests are now passing! If there are no further changes needed here, I think this is safe to merge. I'll begin working on the major refactoring plan following these changes.

@Zeitsperre Thanks for getting this to work. I will have a look on Friday morning whether there was sth to do before the merge.

@Zeitsperre
Added the changes to HISTORY.rst and made the last changes I planned in this PR. So should be fine to merge now.


Add "Connection success" message

John, I suggest including some sort of success message after connection is finished. I struggled a little to find out whether it was connected or not. For instance, I get the message:
>>> import mph
>>> client=mph.start()
Did not find Comsol executable.

Although the connection to the server was successful. I figured it out by running the discovery.py functions separately.

The logic is, if mph.start() does not raise an exception, then it worked and the client is up and running. And connected to a server if you're on macOS, as I think you are.
What you're seeing is not an exception, it's an error reported by the logger, which by default is printed to the console. This is standard behavior when using Python's logging module, which MPh does. That log output could also be redirected to a file, or be suppressed entirely. Like so:
import mph
import logging

logging.getLogger().setLevel(logging.CRITICAL)
client = mph.start()

I don't know what causes this log error to appear. I can only reproduce it if I create an empty Comsol folder besides an existing one. Like on my Linux machine, Comsol is installed in /usr/local/comsol55. (It would be in /Applications/COMSOL55 on macOS.) If I create /usr/local/comsol54 as well as an empty multiphysics folder inside that, I get that log error. Which is a warning, rather, as it would still find the Comsol executable in the 5.5 folder and therefore does not raise an exception. So this reproduces the behavior you're seeing. Maybe you have an orphaned Comsol folder from an installation that you've since removed? I can't think of any other explanation.

 Maybe there is a way to install COMSOL without the libraries for scripting? I would have to run the installer and see if this option exists

The error is reported by this part of the code in discovery.py:
comsol = root/'bin'/'maci64'/'comsol'
if not comsol.exists():
    logger.error('Did not find Comsol executable.')
    continue

Where root is Applications/COMSOL*/Multiphysics. So it's definitely just the executable comsol that is missing in the expected location. It can't have anything to do with the libraries. Because then the (log) error would have a different message.

So in MPh 0.9.1, released today, I changed the level for these kind of log messages from ERROR to DEBUG. This way they will no longer pop up in the console by default.
@gwiederhecker I'd still be interested to know if my explanation attempt was correct. Only to have an idea if this kind of thing is commonplace or a rare exception.


Proposal: using RouterFileSytem to co-exist with existing underlying …

A proposed solution for #2625
This PR intends to put an idea in writing (before even testing it) and start a discussion about it.

Very interesting and clear! Thank you.
I agree that implementing and maintaining a Spark library is hard and I would even say we should avoid this until we really can't.
If we decide to go with a Spark library, this sounds like a very good direction to go.

lgtm, two things that will probably get at some point:

replace regexp will probably will require group match
document limitations for operations while working this package. How operations will execute when there are overlapping configurations. it can be mapping one file system under another.
Rx: delete recursive, /data, while one file system is the complete package, and the /data/lakefs is mapped to lakeFS.

Thanks for this @talSofer ! This seems to be the right approach. There's some prior art that we can look at, as it appears we're not the first project to think about providing a wrapper for a Hadoop Filesystem.
Hadoop itself provides FilterFilesystem which seems to do something similar ("A FilterFileSystem contains some other file system, which it uses as its basic file system, possibly transforming the data along the way or providing additional functionality. ").
There's even an example of how to override it: in this case, to write a checksum for every file that's written; see ChecksumFileSystem.

Thanks @ozkatz!
I took a look at  FilterFilesystem, and I don't think it's the right fit for us. FilterFS is initialized with another (single) file system, but we need to route file system operations to multiple file systems. Extending FilterFilesystem will be similar to extending FileSystem because in both cases, we will need to do extra work to access more than a single file system at runtime.
WDYT?

lgtm, two things that will probably get at some point:

replace regexp will probably will require group match
document limitations for operations while working this package. How operations will execute when there are overlapping configurations. it can be mapping one file system under another.
Rx: delete recursive, /data, while one file system is the complete package, and the /data/lakefs is mapped to lakeFS.

@nopcoder great points (especially 2 which is scary!), I'm adding this as a con to the proposal and we will need to do some discovery work there.


Show exception details in frontend

Description
If you are developing a new plugin and something you are doing is wrong which then throws an exception yet the only output in the frontend I'll get is a message like Unfortunately, something went wrong.
At this point I would except the normal behaviour of the Symfony ErrorHandler. My current workaround is the following in src/Storefront/Framework/Routing/StorefrontSubscriber.php.
127      public function showHtmlExceptionResponse(ExceptionEvent $event): void
128      {
    +        return;
129          if ($event->getRequest()->attributes->has(PlatformRequest::ATTRIBUTE_SALES_CHANNEL_CONTEXT_OBJECT)) {

Example

While APP_ENV=dev the Symfony ErrorHandler should be used instead of the customization from Shopware.

thanks for your contribution @tyurderi :+1: :tada: :blue_heart:


Use kubectl diff instead of helm diff

Helm diff does not compare the manifests against the actual state of the cluster, only against the installed releases, looks like there are no plans to change that: https://github.com/databus23/helm-diff/issues/176
I suggest using kubectl diff instead to show the real diff compared to the actual state of the cluster.

@dudicoco Honestly I don't know if kubectl diff is able to produce correct diff for Helm 3 which uses three-way merge patch. Is it the correct way to do this?

@mumoshu the Helm 3 three-way merge patch is used when applying the configuration.
helm-diff doesn't perform a "three-way merge diff" which is exactly my point - it's not comparing the current state to the cluster state, kubectl diff does this so its behaviour is inline with the helm three-way merge patch and will reflect the true diff before applying the new configuration.

@dudicoco Gotcha! Thanks. I'm now inclined to add this. Would you mind submitting a PR?

@mumoshu I would be happy to give it a shot but since I have no experience with programming in go it would be helpful to get some direction to the right places in the code :)

@dudicoco The smoothest path would be to implement a helm-diff-like diff command based on kubectl diff by changing this function: https://github.com/roboll/helmfile/blob/master/pkg/helmexec/exec.go#L299-L327 😃

@mohsen0 yeah we've tried it already but there are a couple of issues that need to be fixed before it can be usable.
In any case I think this issue can be closed.


Implement handleGetFrames

Is this the intended use of this endpoint? This would be useful for me.
$ curl :10101/index/benchindex/frame

{
    "index": {
        "frames": [
              {
                "name": "slice-height",
                "views": [
                    {
                        "name": "standard"
                    }
                ]
            },
            {
                "name": "zipf",
                "views": [
                    {
                        "name": "standard"
                    }
                ]
            }
        ],
        "name": "benchindex"
    }
}

Coverage decreased (-0.02%) to 66.677% when pulling 72f21f87e54e8ace97237820eb1af95253f9dcac on alanbernstein:get-frames into 4a14e38ebeffe1a1a3f1a38069f21e7394e7032e on pilosa:master.

I would expect the output you suggested to be here instead:
router.HandleFunc("/index/{index}", handler.handleGetIndex).Methods("GET")

I think that handleGetFrames should return a list of frames for that index. whether that list includes the full schema or not i don't know, but it should include a list of endpoints to each frame.
@codysoyland do you have an idea on expectations for these unimplemented endpoints?

I agree with @travisturner that the output in @alanbernstein's comment should live at /index/{index} instead of /index/{index}/frame. I'm not sure that we need to implement /index/{index}/frame because it becomes redundant if both endpoints returned the list of frames. I would probably leave it alone. If we do choose to implement this, let's move the "frames" key to the top level and remove the "Not implemented" comment.

/schema and /status currently return big JSON objects containing all index and frame names, plus some frame info that doesn't seem to fit either endpoint well.
/status returns the result of ClusterStatus() which makes sense from an internal perspective, but it's weird from a user perspective.
Are there any design docs or opinions about how this stuff should be organized? I have a few suggestions I can write up (as a counterpart to #316)

@alanbernstein I'm happy for you to write up your suggestions. Those two endpoints were never really thought through very well, and so far have been primarily used to provide information to the WebUI.

I'm working on a proposal that will likely affect this, closing for now.


Added Marmara University

Official University Website: https://www.marmara.edu.tr/en
Official Computer Engineering Website: http://cse.eng.marmara.edu.tr/en
The confirmation for the usage of "marun.edu.tr" by university students: https://bidb.marmara.edu.tr/en/frequently-asked-questions-student-e-mail

@yusufdmn I am sorry, the domain marun.edu.tr is removed from our trusted domains list because it was abused in the past. If you are the student of this school, please enroll to GitHub Student Developer Program. Then apply for JetBrains educational subscription with your GitHub account (GitHub tab at https://www.jetbrains.com/shop/eform/students).


adding orders

I managed to edit and save a new layout
but how do I add an order?

This should be possible action in the context menu when you right click inside the orders window.


Remove JWT dependency on org.bouncycastle.bcpkix-jdk15on, replace com.nimbusds.nimbus-jose-jwt

Removing dependency on BouncyCastle
The demo code uses PKCS#1 RSA SHA-256 certs. That works with security providers on all modern JDKs; we don't need EllipticCurve (EC) support for non-Oracle JDKs via BouncyCastle.
I successfully removed it and generated 2048 bit RSA certificate stored as PKCS#8 pem and it works on OpenJDK 11 on Linux. Will try on Windows too.
Replacing Nimbus-jose-jwt
I would like to replace this library with something else. The reason is that it has a hard dependency on Jackson JSON called Nimbus json-smart and poisons the MicroProfile demo app's classpath with it.
I believe MicroProfile applications should use JSONB as that is in the spec.
I successfully replaced the implementation with vertx-auth-jwt, i.e. the diff for the demo app pom is:
- com.nimbusds.nimbus-jose-jwt
- org.bouncycastle.bcpkix-jdk15on
+ io.vertx.vertx-auth-jwt

It works with Quarkus. I will be testing it with all other servers to see if there are any problems. If there are glitches, I pick something else. The main point is not to use verx, the point is to get rid of json-smart.
Motivation
I am working on Quarkus support and while it can work with Nimbus-jose-jwt + Nimbus json-smart in Java mode, it cannot build a native executable from it, because of what json-smart does at runtime. I believe that as long as it works for all servers, there is no reason to explicitly depend on Nimbus' package.
PR demonstrating the change comes shortly.
I would like to ask for opinions, especially @rdebusscher as he is the JWT demo author.

The PR is ready now #208

nimbus-jose-jwt is certainly not a must, it is the framework I know best. And indeed, now with having JSON-B, the json-smart dependency should be made swappable.
BouncyCastle is probably a framework everyone uses with features not available in plain Java. +0 on removing that one.

I'll try to find some time on Monday to have a look at the PR.


Can't get this to work.

I think I have followed the instructions to the letter but can't get it to work. Every time I try to give Alexa a command I get "There is a problem with the requested skills response"
I don't know where to look. Any assistance would be appreciated.

@rshames That's usually a problem with the Lambda function. To debug that, go to Lambda, open your code, and click on the Monitoring tab. Click "View Logs In Cloudwatch" and click one of the entries. There, you'll likely see an error. Let me know if you see anything.

Thanks,
The error is below
Syntax error in module 'main': Non-ASCII character '\xe2' in file /var/task/main.py on line 15, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details (main.py, line 15)
But line 15 is the myQ username.
USERNAME = “rob%40shamesnet.com”

Try replacing the "%40" with "@"

I tried that with no luck. But Interestingly , I retyped the quotes around the username and got a different error.
No JSON object could be decoded: ValueError
Traceback (most recent call last):
File "/var/task/main.py", line 32, in lambda_handler
login()
File "/var/task/main.py", line 95, in login
response = requests.get(login_uri()).json()
File "/var/task/requests/models.py", line 808, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib64/python2.7/json/init.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Looks like a problem in your login endpoint, can you add something at line 95? Instead of this:
def login():
    response = requests.get(login_uri()).json()

make it say this:
def login():
    print("Login URL: " + login_uri())
    response = requests.get(login_uri()).json()

That will print the login URL. Careful when posting the error logs for this one, that line you add will print your password. Don't share the password here but share that Login URL with the password removed or change to something harmless.

this is the response in the log. (I replaced the password with ********, thanks for the warning)
Login URL: https://myqexternal.myqdevice.com/Membership/ValidateUserWithCulture?appId=Vj8pQggXLhLy0WHahglCD4N1nAkkXQtGYpq2HrHD7H1nvmbT55KqtN6RSF4ILB%2Fi&securityToken=null&username=rob%40shamesnet.com&password=********&culture=en

Weird, I'm getting the same error now too. Wondering if Chamberlain changed something. Digging in now, thanks for the catch. I'll update this when I know more.

Thank you so much!!

There's an issue on the MyQ side -- all requests are getting denied :( Do you have the Chamberlain app on your phone? Mine quit working a few days ago, wondering now if that's related to this issue as well.

My Chamberlain mobile app is still working.

Ok thanks. I'll keep this open and will try to track down the new API url.

Ok, Let me know if you would like me to test anything.

@tigerbrain Does the code still work for you?

@jbnunn Hello, not using this project but another that reaches the endpoints the same way and I am seeing the same error and getting the same request denied message. Just an FYI.

The gateway is reaching out to <IP_ADDRESS> port 8883 and wireshark shows a protocol of mqtt, is the mqtt part new or has it always been that?

Hi @jbnunn, No this code has stopped working for me as well.  I have a feeling Chamberlain did something to stop it from working or the APP_ID has changed which is so sad since I just finished modifying your code so that it can support opening and closing two doors without having the need to modify main.py.  This just happened this morning as yesterday I had @steve772 test out the modified code and it was working properly for him.  The other thing is that the server might just be down but if that was the case, their app itself should not work as well.

Thanks @tigerbrain, sorry. Your hard work isn't lost, we'll figure something out.
@arraylabs the MQTT part is new, it's only been using an HTTP GET until now. That's a good sign though, it might show Chamberlain is making some updates and will release something official. What does wireshark show when the official mobile app makes a call back to chamberlain? anything we can learn from that?

@jbnunn the mqtt is from the capture between the gateway device and the myq servers. I just looked at the capture from my iphone while moving the door and see it going to the resolved ip for myqexternal.myqdevice.com. Mostly all https so not sure thats going to be much help, i'm new to wireshark so guessing without ssl keys not much going to happen there.

@jbnunn I'm talking again with the API like this:
<EMAIL_ADDRESS>

Looks like they changed the Device_List_Endpoint to all lower case. I made the following two changes and I can get the status of my garage:
LOGIN_ENDPOINT = "api/user/validatewithculture" DEVICE_LIST_ENDPOINT = "api/userdevicedetails"
I'm sure the Set and Status are probably updated too.

@channingcorn Thanks for figuring this out.  I will give this a try after I get home.

Works! Thanks @channingcorn and @arraylabs. How'd you find this out? I'd like to learn how for the next time this happens.

@arraylabs did most of the work. I noticed the URL he posted was lowercase, so once I got past login with his supplied URL, I saw that "device_list_uri" was failing so I tried changing it to all lowercase.
Dumb luck, but it worked. I'm sure sniffing with Wireshark might have worked, but I tried the simple approach first.

FYI just uploaded the most recent code to Lambda and it works. :-)

@jbnunn @channingcorn glad to help! Honestly I just stumbled across another python project that was using different endpoints and starting poking at them with Postman till I got a response.

Nice work everyone.  Thanks for sharing your discoveries and keeping this luxury alive.  It is really such a shame that we could not get MyQ to just support this work.  It benefits them in the end rather than losing their sales to another competitor that supports Amazon Echo.

Anyone having issues again?

Yes, they changed again sometime yesterday. I tracked down the valid ones and updated my project earlier today. @jbnunn you can pull them from:
https://github.com/arraylabs/myq

@arraylabs is that repo private?

@r34220 no, i just initially linked it wrong.

Are people testing the set_state() functionality?  The equivalent in my code (Indigo HA plugin) keeps returning a response of
 {"ReturnCode":"217","ErrorMessage":"Error processing request (217)","CorrelationId":"dba949ea-986d-4975-b063-3705316f24e8"}
I swear my code is including the exact same parameters, etc, as @arraylabs's code.

@FlyingDiver Mine was tested earlier today. Looks like you are sending alot of extra params and not sure about the post data, my params are different as well as my header items.

Are you looking at my GitHub repository?  My current code is not uploaded yet.  Is there a way to get requests to spit back all the info that went into the request, so I can compare the actual calls?

OK, latest commit has my current code.  https://github.com/FlyingDiver/Indigo-MyQ/commit/7d1b89f8b9db55c9df7d51777ffbbc95f02b9405
Still getting the 217 error whenever I try to execute my changeDeviceAction function ("DeviceAttribute/PutDeviceAttribute" call).  Authentication works fine, as does the "userdevicedetails/get" call.

@FlyingDiver did you try with params=payload instead of json=payload? I'm only posting json on the authenticate, others are get or put.

It's actually data=payload now, which matches what you have.

Ok all, I updated the code based on @arraylabs findings. Working again -- please let me know if it is for you as well.


Added combineTypedReducers()

This is needed for the times when Dart 2 can't infer types
and is unable to properly convert from Iterable<TypedReducer<S,A>>
to Iterable<Reducer<S>>, which can occur in situations like this:
typedef T F<T>(T t);

F<T> foo<T>(Iterable<F<T>> iter) => iter.first;

class C<T> {
  T call(T t) => t;
}

List<C<String>> getCs() => [new C<String>()];

void main() {
  foo(getCs());
}

That code works fine in Dart 1 but fails to compiled in Dart 2

@brianegan I know we removed combineTypedReducers in 3.0.0, should we consider eventually replacing combineReducers with this version?

Hey, thanks so much for the idea! I'd like to ask if this is really needed? Overall, I think it'd be great if we can keep the API as simple as possible.
For example, you can fix the type error in your example above by changing this line:
List<C<String>> getCs() => [new C<String>()];
To this: (using the abstract typedef rather than an implementation)
List<F<String>> getCs() => [new C<String>()];
In Redux Terms, you can use normal combineReducers in the test case you've provided by changing this line:
List<TypedReducer<String, dynamic>> getTypedReducers() {
to this:
Iterable<Reducer<String>> getTypedReducers() {
Therefore, I feel like this can be solved by returning Iterable<Reducer<State>> from functions without increasing the API surface of this library.

@brianegan for context, I discovered this while trying to update redux within Google's internal repository, and it broke project X, which is not my project.

For example, you can fix the type error in your example above by changing this line:
List<C<String>> getCs() => [new C<String>()];
To this: (using the abstract typedef rather than an implementation)
List<F<String>> getCs() => [new C<String>()];

In trying to fix the errors, I tried that, but it broke other things that were relying on the methods returning List<TypedReducer<AppState, dynamic>>.  Since it wasn't my project, I didn't want to tug too hard on that thread.
This is likely not going to be needed forever.  I asked the Dart team for advise, and they filed https://github.com/dart-lang/sdk/issues/32864.  So it's possible that at some point in the future, the type inference may just work...  /cc @leafpetersen
I totally agree with you on keeping the API surface to a minimum.  I generally loathe doing this, but we could consider adding the function as deprecated from day 1, with a pointer to https://github.com/dart-lang/sdk/issues/32864

@tvolkert Thanks for the info and glad that you've filed a bug for this case :)

Since it wasn't my project, I didn't want to tug too hard on that thread.

Would it be possible to coordinate with the Project X maintainers to make the required changes?The upgrade will require Project X to make changes to support Redux 3.0.0, and this might be the "thread" you were running into.
Alternatively, if this function solves the problem for Project X, would it make sense to only include it in that project?
I kinda feel like adding this function to this package is the wrong place to fix the problem. For us to release then remove the function, we'd be required to release Redux 4.0.0 as it would be a breaking change. Revving versions is a decent amount of work b/c it requires the community and all libs to upgrade.

Interesting discussion! After some more thought, List<TypedReducer<String, dynamic>>  should definitely be a subtype of Iterable<Reducer<State>> and this wouldn't be an issue. It sounds like dart-lang/sdk#32864 might be an easy fix... maybe it will be resolved before the final dart 2 release?

Alternatively, if this function solves the problem for Project X, would it make sense to only include it in that project?

This might make the most sense, it doesn't seem like typical users are very likely to hit this issue. Generally, the pattern is to take a  List<TypedReducer> and pass them directly into combineReducers, which works as-is.

Revving versions is a decent amount of work b/c it requires the community and all libs to upgrade.

I agree we should try to reduce the number of breaking changes.  That being said, I suspect we will need to make breaking changes at some point in the near-ish future, and we can group breaking changes into a single 4.0.0 release.
At the end of the day i have no strong feelings one way or the other but I would prefer to wait and see what happens with dart-lang/sdk#32864

Sounds good.  I'm gonna close this, and we can always revisit if it proves to be more common.  Thanks all!


Made some Improvements

Hello.
Thanks for a great class!
I've made a few improvements that may be of interest for some.
1
In order to get the parallax to gradually disappear when the user holds her phone steady
I Added to MotionGesture:
        manager.subscription = passthrough
            .receive(on: RunLoop.main)
            .sink {[weak manager] data in
                guard let rotation = data?.rotationRate else { return }
                manager?.rotationOffset.width += rotation.y
                manager?.rotationOffset.height += rotation.x
                manager?.rotationOffset.width *= 0.99 // new line
                manager?.rotationOffset.height *= 0.99 // new line
            }

2
I added the ability to use BOTH motion and touch gesture parallax, since that's what I wanted for my project.
In Environment.swift, replace the ParallaxOffset with
struct ParallaxOffsetMotion: EnvironmentKey {
    static var defaultValue: CGSize = .zero
}

struct ParallaxOffsetTouch: EnvironmentKey {
    static var defaultValue: CGSize = .zero
}

and in EnvironmentValues.swift, add this
var parallaxOffsetMotion: CGSize {
        get { self[ParallaxOffsetMotion.self] }
        set { self[ParallaxOffsetMotion.self] = newValue }
    }

    var parallaxOffsetTouch: CGSize {
        get { self[ParallaxOffsetTouch.self] }
        set { self[ParallaxOffsetTouch.self] = newValue }
    }

Then in Parallax, change to:
        let rotationDampened = CGSize(
            width: (parallaxOffsetMotion.width + parallaxOffsetTouch.width) * CGFloat(parallaxStrength),
            height: (parallaxOffsetMotion.height + parallaxOffsetTouch.height) * CGFloat(parallaxStrength)
        )

        let result = content
            .environment(\.parallaxLevel, actualLevel + 1)
            .offset(
                x: offsetDamp * (parallaxOffsetMotion.width + parallaxOffsetTouch.width) * CGFloat(actualLevel),
                y: offsetDamp * (parallaxOffsetMotion.height + parallaxOffsetTouch.height) * CGFloat(actualLevel) // FIXME: Z-index offsetting
            )

I then created the file MotionAndTouchGesture.swift:
import SwiftUI

extension View {

    func parallaxTouchAndMotionGesture(strength: Double = 1) -> some View {
        self
            .contentShape(Rectangle())
            .parallaxLayer()
            .modifier(ParallaxMotionGesture(strength: strength))
            .modifier(ParallaxTouchGesture(strength: strength))
    }
}

And an example:
        ZStack {
            AsyncImage(url: URL(string: "image.jpg")) { image in
                image.resizable()
            } placeholder: { Color("HomeBGColor") }
                .frame(width: 100, height: 100)
                .clipShape(RoundedRectangle(cornerRadius: 16))
                .parallaxLayer(zIndex: 0)

            Text(item.name).header()
                .parallaxLayer(zIndex: 1)
        }
        .parallaxTouchAndMotionGesture()
    }

Voila!
zip with the new code:
www.inlovewith.com/_swift_motion_and_gesture_parallax.zip

@thyselius thanks for the improvements!
It looks great.
if you want, maybe you could fork and add the changes to a pull request, I'd love to have more contributors. Otherwise I can do it.

Wonderful. I'm not a GitHub guy but feel free to do it!
Also I'm having trouble with the images getting very pixly when using the parallax. Have you noticed this? It looks good in simulator, but on device it looks like half res.
I tracked the problem down to
            result
            /*.rotation3DEffect(
                Angle.degrees(rotationDampened.width * 0.2),
                axis: (0,1,0),
                perspective: perspective
            )*/
            .rotation3DEffect(
                Angle.degrees(-rotationDampened.height * 0.2),
                axis: (1,0,0),
                perspective: perspective

If I removed either one of the rotation3DEffect it looked good on device, and as I can hardly see any difference that's what I'm going with right now, but I wonder if you have the same issue?

Couldn't resist trying to fork the project :)
Added my updates and created a pull request.
Check it out and let me know what you think.
I made another, like having separate strengths for motion and touch.
There's also a invertStrength property that I created in order to have a blurred white circle move in the opposite direction to create a light reflection, it looks really neat.
https://github.com/louis1001/layer-explosion/assets/386502/ed7bb0bd-b3b6-4ed3-9bc7-59df2de916ad

Leaving a comment to remind myself I have to merge your Pull Request.
That UI looks really good BTW. The inverse 3d effect really adds a lot.


Request: Behance Icon

Hi, I noticed no Behance Icon has been added with the social icons, could this be added soon possibly in the next version? I'm going to have to use SVG icons for my social page for now, rather than being inconsistent having Ionicons for some and an SVG for another. This would be a really beneficial social icon for my use and probably a lot of other, thanks!

Just a side note, you can use icomoon.io to convert your SVG icons into a font, and combine them with ionicons.


Any way to obtain "window" object in wasm spawned in Web Worker such as in the case of demo?

So in the demo, we load wasm-worker.js by spawning a new Worker, which in turn patches the actual multhread-able wasm program.
I wonder how we can obtain web_sys::Window object inside that wasm program. I keep seeing that web_sys::window() returns None. wasm-bindgen repository has an example of running wasm-enabled program in Worker but the way they applied is a bit different from what's laid out in the demo. Can I get some ideas?

Right, you can't because Web Workers don't have access to the window / document.
Instead, the idea in this setup is that you expose pure Rust as exports from Wasm, and do any UI work and window manipulation from JS instead, which should also be more efficient as it avoids extra layer and conversions between DOM -> JS -> Wasm (Rust).
Since the demo uses Comlink, another setup you could try is look into exposing some methods from the main thread (or even window itself) via Comlink to the worker, but if that's the route you want to take, I'm afraid it's something you'd have to look into on your own.


⚠ CDN (Staging) has degraded performance

In d507339, CDN (Staging<EMAIL_ADDRESS>experienced degraded performance:

HTTP code: 200
Response time: 1254 ms

Resolved: CDN (Staging) performance has improved in d15e8ef.


Can you update the code for .Net Maui (Android)?

I added this package to a .Net Maui project and it works great when I run it on a Windows device emulator. When I try to run it on an Android emulator (Samsung Tablet) I get an error. The tablet is paired to my Star TSP100 POS Printer. Like I said, it works fine and prints on the same printer when I run it as a windows device. I need to get this to print from an Android tablet for a POS system I'm planning on building.
System.DllNotFoundException: 'winspool.Drv'
Any help would be greatly appreciated. Thanks!

I was able to print in .Net Maui.
private void btnPrint_Clicked(object sender, EventArgs e)
{
Printer printer = new Printer("TSP100-1");
System.Text.EncodingProvider ppp = System.Text.CodePagesEncodingProvider.Instance;
Encoding.RegisterProvider(ppp);
        printer.TestPrinter();
        printer.FullPaperCut();
        printer.PrintDocument();
    }

@datadumped I tried your code, still giving 'winspool.Drv' error  from Android. Can you pls help?


Object Detection (SSD-VGG): Loss nan

Hi,
I trained the SSD for object detection using VGG instead of MobileNet and get loss nan as below:

Pre-trained model VGG was downloaded from here .I trained my own data on Jetson Xavier. It's ok if training using MobileNet as in instruction. Any suggestion to make it work with VGG?
Thanks!

Hi @hoangcuongbk80, I have not tested SSD-VGG - you might want to check the upstream repo here:  https://github.com/qfgaohao/pytorch-ssd

Thank you!


Run make gen

/hold let's see if the automator does its job

@luksa that job will only run when something merges in 1.0,1.1 or 2.0 branches

(or playground)

Yeah, that's why I'm pushing things to the playground branch :)
But the automator doesn't seem to work - make update-1.0-charts can't checkout the 1.0 branch for some reason. Investigating...

Yeah, I can reproduce the error locally, for all branches.

Let's merge this so it doesn't block others. We can investigate/fix the automator job playing in the playground

Yeah, I can reproduce the error locally, for all branches.

Weird. I couldn't reproduce it on my machine. Not by simply running the steps that the automator script runs. But I didn't actually run the script.

Do a fresh clone in an empty dir, just like the Automator does.
On Wed, Sep 8, 2021 at 8:41 AM Marko Lukša @.***> wrote:

Yeah, I can reproduce the error locally, for all branches.
Weird. I couldn't reproduce it on my machine. Not by simply running the
steps that the automator script runs. But I didn't actually run the script.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/maistra/istio-operator/pull/840#issuecomment-915204009,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAA6WP7MT3O7WLGS7KL75ODUA5KX3ANCNFSM5DUGHOFQ
.

--
Jonh Wendell
https://bani.com.br

@jwendell I did, but I cloned the entire repo, I didn't use the --single-branch --branch maistra-2.1 options, so this is what's causing the issue.


RetryableIterator did not work... this should!

Codecov Report

Merging #544 (1dc4586) into master (94923c9) will increase coverage by 0.01%.
The diff coverage is 77.77%.

@@            Coverage Diff             @@
##           master     #544      +/-   ##
==========================================
+ Coverage   71.33%   71.34%   +0.01%
==========================================
  Files         118      118
  Lines       10681    10693      +12
==========================================
+ Hits         7619     7629      +10
- Misses       3062     3064       +2

Impacted Files
Coverage Δ

assemblyline/datastore/stores/es_store.py
59.27% <77.77%> (+0.19%)
:arrow_up:

assemblyline/datasource/alert.py
88.46% <0.00%> (-3.85%)
:arrow_down:

assemblyline/odm/randomizer.py
94.29% <0.00%> (ø)

assemblyline/common/heuristics.py
81.70% <0.00%> (+2.43%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 94923c9...1dc4586. Read the comment docs.


CLI interface

Because no standard tool can unpack encrypted archives locally, it'd be really great to have a basic CLI interface which can:

Unpack all files from the archive, allowing to use plaintext password and take care about converting password to key (like superviser does).
Pack files, with gzip or without, encrypted or not.
Detect if gzip should be used based on file extension.
Maybe specify buffer size as well.

I can put some PR if you're happy with that.

work for me, feel free to create a PR for that


Add an attribute to configure a "theme"

Currently, Asciidoctor Web PDF relies on the attribute stylesheet to configure the styles.
For reference, if the attribute stylesheet is defined, it will override all the styles.
In other words, if you want to add additional styles but still use the default styles, you will need to manually add them.
Since some styles are conditionally added, it can quickly get complicated.
To simplify things, we could support the attribute pdf-theme (and pdf-themesdir) to configure a theme (similar to what Asciidoctor PDF does):
asciidoctor-web-pdf -a pdf-theme=default.css

The pdf-theme attribute should support multiple values separated by colons.
And we should also probably honor a pdf-link-css (or pdf-linkcss) attribute to either use a <link> element or inline the content of the CSS into the HTML page.

For reference, Asciidoctor reveal.js is using the attribute customcss. This attribute only supports a single value. For instance:
:customcss: dark.css

Another suggestion by @mojavelinux:
stylesheet=+custom.css

The main benefit is that we don't introduce (yet) another attribute and it reads nicely.
It also works when we define multiple style sheets:
stylesheet=+custom.css,dark.css


Timestamp differs on localhost and remote client

Running driftsync server locally, and two driftsync C# clients: one on the same machine as the server and another one on a remote LAN machine.
I'm having a noticable difference in the global timestamp value on these two for some reason - about 40-50ms at least.
Please see this video: https://www.dropbox.com/s/z5578xmi9nyzabz/DriftSync_LocalAndRemote.MOV?dl=0
(the left one is the server)
I'm using SECONDS scale.
The time being sent from the server is starting from 0. Using simple code like that: packet.remote = localTime() - start_time; where uint64_t start_time = localTime(); before the server loop start.
Do you have any ideas on why that's happening?

Hi there
I'm not entirely sure I understand what you mean by your code sample and "time being sent from the server". Usually you would connect multiple DRIFTsync clients to the same DRIFTsync server and then use sync.globalTime() on all of them and that global time should be the same across all of the clients. The localTime is really the local clock and is not kept in sync. You can use the localTime as a reasonably high resolution clock source for relative timing, or it can be used to calculate offsets between the globalTime and the localTime where this is useful. Or do you simply mean you adjusted the server code to start from 0 by initially caching the localTime and subtracting it on each packet?
In case your visualization indeed shows the global time (i.e. from sync.globalTime()), then it should definitely be closer. Can you check what the median round trip time is for both clients? Is it unexpectedly high, possibly leading to less accuracy? In a wired LAN I'd expect it to be around 1ms.
Something to watch out for: The display output path of devices can be quite different and therefore cause different display latency. This can happen due to different configuration, i.e. only one side using double buffering or the like, or it can happen due to the display hardware. If it is indeed the output path, it is usually a fixed and constant difference that you would correct for with a simple offset configured per device. 40-50ms is somewhat high to be caused by this however.
Audio can be a more accurate mean of checking for sync, as audio latency is often known and can therefore be accounted for. Measuring it can be done by producing beeps from multiple devices, recording them and then inspecting the recording closely.

Hi!
Some more results. Same as before left computer is running driftsync server and also a client instance.

Showing globalTime with python client
https://www.dropbox.com/s/q5pevrwl6svdv7m/driftsync_python.MOV?dl=0
The difference is changing from 5-30 ms through time. No constant difference which could be caused by the presentation time difference.

Running C# client on left computer, and C# + C client (under WSL) on the right one.
The medianRoundTripTime on right is 1-2ms (not visible on the video)
https://www.dropbox.com/s/b96ew33mg4fzr37/driftsync_unity_vs_c_2_machines.MOV?dl=0
The difference is also not constant, but is being kept in same range more or less.
Right WSL c client: +10+-5ms relative to the Left C# client
Right C# client: - 20+-10ms relative to the Left C# client

Running WSL c client VS C# client on the left computer showed that WSL c is being significally left behind
for some reason.
https://www.dropbox.com/s/uagynb8m4qkglwr/driftsync_unity_vs_c.MOV?dl=0
But that's not a real problem - most interesting cases are 1-2 and the difference is indeed changing through time

Just for a sanity check, I've tested my current syncing implementation on the same setup today (bottom - server, top - client):
https://www.dropbox.com/s/v944k3tbh83ldzx/UDP.MOV?dl=0
It's just a UDP server which sends current timestamp 120 times a second to all the clients.
It's not continious, so it's difficult to use it for smooth playback (more frequent updates lead to jittering anyway), that's why I use that value just for correcting the playback position in case of significant differences between local and remote clock on the client.

Hi there
Thank you very much for the many test cases, this is indeed very interesting.
Unfortunately the video compression makes the numbers somewhat difficult to read in the Unity test cases, so it is harder to compare the numbers over a period of time. The python video is very crisp though and I've stepped the video and wrote out the numbers. The resulting graph is interesting and shows that the local clock apparently runs differently across both machines:

The left machine always does roughly steps of 16 ms, but every ~14 steps it jumps by about 31 ms. The right machine on the other hand updates in steps of around 18 ms and then drops to one step of 9 ms every ~9 steps. The time period of this sampling is small enough that the actual sync process should not be involved at all (unless you reduced the interval), which lets me conclude that the error has to come from the accuracy of the localTime value.
I can't really explain why the two machines apparently run at different internal intervals, but this might be inherent to the clock source being used. The accuracy of sleeping under Windows is generally relatively bad however (i.e. the wakeups happen at irregular intervals) and therefore what's visible here might just be an effect of the sleep loop in the Python client not waking up as regularly as desired. It might also be related to power save features, can you check if both machines are set to the same power profile, and ideally set both to performance?
The good news is that the two machines stay within bounds that would be somewhat expected given the above and average at an offset of 5 ms. So at the very least, the sync itself is within a frame.
I have no idea why the WSL C client is off by 1.1s, that doesn't make sense to me at all. Is this offset constant or does it vary wildly?
The last video you uploaded with your own reference seems to be gone, but I'd be interested in seeing this.
As an additional test, can you try an external reference server? There is a public server hosted at driftsync.org that you can work with. This should take out the difference from one machine using a local server. The public server also runs under Linux, which may or may not be relevant. If the local clock source under Windows indeed jitters significantly, using it as a server may be problematic. Generally the low update frequency and length of integration time should make this factor insignificant however.

Another thing that just came to mind: The Unity rendering is pipelined by default and the various frame *Update hooks will sometimes be called with jitter resulting from that. At low refresh rates this may introduce quite a bit of variation. Also, the clock reference for localTime that DRIFTsync uses is not tied into the render timing of Unity and therefore isn't corrected for some of that variation. This blog post is quite revealing of what is going on under the hood and how this may interact here:
https://blog.unity.com/technology/fixing-time-deltatime-in-unity-2020-2-for-smoother-gameplay-what-did-it-take
Maybe you can try lowering QualitySettings.maxQueuedFrames to 1 so that pipelinging is effectively disabled and then see if this improves things? Also try using VSync in that combination as it should further stabilize the timing of the hooks being called.
Which of the *Update hooks do you currently use for updating the timestamp?

Hi!
Wow, nice analysis! I'm not fully understanding it yet, but I will try to read again, while I'm doing more tests.
I tend to believe that clock under windows can be not as precise as under linux. However it's "drifting" a little bit too much in my opinion even for a not 100% ideal clock implementation :P

Unfortunately the video compression makes the numbers somewhat difficult to read
It's most likely dropbox-introduced artifacts. Will post links to download videos. They will have better quality compared to dropbox player.

The last video you uploaded with your own reference seems to be gone, but I'd be interested in seeing this.
Sorry for providing an invalid link. Here is a new one: https://www.dropbox.com/s/bu5g2nb9lctiyjz/UDP.mov?dl=1

can you check if both machines are set to the same power profile, and ideally set both to performance?
I got my second desktop comuter from service yesterday, so I can run 2 remote clients now. Both are running on high performance power profile.

Connected to windows-based LAN driftsync server: https://www.dropbox.com/s/es27cdobsdkpugc/driftsync_2remotes_win_srv.MOV?dl=1
Still poor syncing, even when both clients are remote.
Public driftsync.org server: https://www.dropbox.com/s/loat6sx71k2rb8z/driftsync_2remotes_public_srv.MOV?dl=1
I would call that somewhat better syncing, although it's still far from ideal

I have no idea why the WSL C client is off by 1.1s, that doesn't make sense to me at all. Is this offset constant or does it vary wildly?
The offset is varying in approximately 0.5-1.2s range which is kinda crazy for me. Should be some WSL specific thing :\

I will see if I can make some tests using Linux next days, finally, just to validate that I get perfect sync in same environment driftsync was developed on (with driftsync.org, probably), if that makes sense (I assume nobody tried it on windows before).

Hi there,
Posting fresh Ubuntu install tests results below. Please take a look.
Connecting to driftsync.org using the default update interval.

Python client [code]
https://www.dropbox.com/s/5a9bmgjbi27418i/driftsync_ubuntu_python.MOV?dl=1
C client [code]
https://www.dropbox.com/s/p20iabnkpj25r5i/driftsync_ubuntu_c.MOV?dl=1

Cheers

Hi
I've investigated this some more and come to an interesting conclusion: It is probably indeed a drift on the display side. I can reproduce the apparent issue in a similar way. It does not stem from an actual synchronization issue however. To verify this, I've written a small tool that takes in timestamps from arbitrary sources and compares them:
https://gist.github.com/mmlr/0610f53dc51b5c9f3503b9f63327e234
It can be used to run a local and a remote DRIFTsync client in stream mode and verify the sync quality (it can also be fed from multiple local clients, for example in different implementations or fed data simply from a UDP socket via netcat). It corrects away the difference in arrival time of timestamps from all its sources but otherwise does nothing else than compare them. Obviously this only works when the network latency is low and stable.
On my network, this gives results in the low one digit milliseconds, most of that certainly stemming from spurious network latency jitter. This is stable across various sources, multiple local clients, local vs. remote machine via SSH or UDP, even local client vs. one running on an Android phone.
The display of the same data across two machines however exhibits a similar pattern to what we saw earlier. Looking at the boundaries of the jitter pattern, it is indeed bounded by one frame time and follows the two fundamental frequencies and their spikes.
I also have a theory as for why you don't see it in your reference video with sending a timestamp out via UDP. This is based on the assumption, that you are sending out a single timestamp over the course of a frame, i.e. you send it multiple times but always with the same value and that value only changes once every frame of the server. If this is indeed how it works, it is easy to explain why you see the same value on both sides: there is only one value to begin with, the one from the server. This is in contrast to DRIFTsync where the timestamp is generated locally (albeit aligned with the global reference) and there indeed being two independent values.
DRIFTsync simply shows the actual difference between the two sides while the UDP timestamp doesn't. In the UDP case, one can still see this difference in the video, but it's shown by the difference of when the value becomes visible on each display instead of by the value itself. It can clearly be seen, that the values don't actually change at the same time on both displays, for example in the case of the value 0,133 seen 3.17 seconds into the video where the client actually shows 0,013 first for a couple of video frames until the server catches up. After that they appear somewhat in sync for a couple of values but then the 0,300 is again slightly quicker to complete on the client for example. Even at 4x or 8x slowdown, this variation is tiny, so it's not that obvious as actually seeing different values.
As for what this actually means: There is a difference in resolution between the two methods. If you were to reduce the DRIFTsync resolution, i.e. by taking the values modulo your frametime, the values should start to match up again. Conversely, if you started to send sub-frame timestamps out via UDP (for example by using the tick counter as a source), you should see the same difference in values showing up between the displays.
In general, the time synchronization is actually the easy part, the hard part being applying this timestamp consistently to something frame based. That is also the reason why DRIFTsync generally operates "slowly" by integrating differences over a larger window of time to make the small variances insignificant. It is also the reason why it provides the suggestPlaybackRate API and not a frame based one. Aligning the initial timestamp and then keeping the playback rates the same between the local and global clock is simpler and more accurate than to try to keep them in sync frame by frame. But it does mean that your clients won't necessarily drop/repeat the same frames to stay in sync.

Hi!

I also have a theory as for why you don't see it in your reference video with sending a timestamp out via UDP. This is based on the assumption, that you are sending out a single timestamp over the course of a frame, i.e. you send it multiple times but always with the same value and that value only changes once every frame of the server.

That's right! I also think I can understand the analysis you've done and the tool does the verification in a correct way, unlike my test setup. I'll give it a try to see the actual network jitter in my case. Thanks for all the work! Really appreciate
Just to finish the "incorrect" tests and move forward, I'm posting couple more videos.
After reading the article you posted I've switched to Unity 2020, as it seems to have some fixes for the main game thread and deltaTime.
The video being played is now 60fps instead of the old 30fps one.
I also seem to invent my own verfication way in parallel to you, although I'm not so sure if it's valid or not. I'm running an independent local System.DateTime.Now.Ticks based clock (it has 100ns precision) instance on each client. Then I on each frame calculating the difference between the that clock instance and the globalTime value. What I see so far is that the difference is also jittering only within a 1-millisecond range. So maybe that's also a good indication? It's called globalTime-simpleTime on the videos.

Max Queued Frames = 1
https://1drv.ms/v/s!AhRsdPdtJl1uhpYVKML6tZho1p8z3A?e=gTh4zj
Max Queued Frames = 2
https://1drv.ms/v/s!AhRsdPdtJl1uhpYWp6UsVynwkFhDSA?e=xfygwP

I don't see why one is better than another here. But it's nice to have the latency control and it might be useful in future when using the suggestPlaybackRate approach.
Btw: even the frame based sync looks really good here