[BUG] v2 snapshot creation time will change in UI from time to time

Describe the bug

The reason is that, the caller randomly picks up a replica and its snapshot map each time when the UI requires snapshot info. And there may be 1 second gap when creating a snapshot among different replicas.
I am trying to ask the engine to collect the info and always pick up the replica that holds the earliest ancestor as the info provider. But the following is, things still change once all replicas get rebuilt.
For example:

Before rebuilding
Replica1: snap1 (time=5) -> snap2 (time 7) -> volume-head (time 12)
Replica2: snap1 (time=4) -> snap2 (time 8 ) -> volume-head (time 13) <== pick up this
Rebuilding replica 1
Replica1: snap1 (time=14) -> snap2 (time 15) -> volume-head (time 15)
Replica2: snap1 (time=4) -> snap2 (time 8 ) -> volume-head (time 13) <== pick up this
Rebuilding replica 2
Replica1: snap1 (time=14) -> snap2 (time 14) -> volume-head (time 14) <== pick up this. Change!
Replica2: snap1 (time=20) -> snap2 (time 21 ) -> volume-head (time 21)

Currently, there are 2 solution candidates:

[ ] Make the creation timestamp as the input of lvol creation API
[ ] Put the creation time in lvol xattr instead

To Reproduce

Create and attach a v2 volume
Create some snapshots
Wait a while, then delete one replica
Detach the volume and trigger the offline rebuilding. And wait for rebuilding complete
Wait a while, then delete all other replicas
Wait for rebuilding complete
Reattach the volume, check the snapshot creation time

Expected behavior
The creation time should not change
Support bundle for troubleshooting

Environment

Longhorn version: v1.6.0-rc1
Impacted volume (PV):
Installation method (e.g. Rancher Catalog App/Helm/Kubectl):
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version:

Number of control plane nodes in the cluster:
Number of worker nodes in the cluster:

Node config

OS type and version:
Kernel version:
CPU per node:
Memory per node:
Disk type (e.g. SSD/NVMe/HDD):
Network bandwidth between the nodes (Gbps):

Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal):
Number of Longhorn volumes in the cluster:

Additional context

cc @derekbit

Creation timestamp is created when taking a snapshot. Unlike v1 volume, the creation timestamp is stored in meta file, so it won't be changed after rebuilding.
After discussing with @shuo-wu, one solution is that we can save creation time got from engine to the xattr of a lvol rather than using the value created when taking a snapshot.

Creation timestamp is created when taking a snapshot. Unlike v1 volume, the creation timestamp is stored in meta file, so it won't be changed after rebuilding.
After discussing with @shuo-wu, one solution is that we can save creation time got from engine to the xattrs of the downstream lvols rather than using the value created when taking a snapshot.

Unfortunately, xattrs are read-only for a snapshot lvol.
I'm thnking if I can add a force option in bdev_lvol_set_xattr method.
WDYT? @shuo-wu @DamiaSan @innobead

I don't know if it is a good idea, if we add this option we must add it not only to spdk_blob_set_xattr but also to spdk_blob_sync_md. Because in spdk_blob_sync_md we have:
	if (blob->md_ro) {
		assert(blob->state == SPDK_BLOB_STATE_CLEAN);
		cb_fn(cb_arg, 0);
		return;
	}

A snapshot has read only metadata and, setting an xattr, make the state of the blob SPDK_BLOB_STATE_DIRTY. So we run into the assert.

For head lvols, you can change the creation time with the RPC bdev_lvol_set_xattr.
For the snapshots, what we can do is to pass a list of xattr to the RPC bdev_lvol_snapshot: in this way we can set the creation_time we want and we can also set the user_created flag when needed.
WDYT? @shuo-wu @DamiaSan @innobead
Let me know because this impact the development I am finishing about the user_created flag.

For the snapshots, what we can do is to pass a list of xattr to the RPC bdev_lvol_snapshot: in this way we can set the creation_time we want and we can also set the user_created flag when needed.

Sounds doable...TBH, the way is a little bit ugly.
Currently, all metadata of a snapshot lvol is always immutable, but I think it should support mutable fields as well.

Why do you think it is not easy to maintain?
If a snapshot is immutable, you are simply writing all your metadata when you create it. SPDK internally already does something similar, writing lvol name and uuid as xattr of the snapshot blob it is creating.

@DamiaSan
Please ignore the previous comment (I already deleted it). I misunderstood your idea before.

For the snapshots, what we can do is to pass a list of xattr to the RPC bdev_lvol_snapshot: in this way we can set the creation_time we want and we can also set the user_created flag when needed.

Yes, this is what we said in above conversation.
BTW, when rebuilding a replica, can we read the source snapshot's creation timestamp and create a destination snapshot with it? Is my understanding correct?

Implemented a generic list of xattr in this issue: #7578

Hello @DamiaSan,
Before moving it to review, need to integrate to go-spdk-helper, longhorn-spdk-engine and longhorn-instance-manager as well.

Related PR for SPDK support: https://github.com/longhorn/spdk/pull/26

@DamiaSan Is it resolved?

I think so, all the works have been done.

Wait... Do we update longhorn-spdk-engine and longhorn-manager?

Wait... Do we update longhorn-spdk-engine and longhorn-manager?

You are right

I was thinking that, if we let a snapshot to store new xattr after its creation (like for example the checksum), this functionality to pass xattr at snapshot creation may become obsolete.
WDYT?

PR for types https://github.com/longhorn/types/pull/12

Do you mean asking the snapshot API in go-spdk-helper to add the creation timestamp xattr? I am fine with it.

We have decided to calculate the snapshot checksum inside SPDK, so there is no need to implement the functionality to add an xattr after the snapshot creation. So the functionality to add xattrs at snapshot creation will continue to be used.

OK. Then, go head remove it.

After types, some works on go-spdk-helper and longhorn-spdk-engine are needed to complete the issue.

longhorn-spdk-engine PR https://github.com/longhorn/longhorn-spdk-engine/pull/162
longhorn-instance-manager PR https://github.com/longhorn/longhorn-instance-manager/pull/534

Hello @DamiaSan
Need to fill the form https://github.com/longhorn/longhorn/issues/7641#issuecomment-1897918543

Hello @shuo-wu, @DamiaSan
Currently v2 volume offline rebuilding is disabled on longhorn master, the reproduce steps are outdated, could you provide other validate steps for this issue? thank you.

The offline rebuilding is disabled but now we should have the online rebuilding. However, whatever is the rebuilding implemented, I think the reproduction steps are the same and so still valid. You have to simply wait for rebuilding complete, right @shuo-wu ?
But, before to proceed with this issue, we must before fix the issue that this development has probably introduced.

The offline rebuilding is disabled but now we should have the online rebuilding. However, whatever is the rebuilding implemented, I think the reproduction steps are the same and so still valid. You have to simply wait for rebuilding complete, right @shuo-wu ? But, before to proceed with this issue, we must before fix the issue that this development has probably introduced.

This ticket needs to wait until the v2 volume online rebuilding is introduced first to validate.

cc @shuo-wu @innobead  👆

The offline rebuilding is disabled but now we should have the online rebuilding. However, whatever is the rebuilding implemented, I think the reproduction steps are the same and so still valid. You have to simply wait for rebuilding complete, right @shuo-wu ? But, before to proceed with this issue, we must before fix the issue that this development has probably introduced.

The issue has been fixed, so the development is ready to be tested once the rebuilding will be enabled again.

@chriscchien  online rebuilding has been merged

Hi @DamiaSan ,
Snapshot created in v1.7.x-head not hit the creation time change issue, but hit this issue if snapshot created in v1.6.2 then upgrade to v1.7.x-head
Steps:

Create v2 volume and have data and sanpshots

>kl get engineimage
NAME          INCOMPATIBLE   STATE      IMAGE                               REFCOUNT   BUILDDATE   AGE
ei-b0369a5d   false          deployed   longhornio/longhorn-engine:v1.6.2   0          63d         4m21s
>
> kl get volume
NAME                                       DATA ENGINE   STATE      ROBUSTNESS   SCHEDULED   SIZE         NODE               AGE
pvc-688e096d-d831-4a10-b2af-ecfeaf72c5d9   v2            attached   healthy<PHONE_NUMBER>   ip-172-31-42-172   91s
>
> kl get snapshot
NAME                                   VOLUME                                     CREATIONTIME           READYTOUSE   RESTORESIZE   SIZE        AGE
f78370a0-926e-4436-ae59-2ae10084fe98   pvc-688e096d-d831-4a10-b2af-ecfeaf72c5d9   2024-07-25T01:48:40Z   true<PHONE_NUMBER>    161480704   44s
3a165f31-e52f-4f0a-8eb3-4f30786be050   pvc-688e096d-d831-4a10-b2af-ecfeaf72c5d9   2024-07-25T01:48:48Z   true<PHONE_NUMBER>    108003328   34s
a0ba1dda-f161-4d37-8242-fd66c17868ec   pvc-688e096d-d831-4a10-b2af-ecfeaf72c5d9   2024-07-25T01:49:02Z   true<PHONE_NUMBER>    108003328   19s

Detach v2 volume, upgrade to v1.7.x-head
Attach the v2 volume

> kl get engineimage
NAME          INCOMPATIBLE   STATE      IMAGE                                    REFCOUNT   BUILDDATE   AGE
ei-b0369a5d   false          deployed   longhornio/longhorn-engine:v1.6.2        0          63d         13m
ei-3c71f773   false          deployed   longhornio/longhorn-engine:v1.7.x-head   0          2d1h        3m27s
>
> kl get volume
NAME                                       DATA ENGINE   STATE      ROBUSTNESS   SCHEDULED   SIZE         NODE              AGE
pvc-688e096d-d831-4a10-b2af-ecfeaf72c5d9   v2            attached   healthy<PHONE_NUMBER>   ip-172-31-37-41   12m

Trigger replica rebuilding serveral times, can see snapshot 3a165f31-e52f-4f0a-8eb3-4f30786be050 creation time changed from 2024-07-25T01:48:48Z to 2024-07-25T02:00:44Z

> kl get snapshot -w
NAME                                   VOLUME                                     CREATIONTIME           READYTOUSE   RESTORESIZE   SIZE        AGE
3a165f31-e52f-4f0a-8eb3-4f30786be050   pvc-688e096d-d831-4a10-b2af-ecfeaf72c5d9   2024-07-25T01:48:48Z   true<PHONE_NUMBER>    108003328   11m
a0ba1dda-f161-4d37-8242-fd66c17868ec   pvc-688e096d-d831-4a10-b2af-ecfeaf72c5d9   2024-07-25T01:49:02Z   true<PHONE_NUMBER>    108003328   11m
f78370a0-926e-4436-ae59-2ae10084fe98   pvc-688e096d-d831-4a10-b2af-ecfeaf72c5d9   2024-07-25T01:48:40Z   true<PHONE_NUMBER>    161480704   11m
a0ba1dda-f161-4d37-8242-fd66c17868ec   pvc-688e096d-d831-4a10-b2af-ecfeaf72c5d9   2024-07-25T01:49:02Z   true<PHONE_NUMBER>    108003328   11m
rebuild-05c69e17                       pvc-688e096d-d831-4a10-b2af-ecfeaf72c5d9                                                                 0s
rebuild-05c69e17                       pvc-688e096d-d831-4a10-b2af-ecfeaf72c5d9   2024-07-25 02:00:41.515614094 +0000 UTC m=+302.713638322   true<PHONE_NUMBER>    2097152     0s
rebuild-05c69e17                       pvc-688e096d-d831-4a10-b2af-ecfeaf72c5d9   2024-07-25 02:00:41.515614094 +0000 UTC m=+302.713638322   true<PHONE_NUMBER>    2097152     39s
rebuild-ce53e846                       pvc-688e096d-d831-4a10-b2af-ecfeaf72c5d9                                                                                0s
rebuild-ce53e846                       pvc-688e096d-d831-4a10-b2af-ecfeaf72c5d9   2024-07-25 02:01:20.725620921 +0000 UTC m=+341.923648130   true<PHONE_NUMBER>    0           0s
3a165f31-e52f-4f0a-8eb3-4f30786be050   pvc-688e096d-d831-4a10-b2af-ecfeaf72c5d9   2024-07-25T02:00:44Z                                       true<PHONE_NUMBER>    108003328   12m
a0ba1dda-f161-4d37-8242-fd66c17868ec   pvc-688e096d-d831-4a10-b2af-ecfeaf72c5d9   2024-07-25T02:00:45Z                                       true<PHONE_NUMBER>    108003328   12m
f78370a0-926e-4436-ae59-2ae10084fe98   pvc-688e096d-d831-4a10-b2af-ecfeaf72c5d9   2024-07-25T02:00:43Z                                       true<PHONE_NUMBER>    161480704   13m
rebuild-ce53e846                       pvc-688e096d-d831-4a10-b2af-ecfeaf72c5d9   2024-07-25 02:01:20.725620921 +0000 UTC m=+341.923648130   true<PHONE_NUMBER>    0           40s
rebuild-891d49dc                       pvc-688e096d-d831-4a10-b2af-ecfeaf72c5d9                                                                                0s
rebuild-891d49dc                       pvc-688e096d-d831-4a10-b2af-ecfeaf72c5d9   2024-07-25 02:02:01.160426788 +0000 UTC m=+382.358451023   true<PHONE_NUMBER>    0           0s

supportbundle_ec67e93d-e081-4db1-9a08-7964ec139b0a_2024-07-25T02-05-53Z.zip

Snapshot created in v1.7.x-head not hit the creation time change issue, but hit this issue if snapshot created in v1.6.2 then upgrade to v1.7.x-head

I think this is expected behavior for the snapshots created in v1.6.2, because they don't have snapshot_timestamp xattrs.
@DamiaSan WDYT?

Updated in https://github.com/longhorn/website/pull/951
cc @chriscchien @DamiaSan @innobead

Snapshot created in v1.7.x-head not hit the creation time change issue, but hit this issue if snapshot created in v1.6.2 then upgrade to v1.7.x-head

I think this is expected behavior for the snapshots created in v1.6.2, because they don't have snapshot_timestamp xattr. @DamiaSan WDYT?

You are right. In 1.6.2 snapshots don't have snapshot_timestamp xattr, so we make fallback on creation_time that change when the snapshot is recreated.

Thanks @DamiaSan's confirmation. @chriscchien I've added the known issue to the important note.

Verified pass on longhorn v1.7.0-rc2(longhorn-spdk-engine c6a703), longhorn master(longhorn-spdk-engine c6a703) with test steps

V2 volume snapshot created in v1.7.0-dev and master will not change creation time while replica rebuilding.
V2 volume snapshot created in v1.6.2 will change creation time while replica rebuilding after upgrade to v1.7.0-dev is expected


engine: print query in case of panic

Recover from panics and print the error in case of a panic.
It looks like the following:
level=error ts=2022-10-05T08:12:27.931977143Z caller=engine.go:146 msg="runtime panic in parser" expr=sum(up) err="runtime error: invalid memory address or nil pointer dereference" stacktrace="goroutine 45 [running]:\ngithub.com/thanos-community/promql-engine/engine.recoverEngine({0x26ff760, 0xc00011c640}, {0x271d740, 0xc00091c600}, 0xc000926b80)\n\t/home/giedrius/dev/promql-engine/engine/engine.go:144 +0xc6\npanic({0x1eb2580, 0x39b01b0})\n\t/usr/lib/go-1.19/src/runtime/panic.go:884 ...

We need to set *promql.Result in advance so that later on it wouldn't return nil in case of a panic.
This is needed so that we could debug problems in production if / when the engine crashes.
Closes #51.
Signed-off-by: Giedrius Statkevičius<EMAIL_ADDRESS>

Nice, is there a way we can add a test for this?

Nice, is there a way we can add a test for this?

We could add tests for helper functions but to test how all of this works holistically, I think we'd have to introduce some kind of fault into the code :smile: would that work?

Hm, what about throwing a panic from storage?

Hm, what about throwing a panic from storage?

Thanks for the suggestion! Added a test that checks for recovery from panics via storage.


Keep the desired code.s3* fields part of the spec

Fixes https://github.com/aws-controllers-k8s/community/issues/1402
The AWS Lambda API doesn't return the original submitted s3Bucket,
s3Key, and s3ObjectVersion when a function is successfully created.
Instead, it returns different values (In Getfunction call) in a
different response field (Code.Location). This is causing the
generated code in ACK to set the spec.Code.s3* fields to nil values.
To keep the experience in ACK lambda-controller consistent in the
Kubernetes-land, we must keep the original submitted code.S3* fields
part of the function's spec.
This patch removes the part that is setting the spec.Code.s3* to nil
and allows the controllers to keep the original values part of the spec.
By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
Signed-off-by: Amine Hilaly<EMAIL_ADDRESS>

/hold

/unhold

/test lambda-unit-test

/lgtm


COMMONSRDF-80: Bumping Jena version 3.5.0 => 3.9.0

The jena dependency is currently tied to the jena-osgi artifact. I think it would be much better to set the dependency to jena-arq. This way, in a non-OSGi context, dependency resolution is easier to manage; in an OSGi context, a developer would still use the jena-osgi module at deploy time.

As another Jena committer, I'd be happier with that too. Thoughts? @stain ? @wikier? Others?

It is not as ideal, as it would make commons-rdf-jena as a Maven package no longer be a sufficient OSGi dependency by itself, as jena-arq is not an OSGi package (for some reason).
But I understand it would make it easier for anyone using the regular jena-arq  dependencies together with Commons RDF to avoid duplication (particularly if you get different versions of Jena).
I think for the OSGi dependencies to be calculated correct for commons-rdf-jena we need to then still list jena-osgi as an <optional> dependency.
While the ideal solution would be for jena-arq etc. to be OSGi packages themselves (avoiding need for jena-osgi overall) I'm OK with the above solution for commons-rdf-jena. OSGi users would simply need to ignore jena-arq and add jena-osgi, we can document that.

OK, changed to apache-jena-libs in ea52c2ff, also documented jena-osgi use
It seems apache-jena-libs also adds in jena-tdb2 and legacy jena-tdb - perhaps this is a bit excessive? Could you verify @ajs6f - it's what is documented on https://jena.apache.org/download/maven.html so presumably this would help the transitive jena-* all be in the same version, or would we need a <dependencyManagement>?

Great, would you be able to update implementations.md on how to use the jena-osgi-features as an alternative? I'll change to jena-arq which should be sufficient.


Device position on map not displaying all devices

We need to provide scroll bar here and also place holder to indicate minimum 3 characters are required to search for device

Fixed in 2.0.1


640x480@50hz

Is it possible an valid to output 640x480 at 50hz?
I built an RGB2HDMI scanner and the limitation is the  memory to store all the video.
If it could be possible to synchronize RGB timings to hdmi timings, I would only need just a few width lines to capture and then render to hdmi.

If you scaled the regular 640x480 60Hz mode down to 50 Hz then the pixel clock would be too slow for the DVI spec (minimum 25 MHz) and most monitors won't accept it. You might be able to design a 640x480 CEA mode that some monitors woluld accept.

Experimentally this seems to work: just add additional hblank to the existing 480p60 mode to increase frame time by 1/5th.
diff --git a/software/libdvi/dvi_timing.c b/software/libdvi/dvi_timing.c
index cff8edc..82ef50b 100644
--- a/software/libdvi/dvi_timing.c
+++ b/software/libdvi/dvi_timing.c
@@ -14,7 +14,7 @@
 // VGA -- we do this mode properly, with a pretty comfortable clk_sys (252 MHz)
 const struct dvi_timing __dvi_const(dvi_timing_640x480p_60hz) = {
        .h_sync_polarity   = false,
-       .h_front_porch     = 16,
+       .h_front_porch     = 16 + 160,
        .h_sync_width      = 96,
        .h_back_porch      = 48,
        .h_active_pixels   = 640,


Not compatible with angular 2+

I am trying to unit test an angular service in my application, which creates socket.io client. My service looks something like this:
export class SocketService {
  private name: string;
  private host: string = window.location.protocol + "//" + window.location.hostname + ":3000";
  socket: io.Socket;

  constructor() { }

  subscribeEvent(name: string): Observable<IJobResp> {
    this.setup(name);
    return new Observable(observer => {
      this.socket.on('job', val => observer.next(val))
    })
  }

  private setup(name: string) {
    this.name = name;
    let socketUrl = this.host + "/" + this.name;
    this.socket = io.connect(socketUrl);
  }
}

So to write my test, I am using the mock-socket library to set up a mock socket.io server. Here is what the test looks like:
describe('SocketService', () => {
  let socket;

  const mockServer = new Server('http://localhost:3000/');
  mockServer.on('connection', server => {
    mockServer.emit('job', 'test message 1');
  });

  (window as any).io = SocketIO;
  beforeEach(() => {
    TestBed.configureTestingModule({
      providers: [JobService]
    });
  });

  it('should xyz, inject([JobService], fakeAsync((service: JobService) => {
    service.subscribeEvent('/').subscribe(val => {
      expect(val).toEqual('test message 1');
    })
  })));
});

However, that test throws the error:

Error: Cannot make XHRs from within a fake async test.

If I don't have the fakeAsync, then the test passes before the assertion in the subscribe() is processed.

+1 also running into the same issue

Additionally If I try to just use async, I get this:
  it('should subscribe to dwnTime events', async(inject([JobService], (service: JobService) => {
    service.subscribeEvent('line/602').subscribe(val => {
      expect(val).toEqual('test 2');
    })
  })));

This doesn't seem to work, I get error: Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.

@ebaker @georgeedwards Thanks for bringing this up. Are either of you able to create a sample repo that I can debug against so I can see what is going on?

@thoov Thanks for getting back to us. Please see this. I have tried to keep this as close to the docs suggestions.

yarn install or  npm i
ng test

Thanks @georgeedwards. I will take a look at that shortly and report back with what I find (and hopefully a solution or fix)

Ok so I figured out what was going on in the sample repo that you sent me @georgeedwards. The issue is that the window object is not properly being stubbed out. This line (https://github.com/georgeedwards/mocksocketdemo/blob/master/src/app/socket-demo.service.ts#L13) is never being swapped out with the MockSocketIO object. Basically you need to do something like this:
import { Injectable } from '@angular/core';
import * as io from 'socket.io-client';

@Injectable()
export class SocketDemoService {
  socket: io.Socket;
  messages: Array<string>;
  constructor() {
    this.messages = [];
  }

  setup() {
    if (window.SocketIO) {
      this.socket = window.io.connect();
    }
    else {
      this.socket = io.connect('http://localhost:8080');
    }
    this.socket.on('chat-message', data => {
      this.messages.push(data);
    });
  }
}

Let me know if you need me to explain if differently

@thoov OK, thanks for finding that!
Would be really nice to not have to change the code you are trying to test. Having separate logic specifically for testing always makes me slightly nervous, but it's a good work around 👍

@georgeedwards 100% agree. Im going to continue to think about your use case and see if I can come up with a better solution. I will post it here if I do

@thoov Awesome, thanks.
One other issue I am facing is how to run multiple tests with different server configurations, I currently have this:
  it('should subscribe to job events', async(inject([JobService], (service: JobService) => {
    mockServer = new Server('http://localhost:3000/line/602');
    mockServer.on('connection', server => {
      mockServer.emit('job', 'test message 1');
    });
    (window as any).MockSocketIo = SocketIO;

    service.subscribeEvent('line/602').subscribe(val => {
      expect(val).toEqual('test message 1');
      mockServer.stop();
    })
  })));

  it('should emit jobresp event on comment complete', inject([JobService], (service: JobService) => {
    mockServer = new Server('http://localhost:3000/line/602');
    mockServer.on('jobresp', server => {
      mockServer.emit('job', 'test message 2');
    });

    service.subscribeEvent('line/602').subscribe(val => {
      expect(val).toEqual('test message 3');
    })

    service.completeJob(('job1' as any))
  }));

but get error:

Error: SocketIO is already in CLOSING or CLOSED state

Any idea where this is going wrong?

@thoov any idea on how to run multiple consecutive tests with different servers, as shown above?

Hi @georgeedwards. Sorry for the delay, I am currently in Europe so don't have a ton of laptop time. There are a few things it might be but first can you verify that mockServer.stop(); is called in the first test? The second thing is that could you verify that if you comment out the first test the second one passes and if you comment out the second test the first one passes?

Any news?


How to change the shape of the curve?

I have drawn a curve grid body with method Bodies.fromVertices , I want to change the shape of the curve when I click a button, there are have some method to update it?

If the vertices are convex you can use Body.setVertices. Depending on your situation, after you have used  Bodies.fromVertices you may be able to just loop through body.parts and use Body.setVertices on each part, so long as you guarantee each part stays convex.
If they're concave then this won't work, you'll need to use Bodies.fromVertices or write your own function that can decompose the vertices and update the body parts. See the source for Bodies.fromVertices to see what I mean.


Playlist Images don't update if updated via spotify

If a playlist creator updates their playlist image, it needs to update in the app.

This involves some work around backgrounds sync (Spotify -> Grüvee). Not exactly sure how we should handle this, but it seems more and more features are going to require some sort of backwards sync like this.


Self punishment

Prevented the currently authenticated user from being able to ban themselves.

nice catch!

@mhurwi Haha, cheers but might need some additional checks as this is only hiding the button and not preventing it from happening (this is my first 24hrs of Ruby existing in my life).

thanks!


Metal complex should be able to set alignment

@andrewtarzia

@andrewtarzia bump - Can I get some more detail here, I'm not 100% what the issue is any more.


NetworkPlugin CNI: Failed to allocate address: No available addresses

Is this a request for help?:
---Yes
Is this an ISSUE or FEATURE REQUEST? (choose one):
---Issue
What version of acs-engine?:
--- 0.15.0

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
Kubernetes
What happened:
After 30 pods are assigned to a node, creating additional pods fail since the IP addresses assigned to a node are exhausted
Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "arena-dqn-linux_default" network: Failed to allocate address: Failed to delegate: Failed to allocate address: No available addresses

Pod sandbox changed, it will be killed and re-created.

Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "arena-dqn-linux_default" network: netplugin failed but error parsing its diagnostic message "": unexpected end of JSON input

This is a linux only cluster, and there are limited number of IP addresses assigned to one node. See image below. The capacity on the other hand is 110 pods which is misleading since only 30 pods can deployed on one node.

What you expected to happen:
Either limit the capacity when the node is created to 30 or allow more ip addresses per node.
In a hybrid cluster the networking is completely different, and a route table is used to forward traffic to a node. Based on the rules I have seen in previous clusters, there are 256 possible IP addresses with /24 subnet. It is baffling to see why its different for linux only cluster.
How to reproduce it (as minimally and precisely as possible):
Create 33 pods for one node on a linux only cluster with one node.

+1. Facing the same issue

1
Is it possible to found some workaround of this ?

You can add more ipconfigs to your interface in the VM provided subnet has more free ips. This can be done via portal or powershell.

"allocatable": {
"cpu": "2",
"ephemeral-storage": "28043041951",
"hugepages-1Gi": "0",
"hugepages-2Mi": "0",
"memory": "7011884Ki",
"pods": "110"
The capacity by default is 110 with new version of acs. But it's limited in 30.

This was happening to me as well and adding ip-configs seems to have helped so far.
Is there an easier way to add lots of ip-configs to a nic?
I ended up just calling this over and over again, increment my name by one each time, but it was slow...
az network nic ip-config create -g XXX --nic-name YYY --name ipconfig37

Facing the same issue. However, I don't have 30 pods in my node.

@advishnuprasad - I opened a support ticket with Microsoft, and was advised to upgrade the CNI plugin to a new version - you might want to try the same.
https://github.com/Azure/azure-container-networking/releases/tag/v1.0.11
The install instructions work for an upgrade too - however, it's advised that after install, you fully restart the node (not just restart kubelet, as the instructions seem to indicate)

Having only 30 IPs per node is really bad. It's worse than a 30 pods per node limit.
Job pods stick around for a while and hold onto their IPs. By default, a Job resource keeps around 3 job pods, for a total of around 4 IPs in use per job resource by default. This means you are limited to 7 job resources per node for your Kubernetes cluster. This design decision effectively makes Job resources unusable with the Azure Networking plugin.
The Azure Networking plugin appears to be buggy, and doesn't seem to always release IPs when pods are done with them. The exact cause is unclear here, but I've had nodes with less than 30 IPs in use by pods giving me this error of no free IPs remaining. Only rebooting the node seems to free the IPs back up. I suspect having job pods greatly aggravates the IP freeing issue, further preventing their practical use on Azure.
The Azure Networking Plugin is responsible for so many stability issues. From the history of problems with the Azure Networking Plugin, I believe acs-engine needs to drop support for the Azure Networking plugin as the default and use a different networking plugin by default.

@sharmasushant I've upgraded to 1.0.12 and it seems more stable, thank you.

@sharmasushant I recently encountered this issue on a cluster with 1.0.12. It seems to have resolved itself for now, but I'd like to understand what happened.
Logs from one node:
azure-vnet-logs.zip

@sharmasushant I'm getting the IP addresses exhausted error again.

@tamilmani1989 ping as well

@tamilmani1989 I raised the limit recently to 100, so that shouldn't be a problem. I was getting a different error when I was hitting the pod limit.

@tamilmani1989 I believe acs engine 0.24.2

That would be Azure CNI 1.0.12

@rocketraman can you send me /var/run/azure-vnet.json and /var/run/azure-vnet-ipam.json?

That would be Azure CNI 1.0.12

Yes it was but I used the manual update procedure after I ran into this issue to update all the nodes to 1.0.13.

@tamilmani1989 Here it is:
varrunjson.zip

@tamilmani1989 Is it significant that every node in the cluster seems to have exactly 30 "Addr" entries in /var/run/azure-vnet-ipam.json?
~$ kubectl get nodes -o name | cut -d / -f 2 | xargs -I{}  -n1 ssh {} 'grep -c "\"Addr\"" /var/run/azure-vnet-ipam.json'
30
30
30
30
30
30

And looking at InUse in /var/run/azure-vnet.json, definitely at least one of these nodes is very close to the limit, and that is also the node showing the "No available addresses" error:
~$ kubectl get nodes -o name | cut -d / -f 2 | xargs -I{}  -n1 ssh {} 'echo 'node ' {}; grep InUse /var/run/azure-vnet-ipam.json | sort | uniq -c'
node k8s-agentpool1-36147952-vmss000000
     14                                                                 "InUse": false
     16                                                                 "InUse": true
node k8s-agentpool1-36147952-vmss000001
     17                                                                 "InUse": false
     13                                                                 "InUse": true
node k8s-agentpool1-36147952-vmss000002
     13                                                                 "InUse": false
     17                                                                 "InUse": true
node k8s-agentpool1-36147952-vmss000003
     17                                                                 "InUse": false
     13                                                                 "InUse": true
node k8s-agentpool2-36147952-vmss000000
      1                                                                 "InUse": false
     29                                                                 "InUse": true
node k8s-agentpool2-36147952-vmss000001
      4                                                                 "InUse": false
     26                                                                 "InUse": true

@tamilmani1989 When this cluster was originally created, it had MaxPods set to the default of 30. I increased that value to 100 as per some instructions provided by @jackfrancis in Slack. But perhaps Azure CNI pre-allocates 30 addresses per that setting as well?

From the state file it looks like, IPs are not get updated to 100. Have you checked in portal if you got 100 ipconfigs for that nic?

Also I'm not sure  "by modifying --max-pods in /etc/default/kubelet"  if this adds ipconfigs to nic. We usually recommend adding ipconfigs in portal.

@tamilmani1989 Just add them manually in the portal? How do I know which IPs I can allocate? I'm concerned about breaking future acs-engine scale or update operations by allocating IPs in a manner that isn't consistent with acs-engine.

can you please verify in portal if the nic got 100 ipconfigs?
you don't need to know about IPs if you select dynamic but anyway lets check why cni doesn't get the update.

Also @jackfrancis Does acs-engine support scaling ipaddresses on the fly (i.e) does it add ipconfigs to nic?
Can you please confirm if specifying  this -> "-max-pods in /etc/default/kubelet " adds ipconfig to nics?

can you please verify in portal if the nic got 100 ipconfigs?

No, they have not. Each node currently has 30 ipconfigs according to the "Connected devices" page of the portal.
It appears in the portal I cannot add ipconfigs manually to the agent nodes even if I wanted to -- they are using VMSS and don't appear to have any option to add ipconfigs, unlike regular VMs.

@rocketraman ok this explains why CNI doesn''t get update. i'm not sure whether acs-engine supports scaling of ip addresses. @jackfrancis has to help here
Can't you access Network interface resource directly and add ipconfigs?

@rocketraman how did you raise the limit from 30 to 100? Did you manually update the runtime configuration on each node? Or did you update the value in an api model and then pass that updated input to acs-engine upgrade?

@jackfrancis Manually based on a discussion with yourself in slack. I guess I should apply the upgrade method here, then?

@jackfrancis Attempted the upgrade... not working because pods aren't moving around the cluster properly due to IP address exhaustion. And because of the drain timeout being too low. And because Azure ARM deployment fails because it says a volume is duplicated or attached to multiple VMs when the portal shows it is not attached anywhere  Goddamn frustrating. And now I don't even have an easy way to revert to a working state, AFAICT.

Pinging @rocketraman and @apeschel..  Does anyone know the permanent fix for this? I am experiencing the issue with IP exhaustion with a self-managed cluster on azure.
Despite not having 30 pods running at one time, (limit is 30) and more than sufficient IP addresses being allocated, K8S cannot spawn new pods due to this:
2019/12/17 09:48:51 [cni-ipam] Plugin azure-vnet-ipam version v1.0.7.
2019/12/17 09:48:51 [cni-ipam] Running on Linux version 4.15.0-1055-azure (buildd@lcy01-amd64-026) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10)) #60-Ubuntu SMP Thu Aug 8 18:29:07 UTC 2019
2019/12/17 09:48:51 [ipam] reboot time 2019-08-20 04:26:51 +0000 UTC store mod time 2019-12-17 09:48:37.885566345 +0000 UTC
2019/12/17 09:48:51 [ipam] Restored state, &{Version:v1.0.7 TimeStamp:2019-12-17 09:48:37.889826956 +0000 UTC AddrSpaces:map[local:0xc420095140] store:0xc420094e10 source:<nil> netApi:<nil> Mutex:{state:0 sema:0}}
2019/12/17 09:48:51 [cni-ipam] Plugin started.
2019/12/17 09:48:51 [cni-ipam] Processing ADD command with args {ContainerID:c425d454b2d24d2e66b10c4374573c76a0f125ab6928835a565c490b4447187f Netns:/proc/20956/ns/net IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=iotaccelerator;K8S_POD_NAME=iam-keycloak-0;K8S_POD_INFRA_CONTAINER_ID=c425d454b2d24d2e66b10c4374573c76a0f125ab6928835a565c490b4447187f Path:/opt/azure-vnet/bin:/opt/cni/bin}.
2019/12/17 09:48:51 [cni-ipam] Read network configuration &{CNIVersion:0.3.0 Name:azure Type:azure-vnet Mode:bridge Master: Bridge:azure0 LogLevel: LogTarget: Ipam:{Type:azure-vnet-ipam Environment: AddrSpace: Subnet:<IP_ADDRESS>/24 Address: QueryInterval:} DNS:{Nameservers:[] Domain: Search:[] Options:[]} AdditionalArgs:[]}.
2019/12/17 09:48:51 [ipam] Starting source azure.
2019/12/17 09:48:51 [ipam] Refreshing address source.
+2019/12/17 09:48:51 [ipam] Source refresh failed, err:expected element type <Interfaces> but have <html>.
2019/12/17 09:48:51 [ipam] Requesting address with address: options:map[].
2019/12/17 09:48:51 [ipam] Address request completed with address:<nil> err:<nil>.
2019/12/17 09:48:51 [azure-vnet-ipam] Failed to allocate address: No available addresses.+
2019/12/17 09:48:51 [cni-ipam] ADD command completed with result:<nil> err:Failed to allocate address: No available addresses.
2019/12/17 09:48:51 [cni-ipam] Plugin stopped.

The pod is stuck in ContainerCreating on the agent node, and the logs above are from azure-cni on the node where the pod tries to start. We are running v1.0.7.

@flow0787 You're on a really old version of ipam. I think the fix was in 1.0.12 or something like that. I'm currently on 1.0.18.

@rocketraman i know its a long shot, but do you by any chance remember which instructions you used for the CNI upgrade?

@flow0787 Yup, its actually documented: https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md#upgrading-cni-on-existing-kubernetes-cluster-deployed-using-acs-engine


URL filling up with code and oauth

When using TileBoard for a while, its URL tends to evolve into something as this:
https://localhost:8123/local/tileboard/index.html?oauth=1&oauth=1&oauth=1&oauth=1&oauth=1&oauth=1&oauth=1&oauth=1&oauth=1&code=3f995498dd8a4e09a6315da3369c85&code=96b3f2af100146bc9f290934de17de&code=cda2e5e8019527b98ac16b4295b97&code=c15025bcae6a4c5e9e9507b475e531&code=3fb5277426a447d894f6a2e08fc8cdb&code=051c410d71a4438799313c5613450&code=fb63b9021e4b68bb91f49280e23ada&code=a9b14ac91ea94849898643254566b&code=d8a471eecbf4cf1aa6e480f6874b1fa&oauth=1.
It might be related to my unstable VPN connection.

3e504c804a0f28a50e11ab36d66bc7c6bef498b9 is probably gonna make this less of a problem as any random error won't trigger re-authentication. But it would still happen if actual authentication error happens.

Ok. I'll monitor and report back or close the issue. Thanks!


GH-43624: [Go] Add JSON/UUID extension types, extend arrow -> parquet logical type mapping

Rationale for this change

Missing JSON extension type implementation.
Current precedent in C++ (and thereby PyArrow) is that canonical extension types do not require manual registration.
Issues like #43640 and #43624 suggest that we need to expose ways of configuring parquet types written from arrow records, but casting the underlying data presents challenges for a generalized approach.

What changes are included in this PR?

Move UUIDType from internal to arrow/extensions
Implement JSON canonical extension type
Automatically register all canonical extension types at initialization

remove register/unregister from various locations these extension types are used

Add new CustomParquetType interface so extension types can specify their target LogicalType in Parquet
Refactor parquet fieldToNode to split up PrimitiveNode type mapping for leaves from GroupNode composition
Simplify parquet LogicalType to use only value receivers

Are these changes tested?
Yes

Are there any user-facing changes?

UUID and JSON extension types are available to end users.
Canonical extension types will automatically be recognized in IPC without registration.
Users with their own extension type implementations may use the CustomParquetType interface to control Parquet conversion without needing to fork or upstream the change.

GitHub Issue: #43624

Thanks for the thorough review @pitrou. The UUID extension was moved from a previously internal part of the repo, but I clearly didn't do the diligence to confirm it did in fact meet the current specification. Let me know if you have any other thoughts.


autogenerate snap sync system ABI

Right now the snap sync ABI is manually created and put into a local file in the templates. We should autogenerate this and import it from the snap sync module. (See #764)

Is this still needed?


[bug] unable to switch to elevation prompt window by cklicking on it in the taskbar

I'm unable to switch to the elevation prompt of a program opened from Cairo. I click there and nothing happens. I can alt-tab with no problem. Explicitly running Cairo as administrator changes nothing.

@beppe9000 This should actually be fixed in the latest build (I missed updating this issue when it was released). Is this working for you now?

i'll let you know as soon as I test that

@beppe9000 Were you able to test this?

i was not able to reproduce it, so I think it's fixed


feature-request: saner nftables chain priority  and counter

Hello,
thanks for crowdsec and the bouncer.
I have to two options I'd really like to have configurable in nftables, Atm I work with "set-only: true" which is great but quite some work.
chain priority
The default created chains are e. g.:
	chain crowdsec6-chain {
		type filter hook input priority filter; policy accept;
		ip6 saddr @crowdsec6-blacklists drop
	}

priority filteris "0" and this is the priority used by many people and FW frameworks. If two chains have the same priority in nftables it's not deterministic which is used first. E. g. sshguard therefore uses:
	chain blacklist {
		type filter hook input priority filter - 10; policy accept;
		ip6 saddr @attackers drop
	}

to have its chains processes before the "normal" filter priority.
Making the priority configurable and have it set to a imho saner default of e .g. "-10" would be great.
This is not a security issue, because nftables behaves quite unintuitive if multiple chains are present with the same hook and family and "first-match" only applies for DROP rules (see https://github.com/openwrt/openwrt/issues/9981 for details).
counter
It'd be great to have the option to easily get counters for the drop rules:
- ip6 saddr @crowdsec6-blacklists drop
+ ip6 saddr @crowdsec6-blacklists counter drop

to see if a DoS is happening or things are out of normal but where no logging is needed.
Thanks a lot!

@zxyz  thanks for feature request. The counter feature is included in upcoming release and we have a patch to add priority option now.


Please close the following ideas as resolved since as per the documentation there are no egress charges when the Power BI tenant is deployed in the same Azure region as the data sources

Please close the following ideas as resolved since there are no egress charges when the Power BI tenant is deployed in the same Azure region as the data sources:
https://ideas.powerbi.com/forums/265200-power-bi-ideas/suggestions/11571870-egress-charges-from-scheduled-refresh-against-sour
https://ideas.powerbi.com/forums/265200-power-bi-ideas/suggestions/10855002-guarantee-scheduled-refresh-data-processing-is-ini
This is confusing the customers as these two unresolved ideas come up in the search results as open.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b0577c02-6c3e-870c-ddee-d93cafef9206
Version Independent ID: b9c65245-32e4-8bc9-8971-e76bff0ad837
Content: Power BI and Azure egress - Power BI
Content Source: powerbi-docs/service-azure-egress-power-bi.md
Service: powerbi
Sub-service: powerbi-service
GitHub Login: @davidiseminger
Microsoft Alias: davidi

Hi @mpsingh24 -- thanks for raising this issue! Unfortunately we can't close items on ideas.powerbi.com. Can you add this note to the ideas themselves?
thanks!


strict: true should enable all strict mode options (aka, be like typescript)

What version of Ajv you are you using?
v7
What problem do you want to solve?
Whenever I enable strict mode I always pass strict: true, only to then remember that that doesn't do anything, as I've got to actually set two other properties to true.
What do you think is the correct solution to problem?
When I do strict: true, use that as the default for all strict-mode options - that way all strict-mode options will be enabled when I pass strict: true while allowing me to turn off the ones I don't want by explicitly setting them to false or log.
This is the behaviour I've come to expect from having worked with TypeScript for so long, as it's strict mode is pretty much the same as ajvs.
Will you be able to implement it?
Yes :)

@G-Rath is the proposal to change strict: true to enable strictTypes and strictTuples, but keep strict: undefined to mean what it means now?
It probably needs to be better defined then and I think it may be better to postpone till v8 release in a couple of months...
Currently there is:

strict - includes mostly preventing ignored keywords in schema and prevents ignored defaults and strictNumbers (that does feel an odd one)
strictTypes - on top of strict: true
strictTuples - on top of strict: true)

Possibly, with the change there would be:

strictSchema (true by default) = strictKeywords (v6) + strictDefaults (v6) = strict (current) - strictNumbers (v6)
strictNumbers (true by default) - as in v6
strictTypes (log by default)
strictTuples (log by default)
strict: true - set all above to true
strict: false - set all above to false

Does it make sense?

@G-Rath is the proposal to change strict: true to enable strictTypes and strictTuples, but keep strict: undefined to mean what it means now?
It probably needs to be better defined then and I think it may be better to postpone till v8 release in a couple of months...
Currently there is:

strict - includes mostly preventing ignored keywords in schema and prevents ignored defaults and strictNumbers (that does feel an odd one)
strictTypes - on top of strict: true
strictTuples - on top of strict: true)

Possibly, with the change there would be:

strictSchema (true by default) = strictKeywords (v6) + strictDefaults (v6) = strict (current) - strictNumbers (v6)
strictNumbers (true by default) - as in v6
strictTypes (log by default)
strictTuples (log by default)
strict: true - set all above to true
strict: false - set all above to false

Does it make sense?

hi @epoberezkin ,
how to throw error if particular property value is "NA" or "undefined"?
sample data: {
"a":"NA",
"b":"undefined",
"c":"some value"
}
Now i want to show error as "a" and "b" have inappropriate value. Also how to compare two date property? thnks.

hi @epoberezkin ,
how to throw error if particular property value is "NA" or "undefined"?
sample data: {
"a":"NA",
"b":"undefined",
"c":"some value"
}
Now i want to show error as "a" and "b" have inappropriate value. Also how to compare two date property? thnks.

it may be better to postpone till v8 release in a couple of months

If a new major is coming out soon, then 💯 agree & am happy for it to wait until then since it'll give us more freedom.

Does it make sense?

Totally! I didn't realise strict by itself was enabling a few things.
Ultimately this boils down to "I want to have one property I can set to true or log to have it enable all (or most) strict-mode validations".
Then, thinking on how to go about that, it brings to mind the way TypeScript handle their strict mode and flags, which I think would be a good fit here too.
So yes the list of properties you've written up makes sense and I think would be good. The only adjustment I'll make would be on those last two:

strict: true - set all above to true, unless explicitly provided
strict: false - set all above to false, unless explicitly provided

This might have been what you were meaning, but want to be clear :)

Some examples
strictSchema (true by default) = strictKeywords (v6) + strictDefaults (v6) = strict (current) - strictNumbers (v6)
strictNumbers (true by default) - as in v6
strictTypes (log by default)
strictTuples (log by default)
strict: true - set all above to true
strict: false - set all above to false

{
  strictSchema: true,
  strictNumbers: true,
  strictTypes: true,
  strictTuples: true,
  strict: true,
}

{
  strict: true,
}
// is equivalent to
{
  strictSchema: true,
  strictNumbers: true,
  strictTypes: true,
  strictTuples: true
}

{
  strictTuples: true,
  strict: false,
}
// is equivalent to
{
  strictSchema: false,
  strictNumbers: false,
  strictTypes: false,
  strictTuples: true
}

{
  strictTypes: 'log',
  strict: true,
}
// is equivalent to
{
  strictSchema: true,
  strictNumbers: true,
  strictTypes: 'log',
  strictTuples: true
}

it may be better to postpone till v8 release in a couple of months

If a new major is coming out soon, then 💯 agree & am happy for it to wait until then since it'll give us more freedom.

Does it make sense?

Totally! I didn't realise strict by itself was enabling a few things.
Ultimately this boils down to "I want to have one property I can set to true or log to have it enable all (or most) strict-mode validations".
Then, thinking on how to go about that, it brings to mind the way TypeScript handle their strict mode and flags, which I think would be a good fit here too.
So yes the list of properties you've written up makes sense and I think would be good. The only adjustment I'll make would be on those last two:

strict: true - set all above to true, unless explicitly provided
strict: false - set all above to false, unless explicitly provided

This might have been what you were meaning, but want to be clear :)

Some examples
strictSchema (true by default) = strictKeywords (v6) + strictDefaults (v6) = strict (current) - strictNumbers (v6)
strictNumbers (true by default) - as in v6
strictTypes (log by default)
strictTuples (log by default)
strict: true - set all above to true
strict: false - set all above to false

{
  strictSchema: true,
  strictNumbers: true,
  strictTypes: true,
  strictTuples: true,
  strict: true,
}

{
  strict: true,
}
// is equivalent to
{
  strictSchema: true,
  strictNumbers: true,
  strictTypes: true,
  strictTuples: true
}

{
  strictTuples: true,
  strict: false,
}
// is equivalent to
{
  strictSchema: false,
  strictNumbers: false,
  strictTypes: false,
  strictTuples: true
}

{
  strictTypes: 'log',
  strict: true,
}
// is equivalent to
{
  strictSchema: true,
  strictNumbers: true,
  strictTypes: 'log',
  strictTuples: true
}

Ok, yes - that’s what I meant - strictSomething overrides strict
atm strict: true indeed does not change anything - it’s just default. strictSchema and strictNumbers is just something that will be added to retain the current strict: true behaviour.
another options would be to allow strict as object with properties (in addition to true/false/log values) - maybe it’s better. in this case you either set strict or individual settings but not both, so it’s easier and less confusing.

Ok, yes - that’s what I meant - strictSomething overrides strict
atm strict: true indeed does not change anything - it’s just default. strictSchema and strictNumbers is just something that will be added to retain the current strict: true behaviour.
another options would be to allow strict as object with properties (in addition to true/false/log values) - maybe it’s better. in this case you either set strict or individual settings but not both, so it’s easier and less confusing.


mod_xml2enc: avoid corrupting Microsoft Office documents

The Microsoft OOXML format uses xml packaged into a zip file, and has
mimetypes like:
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
This mimetypes contains 'xml', but is unfortunately not an xml file.
xml2enc processes these files (in particular, when mod_proxy_html is
used), typically resulting in them being corrupted as it seems to
attempt to perform a ISO-8859-1 to UTF-8 conversion on them.
Update the check to look for '+xml', which hopefully reflects only
mimetypes that actually contain xml.
closes https://bz.apache.org/bugzilla/show_bug.cgi?id=64339

Thanks for the contribution!


Language detection of raw file data without file on disk

Linguist::FileBlob operates on the path to a file. In some cases the
file is not available on disk. Hence, the FileContents mimics FileBlob
and enables language detection based on the raw file data and
the name of the file.

Thanks for the pull request! Since there isn't anywhere in linguist where we require the content to be a FileBlob, you can pass in an object that "quacks like a blob". We actually use this technique on GitHub.com for all of our blob content.
For example, you could use an OpenStruct:
require "ostruct"
blob = OpenStruct.new(:name => "foo.rb", :data => "puts 'hello world'")
Linguist::Language.detect(blob)
#=> #<Linguist::Language name=Ruby>

blob.extend(Linguist::BlobHelper)
blob.language
#=> #<Linguist::Language name=Ruby>

Alright, we can live with OpenStruct. Thanks for the hint! :+1:


Change changelog task to for direct invocation from external command

The commit message has more details, but this is a start to allow ember-cli-release to call the changelog task directly instead of relying on a hook in the release config. Intended to work with this branch of ember-cli-release.
Right now this is totally untested 🙈

@slindberg we should probably chat about this, I suspect running with mods to release-with-changelog is going to work easier.

I think my overall impression is that there are some good ideas in this PR, especially for code organization and flow, but I think it also missed too many things for me to accept as is.  Ping me on slack if you want to run through these changes with me later, I think for me to have a better understanding of exactly how you intend it to be used by ember-cli-release is important.

Sorry, I should have added a 'WIP' to the description. By battery was running low and I rushed to push out my progress so you could look at it. Definitely not mergable as-is. Let's definitely chat about it.


Add support for RSA encryption

This PR adds the possibility to use fingerprint authentication with RSA public-private key authentication. Doing so allows to encrypt values without fingerprint authentication that can then later only be decrypted with fingerprint authentication.
Numerous things were refactored for this PR to allow for both the new RSA crypto and the previous AES crypto to work side by side.
RxFingerprint gains a two new methods:

RxFingerprint.encrypt(EncryptionMethod, Context, String, String)
RxFingerprint.decrypt(EncryptionMethod, Context, String, String)

EncryptionMode gives the choice to use either AES as previously or RSA. Keys can only be encrypted and decrypted with the same method, otherwise an error will be thrown. Calling encrypt or decrypt without the EncryptionMethod parameter will continue to use the previous AES encryption.
Resolves #26

👍 appreciate the work


Hoare logic symbols

Before requesting a feature, please search for existing issues.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
I am writing notes for my logic and computation class with KaTeX but have been unable to type the parentheses required for Hoare triples.
Describe the solution you'd like:
A clear and concise description of what you want to happen.
The \llparenthesis and \rrparenthesis symbols or similar, similar to the \llbracket and \rrbracket already supported in KaTeX.

Source
Link to or name of a (La)TeX package that provides the same feature:
KaTeX's general goal is to render as LaTeX does.
stmaryd
Describe alternatives you've considered:
A clear and concise description of any alternative solutions or features you've considered.
Using \{ P \} C \{ Q \} notation, although it is much less common.
Additional context:
Add any other context or screenshots about the feature request here.

Would be great to have the banana brackets in KaTeX!


does not seem to compile android when I try     "android": "cd apps/expo && expo android" script

Hi there, I am trying to use this template for my monorepo.
followed the instruction on readme and it works on Expo client when I run the script web or native however, when I added a new script target on the root package.json
    "android": "cd apps/expo && expo android"
and then if I run yarn android it doesnt compile.
it throws following error
`

Task :react-native-gradle-plugin:compileKotlin FAILED
'compileJava' task (current target is 1.8) and 'compileKotlin' task (current target is 11) jvm target compatibility should be set to the same Java version.

FAILURE: Build failed with an exception.

What went wrong:
Execution failed for task ':react-native-gradle-plugin:compileKotlin'.

Failed to query the value of task ':react-native-gradle-plugin:compileKotlin' property 'compilerRunner$kotlin_gradle_plugin'.
Kotlin could not find the required JDK tools in the Java installation. Make sure Kotlin compilation is running on a JDK, not JRE.

Try:

Run with --stacktrace option to get the stack trace.
Run with --info or --debug option to get more log output.
Run with --scan to get full insights.

Get more help at https://help.gradle.org

BUILD FAILED in 10s
1 actionable task: 1 executed
Error: /Users/hiteshkhatri/Documents/personal/projects/my-solito-app-2/apps/expo/android/gradlew exited with non-zero code: 1
Error: /Users/hiteshkhatri/Documents/personal/projects/my-solito-app-2/apps/expo/android/gradlew exited with non-zero code: 1
at ChildProcess.completionListener (/Users/hiteshkhatri/Documents/personal/projects/my-solito-app-2/node_modules/@expo/spawn-async/build/spawnAsync.js:52:23)
at Object.onceWrapper (node:events:628:26)
at ChildProcess.emit (node:events:513:28)
at maybeClose (node:internal/child_process:1100:16)
at Process.ChildProcess._handle.onexit (node:internal/child_process:304:5)`
any ideas, what I am doing wrong?
I can compile the vanilla EXPO project created with expo cli with android script without any issues.
just to verify that i have all the required Java and JDK /JRE is installed etc.

you need to use expo run:android to generate a dev client if you’re using native libraries. then you can start a dev server with expo start —dev-client


I want you to teach me kaggle.

My name is Taiyo Suzuki. I live in Yokohama, Japan, and I speak English.
I'm studying data science to seriously accelerate the Singularity.
I want you to teach me kaggle.
I don't know how to build a model.
I'm an intern with no major in machine learning.
Thank you for your time.

Please do not open spam PRs; github PRs are not for soliciting help on unrelated problems. And for getting started on kaggle there is a whole section on their website where you should find all the starting material you need:
https://www.kaggle.com/getting-started


Update PPP

Change to full name

Codecov Report

Merging #297 (d080f27) into master (dafdd57) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #297   +/-   ##
=========================================
  Coverage     78.68%   78.68%
  Complexity      978      978
=========================================
  Files           131      131
  Lines          2656     2656
  Branches        331      331
=========================================
  Hits           2090     2090
  Misses          465      465
  Partials        101      101

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dafdd57...d080f27. Read the comment docs.


Release R156

Can we maybe wait for https://github.com/snowplow/iglu-central/pull/1397 ? 😄

@AlexBenny Can you cherry pick this one since it has been approved https://github.com/snowplow/iglu-central/pull/1397  ?
Thanks folks


Catch up with sway

Would be nice to keep up with upstream - I'm going to add a couple of things to our sway fork (then upstream) - it would be good to start fresh.

LGTM. I forgot to sync with master.


Clear application state on exchange disconnect

About
In the past, a refresh of the browser window would cause a restart of the devtools panel.
With native, this is not the case. We need to clear/reset the state when the exchange disconnects.

Fixed by #cb8b94fb596c3b79cf07a3ceb572a01a1ea5b492


2.0.0: FIPS 202 SHA-3 standard hash algorithms and SHAVS test vectors

Closes #14

Ready to merge and release.
cc: @FooBarWidget

TL;DR Keccak reference implementation -> FIPS 202 patch
--- KeccakSponge.c.orig	2016-03-21 00:20:19.000000000 -0700
+++ KeccakSponge.c	2016-03-20 11:07:20.000000000 -0700
@@ -200,13 +200,13 @@
 {
     // Note: the bits are numbered from 0=LSB to 7=MSB
     if (state->bitsInQueue + 1 == state->rate) {
-        state->dataQueue[state->bitsInQueue/8 ] |= 1 << (state->bitsInQueue % 8);
+        state->dataQueue[state->bitsInQueue/8 ] |= 6 << (state->bitsInQueue % 8);
         AbsorbQueue(state);
         memset(state->dataQueue, 0, state->rate/8);
     }
     else {
         memset(state->dataQueue + (state->bitsInQueue+7)/8, 0, state->rate/8 - (state->bitsInQueue+7)/8);
-        state->dataQueue[state->bitsInQueue/8 ] |= 1 << (state->bitsInQueue % 8);
+        state->dataQueue[state->bitsInQueue/8 ] |= 6 << (state->bitsInQueue % 8);
     }
     state->dataQueue[(state->rate-1)/8] |= 1 << ((state->rate-1) % 8);
     AbsorbQueue(state);

There are simpler, shorter implementations out there as well.
References

Keccak Round 3 finalist submission (zip) from SHA-3 Competition, Round 3
FIPS 202 Standard (pdf)
from FIPS 202 Standard (landing)
[SHAVS test vectors for byte-length messages (zip)] (http://csrc.nist.gov/groups/STM/cavp/documents/sha3/sha-3bytetestvectors.zip)
from SHAVS test vectors (landing)

Sorry for the very long delay on this PR.
I completely agree with #14 that this library is a bit of a misnomer. There are consumers downstream that are using it intentionally to provide the Keccak variant. Even with a major version bump, if the library suddenly begins to produce a different hash, it could cause some trouble. To that end, I propose an iterative approach vs a rewrite.
Since the discussion on #14 is dated and closed, I'll create some follow-up issues to try and establish a plan for how to move forward, and we'll make this happen!

Created #36 as a follow-up for a phased rollout of SHA-3.


--main-path option not recognized in Nuitka 1.6

After carefully reviewing the Issue Guidelines I allow myself to report a bug with the following information:

Noise, I had to go through

Please check whether the bug was already reported or fixed.

I checked and I could not find one.

Please check out if the develop version of Nuitka works better for you.

No, I want to use a stable version of the software.

How did you install Nuitka and Python

I used pip install nuitka to install nuitka using pip.

Many times when you get an error from Nuitka, your setup may be special

A print("Hello world") program does work.

Also supply a Short, Self Contained, Correct, Example

I considered it and found it not relevant to the problem

Provide in your issue the Nuitka options used

I state the command line.

Avoid unnecessary options

Please tell me which options are unnecessary  and I will remove them

Note if this is a regression

I have no observations on regressions for this project and do not plan to do so.

Consider getting commercial support

I will consider getting commercial support once I determined it suits all my needs.

Description:
I am attempting to compile multiple Python scripts into a single binary using the --main-path option as described in the Nuitka user manual. However, when I run the command, I receive an error stating that --main-path is not a recognized option.
Environment:

Nuitka version: 1.6
Python version: 3.10.11 (main, Apr  5 2023, 14:15:10) [GCC 9.4.0]
Operating system: Linux, Ubuntu (based on Debian) 20.04.4

Steps to reproduce:

Create a Makefile with the following content:
# Declare variables
PYTHON = python
NUITKA = -m nuitka
FLAGS = --follow-imports --nofollow-import-to=numpy output-dir=bin
A_SCRIPT = src/python/apps/script_a.py
B_SCRIPT = src/python/apps/script_b.py

# Declare phony targets
.PHONY: build

build:
    $(PYTHON) $(NUITKA) $(FLAGS) --main-path=$(A_SCRIPT ) --main-path=$(B_SCRIPT )

Run make build.

Expected outcome:
Successful compilation of the two Python scripts into a single binary.
Actual outcome:
The compilation process fails with the error message: __main__.py: error: no such option: --main-path.
Additional notes:
From my understanding based on the Nuitka user manual, --main-path should be a valid option that can be used to specify multiple main scripts for the compilation process. Any assistance in resolving this issue would be greatly appreciated.

Here are the specific PyPI names and versions
aiocache==0.12.0
anyio==3.7.0
arel==0.2.0
astroid==2.15.5
async-asgi-testclient==1.4.11
beautifulsoup4==4.12.2
bidict==0.22.1
blinker==1.6.2
boxx==0.10.0
bpy==3.4.0
certifi==2019.11.28
cffi==1.15.1
chardet==3.0.4
charset-normalizer==3.1.0
click==8.1.3
contourpy==1.0.7
cryptography==41.0.1
cycler==0.11.0
Cython==0.29.35
dacite==1.8.1
dbus-python==1.2.16
debugpy==1.6.7
dill==0.3.6
distinctipy==1.2.2
distro-info===0.23ubuntu1
exceptiongroup==1.1.1
exclusiveprocess==0.9.4
executing==1.2.0
fastapi==0.96.0
ffmpeg-python==0.2.0
Flask==2.3.2
Flask-Cors==3.0.10
fonttools==4.39.4
future==0.18.3
gitdb==4.0.10
GitPython==3.1.31
giturlparse==0.10.0
h11==0.14.0
httptools==0.5.0
idna==2.8
imageio==2.30.0
imutils==0.5.4
iniconfig==2.0.0
isort==5.12.0
itsdangerous==2.1.2
Jinja2==3.1.2
json-merge-patch==0.2
jsonpatch==1.32
jsonpointer==2.3
kiwisolver==1.4.4
lazy-object-proxy==1.9.0
lazy_loader==0.2
loguru==0.7.0
markdown-it-py==2.2.0
MarkupSafe==2.1.3
mathutils==3.3.0
matplotlib==3.7.1
mccabe==0.7.0
mdurl==0.1.2
minexr==1.0.2
multidict==6.0.4
networkx==3.1
Nuitka==1.6
numerize==0.12
numpy==1.24.3
opencv-python==<IP_ADDRESS>
ordered-set==4.1.0
packaging==23.1
pandas==1.5.2
Pillow==9.5.0
platformdirs==3.5.1
pluggy==1.0.0
protobuf==3.20.3
psutil==5.9.5
PTable==0.9.2
pycparser==2.21
pydantic==1.10.8
pydicom==2.3.1
Pygments==2.15.1
PyGObject==3.36.0
pylint==2.17.4
pynrrd==0.4.3
PyOpenGL==3.1.7
pyparsing==3.0.9
pytest==7.3.1
python-apt==2.0.1+ubuntu<IP_ADDRESS>
python-dateutil==2.8.2
python-dotenv==0.21.1
python-json-logger==2.0.7
python-magic==0.4.27
python-multipart==0.0.5
pytz==2023.3
PyWavelets==1.4.1
PyYAML==6.0
requests==2.31.0
requests-toolbelt==1.0.0
requests-unixsocket==0.2.0
rich==13.4.1
scikit-image==0.18.3
scikit-video==1.1.11
scipy==1.10.1
seaborn==0.12.2
Shapely==1.8.5.post1
SimpleITK==2.2.1
six==1.14.0
smmap==5.0.0
snakeviz==2.2.0
sniffio==1.3.0
soupsieve==2.4.1
starlette==0.27.0
stringcase==1.2.0
supervisely==6.72.28
tifffile==2023.4.12
tomli==2.0.1
tomlkit==0.11.8
tornado==6.3.2
tqdm==4.65.0
trimesh==3.21.7
typing_extensions==4.6.3
tzdata==2023.3
unattended-upgrades==0.1
urllib3==1.26.15
uvicorn==0.22.0
uvloop==0.17.0
varname==0.11.1
watchfiles==0.19.0
watchgod==0.6
websockets==10.4
Werkzeug==2.3.4
wrapt==1.15.0
zstandard==0.21.0

The docs don't match the code, the option was changed to --main at some point during development I guess. Updating the docs to reflect that.

The docs don't match the code, the option was changed to --main at some point during development I guess. Updating the docs to reflect that.

Sir, thank you for your timely response. Indeed, using --main leads to the expected behavior.
If you are updating the docs, please note that I also could not find the --main nor the --main-path option from the help.
Running python -m nuitka --help | grep -- '--main' returns no results

I guess help=SUPPRESS_HELP does not help it being there, that doesn't mean it doesn't exist though. I have to admit, not making it easy to use.

Going to add help. I guess I was too scared to even attempt to describe what it is, because it is orthogonal to standalone/onefile/accelerated mode. The multidist name is more a goal, namely standalone of multiple programs, and not exactly what it does.
But alas, adding the help output now.

Yes, that's true. Windows symlinks, CMD files, shell scripts, posix symlinks, they all could be created indeed. But maybe that's for somebody else to PR into Nuitka. It's not for lack of ideas that this is not done. :-)

Fair.
So from my point of view the issue is resolved but I leave it up to you to close it, once everything is updated.
In case someone in the future runs into the same problem for some reason, I am going to leave this here:
✨ SOLUTION ✨
Works for 1.6
Use --main instead of --main-path

So, the doc updates will be added to 1.6.2 and this will get a factory tag once it has a fix, and will be closed, once the 1.6.2 release is done. Which may not be off that far in the distance.

This is part of the hotfix release 1.6.2 that I just made.


bring prefix config param in dynamic-path-parser

I signed it!

I signed it!

High @magemello, thanks for helping out! However, we're sticking to the style guide as much as possible in the directory structure and the app folder is a part of that.


Plugin exception when running web app

Hi there, I'm testing the plugin on a web application I develop and a I came across this exception when trying to set up the app:
Error: MissingPluginException(No implementation found for method setSdkVerbosity on channel datadog_sdk_flutter)
That's my main code:
final configuration = DdSdkConfiguration(
        clientToken: ***,
        env: injection.flavor.environment.name,
        site: DatadogSite.us1,
        trackingConsent: TrackingConsent.granted,
        nativeCrashReportEnabled: true,
        loggingConfiguration: LoggingConfiguration(
          sendNetworkInfo: true,
          printLogsToConsole: true,
        ),
        rumConfiguration: RumConfiguration(
          applicationId: ***,
        ),
      );

      final ddsdk = DatadogSdk.instance;
      ddsdk.sdkVerbosity = Verbosity.verbose;

      await DatadogSdk.instance.initialize(configuration);

      FlutterError.onError = (FlutterErrorDetails details) {
        FlutterError.presentError(details);
        ddsdk.rum?.handleFlutterError(details);
      };

      runApp(
        AppWidget(
          ...
        ),
      );

Even without setting the sdkVerbosity or providing a logging configuration, the exception happens
Additional context

Dart/Flutter version:

Flutter 3.7.8 • channel stable • https://github.com/flutter/flutter.git
Framework • revision 90c64ed42b (2 weeks ago) • 2023-03-21 11:27:08 -0500
Engine • revision 9aa7816315
Tools • Dart 2.19.5 • DevTools 2.20.1

Datadog SDK version:
  datadog_flutter_plugin: ^1.3.2

Hey @jorgelrj
This one really stumps me. If you're not setting sdkVerbosity there should be no reason for you to end up at that call stack. More importantly, on web we should be replacing the Platform in such a way that it should never use the method channel.
I have one other report of this, but it's with Release mode only. Is that true for you as well?

@fuzzybinary Yes, sorry I forgot to mention it. Only happens on the deployed app built in release mode. Testing it locally works fine

@jorgelrj Does testing the same build work locally, but not when deployed?
If you run:
flutter build web --release --web-renderer canvaskit --pwa-strategy=none
cd build/web
python3 -m http.server 8080

And go to the site locally, does it work? If so, I'm wondering if where you're deploying is doing something weird.

@fuzzybinary
I did two tests here, I built the app running only flutter build web and it it worked on the local server, but when I added the --source-maps flag that I use on my ci flow, I got the same exception again

@jorgelrj I can't reproduce this, but it looks like it might be something related to issues with the generated_plugin_registrant.dart file, similar to this issue https://github.com/fluttercommunity/plus_plugins/issues/1270
The fix for them was updating the plugin to specify versions of Flutter > 2.11, but at the moment we support 2.8/
If you're so inclined, you can pull the repo locally, change the pubspec.yaml to set flutter: ">=2.11.0, depend on that module from the path and see if that fixes your issue.
If it does let me know and I'll see if we can update to support 2.11.0+ and publish a new version.

Hey @fuzzybinary,
Just test it here and setting flutter: ">=2.11.0" did get it to work 👍

Okay, thanks @jorgelrj .  I'll see if we can update our min-version for 1.4

@fuzzybinary I'm getting the exact same error in web only in release mode (not in debug mode)
My initialization code:
final configuration = DatadogConfiguration(
  clientToken: ***,
  env: 'staging',
  service: ***,
  site: DatadogSite.us1,
  nativeCrashReportEnabled: true,
  loggingConfiguration: DatadogLoggingConfiguration(),
);
await DatadogSdk.instance.initialize(configuration, TrackingConsent.granted);

Additional context:
Flutter version: 3.16.2
Dart version: 3.2.2
Environment: sdk: '>=3.1.2 <4.0.0'
Datadog package version: datadog_flutter_plugin 2.3.0

Weird, I wonder if something regressed on the Flutter side. I'll reopen this and take a look as soon as I can.

Hi @lirantzairi,
I just checked this quickly on Flutter 3.19.2 and 3.16.2 didn't receive the error.  How are you trying to access your Flutter web app? Opening the html usually does not work, so you have to run a web server in your build/web directory (I usually just use the built in python web server for testing).
If you're still having the issue, can you give me any more information about your setup that might help me determine what's going wrong?

Same issue with flutter 3.19.3 and
datadog_flutter_plugin: ^2.3.0
datadog_tracking_http_client: ^2.1.0

Hi @lirantzairi,
I just checked this quickly on Flutter 3.19.2 and 3.16.2 didn't receive the error. How are you trying to access your Flutter web app? Opening the html usually does not work, so you have to run a web server in your build/web directory (I usually just use the built in python web server for testing).
If you're still having the issue, can you give me any more information about your setup that might help me determine what's going wrong?

Hi @fuzzybinary, I use Firebase Hosting for deploying my web app. By "release mode" I meant to say that it happens in a fully deployed web app.

Hi folks --
Sorry for the delay here, but I still cannot reproduce.
After building, can you check that your generated_plugin_registrant.dart contains the following:
DatadogSdkWeb.registerWith(registrar);
registrar.registerMessageHandler();

Also, I'm seeing reports from others with similar issues that a clean rebuild occasionally fixes the issue for them. Can you confirm that doesn't work?

Going to close this as I was unable to reproduce and haven't heard of anyone else having this issue.


ModuleNotFoundError: No module named 'fastreid'

I didn't add any arguments for --fastreid, but it gives this error.
why is this happening?

Because fastreid is used in the files  init.py, deep_sort.py and feature_extractor.py, and these three files are in deep_sort folder. So, if you comment out all the codes about fastreid in these three files, the error will be solved.

Because fastreid is used in the files init.py, deep_sort.py and feature_extractor.py, and these three files are in deep_sort folder. So, if you comment out all the codes about fastreid in these three files, the error will be solved.

but shouldn't below code in deepsort.py skip fastreid when I don't set  arguments for --fastreid?
  if args.fastreid:
      cfg.merge_from_file(args.config_fastreid)
      cfg.USE_FASTREID = True
  else:
      cfg.USE_FASTREID = False

Because fastreid is used in the files init.py, deep_sort.py and feature_extractor.py, and these three files are in deep_sort folder. So, if you comment out all the codes about fastreid in these three files, the error will be solved.

but shouldn't below code in deepsort.py skip fastreid when I don't set arguments for --fastreid?
  if args.fastreid:
      cfg.merge_from_file(args.config_fastreid)
      cfg.USE_FASTREID = True
  else:
      cfg.USE_FASTREID = False

in this file, it include a module named 'fastreid', so even if you do not set argument for fastreid, it also can't find this fastreid.

Because fastreid is used in the files init.py, deep_sort.py and feature_extractor.py, and these three files are in deep_sort folder. So, if you comment out all the codes about fastreid in these three files, the error will be solved.

but shouldn't below code in deepsort.py skip fastreid when I don't set arguments for --fastreid?
  if args.fastreid:
      cfg.merge_from_file(args.config_fastreid)
      cfg.USE_FASTREID = True
  else:
      cfg.USE_FASTREID = False

in this file, it include a module named 'fastreid', so even if you do not set argument for fastreid, it also can't find this fastreid.

I see and problem solved. Thank you.
One more question:  should the code be modified somehow to avoid manual comment out since it offers FASTREID as optional? But I'll close it for now.

great!


Fix a containerd deadlock.

I hit this issue during cluster bootstrapping.
If containerd is terminated with pkill containerd before it is fully bootstrapped, it may fall into a deadlock.
The reason is that the server channel size is 0. If handleSignals exits early, no one is going to consume the message in channel, and serverC <- server will stuck forever.
If this makes sense, we should cherrypick into 1.0. @crosbymichael
Signed-off-by: Lantao Liu<EMAIL_ADDRESS>

Codecov Report

Merging #1907 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1907   +/-   ##
=======================================
  Coverage   47.29%   47.29%
=======================================
  Files          89       89
  Lines        8830     8830
=======================================
  Hits         4176     4176
  Misses       3968     3968
  Partials      686      686

Flag
Coverage Δ

#linux
50.66% <ø> (ø)
:arrow_up:

#windows
42.24% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6393165...0eebf90. Read the comment docs.

LGTM


tensorflow not available in cosmodesi env

Hi, I was unable to train a neural emulator (MLPEmulatorEngine) in the NERSC cosmodesi environment because it appears tensorflow is not installed.
This is not pressing for me, but I wanted to alert to the issue.

tensorflow is now in the cosmodesi environment at NERSC, please let me know if you face issues (https://github.com/cosmodesi/cosmodesiconda would be the best place to raise environment-related issues).

Thank you Arnaud - and noted, regarding the cosmodesiconda repo.


Can't make multiline comments working for Python

Hi,
First thanks a lot for this plugin, it is amazing and helps me a lot.
I just encounter a problem with multiline comments in Python and tried a lot of different indentations with no success..
Here is a screenshot of my issue: The syntax highlighting is highlighting inside the comment for some reason.. Is there any workaround?

Hi Threynaud,
that looks weird...normally multiline phyton comments are supported (see https://enlighterjs.org/Language.Python.html). Maybe its a bug within the tokenizer engine.
could you please provide the testcode above ?
br, Andi


Running rootless containers on ubuntu bionic doesn't work

kind bug
Description
After having installed podman on an ubuntu bionic system using the PPA and attempting to run a container, I got:
* Error committing the finished image: error adding layer with blob "sha256:32802c0cfa4defde2981bec336096350d0bb490469c494e21f678b1dcf6d831f": ApplyLayer exit status 1 stdout:  stderr: lchown /etc/gshadow: invalid argument

The machine in question is a recently installed from scratch Ubuntu Bionic with not a ton installed.
Steps to reproduce the issue:

Install podman:

sudo add-apt-repository ppa:projectatomic/ppa
sudo apt-get update
sudo apt-get install podman

Install a registries.conf file

# This is a system-wide configuration file used to
# keep track of registries for various container backends.
# It adheres to TOML format and does not support recursive
# lists of registries.

# The default location for this configuration file is /etc/containers/registries.conf.

# The only valid categories are: 'registries.search', 'registries.insecure',
# and 'registries.block'.

[registries.search]
registries = ['docker.io']

# If you need to access insecure registries, add the registry's fully-qualified name.
# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.
[registries.insecure]
registries = []

# If you need to block pull access from a registry, uncomment the section below
# and add the registries fully-qualified name.
#
# Docker only
[registries.block]
registries = []

Try to run a container:

podman run -it --rm ubuntu /bin/sh

Describe the results you received:
Trying to pull docker.io/ubuntu:latest...Getting image source signatures
Copying blob sha256:32802c0cfa4defde2981bec336096350d0bb490469c494e21f678b1dcf6d831f
 30.62 MB / 30.62 MB [======================================================] 7s
Copying blob sha256:da1315cffa03c17988ae5c66f56d5f50517652a622afc1611a8bdd6c00b1fde3
 847 B / 847 B [============================================================] 0s
Copying blob sha256:fa83472a3562898caaf8d77542181a473a84039376f2ba56254619d9317ba00d
 556 B / 556 B [============================================================] 0s
Copying blob sha256:f85999a86bef2603a9e9a4fa488a7c1f82e471cbb76c3b5068e54e1a9320964a
 162 B / 162 B [============================================================] 0s
Copying config sha256:93fd78260bd1495afb484371928661f63e64be306b7ac48e2d13ce9422dfee26
 3.32 KB / 3.32 KB [========================================================] 0s
Writing manifest to image destination
Storing signatures
ERRO[0011] Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: lchown /etc/gshadow: invalid argument
Failed
unable to pull ubuntu: 1 error occurred:

* Error committing the finished image: error adding layer with blob "sha256:32802c0cfa4defde2981bec336096350d0bb490469c494e21f678b1dcf6d831f": ApplyLayer exit status 1 stdout:  stderr: lchown /etc/gshadow: invalid argument

Describe the results you expected:
To be in a shell in the container.
Additional information you deem important (e.g. issue happens only occasionally):
Happens for other containers - it's not just an ubuntu container.
Output of podman version:
Version:       0.11.2-dev
Go Version:    go1.10.4
OS/Arch:       linux/amd64

Output of podman info:
host:
  BuildahVersion: 1.5-dev
  Conmon:
    package: 'cri-o-1.12: /usr/lib/crio/bin/conmon'
    path: /usr/lib/crio/bin/conmon
    version: 'conmon version 1.12.4-dev, commit: '
  Distribution:
    distribution: ubuntu
    version: "18.04"
  MemFree: 190324736
  MemTotal:<PHONE_NUMBER>
  OCIRuntime:
    package: 'cri-o-runc: /usr/sbin/runc'
    path: /usr/sbin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 483840000
  SwapTotal:<PHONE_NUMBER>
  arch: amd64
  cpus: 4
  hostname: solace
  kernel: 4.15.0-29-generic
  os: linux
  rootless: true
  uptime: 73h 46m 19.84s (Approximately 3.04 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
store:
  ContainerStore:
    number: 0
  GraphDriverName: vfs
  GraphOptions: []
  GraphRoot: /home/mordred/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 0
  RunRoot: /run/user/1001

Additional environment details (AWS, VirtualBox, physical, etc.):
Lenovo Thinkpad Laptop

Is this running Podman without root?

Yes - totally just running as a normal user.
I went and tried with root and it works. (yay!) So I suppose rootless
podman is the thing that isn't working.
I have my user listed in /etc/subuid and /etc/subgid as per the docs.
On Fri, Nov 30, 2018 at 3:06 PM Matthew Heon<EMAIL_ADDRESS>wrote:

Is this running Podman without root?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/containers/libpod/issues/1908#issuecomment-443339368,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAFztD5BUoHiCE5zk6liWm1uX6-hE8-hks5u0Z2_gaJpZM4Y8YCY
.

What entry do you have in /etc/subuid and /etc/subgid?
This could be attempting to add an UID/GID that is not in the range of your usernamespace.

@mheon BOOM. There it is. I installed that and it worked. That's super exciting!
Perhaps uidmap should go into the Depends for the package? I'll go comment on the packaging issue

For completeness, the following is what is needed to get rootless podman working on ubuntu bionic:
sudo add-apt-repository ppa:projectatomic/ppa
sudo apt-get update
sudo apt-get install podman uidmap
echo "$(whoami):10000:65536" | sudo tee /etc/subuid
echo "$(whoami):10000:65536" | sudo tee /etc/subgid
echo -e "[registries.search]\nregistries = ['docker.io']" | sudo tee /etc/containers/registries.conf

@lsm5 We might want to make uidmap a dependency of Podman in the PPA, so we can have rootless work out of the box

this PR should avoid the kind of isses you've reported: https://github.com/containers/libpod/pull/1923

@giuseppe I think I have configured steps mentioned above I still get the error:
abhi@f12n12:~$ podman run -it ubuntu bash
Trying to pull docker.io/library/ubuntu...Getting image source signatures
Copying blob f12d17d7baa6 done
Copying blob fce39298a87b done
Copying blob 404b73f949d7 done
Copying blob cecce6095d35 done
Copying config 4e6d115184 done
Writing manifest to image destination
Storing signatures
ERRO[0004] Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: lchown /etc/gshadow: operation not permitted
ERRO[0004] Error pulling image ref //ubuntu:latest: Error committing the finished image: error adding layer with blob "sha256:cecce6095d35c15ae2a27f2e80fbc0a0806d92aa38fbd27d004a9168f6b3aa58": ApplyLayer exit status 1 stdout:  stderr: lchown /etc/gshadow: operation not permitted
Failed
Error: unable to pull ubuntu: 1 error occurred:
* Error committing the finished image: error adding layer with blob "sha256:cecce6095d35c15ae2a27f2e80fbc0a0806d92aa38fbd27d004a9168f6b3aa58": ApplyLayer exit status 1 stdout:  stderr: lchown /etc/gshadow: operation not permitted
can you please help?
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.2 LTS
Release:	18.04
Codename:	bionic
abhi@f12n12:~$ podman --version
podman version 1.4.4

@asm582 can you try to do podman system migrate first?  If it doesn't still work, please attach the output of: podman unshare cat /proc/self/uid_map

@giuseppe below is my output:
asmalvan@f12n12:~$ podman unshare cat /proc/self/uid_map
0       2287          1
1     100000      65536

can be AppArmor blocking lchown?

@giuseppe I am not sure, Sorry I missed sharing this information.  I am running podman on NFS with root squash enabled, should we take this discussion on a git new issue?

ah then that is the issue.  NFS cannot be used for rootless containers: https://github.com/containers/libpod/blob/master/rootless.md (look for NFS in the page)

Should we add a troubleshooting note that moving directories out of /home will solve this?

I'm in the midst of updating the troubleshooting guide for the overlayfs build woes, I'll tag this on too.

Maybe we should update containers/storage overlay checking to check if the upper layer is NFS, and the ignorechownerrors flag is not turned on return and error.

Found this : https://www.redhat.com/sysadmin/rootless-podman-nfs
Tried but still
myuser@bionic:~$ podman unshare cat /proc/self/uid_map
         0      82623          1
         1      10000      65536

myuser@bionic:~$ cat /etc/containers/{registries.conf,storage.conf}
[registries.search]
registries = ['docker.io']
# storage.conf is the configuration file for all tools
# that share the containers/storage libraries
# See man 5 containers-storage.conf for more information

# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/run/user/82623"

# Primary read-write location of container storage
graphroot = "/tmp/containers/storage"

[storage.options]
# AdditionalImageStores is used to pass paths to additional read-only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Size is used to set a maximum size of the container image.  Only supported by
# certain container storage drivers (currently overlay, zfs, vfs, btrfs)
size = ""

# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"

myuser@bionic:~$ podman run --rm -it ubuntu /bin/bash
Trying to pull docker.io/library/ubuntu...
Getting image source signatures
Copying blob 651c9d2d6c4f done
Copying blob c63719cdbe7a done
Copying blob 19a861ea6baf done
Copying blob 5c939e3a4d10 done
Copying config ccc6e87d48 done
Writing manifest to image destination
Storing signatures
ERRO[0006] Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: lchown /etc/gshadow: operation not permitted
  ApplyLayer exit status 1 stdout:  stderr: lchown /etc/gshadow: operation not permitted
Error: unable to pull ubuntu: 1 error occurred:
	* Error committing the finished image: error adding layer with blob "sha256:5c939e3a4d1097af8d3292ad3a41d3caa846f6333b91f2dd22b972bc2d19c5b5": ApplyLayer exit status 1 stdout:  stderr: lchown /etc/gshadow: operation not permitted

Yes we have a PR for this in container storage, but have not been able to get back to it.
https://github.com/containers/storage/pull/518


increase the timer for reading k8s event in test

Changes
This commits extends the current timer from 10 millisecond to wait.ForeverTestTimeout and tried to fix flake test #5748. The 10 ms timer may be too short to receive events from channel in tests.
/kind flake
Signed-off-by: Yongxuan Zhang<EMAIL_ADDRESS>
Submitter Checklist
As the author of this PR, please check off the items in this checklist:

[ ] Has Docs included if any changes are user facing
[ ] Has Tests included if any functionality added or changed
[x] Follows the commit message standard
[x] Meets the Tekton contributor standards (including
functionality, content, code)
[x] Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
[ ] Release notes block below has been updated with any user facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings)
[ ] Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes
NONE

/test all

/lgtm
thanks @Yongxuanzhang I am happy to merge this as is since its addressing a flake but the wait time could have been made configurable 🙃

/lgtm
thanks @Yongxuanzhang I am happy to merge this as is since its addressing a flake but the wait time could have been made configurable 🙃

yeah sure! I think that makes sense!


Bot Emulator Connectivity issue

I am trying to connect just a Hello world bot application and its getting the error POST connect ECONNREFUSED <IP_ADDRESS>:3978, on the emulator, i have tried with valid MSA id and secret and also blank ones. but still receiving the same.

Resolved this issue by downloading and unzipping and copying the Visual Studio Bot Application template from https://docs.botframework.com/en-us/downloads/

I'm glad you found a solution!


polling does not work with Fortigates with certificates signed with custom CA

workaround :
I forked the repository and replaced the ca-certificates.crt in the Dockerfile.
COPY ca.crt /etc/ssl/certs/ca-certificates.crt
Error is :
`2020/10/08 13:29:53 Loaded 1 API keys
2020/10/08 13:29:53 Fortigate exporter running, listening on ":9710"
2020/10/08 13:30:57 Error: Get "https://fgt.domain.net/api/v2/monitor/system/status": x509: certificate signed by unknown authority
2020/10/08 13:30:57 Probe of "https://fgt.domain.net" failed, took 0.148 seconds
`

You should be able to provide your own ca-certificates.crt using SSL_CERT_FILE environment variable as far as I know.
Can you try putting your custom CA root somewhere as a custom file and setting that environment variable?

indeed, missed this, I will try it and tell you

I tried, and it does not work :
Error: Get "https://fgt.domain.net/api/v2/monitor/system/status": x509: certificate signed by unknown authority

Hello,
If the CA is trusted by the system, then it works.
For exemple in CentOS you put the CA certificates in /etc/pki/ca-trust/source/anchors/ then you issue the command update-ca-trust.
After that you should not have the  "x509: certificate signed by unknown authority" error message.
Best Regards,

You can now use both -extra-ca-certs as well as the panic button -insecure if you want to skip validation altogether.
This should close this, let me know if you have any further issues!


Refactor Home Page UI

I feel like there is still room for improvement in the application UI part. I would love to make it a little better and more accessible.

@Kritika30032002 Can you please assign this to me?

I'm no longer looking forward to work on this. If anyone wants to take this, please go ahead.

Can I work on this issue?

Hey @Kritika30032002  as no one is working on this issue, I would like to get assigned for this.

Thanks @Kritika30032002 I'll soon start work on it


Sub-canvas is narrower

I have a situation where I overlay a canvas over some HTML in order to draw lines between some of the HTML boxes.
This works fine in the browser.
However, when I render the combined HTML and canvas using html2canvas, the overlaid canvas seems to be slightly narrower than it is in the browser. This means that the lines to 'join' the boxes properly.
Any ideas what would cause this and what I can do about it?

Does the canvas have dimensions set explicitly?

Yes, the canvas has margin-top, margin-left height and width set explicitly in order to overlay the HTML boxes.
Here is what it looks like in the browser:

And here is the rendered PNG file from html2canvas:

See how the lines joining the boxes are offset to the left.
To see this in action, register at https://strategiclearningapp.com, then select Analysis -> Add -> McKinsey 7S. This will show you the browser version. To convert to PNG, click on the 'gears' icon (top right) and select 'download' (however, you will note that I have currently excluded the lines from the download, rather than including them incorrectly.)
Many thanks.

Note, in the examples posted above, it is intended that the buttons shown on the browser rendered version are not shown in the html2canvas PNG version.

Is this still an issue with v1.0.0? If so, could you please share an example on jsfiddle.


Add samples for turbo replication (15 minute RPO)

Samples need to be added for turbo replication (client library request).
Please see NodeJS example here: https://github.com/googleapis/nodejs-storage/pull/1618
Please wait for approval from myself before merging or releasing this change so we can coordinate around release dates.

@shivgautam assigning to you so you can reassing appropiately.
I can take it if you don't have the bandwidth.

@amanda-tarafa I'll get it done. Just got the timelines from Denis. This should be doable.

Samples done and merged


Get-DSCBlock doesn't properly handle arrays of CIMInstance

If the parameter passed to the Get-DSCBlock function is an array of CIMInstance, the the output is just a sequential string of all instances. No @() and commas between. This is causing some issues to certain ReverseDSC Orchestrator Scripts (e.g. Web site bindings in the the xWebSite resource of the xWebAdministration Module).

Fixed in <IP_ADDRESS>


Update scraper status page

Since the workflow was simplified, the scraper status page is not up-to-date. The column with the GitHub status should be removed.

on it.


Change PR to pull request

There was an inconsistent use of "PR" and "pull request" throughout the
document. I believe that I read an article from GitHub that they suggested
the use of "pull request" instead of "PR" for the purposes of clarity.

Thanks!

I have some additional - slightly more drastic - changes to submit too. Some are simply formatting changes to the markdown (mostly all consistency related) and other wording changes that I didn't want to include in one big PR. Ill submit that soon.


MRTK assemblies are built against the "split" Unity assemblies

Ported to Github, reference 21543183
MRTK assemblies are built against the split Unity assemblies, such as UnityEngine.CoreModule.dll. I believe these are only supposed to be used at runtime so that Unity can load smaller assemblies. At compile time, it seems that the intention is that you reference the top level UnityEngine.dll. The following pages suggest this is the case:
https://docs.unity3d.com/Manual/UsingDLL.html
https://forum.unity.com/threads/unityengine-dll-vs-unityengine-coremodule-dll-unityengine-animationmodule-dll-etc.509478/
This is what we did in MRW, but what that means is that our code thinks things like MonoBehaviour is the type defined in UnityEngine.dll, while MRTK code thinks it is defined in UnityEngine.CoreModule.dll, which at compile time are effectively two different types, which makes MRTK code incompatible with MRW code when compiling from the context of MSBuild.
I believe the right change here is to ensure that MRTK is built against UnityEngine.dll.

high priority for MRW.
MRTK can be used in individual apps, but this blocks building a shared tech for all apps to consume.
Need to find an owner to address this week. (david/Nial are the expert here)

@davidkline-ms Let's chat today, may be a dependency related change

I am not familiar with our nuget package build system enough to spot the changes that need to be made.
Assigning @andreiborodin and @MenelvagorMilsom who should have a better understanding.

fixed.