[SPARK-22575][SQL] Add destroy to Dataset

What changes were proposed in this pull request?
In the Dataset API we may acquire resources which we cannot deallocate. This happens for broadcast joins. The broadcasted object is never destroyed and we rely on the garbage collection of broadcasted object to free it. In a general use case, this is a safe assumption, but when dynamic allocation is enabled, the current approach can lead to resource leakage.
In particular, when a Spark application is submitted on YARN with dynamic allocation enabled, we may leak disk space. Indeed, in such a scenario, when query with a broadcast join is executed, it is likely that we ask for new containers. These containers are used for the execution of the query and then killed. They may be killed before the broadcast object is GCed. In this case, the files which have been written are never removed (as the container is not alive anymore to remove them and YARN removes them only when the application ends).
In order to solve the above-mentioned issue, the PR proposes to add a destroy method to the Dataset class, which can be used to free all the resources which have been acquired in the plan execution. Eagerly destroying the acquired resources, they are freed before the containers are killed, avoiding (or at least reducing considerably) the problem.
How was this patch tested?
added UT

Test build #91504 has started for PR 21502 at commit 147bd08.

Test build #91504 has finished for PR 21502 at commit 147bd08.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91504/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3823/
Test PASSed.

Test build #91505 has started for PR 21502 at commit ec365d6.

Test build #91505 has finished for PR 21502 at commit ec365d6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91505/
Test FAILed.

Test build #91524 has started for PR 21502 at commit 789168e.

Test build #91524 has finished for PR 21502 at commit 789168e.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91524/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3832/
Test PASSed.

Test build #91528 has started for PR 21502 at commit 4d080cf.

Test build #91528 has finished for PR 21502 at commit 4d080cf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91528/
Test PASSed.

How does this solve the problem you described? If the container is gone, the process is gone and users can't destroy things anymore.

@rxin this can reduce the problem since the destroy operation is invoked when the execution of the query is terminated (in STS), instead of some time later when the cleaner thread does it. In this way, when dynamic allocation is enabled, it is very likely that during the delay introduced by waiting for the cleaner thread to remove the blocks a lot of containers are killed, while if we destroy it immediately after the query terminates, the containers are very likely to not have been killed yet, so they can clean up the blocks.

Is it the best to add a new api for that? Is that a bad idea to invoke a monitor thread to check that kind of things?

@maropu I think a monitor thread would be useless. Once the container is gone, there is nothing we can do. Another solution which may be worth to investigate is to clear the block manager for an executor before killing it. But I am not sure about this, as it introduces an overhead during the scale-down of containers.

Well, it happens especially with dynamic allocation, but there may be other causes like YARN preemption. Anytime a container is killed we can face this issue. Anyway, I plan to check the feasibility of this other approach (it may take some time as I'm not very familiar with that part of the codebase).

aha, thanks for the kindly explanation!


Add interface support for "virtual" cable types

Environment

Python version:  3.6.7
NetBox version:  2.5.3

Proposed Functionality
Add a "virtual" cable type

Use Case
Most of my environment is using NSX. I have found a workaround for our virtual switches and routers by creating them as "devices" (as non-rackable devices) rather than virtual machines. This way I can create a actual cable from one device to another, which I can't do as a VM since all interfaces are virtual interfaces (which cant be connected).
With a virtual cable (wire, as NSX describes it) I'll be able to use the filters in the sidebar instead of leaving the blank and losing out on that function. It's also a step towards some SDN support without extending the functionality too much (Too many products out there to account for).

Database Changes
"virtual" as a new category of cable type

External Dependencies

This was requested and rejected in #2797 due to the current backlog.  In the future, once the backlog has cleared, there might be time to revisit this.


Adding the AudioVideoPlayerBot

This bot can stream audio and video using the AudioVideoFramePlayer

@waboum,
Thanks for your contribution.
To ensure that the project team has proper rights to use your work, please complete the Contribution License Agreement at https://cla.microsoft.com.
It will cover your contributions to all Microsoft-managed open source projects.
Thanks,
Microsoft Pull Request Bot

@waboum,
Thanks for your contribution.
To ensure that the project team has proper rights to use your work, please complete the Contribution License Agreement at https://cla.microsoft.com.
It will cover your contributions to all Microsoft-managed open source projects.
Thanks,
Microsoft Pull Request Bot


Wrong expression to detect production mode

In expression:
const inProduction = (typeof process !== "undefined" && process.env.NODE_ENV === "production") ||         verifyMinified.name === "verifyMinified"
Second condition:
verifyMinified.name === "verifyMinified"
is inappropriate.
It should probably be verifyMinified.name !== "verifyMinified" if I understand correctly ('===' changed to '!==').

Fixed in 1.0.3. Thanks for noticing!


Tracing of network-helpers.mine() event

I'm trying to simulate multiple txs in the same block.
I send txs to "mempool" consequently and finally run  network-helpers.mine() to mine the block.
On the event of sending, I see logs like:
Hardhat Tracer wanted to print trace, but lastTrace is undefined.
This only works on hardhat network, if you are running your script over RPC provider then VM data is not available.

I'd like to trace txs on the mining event. How can I do it?
One way is to run debug_traceTransaction for every mined txs and then convert them to readable format. Can I use this module to do it?

Or, can I use this tracer to convert result of "debug_traceTransaction" to readable format?

Thanks for reporting. This plugin wasn't tracing when block was being authored through evm_mine, just tried to fix this behaviour.
Now the trace should be printed when you mine the block.
Please try<EMAIL_ADDRESS>


docs: Update README

Adjust language from past tense in several sentences.

What type of PR is this?

What this PR does / why we need it:
Which issue(s) this PR fixes (optional, using fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when the PR gets merged):
Fixes #
Special notes for your reviewer:
TODOs:

[x] squashed commits (n/a)
[x] includes documentation (n/a)
[x] adds unit tests (n/a)

NOTE: update the VAP sentence to something "ensure VAP is enabled".

/retitle docs: Update README

@aramase CLA done 👍


transaction.ready webhook missing from SUPPORTED_WEBHOOKS

See here:
https://github.com/websideproject/django-paddle-billing/blob/4a87c501288984e4552e294acae82611ced37e5a/src/django_paddle_billing/views.py#L61-L69
Compared to the list of webhooks on the paddle destination page:

Thanks, I have added.
Fixed in v0.1.5


Terminology

Need to think about the terminology ...

Group vs. team
Organization vs. Namespace
etc.

Here are the terminologies used by a few providers:

GitHub
BitBucket
GitLab

Organization
Team
Group/Namespace

Team
Group
?

My current inclination is:

Namespace - Sounds fairly neutral (as opposed to organization or team)
Team/Group ... not sure

For more info, see

http://doc.gitlab.com/ce/workflow/groups.html
https://confluence.atlassian.com/bitbucket/bitbucket-cloud-teams-321853005.html

Currently, the focus of Gitomator is to provide a convenient "Swiss army knife" for managing GitHub organizations. Therefore, I am not focusing on implementing different providers.
If/when someone implements other hosting providers, they will re-visit this issue.


tsu command only works with root

tsu only works on root and is even worse when passed username
at least same username should result same result
I am trying to back to user of Termux from root. i have tried HOME=/data/...../home PATH=/data/...../sbin  su u0_a194 -s /data/...../bash but doesn't seem to work properly
port 22 for ssh can not be used without root and when start sshd as root I login in root from ssh.

There is no -c option in tsu..  And the -s isnto change the shell like bash/sh
You can use sudo  which comes with tsu to run single commands as root with proper PATH passed in..
Should prolly include this in help text..

This didn't work for me. I have already tried reinstalling tsu

Ignore wast comment, I was using Termux tsu version 8.3.2
after cloning repo - version 8.4.2 :

now works slightly better but still not works as it should:
(port 22 cant be used without root)

this  HOME=$HOME bash works on my linuxls PC

Ah, I can see the problem..  Im not checking if Termux user can access the Termux binaries..
Also I think sshd cant automatically start with different user.. cause its patched in termux to run in single user mode


Create GitHub labels

I just added website content & website format as labels; are we also looking to add other items on our timeline (e.g., scheduled presentations, etc.?) @AHolder1

Awesome, thanks @cclatterbuck! I think we should discuss what we want to include in the timeline - I'm down for including scheduled presentations so we can be transparent about everything we are doing, but am open to using the timeline differently if that makes sense to the rest of the team.

Let's review current labels at an upcoming Seaside Chat and add any that we think we'll need - and then close this comment.

Review complete.


Launch dev jvm image under my user

I am actually very interested to use dev jvm Docker image on my linux machine cause otherwise the binaries I built are linked to a very new GLIBC and they cannot be used on my Spark cluster
But the thing is on linux Docker behaves differently from other platforms.
One thing which is different is that on linux before this change the Docker container would produce a root owned files. This makes it difficult to switch from local dev to this container and back.
In this change I employ entrypoint.sh script which other Docker images use which create a user inside the container and it uses gosu to launch a build|test job under this user. This leads for the script to produce files which are owned by my host machine user.
I also changed a bit the way I share a maven M2 cache cause after this change you can't easily use /root folder
exec ensures that the last command is not launched in a subshell

I forgot to reference the relevant pull request: #4640
cc @thesuperzapper

@trams have you tested that the new Dockerfile works with .\dev\build-linux.cmd on Windows?
And that --skip-tests still works?

I just tested the build-linux.cmd on Windows and it worked. But note that I did not actually changed the Windows part at all

On Windows I testsed

@thesuperzapper can you help to review this?

I forgot to tell that on Linux I made sure that --skip-tests still work

@thesuperzapper, please help review this change


docs: update port number in k8s-health-check

Change Description
I changed the value of port in k8s-health-check, because the default value is 9090.
Checklist

[x] Make sure to open an issue as a
bug/issue
before writing your code!  That way we can discuss the change, evaluate
designs, and agree on the general idea.
[x] Ensure the tests and linter pass
[x] Appropriate documentation is updated (if necessary)

Relevant issues:

Fixes #1635

Great catch! Thanks @ryoheinagao for the issue and PR 😄 We always appreciate the community helping to make the proxy better and more easy to use! Much appreciated, have an awesome day.


Unable to resolve mapper when using Factory.Create for unit testing

I am writing a unit test for this very simple endpoint
public class Endpoint : Endpoint<GetProductTagRequest, ProductTagResponseModel, Mapper>
{
    private readonly ILogger<Endpoint> logger;
    private readonly IProductTagRepository productTagRepository;

    public Endpoint(ILogger<Endpoint> logger, IProductTagRepository productTagRepository)
    {
        this.logger = logger;
        this.productTagRepository = productTagRepository;
    }

    public override void Configure()
    {
        Get("/api/v2/producttag/{id}");
        AllowAnonymous();
    }

    public override async Task HandleAsync(GetProductTagRequest req, CancellationToken ct)
    {
        logger.LogInformation("Getting product tag by request.id: {id}", req.Id);

        var productTag = await productTagRepository.GetDocumentByIdAsync(req.Id);

        if (productTag == null)
        {
            await SendNotFoundAsync(ct);
        } else
        {
            var response = Map.FromEntity(productTag);
            await SendOkAsync(response, ct);
        }
    }
}

Here is my unit test
    public class GetProductTagEndpointTests
    {
        [Fact]
        public async Task When_GetProductTagEndpoint_Succeeds()
        {
            // Arrange
            var fakeLogger = A.Fake<ILogger<Endpoint>>();
            var fakeRepository = A.Fake <IProductTagRepository>();
            var fakeProductTag = new Faker<Domain.Entities.ProductTag>()
                                    .RuleFor(x=> x.Id, Guid.NewGuid())
                                    .Generate();

            A.CallTo(() => fakeRepository.GetDocumentByIdAsync(A<Guid>.Ignored, default))
             .Returns(Task.FromResult(fakeProductTag));

            var su = Factory.Create<Endpoint>(
                fakeLogger,
                fakeRepository);

            var fakeRequest = new GetProductTagRequest()
            {
                Id = new Guid()
            };

            // Act
            await su.HandleAsync(fakeRequest, default);

            var response = su.Response;

            response.ShouldNotBeNull();
        }
    }

When I run the test
I am getting this exception
System.ArgumentNullException : Value cannot be null. (Parameter 'provider')

Map is null
BTW I'm using FastEndpoints 5.2.0

you have to inject the mapper as a dependency in the endpoint constructor.
example unit test: https://github.com/FastEndpoints/FastEndpoints/blob/09e213399c101c1b873babdbc0d04794010207a2/Tests/UnitTests/FastEndpoints.UnitTests/WebTests/UnitTests.cs#L12-L33
endpoint:
https://github.com/FastEndpoints/FastEndpoints/blob/09e213399c101c1b873babdbc0d04794010207a2/Web/[Features]/TestCases/MapperTest/Endpoint.cs#L7-L11

That works @dj-nitehawk ! Thanks
I was under the impression that it will be automatically resolved same as running the API.
Since its part of the endpoint constraint
public class Endpoint : Endpoint<GetProductTagRequest, ProductTagResponseModel, Mapper>

I guess, doing  Endpoint<GetProductTagRequest, ProductTagResponseModel> will also work?
Not sure if this is documented or if I just missed it.

it would resolve automatically if doing integrations testing with WAF.
for unit tests, it has to be done like above.
unit testing documentation is still a work in progress.
when in doubt, take a look at the unit test project linked above.

Gotcha, great work on this project btw! Again thanks :)


fix standalone snapshot to be idempotent

Standalone_snapshot gives error in case the fs0 device has already been detached . Closes https://github.com/openstack-k8s-operators/data-plane-adoption/issues/105

/ok-to-test

/override rdoproject.org/github-check


Consider using Click's environment variable features, rather than a config file

https://click.palletsprojects.com/en/7.x/options/#values-from-environment-variables
https://click.palletsprojects.com/en/7.x/arguments/#environment-variables
The many different options for setting the DB URL in Kingfisher Process/Views is probably overkill for Kingfisher Archive. If we can just settle on a command-line option and env var (both supported by Click), I think that'll be better.
From https://github.com/open-contracting/kingfisher-archive/pull/16#pullrequestreview-482769598

This will involve updates to setup.py


Peripherals block/activate support

Hello, in mobile app I'm able to block and then activate back any peripheral (Settings->Peripherals). I'm using it for example for guest room right now (motion sensor device), which is under one section. When we have guests, I block it in the app, so I can arm the section as usually, but blocked device will not trigger alarm, even when the section, in which the device is, is armed.
It would be good to be able to block/activate it via bridge also.
At least get the list of the peripherals would be really useful, so I can make some reminder, that I have some blocked (because sometimes I forget to activate it again).
Thank you.

Thanks for reporting, I have never tried to do it but since it's possible in the app it should also be possible in Homekit. But it might become very messy as we would need to create a switch per device. I'll explore the API for possibilities next time when I'm in the API.

Yea, I didn't think about it. You will see what's possible ;).


[HIVE-20683] Add the Ability to push Dynamic Between and Bloom filters to Druid

Implementation details -

Hive generates and passes the filters as part of 'filterExpr' in TableScan.
DruidQueryBasedRecordReader gets this filter passed as part of the conf.
During execution phase, before sending the query to druid in DruidQueryBasedRecordReader we will deserialize this filter, translate it into a DruidDimFilter and add it to existing DruidQuery. Tez executor already ensures that when we start reading results from the record reader, all the dynamic values are initialized.
Explaining a druid query also prints the query sent to druid as druid.json.query. We also need to make sure to update the druid query with the filters. During explain we do not have the actual values for the dynamic values, so instead of values we will print the dynamic expression itself as part of druid query.

@b-slim : Updated based on your comments, please check.

@b-slim : Updated patch


Is there any chance we could configure Business Central on-premise to use OKTA for MFA ?

Is there any chance we could configure Business Central on-premise to use OKTA for MFA ?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 24c36d03-7f72-72d0-bbe1-d60f881a9e66
Version Independent ID: c5775944-717d-d054-c016-a82a2efed79c
Content: Multi-Factor Authentication - Business Central
Content Source: dev-itpro/security/multifactor-authentication.md
Service: dynamics365-business-central
GitHub Login: @jswymer
Microsoft Alias: jswymer

Hi, thanks for your question - please ask through your product/support channel. Thanks.


CFA: excessive memory usage

In private builds (i.e. can't link to them here) we see excessive memory usage (>50GB) in the bitbake workers which (most likely) executes the CFA process_filesystem() code. Builds never completed without getting killed by the kernel OOM killer and we haven't had time to investigate, so this might be both a problem with the code or the image that is getting analyzed.
However, even if the image is just producing too many reports, the code should be able to handle that without this excessive memory usage.

Should be much better with this commit: https://github.com/01org/meta-security-isafw/commit/2e05daeb5a64035150301b11a952234152ce7382

@pohly, do you still see this issue now?

I haven't enabled cfa checks again. Let me do that now and then we'll see whether it really works better where it previously failed.

I am closing this since we have seen the issue anymore


Possibility of elastic animation with boundaries?

I was wondering if it was possible to create an element dragging animation/interaction with react-motion to simulate a pinned down state with elastic boundaries.
Imagine you have an element in the middle that is pinned to a fixed position. You can start dragging it but the further away you try to drag it from the pin position the least it moves: like if it was tied to a rubber band. When you release it goes back to the pin position with the spring animation.

Yes, I would implement the motion you described in two parts:

the dragging motion when the item follows the pointer with a certain friction;
the release motion when it gets back;

On component mount I would store the initial coordinates of the item, and set the component state to be "returning" (a plain string for the current part of the animation).
On pointer down I would set the state to "dragging", and set the current desired coordinates to the pointer coordinates in the component state. If you'd like to animate the initial transition from the rendered coordinates to the current effective coordinates, you'd need an additional flag to fade out such animation when the dragging takes over.
On pointer move, if the state is "dragging", I would update the current desired coordinates with the pointer coordinates. In render, if the state is "dragging", the 1) would be a direct mapping of the current desired coordinates to the item coordinates with an euclidean-distance-based friction coefficient, no spring required unless you'd like to animate the initial move to the pointer coordinates.
On pointer release I would update the state to be "returning" and the current desired coordinates to equal the initial coordinates. In render, if state is "returning", the 2) would be a regular spring motion to the initial item coordinates.

Thank you @sompylasar, that's really helpful.
I managed to get most of this done:
http://codepen.io/mares0909/pen/jVYrye
The only thing I'm not sure how to implement is the euclidian distance-based friction. Would you be able to fork that pen and add it to the example? Or push me into the the right direction with that?

Sort of:
dx = (pointerX - initialX);
dy = (pointerY - initialY);
distance = Math.sqrt(dx*dx + dy*dy);  // https://en.wikipedia.org/wiki/Euclidean_distance
angle = Math.atan2(dy, dx);
decay = 0.05;
actualX = Math.cos(angle) * distance * decay;
actualY = Math.sin(angle) * distance * decay;

Yes, it works (slightly edited my comment): http://codepen.io/anon/pen/zopWGQ

@sompylasar , that is perfect, thank you!!


Unschedulable Pod Metric

Tell us about your request
We used this metric with the cluster_autoscaler cluster_autoscaler_unschedulable_pods_count.
Would it be possible to have Karpenter expose a similar metric?
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
How to alert when Karpenter is unable to scale out
Are you currently working around this issue?
Looking at Kubernetes Scheduler metric scheduler_pending_pods, however if you have multiple schedulers this gets more complicated.
EKS also does not expose metrics from the Kubernetes Scheduler.
Additional Context
We might have been abusing this metric a bit as the OSS Autoscaler would also observe entire cluster state.
We have 2 Autoscalers running (CAS and an Internal autoscaler), so we were somewhat overloading on this metric since CAS still took a whole cluster view even if it wasn't scaling most of our nodes.
Attachments
No response
Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Actually this metric might work for our uses: kube_pod_status_unschedulable
I'll leave this open for a bit to see if anyone else has thoughts.

@sidewinder12s We're looking over all of our metrics right now to get a more comprehensive idea of where we have gaps so this is good feedback for that!

I can see adding this as a gauge, but I do believe you should be able to get the equivalent with some sum(kube_pod_status_unschedulable) as well like you said.


Error log if AZDCAP_DEBUG_LOG_LEVEL env var is not set

If the AZDCAP_DEBUG_LOG_LEVEL environment is not set, the following error log is printed:
Azure Quote Provider: libdcap_quoteprov.so [ERROR]: Could not retreive environment variable for 'AZDCAP_DEBUG_LOG_LEVEL'
I believe this is undesirable behaviour and the library should set a sensible default value for the log level that could be overridden with AZDCAP_DEBUG_LOG_LEVEL.

That should be a warning/info, it's definitely not an error. Filed https://github.com/microsoft/Azure-DCAP-Client/pull/115 for the typo.


"Value cannot be null. Parameter name: element" when reflecting attributes

The following code works fine on dotnetfiddle but throws in Bridge/Deck.net:
https://dotnetfiddle.net/YYkcrU
https://deck.net/0dfa5632c667489768865c1b6633ca71
using System;
using System.Linq;

public class Program
{
	public static void Main()
	{
		var a = Status.Item1.GetAttribute<DisplayAttribute>();
	}

	public enum Status
	{
		[Display]
		Item1
	}

	class DisplayAttribute : Attribute
	{
	}
}

public static class EnumExtensions
{
	public static T GetAttribute<T>(this Enum value)
	where T : Attribute
	{
		return value
		.GetType()
		.GetField(value.ToString())
		.GetCustomAttributes(typeof(T), false)
		.FirstOrDefault()
		as T;
	}
}

The error is:

bridge.min.js?<IP_ADDRESS>:8 Uncaught
ctor {message: "Value cannot be null.↵Parameter name: element", innerException: null, errorStack: Error at ctor (https://deck.net/resources/js/bridge/bridge.min.js?<IP_ADDRESS>:8:63274) at ctor …, data: ctor, $initialized: function…}

I've tried adding [Reflectable] to both the enum and the attribute, but that doesn't seem to make any difference.
Is this a bug or do I need to do something differently?

The same issue
https://github.com/bridgedotnet/Bridge/issues/2622


Make PrimsJS return multiple lines front a given single line

I implemented PrismJS on an Angular 2 app, but I was not able to make the code be more complexed organized. Is there some plugin that prettify the code like the example above (especially on SQL language)?
If I give to prismjs this single line code
select `title`, `author`, `year` from `books` group by `title`, `author` limit 10

PrismJS will print this
select `title`, `author`, `year`
from `books`
group by `title`, `author`
limit 10

Hi @calebeaires.
No, PrismJS has no formatting ability, and AFAIK this is not on the agenda.


Document changes added in v0.4.0

https://github.com/archtechx/laravel-seo/releases/tag/v0.4.0

Also document changes from https://github.com/archtechx/laravel-seo/pull/29


Fix a merge mistake in engne_list_add

master version increments the struct_ref early
and needs to decrement the struct_ref on error,
while 3.1 and 3.0 increment the struct_ref later.

Checklist

[ ] documentation is added or updated
[ ] tests are added or updated

I think this is urgent, as the branch is broken recently

@bernd-edlinger - if you also agree this is urgent can you merge immediately? Or otherwise let me know and I can do it.

Ok will do the merge immediately.

Merged to openssl-3.0 as 7f81dec and openssl-3.1 as bb67dff
Thanks!


Chore - Add code coverage analysis support

Context
Add support for a code coverage analysis tool as part of CI. One alternative is Coveralls as they provide free-tier for open source projects.

Definition of Done
Code Coverage stats available and updated as part of CI task.

Frozen due to known issues with JaCoCo and Kotlin: https://discuss.kotlinlang.org/t/codecoverage-with-jacoco-fails/2131


Kubernetes is not Resolving External Domain Names

Kubernetes should be able to resolve External Domain Names our k8s e2e test also show that this is failing.

On this issue specify which cloud, even if it's all.

Unsure what clouds are effected at the moment we will need to investigate further but this isn't causing any troubles yet so adding to the Backlog.

Switched out Kube-DNS with CoreDNS and External names are now resolving on aws/packet.


CI pipeline

https://github.com/marketplace/actions/flutter-action

As of 7c6b04d06de62f3f415b18aa292f4e6da5e104c1 we started to Use Github Actions for Compile, Build and Release our Rust and Flutter Code 🎉 !


Update expressroute-howto-coexist-resource-manager.md (#1)

Fixed typo in page: "ExressRoute" to "ExpressRoute"

@charwen @cherylmc - Would you review the proposed changes?  Thanks

#sign-off

Good catch! Thanks, @gfilicetti

@cherylmc @GitHubber17 @gfilicetti I am an Engineer working on the PRMerger automation tool. I am observing that the correct labels have not been applied to this PR because the .md file which was submitted has an unknown character which is preventing the correct rendering of the meta data yaml headers. Can you please correct the same ? Below is a screenshot...

@gfilicetti : Thanks for your contribution to the Azure documentation! The author, @charwen, has been notified to review your proposed change.

Thanks @seshenoy, file corrected.

Excellent.. I can already see the correct labels.


population EDA and ML jupyter notebook from fork

2021.01.27 - JA

after fork, clone, branch
added request to pipfile - using 'pipenv install requests'
added pandas to pipfile - using 'pipenv install pandas'
no need to add ast to pipfile - already exists
added sklearn to pipfile - using 'pipenv install sklearn'

Very smart of you to use scaling to work this issue out.

reviewed just needed to be approved

this was included with the dummy endpoints


Docker Compose Documentation errors out

In the root Readme file, the instruction to run Docker Compose produces an error:
$ docker-compose MYSQL_PASSWORD='pass@word1' docker compose up -f dist/docker-compose/docker-compose.yml

unknown shorthand flag: 'f' in -f

Closed by #446


Feature request, collection showRecordId as boolean or string or objectid?

Would it be possible to add, for the options, showRecordId as boolean or string or objectid? Right now it is only boolean.
The reason I'm asking is that, most of the time, for the front end, we have to convert the ObjectID back to a string to have a "flat" index key. Our API handles both responses from a SQL or NoSQL dbs, but for MongoDB we have to cycle through the documents just to convert the object id. I would be nice if the library could return the _id as string.

Would it be possible to add, for the options, showRecordId as boolean or string or objectid? Right now it is only boolean.

I think you're misunderstanding the purpose of the showRecordId option. This is a boolean option for the find command that, if true, instructs the server to inject a $recordId field into each result document. The PHP driver/library does nothing with showRecordId other than allowing an application to specify it when executing a query (i.e. find command). The documentation for the mongo shell's cursor.showRecordId() method demonstrates the effect of using this option. AFAIK, the record ID is an integer and has nothing to do with a document's _id field, which is typically an ObjectId.
It sounds like your underlying request is for the driver to provide some functionality to automatically convert ObjectIDs in _id fields to a string. If so, I think this is better implemented in your application. Types other than ObjectId may legitimately be used in the _id field of a document, so I'd be hesitant to add a feature in the driver that makes the actual BSON type of the _id field ambiguous.
There are several ways to accomplish this in your application:

Cast _id to a string when working with returned documents. This will invoke ObjectId::__toString() as needed.
Wrap the cursor returned by find() with an iterator/generator that performs the string cast behind the scenes when accessing the "current" element (i.e. document). Wrapping a Normal Cursor in the library documentation has an example of how to accomplish this using IteratorIterator. Your question suggests that your application is database-agnostic, so I would assume that you're not actually relying on iterating a MongoDB\Driver\Cursor object, so wrapping it with another iterable should not pose a problem.
Leverage the driver's type map functionality to have BSON documents deserialized into a custom class. By default, the driver deserializes BSON documents as stdClass objects. The library specifies its own type map by default to return documents as MongoDB\Model\BSONDocument objects (see: BSON classes. You can specify your own class (optionally extending BSONDocument in the library) and convert ObjectIds in bsonUnserialize() when the document is first processed (or later through some other class method if you wish). The driver's API for type maps and deserialization is discussed in more detail in Deserialization from BSON.

thanks I appreciate your answer. The thing is I was trying to not loop thought documents again just to convert _id.


Consolidate shared prefs to a single store

Description (required)
Fixes #2607
Fixes #906
What changes did you make and why?
Tests performed (required)
Tested on prodDebug variant.

Codecov Report

Merging #2613 into master will decrease coverage by <.01%.
The diff coverage is 7.89%.

@@            Coverage Diff            @@
##           master   #2613      +/-   ##
=========================================
- Coverage     2.7%    2.7%   -0.01%
=========================================
  Files         258     258
  Lines       12266   12255      -11
  Branches     1107    1107
=========================================
- Hits          332     331       -1
+ Misses      11908   11898      -10
  Partials       26      26

Impacted Files
Coverage Δ

...ons/explore/categories/SearchCategoryFragment.java
0% <ø> (ø)
:arrow_up:

...fr/free/nrw/commons/settings/SettingsFragment.java
0% <ø> (ø)
:arrow_up:

...w/commons/category/CategoryImagesListFragment.java
0% <ø> (ø)
:arrow_up:

...ain/java/fr/free/nrw/commons/mwapi/LogBuilder.java
0% <ø> (ø)
:arrow_up:

...bookmarks/locations/BookmarkLocationsFragment.java
0% <ø> (ø)
:arrow_up:

...ommons/contributions/ContributionsSyncAdapter.java
0% <ø> (ø)
:arrow_up:

...ava/fr/free/nrw/commons/nearby/NearbyFragment.java
0% <ø> (ø)
:arrow_up:

...java/fr/free/nrw/commons/upload/FileProcessor.java
3.94% <ø> (ø)
:arrow_up:

...mmons/contributions/ContributionsListFragment.java
0% <ø> (ø)
:arrow_up:

...rw/commons/explore/images/SearchImageFragment.java
0% <ø> (ø)
:arrow_up:

... and 26 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1d225c5...5b56c91. Read the comment docs.


Add variable for ETS & sectoral CO2 emissions

This PR implements one aspect of #18: CO2 emissions by ETS/non-ETS and at the sectoral level.
cc @ClaKet
Two questions:

We could add an additional hierarchy-level like `Emissions|CO2|{Sector}|{Final Energy Carrier}
We could add an additional variable Gross Emissions|CO2 (and subcategories) to clearly differentiate between the amount of CO2 released to the atmosphere versus CO2 captured in a sector, see this usage in the openentrance project here

Of course, both additions could be implemented later...

Will merge and we can add more detailed CO2 emissions variables and "gross emissions" (for reporting emissions captured and requested/used at the source) later.


Comment on a column in a migration is at the wrong place, causing a syntax error

What steps will reproduce the problem?
Creating a table-column with a "check" and a "comment".
Either by putting it inside of the initial creation or by adding it afterwards.
This is in a default yii-migration-class:
$this->createTable('something', [ 'test' => "longtext NULL DEFAULT NULL COMMENT 'this works' CHECK(json_valid(`test`))" ]);
$this->addCommentOnColumn('something', 'test', 'this breaks');
$this->createTable('something2', [ 'test' => $this->json()->comment('this breaks') ]);
//Created Type: longtext NULL DEFAULT NULL CHECK(json_valid(`test`)) COMMENT 'this breaks'

What is the expected result?
The both not working commands should create MySQL similar to the working example included above.
In the end the column should have the check and the comment.
What do you get instead?
A Syntax error from MariaDb (and probably MySql if I used that). According to the MySql-reference, comment should be way earlier than check_constraint_definition:
https://dev.mysql.com/doc/refman/8.0/en/create-table.html
Additional info

Q
A

Yii version
<IP_ADDRESS>

PHP version
7.3

Operating system
Linux

Do you want to help by working on a pull request?

I tried to make an pull request, but the test build script failed due to some expected  mysql-5.5 depency.
https://github.com/yiisoft/yii2/pull/17454

Btw, sorry for the double commit. Should I merge the two commits to one?

No, that's fine.

Sadly this doesn't work.
For some reason comments (in mysql) are added via additional alter-commands after table-creation.
I've fixed this problem locally and will create a Pull-Request that work's for me.

Also CHECK statement goes after AFTER statement
$this->addColumn('table', 'column', $this->json()->null()->after('previous_column'));

Just stumbled upon extension that sets comment on migration and then Query tries to do it MySQL way. In PostgreSQL the comment is attached later in different statement.
Instead of
	"id" serial NOT NULL PRIMARY KEY,
	"some_column" int8 COMMENT "Some comment"
    ....
)```

it is
```CREATE TABLE "oauth2_client" (
	"id" serial NOT NULL PRIMARY KEY,
	"some_column" int8
    ....
)

COMMENT ON COLUMN some_column IS "Some comment";

is there a way to disable comments globally in migration?

There is no easy way. Looks like comments for PostgreSQL should be done separately. Could you be able to prepare a PR?

I wish I could do it. DAO and its intertwined components are a beast.
I will try to have a look at it though and see if I can try to fix it.
Do you know where the comments are actually parsed? I tried to quickly find it and I could not!

Right now basic comment() is implemented here. The trick is to extract that call to a separate SQL statement.

I checked that one but actually it was just assigning it. I wanted to get where it actually processes the comments to create the Query. Am sure there is where I can do something. but could not find. That might be because am unfamiliar on how the whole database access object works under the hood


Remove dead EcoCare link from blogs

This PR resolves the issue reported in https://github.com/corona-warn-app/cwa-website/issues/3415#issuecomment-1455528374.

Dead link to https://buergertest.ecocare.center/#c734 in blogs:

https://www.coronawarn.app/de/blog/2021-05-11-how-to-rapid-test-integration/

https://www.coronawarn.app/de/blog/2021-05-02-corona-warn-app-version-2-1

https://www.coronawarn.app/en/blog/2021-05-02-corona-warn-app-version-2-1

It removes the link https://buergertest.ecocare.center/#c734 from being actively used and adds a note to say that the website is no longer in operation.
This allows npm run test:links and the weekly link check to report success again.

PR failed due to HTTP 502 (bad gateway):
[RESPONSE 502] https://plotly.com/ hostname: plotly.com on 'http://localhost:8000/en/science/2021-07-08-science-blog-2'
[RESPONSE 502] https://plotly.com/ hostname: plotly.com on 'http://localhost:8000/en/science/2021-07-08-science-blog-2'
[RESPONSE 502] https://plotly.com/ hostname: plotly.com on 'http://localhost:8000/en/science/2021-07-08-science-blog-2'

retrying.

Repeat attempt was successful - https://github.com/MikeMcC399/cwa-website/actions/runs/4342665533.
This PR is ready for merge.

To other blogs that were updated, like https://www.coronawarn.app/en/blog/2021-12-20-cwa-2-15, an "Updated on" date was added. Should this also be done here?

Yes, that'd be nice. Please also kindly include a short description of what has changed.
Please let me know if you do not want to, for any reason - I will take care of it then.

@Ein-Tim

To other blogs that were updated, like https://www.coronawarn.app/en/blog/2021-12-20-cwa-2-15, an "Updated on" date was added. Should this also be done here?

The updated-on is already included.

These blog are only of historical interest. The contents are generally inaccurate.
Each change is accompanied by a date and an explanation. Are you expecting more than this? For example:

IMO, this date needs an update:

dm drogerie-märkte: Schnelltest INOPAI (dm.de) shows a blank map
EcoCare (Lidl and Kaufland): EcoCare - Bürgertest: EcoCare - Bürgertest shows TYPO3 Page Not Found
testbuchen.de/No-Q (participating pharmacies, general practitioners and other test centers): Finde deinen Testtermin (testbuchen.de) still working, but most test centers show "Nicht verfügbar"
Bavarian Red Cross: Mein Corona-Test | BRK shows only one test station (91327 Gößweinstein) in the whole of Germany

So if you want to add an "updated" at the top of the article, then you would need to update these other entries as well.

The alternative to taking the bad link out of the article (which is my preferred solution) would be to block the link-checker from checking the domain buergertest.ecocare.center. That would stop the link test from failing and it would leave the bad link in the blog articles.

I'll leave this to @brianebeling & team to decide! Thank you @MikeMcC399!

Looking good. I think not just removing, but keeping the original link for historical sake is the right call.
I'm traveling right now, the mobile view is.. difficult to work with. You will receive a Review tomorrow, thank you for the PR!

@Ein-Tim

To other blogs that were updated, like coronawarn.app/en/blog/2021-12-20-cwa-2-15, an "Updated on" date was added. Should this also be done here?

The updated-on is already included.

My bad. A date and a description is indeed already there.
When testing I also noticed that the link https://corona-schnelltest-zentren.dm.de/o/dm/login is showing a blank frame.
I'd suggest that we also add a callout to those links and let the user know that they became non-functional or might now work in the future.
That the other links show mostly "Nicht Verfügbar" or only a few test stations is ok for now.
I'd suggest adding a callout to the very top of the page to let the users know that the content may be outdated.
As I'm currently loaded with other tasks and might not get around to it in a timely manner. It'd be much appreciated if we can count on you again - no problem if not - I will get around to it eventually.

@brianebeling
Unfortunately this PR is becoming problematic. Changing the one entry, with the shut-down website, solves one issue, however it leaves the other references uncommented, even though we know that two others are essentially shut down in terms of content as well.
The general principle of blogs is that that they are published and left untouched after publication, unlike FAQ articles which are kept up to date. So if you start editing blogs because they are out of date then you put yourself onto a slippery slope where you would need to edit other similar blogs because their information has become out of date as well.
The reference website for finding test stations is https://map.schnelltestportal.de/. Any general changes about how to find a place to get tested should be announced and shown there. (See also https://github.com/corona-warn-app/cwa-map-public-frontend/issues/35.)
I suggest two alternatives:

Merge this PR as it is with no further changes
Close the PR and open a new PR to add the hostname buergertest.ecocare.center to the list of excepted websites which are not checked by the link checker.

The replacement for this PR is #3418

Closing this PR as the alternative has been merged, I don't want to rule out that we will update the blogs in one way or another, but it seems more likely that they will stay as is.
Thanks for the PR and especially going the mile to also provide an alternative!


chrome unable to access localhost site when using proxy-login-automator as proxy server

Awesome work!  Thank you!! I ran into one issue...
I want to access http://localhost:3000 with google chrome (i'm running a local web application)...
I set google chrome's proxy server to the proxy-login-automator server.  When I attempt to go to the "http://localhost:3000" in the browser, a popup occurs asking for authentication from "cfauth.com"
Any ideas?  Keep up the great work!

Thanks for your feedback.
Are you using PAC?
You can specify a bypass list in chrome command line to avoid using proxy
for localhost.
Regards
2017年11月29日(水) 0:58 ZodsPlan<EMAIL_ADDRESS>
Awesome work! Thank you!! I ran into one issue...
I want to access http://localhost:3000 with google chrome (i'm running a
local web application)...
I set google chrome's proxy server to the proxy-login-automator server.
When I attempt to go to the "http://localhost:3000" in the browser, a
popup occurs asking for authentication from "cfauth.com"
Any ideas? Keep up the great work!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/sjitech/proxy-login-automator/issues/11, or mute the
thread
https://github.com/notifications/unsubscribe-auth/AE_djO_fz5c_iQ2UOJqbBArXJNupapjwks5s7C2fgaJpZM4QteNc
.

Thanks for the feedback.  I am not sure if I am using PAC.  To my understanding, PAC would be something configured at the server level?  In that case, I am behind a corporate proxy so I do not have access to those configurations.  I tried to specify --proxy-bypass-list="localhost:3000" and --proxy-bypass-list="localhost" with no luck.

You can check who is listening at port 3000, just by “netstat -an”  or
similar linux cmd, it might exists two or more processes catching 3000, and
some listen the ipv6, some ipv4, so maybe an application happened to listen
3000 it overrided the existing one which really needed to do underline
proxy work.
2017年12月7日(木) 0:10 ZodsPlan<EMAIL_ADDRESS>
Thanks for the feedback. I am not sure if I am using PAC. To my
understanding, PAC would be something configured at the server level? In
that case, I am behind a corporate proxy so I do not have access to those
configurations. I tried to specify --proxy-bypass-list="localhost:3000" and
--proxy-bypass-list="localhost" with no luck.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/sjitech/proxy-login-automator/issues/11#issuecomment-349668191,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE_djBNbPoH1Nj0G4k8HUG22SPm8ZF-Uks5s9q5ygaJpZM4QteNc
.

Also check 8080 etc.
2017年12月7日(木) 8:16 Jin Qian<EMAIL_ADDRESS>
You can check who is listening at port 3000, just by “netstat -an”  or
similar linux cmd, it might exists two or more processes catching 3000, and
some listen the ipv6, some ipv4, so maybe an application happened to listen
3000 it overrided the existing one which really needed to do underline
proxy work.
2017年12月7日(木) 0:10 ZodsPlan<EMAIL_ADDRESS>
Thanks for the feedback. I am not sure if I am using PAC. To my
understanding, PAC would be something configured at the server level? In
that case, I am behind a corporate proxy so I do not have access to those
configurations. I tried to specify --proxy-bypass-list="localhost:3000" and
--proxy-bypass-list="localhost" with no luck.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/sjitech/proxy-login-automator/issues/11#issuecomment-349668191,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE_djBNbPoH1Nj0G4k8HUG22SPm8ZF-Uks5s9q5ygaJpZM4QteNc
.

The PAC solution worked.  I found out the PAC file being used here and configured everything according to your instructions for the PAC file.  Worked like a charm.  Good work!  Thank you.


Build broken

Electron update broke the build, this commit https://github.com/Microsoft/vscode/commit/a7095d7b6f18ae3e5fceae2a93f3729a391df96a
Erros can be found here https://monacotools.visualstudio.com/Monaco/_build/results?buildId=40314
Seems like for some reason he fails to download the node-debug / node debug-2 extension, even though there is nothing wrong with them.
Please note that building locally works for me, error only happens in the build agent.
fyi @weinand

@Tyriar I have reverted the PR related to electron update so that we can have builds. - https://github.com/Microsoft/vscode/pull/54854

Also just be aware to handle distro commit correctly. I see that the PR merged an old distro commit.

After bit more investigation, it looks like the build is failing while downloading the Electron v2.0.5 from the repo. Looks like the repo does not has it hosted.

Thanks for reverting, closing this in favor of https://github.com/Microsoft/vscode/issues/45542 and https://github.com/Microsoft/vscode/pull/54873


Xarray reader

ping @geospatial-jeff @kylebarron @TomAugspurger @rabernat

I recommend using this dataset: https://pangeo-forge.org/dashboard/feedstock/3
import xarray as xr

store = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/noaa-coastwatch-geopolar-sst-feedstock/noaa-coastwatch-geopolar-sst.zarr'
ds = xr.open_dataset(store, engine='zarr', chunks={})
ds

This seems to work:
import mercantile
import rioxarray
import xarray as xr
import numpy as np

def clip_to_tile(tile: mercantile.Tile, ds: xr.DataArray):
    bounds = mercantile.xy_bounds(tile)
    wgs84_bounds = mercantile.bounds(tile)

    # Create source array by clipping the xarray dataset to extent of the tile.
    source_arr = ds.rio.clip_box(*wgs84_bounds)

    # Create the target array.
    x = np.linspace(bounds[0], bounds[2], 256)
    y = np.linspace(bounds[1], bounds[3], 256)
    target_arr = xr.DataArray(
        np.zeros((ds['time'].shape[0], 256, 256), data_array.dtype),
        [
            ("time", ds['time'].data),
            ("y", y),
            ("x", x)
        ]
    )
    target_arr.rio.write_crs(3857, inplace=True)

    # Map source to target
    matched = source_arr.rio.reproject_match(target_arr)
    return matched

# Create a web mercator tile.
x = 0
y = 0
z = 10
tile = mercantile.tile(x, y, z)

# Open xarray dataset.
store = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/noaa-coastwatch-geopolar-sst-feedstock/noaa-coastwatch-geopolar-sst.zarr'
ds = xr.open_dataset(store, engine='zarr', chunks={})
data_array = ds['analysed_sst']

# Add CRS so rioxarray doens't complain.
data_array.rio.write_crs(4326, inplace=True)

# Only fetch one time slice.
data_array = data_array[:1]

arr = clip_to_tile(tile, data_array)

Here's how you would make a cached zarr store
import zarr
url = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/gpcp-feedstock/gpcp.zarr'
store = zarr.storage.FSStore(url)
cache = zarr.LRUStoreCache(store, max_size=2**28)
ds = xr.open_dataset(cache, engine='zarr')

with the latest commit
with xarray.open_dataset(
    "https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/noaa-coastwatch-geopolar-sst-feedstock/noaa-coastwatch-geopolar-sst.zarr",
    engine="zarr",
    decode_coords="all"
) as src:
    ds = src["analysed_sst"][:1]
    ds.rio.write_crs("epsg:4326", inplace=True)
    with XarrayReader(ds) as dst:
        img = dst.tile(1, 1, 2)

An example using Daymet on Azure:
import rioxarray
import xarray
from matplotlib.pyplot import imshow
from rio_tiler.io import XarrayReader
import requests

credential = requests.get("https://planetarycomputer.microsoft.com/api/sas/v1/token/daymet-daily-pr").json()["token"]

url = "az://daymet-zarr/annual/pr.zarr"
storage_options = {
    "account_name": "daymeteuwest",
    "credential": credential
}
src = xarray.open_dataset(url, engine="zarr", decode_coords="all", storage_options=storage_options)

with XarrayReader(src["prcp"][:1]) as dst:
    img = dst.tile(1, 1, 2)
    imshow(img.data_as_image())

With decode_coords="all", rioxarray is able to correctly construct a CRS (I think using CF-conventions and the grid_mapping_name  attribute).

with latest commit I've removed XarrayReader from rio_tiler.io.__init__ so you need to do from rio_tiler.io.xarray import XarrayReader

here is a simple tile server based on XarrayReader

import urllib
from enum import Enum
from typing import List, Optional, Tuple

import xarray
from fastapi import FastAPI, Query
from fastapi.responses import Response
from pydantic import BaseModel, Field, root_validator
from rio_tiler.io.xarray import XarrayReader
from starlette.requests import Request

class SchemeEnum(str, Enum):
    """TileJSON scheme choice."""

    xyz = "xyz"
    tms = "tms"

class TileJSON(BaseModel):
    """
    TileJSON model.

    Based on https://github.com/mapbox/tilejson-spec/tree/master/2.2.0

    """

    tilejson: str = "2.2.0"
    name: Optional[str]
    description: Optional[str]
    version: str = "1.0.0"
    attribution: Optional[str]
    template: Optional[str]
    legend: Optional[str]
    scheme: SchemeEnum = SchemeEnum.xyz
    tiles: List[str]
    grids: Optional[List[str]]
    data: Optional[List[str]]
    minzoom: int = Field(0, ge=0, le=30)
    maxzoom: int = Field(30, ge=0, le=30)
    bounds: List[float] = [-180, -90, 180, 90]
    center: Optional[Tuple[float, float, int]]

    @root_validator
    def compute_center(cls, values):
        """Compute center if it does not exist."""
        bounds = values["bounds"]
        if not values.get("center"):
            values["center"] = (
                (bounds[0] + bounds[2]) / 2,
                (bounds[1] + bounds[3]) / 2,
                values["minzoom"],
            )
        return values

    class Config:
        """TileJSON model configuration."""

        use_enum_values = True

app = FastAPI()

@app.get("/tiles/{z}/{x}/{y}", response_class=Response)
def tile(
    z: int,
    x: int,
    y: int,
    url: str = Query(description="Zarr URL"),
    variable: str = Query(description="Zarr Variable"),
):
    with xarray.open_dataset(url, engine="zarr", decode_coords="all") as src:
        ds = src[variable][:1]

        # Make sure we are a CRS
        crs = ds.rio.crs or "epsg:4326"
        ds.rio.write_crs(crs, inplace=True)

        with XarrayReader(ds) as dst:
            img = dst.tile(x, y, z)

        content = img.render(img_format="PNG")

    return Response(content, media_type="image/png")

@app.get(
    "/tilejson.json",
    response_model=TileJSON,
    responses={200: {"description": "Return a tilejson"}},
    response_model_exclude_none=True,
    tags=["API"],
)
def tilejson(
    request: Request,
    url: str = Query(description="Zarr URL"),
    variable: str = Query(description="Zarr Variable"),
):
    """Handle /tilejson.json requests."""
    kwargs: Dict[str, Any] = {"z": "{z}", "x": "{x}", "y": "{y}"}
    tile_url = request.url_for("tile", **kwargs)

    qs = [
        (key, value)
        for (key, value) in request.query_params._list
        if key not in ["tile_format"]
    ]
    if qs:
        tile_url += f"?{urllib.parse.urlencode(qs)}"

    with xarray.open_dataset(url, engine="zarr", decode_coords="all") as src:
        ds = src[variable][:1]

        # Make sure we are a CRS
        crs = ds.rio.crs or "epsg:4326"
        ds.rio.write_crs(crs, inplace=True)

        with XarrayReader(ds) as dst:
            return dict(
                bounds=dst.geographic_bounds,
                minzoom=dst.minzoom,
                maxzoom=dst.maxzoom,
                name="xarray",
                tilejson="2.1.0",
                tiles=[tile_url],
            )

curl http://<IP_ADDRESS>:8081/tilejson.json?url=https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/noaa-coastwatch-geopolar-sst-feedstock/noaa-coastwatch-geopolar-sst.zarr&variable=analysed_sst | jq

{
  "tilejson": "2.1.0",
  "name": "xarray",
  "version": "1.0.0",
  "scheme": "xyz",
  "tiles": [
    "[http://<IP_ADDRESS>:8081/tiles/{z}/{x}/{y}?url=https%3A%2F%2Fncsa.osn.xsede.org%2FPangeo%2Fpangeo-forge%2Fnoaa-coastwatch-geopolar-sst-feedstock%2Fnoaa-coastwatch-geopolar-sst.zarr&variable=analysed_sst](http://<IP_ADDRESS>:8081/tiles/%7Bz%7D/%7Bx%7D/%7By%7D?url=https%3A%2F%2Fncsa.osn.xsede.org%2FPangeo%2Fpangeo-forge%2Fnoaa-coastwatch-geopolar-sst-feedstock%2Fnoaa-coastwatch-geopolar-sst.zarr&variable=analysed_sst)"
  ],
  "minzoom": 0,
  "maxzoom": 2,
  "bounds": [
    -180.00000610436345,
    -89.99999847369712,
<PHONE_NUMBER>0436345,
    89.99999847369712
  ],
  "center": [
    0,
    0,
    0
  ]
}

This is fantastic, I'm going to try to find some time to test it.
🚀

In case it's helpful for testing, I put a Zarr store up at "https://pangeo.blob.core.windows.net/pangeo-public/daymet-rio-tiler/na-wgs84.zarr/" that's been reprojected to WGS84 and chunked into 256x256 pixel tiles. There's a single time slice, and one variable tmax.
It's Azure's West Europe region.
import xarray
import matplotlib.pyplot as plt
from rio_tiler.io.xarray import XarrayReader

ds = xarray.open_dataset(
    "https://pangeo.blob.core.windows.net/pangeo-public/daymet-rio-tiler/na-wgs84.zarr/",
    engine="zarr",
    decode_coords="all",
    consolidated=True,
)
da = ds["tmax"]

with XarrayReader(da) as dst:
    img = dst.tile(1, 1, 2)

plt.imshow(img.data_as_image());

We are getting closer to the first release of rio-tiler 4.0
I think the XarrayReader is in a good place, we just need to add tests now to 🚢

🙏 thanks everyone
I'm merging this to the main PR and will work on updating the documentation before publishing a new version of rio-tiler with the new XarrayReader.

Thanks Vincent!

Hi @kylebarron @TomAugspurger again,
I wonder why the axes are always 0-250 (my Zarr store is SST ocean data around Australia's region)

and also for tile settings, I wonder if there is somewhere document on how to pick the numbers. I can only get it works with (1,1,1)
When I ran my Zarr store with the XarrayReader fastapi server, the json returned looks like this :
{
"tilejson": "2.1.0",
"name": "xarray",
"version": "1.0.0",
"scheme": "xyz",
"tiles": [
"[http://<IP_ADDRESS>:8000/tiles/{z}/{x}/{y}?url=s3%3A%2F%2Fimos-data-pixeldrill%2Fvhnguyen%2Fplayground%2Fmulti-years&variable=sea_surface_temperature](http://<IP_ADDRESS>:8000/tiles/%7Bz%7D/%7Bx%7D/%7By%7D?url=s3%3A%2F%2Fimos-data-pixeldrill%2Fvhnguyen%2Fplayground%2Fmulti-years&variable=sea_surface_temperature)"
],
"minzoom": 1,
"maxzoom": 5,
"bounds": [
70.00000213595068,
-69.99999786350668,
190.00000549344387,
19.999999770855315
],
"center": [
130.00000381469727,
-24.999999046325684,
1
]
}

Thank you.

@vietnguyengit is it possible that your X coordinates goes from 0 to 360?

Thanks @vincentsarago
Actually, I understood what those numbers are after reading the docs here: https://cogeotiff.github.io/rio-tiler/api/rio_tiler/io/xarray/
And the results so far are sweet! :tada:

Interactive tools helping me pick XY coordinates values: https://www.maptiler.com/google-maps-coordinates-tile-bounds-projection/

Sample results:
with random XY coordinates around Australia's coastal regions
zoom level 2

zoom level 4

zoom level 5

Ping @alexgleith

Hi @vietnguyengit, nice work! I'll come find you today, let's see if we can get this showing on a webmap... maybe with a timeslider too!

In case it's helpful for testing, I put a Zarr store up at "https://pangeo.blob.core.windows.net/pangeo-public/daymet-rio-tiler/na-wgs84.zarr/" that's been reprojected to WGS84 and chunked into 256x256 pixel tiles. There's a single time slice, and one variable tmax.
It's Azure's West Europe region.
import xarray
import matplotlib.pyplot as plt
from rio_tiler.io.xarray import XarrayReader

ds = xarray.open_dataset(
    "https://pangeo.blob.core.windows.net/pangeo-public/daymet-rio-tiler/na-wgs84.zarr/",
    engine="zarr",
    decode_coords="all",
    consolidated=True,
)
da = ds["tmax"]

The DataArray has nan values – can they be masked out in tile endpoints or does nodata handling need to be added to XarrayReader.tile?
da
<xarray.DataArray 'tmax' (time: 1, y: 3728, x: 17268)>
array([[[nan, nan, ..., nan, nan],
        [nan, nan, ..., nan, nan],
        ...,
        [nan, nan, ..., nan, nan],
        [nan, nan, ..., nan, nan]]], dtype=float32)

Thanks!

@yoninachmany we didn't implement mask in https://github.com/cogeotiff/rio-tiler/blob/main/rio_tiler/io/xarray.py#L239
I'm not familiar enough with rio-xarray but I feel that it should be possible to create a mask and add it to the ImageData output


Feature/add content blocks

Add an initial implementation of content blocks. This branch should:

Add content blocks of type "title", "text" and "gallery"
Add editing functionality for content blocks
Add content generation support for content blocks
Optional seeding id in order to trigger reseed on system update (will improve demos)

This branch will affect:

Data format for posts and pages

Can argue with this. Fix #11


error on "catkin_make_isolated --install"

Hello Bruno
Can you delete the folders "build_isolated", "devel_isolated", and "install_isolated"?
After that, try to recompile the project...
Regards,
Marcelo


feat(eth-contracts): add multisig to TokenLockerV2 and CKBChainV2

[x] add multisig to TokenLockerV2 contract based on TokenLocker
[x] add new tests to TokenLockerV2
[x] add multisig to CKBChainV2 contract based on CKBChain
[x] add new tests to CKBChainV2
[x] add solidity-coverage to generate coverage reports for eth-contracts
[x] upload coverage reports of eth-contracts to codecov in ci

Codecov Report

Merging #87 (0de83a2) into main (4e9dc3c) will increase coverage by 52.97%.
The diff coverage is 69.49%.

@@            Coverage Diff             @@
##            main      #87       +/-   ##
==========================================
+ Coverage   8.22%   61.20%   +52.97%
==========================================
  Files         28       14       -14
  Lines       4290      750     -3540
  Branches       0      130      +130
==========================================
+ Hits         353      459      +106
+ Misses      3937      291     -3646

Impacted Files
Coverage Δ

eth-contracts/contracts/TokenLocker.sol
52.63% <ø> (ø)

eth-contracts/contracts/libraries/Blake2b.sol
50.00% <ø> (ø)

eth-contracts/contracts/TokenLockerV2.sol
57.14% <57.14%> (ø)

...th-contracts/contracts/libraries/MultisigUtils.sol
80.00% <80.00%> (ø)

eth-contracts/contracts/CKBChainV2.sol
66.45% <94.44%> (ø)

...racts/lib/eth-bridge-lockscript-lib/src/actions.rs

ckb-contracts/types/src/eth_header_cell.rs

...ontracts/types/src/generated/eth_recipient_cell.rs

...h-light-client-typescript-lib/src/adapter/chain.rs

...b-contracts/types/src/generated/eth_header_cell.rs

... and 37 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cf8387b...785a9fb. Read the comment docs.


Document lol widget launchText optionality

In the documentation about the list-of-links widget type, note that launchText is now optional per #469. .

Coverage remained the same at 15.784% when pulling 45ca31ad8be92048afaf469ed462f94bbc18c699 on apetro:doc-lol-no-launch-link-handling into efdee413125f14f1bbe2deee70cd058f0a7a8fa9 on UW-Madison-DoIT:master.

:+1:


package readying pylds

I moved the code to a subdirectory so that it's easier to import in other projects.
I also added support for nonstationary observation noise.

Noice!
Aesthetically I'm not sure about having the branch inside the loop over t, but realistically any branch predictor would nail that. Maybe we can move that around in the future but for now this is a welcome feature.
Thank you!


Csharp template fix

Small fix in cs files comments regarding python instead cs.

Looks good, thanks!


Create previews for branches, not per commit

Sorry did not mean to open this issue in this repo lol


Clock click update function mutates the underlying data before calling the update function, preventing some hooks from working properly

In GitLab by @itamarcu on Sep 9, 2023, 10:19
There's a bug in the following lines:
https://gitlab.com/asacolips-projects/foundry-mods/pbta/-/blob/master/src/module/actor/actor-sheet.js?ref_type=heads#L463-486
The solution is to alter line 463 from:
    let attr = getProperty(this.actor, prop);
to:
    let attr = deepclone(getProperty(this.actor, prop));
Explanation
What the code is currently doing, while the bug is there, is mutating the attr.value value and the attr.steps array in the original actor's  local data while calculating the new value and the new array.  The method is only supposed to create an updated copy of attr, and send it via this._updateActorOrToken(update); for Foundry to handle with its data update code.  But, by altering the value before it's being sent to that method, the current behavior prevents any hooks from being able to compare the old and new values of the attribute.
I found this bug because it's currently preventing my module, https://github.com/shemetz/tokenmagic-automatic-wounds, from working with PBTA systems.  When my code compares the old property value and the new property value, they're always "identical", because the comparison - even though it happens in the preUpdateActor hook - uses the post-update value.

In GitLab by @mclemente on Nov 13, 2023, 20:59
mentioned in commit b04957be8a48c8232e96a67a249422d24b3b6a29


chore: refactor PromQL enforcer

This change renames the Enforcer struct to PromQLEnforcer to clarify its purpose. It also removes the error type casting in favor of error wrapping.

You've got a good point regarding not returning the bad PromQL expression. I'll change this (it will also be consistent with the Prometheus API).


Cppms patch

According to OSGI, Update notifications are only supposed to be sent by ConfigurationAdmin when a configuration object has been updated. A configuration object which has not yet been Updated should not result in an Update Notification. Also, a Removed Notification may only be sent if an Updated notification was previously sent. There were several places in our ConfigurationAdmin code where Updated and Removed notifications were being sent to services in error. This change fixes this.
Each Configuration object contains a changeCount. The changeCount is set equal to 1 when the Configuration object is created and incremented each time the Configuration object is Updated. Updated and Removed notifications will only be sent for Configuration objects with a changeCount > 1. The AddConfigurations method, adds Configuration objects for configurations read from the manifest.json file. These Configuration objects are added to the repository with changeCount = 2 since the construction of these objects is equivalent to a creation and an update.
The Configuration object now has a method called IsUpdated which returns true for Configuration objects with a changeCount > 1 and false otherwise.
ConfigurationAdminImpl::NotifyConfigurationUpdated now accepts a second input parameter called removeUpdate. This is true when the Configuration object has been removed but IsUpdated was true before the object was removed. NotifyConfigurationUpdate is used for both Updated and Removed operations. For a Removed operation the Configuration object has already been removed and IsUpdated cannot be called so this input parameter controls whether or not a Removed notification will be sent.
ConfigurationAdmin has service trackers that monitor services that implement the ManagedService and ManagedServiceFactory interfaces. It receives an AddingService notification when one of these services is registered. ConfigurationAdmin is responsible for notifying services of Configuration objects that were added to the repository (and updated) prior to the service being registered. There was a problem with our code because the configuration repository was being changed while the AddingService methods were accessing the repository. The race condition resulted in two Updated notifications being sent for the same Update. This change fixes this problem by locking the configuration repository at the beginning of the AddingService method so that no changes can occur while the AddingService method is in process.

Codecov Report

Merging #643 (c8d9362) into development (b088a4f) will decrease coverage by 0.88%.
The diff coverage is n/a.

:exclamation: Current head c8d9362 differs from pull request most recent head 11ad0d5. Consider uploading reports for the commit 11ad0d5 to get more accurate results

@@               Coverage Diff               @@
##           development     #643      +/-   ##
===============================================
- Coverage        90.50%   89.61%   -0.89%
===============================================
  Files              250      133     -117
  Lines            13449     8611    -4838
===============================================
- Hits             12172     7717    -4455
+ Misses            1277      894     -383

Impacted Files
Coverage Δ

compendium/LogService/src/LogService.cpp
0.00% <0.00%> (-100.00%)
:arrow_down:

framework/src/bundle/BundleRegistry.cpp
80.72% <0.00%> (-11.25%)
:arrow_down:

...mework/src/service/ServiceReferenceBasePrivate.cpp
87.55% <0.00%> (-5.40%)
:arrow_down:

framework/src/service/ServiceListeners.cpp
95.37% <0.00%> (-4.63%)
:arrow_down:

framework/src/service/ServiceRegistry.cpp
90.72% <0.00%> (-2.58%)
:arrow_down:

framework/src/service/ServiceRegistrationBase.cpp
80.82% <0.00%> (-1.83%)
:arrow_down:

...include/cppmicroservices/detail/TrackedService.tpp
83.87% <0.00%> (-1.62%)
:arrow_down:

framework/src/service/ServiceReferenceBase.cpp
96.02% <0.00%> (-1.33%)
:arrow_down:

framework/src/bundle/BundleResourceContainer.cpp
93.06% <0.00%> (-1.16%)
:arrow_down:

framework/src/bundle/BundlePrivate.cpp
82.37% <0.00%> (-1.07%)
:arrow_down:

... and 122 more

ConfigAdminTests.testServiceUpdated failed in a couple of the builds


Failing on webpack build

Getting the following error when I import my main.go file. Could this be a problem with my environment variables?
ERROR in ./src/main.go
Module build failed (from ./node_modules/golang-wasm-async-loader/dist/index.js):
Error: spawn undefined/bin/go ENOENT
    at Process.ChildProcess._handle.onexit (internal/child_process.js:264:19)
    at onErrorNT (internal/child_process.js:456:16)
    at processTicksAndRejections (internal/process/task_queues.js:84:9)
 @ ./src/index.js 1:0-28 4:25-29

I do get the same error when cloning the repo and running the web example itself

Note to anyone else encountering this issue: need to explicitly define variables GOENV and GOROOT. Does not take defaults. Does not take variables in go env


Add markerfacecolor & markeredgecolor params to _paint_carto_legend

Hi! Awesome work!
I've been using geoplot to make pointplots such as these.

My points are colored with a single color and hence I wanted that my legend would have the same markeredgecolor and facecolor that my points on the map had. I were exploring your codes and it seems that only way to change the legend coloring in _paint_carto_legend was to go and change the source code.
Hence, I would suggest to adding 'markerfacecolor' and 'markeredgecolor' parameters into the legend_kwargs which would allow to adjust those accordinly.
Keep up the good work!
Henrikki

Hey there. Working on that in #34

As of #114, markerfacecolor and markeredgecolor are now propagated to the legend when you pass color/facecolor and/or edgecolor parameters to your plot.
Sorry to have left this issue open for so long. There are a ton of improvements to geoplot coming out in the next 0.3.0 release, which fixes many many longstanding issues like this and others, you should check it out again once it's done. 😄

Following up to say that this is now addressed more directly by #127.
On geoplot@master you can now pass marker* arguments to legend_kwargs (specifically: {marker, markersize, markeredgecolor, markeredgewidth, markerfacecolor}), and those parameters will be passed to the legend markers.
For example, here is a code snippet demonstrating the current defaults:
import geopandas as gpd
import geoplot as gplt
import geoplot.crs as gcrs

continental_usa_cities = gpd.read_file(gplt.datasets.get_path('usa_cities'))
continental_usa_cities = continental_usa_cities.query('STATE not in ["AK", "HI", "PR"]')
contiguous_usa = gpd.read_file(gplt.datasets.get_path('contiguous_usa'))

ax = gplt.pointplot(
    continental_usa_cities, projection=gcrs.AlbersEqualArea(),
    scale='POP_2010', limits=(2, 30), hue='POP_2010',
    legend=True, legend_var='scale', legend_labels=range(1,6)
)
gplt.polyplot(contiguous_usa, ax=ax)

We can fiddle with the legend marker params to get something very different:
import geopandas as gpd
import geoplot as gplt
import geoplot.crs as gcrs

continental_usa_cities = gpd.read_file(gplt.datasets.get_path('usa_cities'))
continental_usa_cities = continental_usa_cities.query('STATE not in ["AK", "HI", "PR"]')
contiguous_usa = gpd.read_file(gplt.datasets.get_path('contiguous_usa'))

ax = gplt.pointplot(
    continental_usa_cities, projection=gcrs.AlbersEqualArea(),
    scale='POP_2010', limits=(2, 30), hue='POP_2010',
    legend=True, legend_var='scale', legend_labels=range(1,6),
    legend_kwargs={
        'markeredgecolor': 'gray',
        'markeredgewidth': 3,
        'markerfacecolor': 'lightgray',
        'marker': '*'
    }
)
gplt.polyplot(contiguous_usa, ax=ax)

Hope that helps. 😄


7830495659 - 1.0 Alef Release

This is the Alpha (Alef) release of the Tiferet Engine framework, v1.

Bereshit barah Elohim...


Bug: Connection ID unknown immediately after connection

Describe the bug
Immediately after connecting to DB using \Config\Database::connect('group_name'), the connID of DB cannot be obtained. You will be able to see the connID after executing some SQL. I found it when using Postgre.
CodeIgniter 4 version
4.0.4
Affected module(s)
CodeIgniter\Database\Postgre
Expected behavior, and steps to reproduce if appropriate
Execute \Config\Database::connect('group_name') to get the public connID of the returned object.
Context

OS: Windows 10 2004 64bit
Web server is [php spark server]
PHP version 7.4.9 x86_64 Thread Safe

This is intentional. It's made that's way to avoid unnecessary database connection. You may have database class loaded globally and an immediate connection is not always needed as we may not use the database at all for some pages. Therefore the connection takes place only when the first query is called.
If you want to connect immediately - before you run any query, just call the initialize() method.
$db = \Config\Database::connect();
$db->initialize();

Thank you for answering.


clk_divider with cocotb test

Motivate of the pull request

[ ] To address an existing issue. If so, please provide a link to the issue:
[X] Breaking new feature. If so, please describe details in the description part.

Describe the technical details
What is currently done? (Provide issue link if applicable)
I have added clock divider rtl design with it's cocotb test-bench

What does this pull request change?

Which part of the code base require a change

[X] RTL
[ ] Gate-level netlists
[X] Design Verification
[ ] Physical Design
[ ] Timing characterization
[ ] Documentation
[ ] Regression tests
[ ] Continous Integration (CI) scripts

Impact of the pull request

[ ] Require a change on Quality of Results (QoR)
[ ] Break back-compatibility. If so, please list who may be influenced.

@hammadshabbir-rs Can you update the documentation? We are close to merge this PR. You can find guidelines.
If you need help, do not hesitate to contact me.


Nodejs Runtime: Use bare minimum base image

Issue Ref:  #750
Description:
Provide a bare minimum docker image for nodejs using distroless.
[PR Description]
An additional Dockerfile Dockerfile.8.opt is added which provides a bare minimum nodejs 8 runtime using the distroless images.
One refactoring is required. All kubeless related files need to be put within a sub folder.
This affects all nodejs dockerfiles and the kubeless.js file.
Currently, this sub folder is named kubeless_rt.
Before, in example the lib folder has been copied to the root directory and conflicts with an existing folder within the distroless base image.
I have tested the refactored nodejs docker images Dockerfile.8 and Dockerfile.8.opt.
The additional image is not yet integrated in the overall make/build process.
TODOs:

[x] Ready to review
[x] Automated Tests
[ ] Docs

great, @Henrike42 do you mind write a few lines of docs explaining this variant in docs/runtimes.md? (in the NodeJS section)

thank you!


Allow use of async / await

Hello. I think with the new ES5 / 6 updates, it would be interesting to use async / await. What do you think? I can work in a PR for this.

You can use the current promise based methods with async/await ;)

You can use the current promise based methods with async/await ;)

Yeah, I realized that after I had already opened the issue. I'm sorry.

And sorry about my english too. I am Brazilian. I'm training.


Temp directory creation for script handling is missing in aadIntegration.sh for Dynamic Cluster offer

Temp directory creation for script handling is missing in aadIntegration.sh for Dynamic Cluster offer

This issue is now resolved.


Upgrade from previous version

Hi Tom, great to see you moved to hithub, this way you have more chances to get help from community :)
BTW would be helpful to have hints on update procedure, especially from previous version published as update set directly on snow forum.

Hi Yuri,
Unfortunately there is no "upgrade" path as the new version is a completely separated scoped application. You would have to install the new version and re-configure the compare configuration. If you have created many and complex configurations I can write a script to transition the records over (the db structure is quite similar), but most probably it's less effort to just manually recreate the configuration. After the transition the old Scoped Application including the "automagically" created Global Scoped script "CrossInstanceCompareGlobal" can be deleted.
Thanks for raising the issue, please let me know if you need help with the transition or if this is fine for you.
Tom

Thanks, it's ok to drop all the data.
I was asking more in terms of installation process, if there are some conflicts possible/special steps required.
So basically we just backout update set with beta version and deploy new one from this repo.
Thanks for reply.


Add exception for missing system requirements

Pre-requisites
Please ensure you have done the following:

[x] I have read the CONTRIBUTING.md document.
[x] If my change requires a change to the documentation, I have updated the documentation accordingly.
[x] I have added tests to cover my changes.

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to change)

Describe changes
Briefly describe the changes you have introduced.
I have modified the check_installation function and the Integration class to look for missing system requirements particularly pertaining to the graphviz integration. The new code raises an Exception during zenml init were it mentions the library whose package is missing from the system. I tried to make it raise the error during zenml example run but for that to be done the changes should be done in example cli and it was mentioned in the following issue #266 that an override for check_installation is preferable.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

I like the implementation using the additional list of system requirements! We might consider using shutil.which(...) to check if a command exists without actually running it using subprocess.run(...), which would allow us to simplify the system requirements to ["dot"] (without the -V).

I like the implementation using the additional dict of system requirements! We might consider using shutil.which(...) to check if a command exists without actually running it using subprocess.run(...), which would allow us to simplify the system requirements to ["dot"] (without the -V).

Thank you for the suggestion, I will change the subprocess to shutil.which .

Makes sense that we have failing tests now, as they also have a dependency on check_installation. For the scope of this bugfix it might make sense to hard code the installation of graphviz on the level of our github actions. In the long run we could then maybe have a feature ticket to implement this in a more generic way for all possible future “System requirements” as well.
Do you need any additional guidance regarding our github actions?

Makes sense that we have failing tests now, as they also have a dependency on check_installation. For the scope of this bugfix it might make sense to hard code the installation of graphviz on the level of our github actions. In the long run we could then maybe have a feature ticket to implement this in a more generic way for all possible future “System requirements” as well.
Do you need any additional guidance regarding our github actions?

So the idea is to now ensure that graphviz is installed on the system that the github actions is testing? Please do correct me if I am wrong.

I have modified the error message, subprocess library and the github actions as well in the new commit.

I have modified the error message, subprocess library and the github actions as well in the new commit.

Thanks Kamalesh. I don’t see any changes on the github actions within your commit. I think you may have forgotten to commit or push your changes within the .github folder?

I have modified the error message, subprocess library and the github actions as well in the new commit.

Thanks Kamalesh. I don’t see any changes on the github actions within your commit. I think you may have forgotten to commit or push your changes within the .github folder?

Yup sorry I forgot to push them. I have modified the commit and pushed it again.

@kamalesh0406 In order to fix these linting issues, we have a script in the scripts folder called format.sh. Run it as  follows:
cd zenml/
poetry run bash scripts/format.sh

or simply:
./scripts/format.sh

You can also run linting as follows:
poetry run bash scripts/lint.sh
poetry run bash scripts/check-spelling.sh

And tests as follows:
poetry run bash scripts/test-coverage-xml.sh

I'll add this to the CONTRIBUTING.md for future reference

@kamalesh0406 Sorry for pushing directly onto your repo, I must have pressed the wrong button, was just gonna add it as a suggestion. I think now there's just a newline missing at the end of the file before we can merge

Okay I will check it and push the changes.

I am not able to recreate the lining issue on my local, can you specify the exact issue? The github actions output is not very clear as well.

I am not able to recreate the linting issue on my local, can you specify the exact issue? The github actions output is not very clear as well.

Locally within your repo you should be able to just run poetry run bash scripts/format.sh - this will fix the issue for you. Then you just need to commit and push the file.

@kamalesh0406 Looks great, thanks for your contribution :pray:


Package Ubuntu / Debian companion

https://askubuntu.com/questions/71510/how-do-i-create-a-ppa

https://blog.packagecloud.io/eng/2015/07/14/using-dh-make-to-prepare-debian-packages/

https://blog.packagecloud.io/debian/debuild/packaging/2015/06/08/buildling-deb-packages-with-debuild/
https://blog.packagecloud.io/eng/2015/05/18/building-deb-packages-with-pbuilder/
https://wiki.debian.org/Packaging/Intro?action=show&redirect=IntroDebianPackaging
https://packaging.ubuntu.com/html/packaging-new-software.html

AppImage seems to work for Chrome and Firefox. Does not work for Snap Chromium unfortunately.

https://saveriomiroddi.github.io/Building-a-debian-deb-source-package-and-publishing-it-on-an-ubuntu-ppa/#building-the-source-package
https://saveriomiroddi.github.io/Learn-to-prepare-PPA-packages-by-setting-up-a-Ruby-PPA/


chore(release): update rbac-backend to v5.2.1

Description
update rbac-backend to v5.2.1
Which issue(s) does this PR fix

Fixes https://issues.redhat.com/browse/RHIDP-4732

PR acceptance criteria
Please make sure that the following steps are complete:

[ ] GitHub Actions are completed and successful
[ ] Unit Tests are updated and passing
[ ] E2E Tests are updated and passing
[ ] Documentation is updated if necessary (requirement for new features)
[ ] Add a screenshot if the change is UX/UI related

How to test changes / Special notes to the reviewer

Works correctly!

Rebased to get e2e tests improvements from the main branch.

/test e2e-tests

Close on favour: https://github.com/janus-idp/backstage-showcase/pull/1869


limits.kernel.nofile and systemd init / PID 1 limit

Required information

Distribution: Ubuntu
Distribution version: 20.04
The output of "lxc info" or if that fails:

Kernel version: 5.4.0-9-generic
LXC version: 3.2.1
LXD version: 3.22
Storage backend in use: dir

Issue description
LXC/LXD documentation mentions using limits.kernel.nofile to increase max. open files for containers but that doesn't allow you to get past 1048576 with container failing to start:
lxc cactuar<PHONE_NUMBER>5049.930 ERROR    conf - conf.c:setup_resource_limits:2597 - Operation not permitted - Failed to set limit nofile
lxc cactuar<PHONE_NUMBER>5049.930 ERROR    start - start.c:lxc_spawn:1792 - Failed to setup resource limits
lxc cactuar<PHONE_NUMBER>5049.930 WARN     network - network.c:lxc_delete_network_priv:2589 - Operation not permitted - Failed to remove interface "eth0" with index 12
lxc cactuar<PHONE_NUMBER>5049.930 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:842 - Received container state "ABORTING" instead of "RUNNING"
lxc cactuar<PHONE_NUMBER>5049.931 ERROR    start - start.c:__lxc_start:1939 - Failed to spawn container "cactuar"
lxc<PHONE_NUMBER>5049.942 WARN     commands - commands.c:lxc_cmd_rsp_recv:132 - Connection reset by peer - Failed to receive response for command "get_state"
Not even with fs.nr_open set to something higher than 1048576.
It seems this is due to the systemd limit as per commit https://github.com/systemd/systemd/commit/6385cb31ef443be3e0d6da5ea62a267a49174688 and also:
$ grep open /proc/1/limits
Max open files            1048576              1048576              files
The fix here is to call prlimit to increase PID 1's limits. Maybe LXD should do this if/when limits.kernel.nofile is configured?
$ sudo prlimit --pid 1 --nofile=7424000
Steps to reproduce

Spin up a new container.
sudo sysctl fs.nr_open 7424000
sudo lxc config set $container$ limits.kernel.nofile 7424000
Stop and start container.
Fails to start.
sudo prlimit --pid 1 --nofile=7424000
Start container.

Information to attach

[ ] Any relevant kernel output (dmesg)
[ ] Container log (lxc info NAME --show-log)
[ ] Container configuration (lxc config show NAME --expanded)
[ ] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
[ ] Output of the client with --debug
[ ] Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)

Deployed a clean Ubuntu 20.04 system and did the following per your instructions:
root@vm02:~# lxc launch images:alpine/edge a1
Creating a1
Starting a1
root@vm02:~# cat /proc/sys/fs/nr_open
1048576
root@vm02:~# echo 7424000 > /proc/sys/fs/nr_open
root@vm02:~# cat /proc/sys/fs/nr_open
7424000
root@vm02:~# lxc config set a1 limits.kernel.nofile 7424000
root@vm02:~# lxc stop a1
root@vm02:~# lxc start a1
root@vm02:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu Focal Fossa (development branch)
Release:	20.04
Codename:	focal
root@vm02:~# lxc exec a1 sh
~ # cat /proc/self/limits | grep "open files"
Max open files            7424000              7424000              files
~ # root@vm02:~#

anything else I'm missing to have this issue happen?

Ah, the steps I used to reproduce this is Bionic and LXD 3.03.
https://pastebin.ubuntu.com/p/Jx8vjZ8YtS/

I'll just close this off for now as we'll just rework our application reducing the max. fds required.


Enabling use of [] tokens in map

Hi,
I've made a simple implementation than the #1 pull request to enable the mappings to use array notation to seek and create arrays.
Examples in test file.
Regards

Awesome. Thank you for this. I merged and pushed out v2.0.0 to npm.
:+1:


fix nilpointer

see https://github.com/raintank/raintank-metric/issues/210 for rationale.
@woodsaj i wanna leave this open until it's been daytime for you to give you a chance to react.
but i hope you'll just agree with me


Add scss support for more powerful stylesheets

Adds scss compilation on startup, as well as refactoring of css into scss

Resolves #28

appengine compilation delegated to #37


feature： Add UT with pytest

What would you like to be added:
Add UT with pytest.
Why is this needed:

@Nathan


Add --open classnames to sidemenu

just a small one,
we show larger buttons there and its currently not possible to do an overflow, because
there is no class when the sidemenu is open.
i added in on both sidemenu and sidemenu__items (&--open). sidemenu__items--open had already been there in a previous version but was removed as it seems.

Welcome back @macrozone! :)
It's released under the 0.4.23 version.


ClrCollectionAccessorSource don't use IClrPropertyGetter from DI

When FW accessing to collection on Entity by ClrCollectionAccessorSource, accessing property "directly". It is possible use IClrPropertyGetter from DI for access to property?
File: ClrCollectionAccessorSource.cs, line: 81, branch: dev
=> var getterDelegate = (Func<TEntity, TCollection>)property.GetMethod.CreateDelegate(typeof(Func<TEntity, TCollection>));

@LonDat can you provide details of the scenario you are trying to achieve? This is not a type we would expect app developers to need to interact with, so it's just setup to work for what our internal implementation needs.

I implemented lazy loading support for our DTO.
Each of our DTO has its own data storage (something like Dictionary <propName, value>) and each property has custom getter something like get {return _dic [propName] ?? (_dic [propName] = LazyLoad ()); }. I have custom implemtation of IClrPropertyGetter, which accessing data directly over "dictionary". The LazyLoad() retrieve a collection of records, which is due ChangeTracking trying to update the same collection that loading (it's ok). But because ClrCollectionAccessorSource accessing property directly, we have stack overflow :)
I have temporary solution by custom implementation of IClrCollectionAccessorSource and our lazy loading implementation work's great.
I don't know, if you design IClrPropertyGetter/IClrPropertySetter as VALUE accessors, but I like idea "one service for DTO properties" and IClrCollectionAccessorSource doing much more then accessing properties.
Thank you for question and sorry for my English :)

@LonDat IClrCollectionAccessorSource is the extensiblity point to use for changing the way collection properties are accessed. It handles the caching and dynamic creation of delegates. If you want to do something that doesn't dynamically create delegates or creates them in a different way, then this is the service to replace.

@ajcvickers Thx for response...so IClrCollectionAccessorSource will not use IClrPropertyGetter/IClrPropertySetter. I understand that correctly?
Thx :)


Add calico v3.22.1 package

What this PR does / why we need it

Added calico v3.22.1 package
Added tests for calico v3.22.1 package
Removed the invalid namespace config for all calico packages
Removed calico-ce package which is also v3.22.1

Which issue(s) this PR fixes
Fixes: #3743
Describe testing done for PR
Ran make test for Calico 3.22.1 package.
Special notes for your reviewer
Package is stored in personal registry and hasn't pushed to projects.registry.vmware.com/tce yet

Hi @joshrosso @stmcginnis , for pr https://github.com/vmware-tanzu/community-edition/pull/3550, I removed this calico-ce package and integrate v3.22.1 back to calico packages, please take a look. If there is any background that we need to keep 2 kinds of calico packages please let me know, thanks!
And perhaps since the v3.22.1 package bundle is already pushed by calico-ce package, when I push the package to the registry, the error below shows. So I am temporarily using a personal registry. Not sure how to handle this, cc @seemiller .
Error: Writing 'projects.registry.vmware.com/tce/calico:3.22.1': Writing image: Retried 5 times: PUT https://projects.registry.vmware.com/v2/tce/calico/manifests/3.22.1: PRECONDITION: Failed to process request due to 'calico:3.22.1' configured as immutable.

The reason that this pr involves harbor package owners is because I removed one blank line in harbor package tests, which was blocking pr lint. No other modifications made.


Bad/Wrong poses after custom training

Hi all, I am trying to train CosyPose on my own dataset.
After playing with TLESS dataset i tried with my own data (one object):

Create "synthetic data" with "start_dataset_recording.py"
Train detector on this snyt dataset
Train coarse pose and refine pose on this synt dataset
Start inference with "webcam" (realsense D435)
Be happy, that the pose estimation looks good for this first try. (The realsense looked on my computerscreen with opened 3d viewer with the trained object)

As my next step i tried to render  better data with "BlenderProc". I adjust the "bop_object_physics_positioning" example to render data.
As before I train detector, coarse and refine pose. But the detection of the pose is now wrong. And i think its not unprecise, i think its wrong. Because after visualizing the result of 6d estimation one corner of the object is always in the center of the bounding box of the object detection. Here is an image as example:

What do you think about it? Is it just a very unprecised detection or is something wrong? Perhaps I have an error in rendering the data? Perhaps there is something wrong with my model, or i do something wrong with the camera parameters? Does anyone have an idea?
Currently i try to render new data with one model (27) of the tless dataset, and I will try to train on this data than.
--> Done. I used the *.ply and *.obj (urdf) files from object 27 of the TLESS dataset, and do all the rest on my own: rendering data, train detector, train coarse/refiner pose. After that pose estimation work (a bit unprecise, but ok).
So my issue must be somewhere in my 3d modelfiles.
The last difference i see is, that my 3d model is always in positives values (x/y/z). The TLESS model is "arround 0/0/0", so it has also negativ values... But that can not be a problem?? Or can it??
Please help!
Here are my used files for rendering data /training:
urdfs-versuch03.zip
models.zip
@azad96 Perhaps you have an idea?

@hannes56a, I am not sure man. I'd look into it, but I no longer have access to my codebase. What is the size of the training set. Do you have enough variety in the training set?  Did you check the evaluation metrics with paper_training_logs.ipynb?

Thank you for your answer. I do not think, that it is an issue in training or size of training set. Because i tried the same (same trainingset size, training parameter, etc...) only with another 3d object model. And with this other object model it works...

Thank you for your answer. I do not think, that it is an issue in training or size of training set. Because i tried the same (same trainingset size, training parameter, etc...) only with another 3d object model. And with this other object model it works...

hello,could you tell me how to train 2D detector ,thanks!
my english is bad ,sorry


Fix connection backup_server_node parameter

This fixes a SyntaxError with the backup_server_node setting. The code attempts to execute: id_port = <IP_ADDRESS> when the desired functionality is id_port = '<IP_ADDRESS>'.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks a lot @lassiraa for fixing this issue!


fix: center user full name for multiple lines

Linked issue
Fixes user's full name not being centered when broken over multiple lines.

:tada: This PR is included in version 0.2.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket: