Removes explicit type declaration

This pull request removes the need to explicitly declare type or force cast from NSManagedObject for find and create methods.
let cat: Cat = Cat.firstOrCreate()
// or
let cat = Cat.firstOrCreate() as! Cat

can be now written as
let cat = Cat.firstOrCreate()

This is sadly not possible for [Self] now. Tests adjusted accordingly.

Thanks @kafejo, it's better now!


Unable to track an entity of type  because alternate key property 'id' is null. EF 3.0 preview.

I am attempting to attach a database entity to track changes and I receive the following exception.
Unable to track an entity of type 'GameSession' because alternate key property 'id' is null. If the alternate key is not used in a relationship, then consider using a unique index instead. Unique indexes may contain nulls, while alternate keys must not.
FullStackTraceException.txt
The line in question fails here:
dbContext.GameSessions.Attach(session);
I am using CosmosDB as my data-provider. I did put a breakpoint at the attach line to stop execution and check the DB to see if the GameSession was properly filled out. Below is the document entry that was created. As you can see there is an id field and it is non-null.
{
    "GameSessionId": "e8c49f74-c125-4fcf-82c1-6b04836c4e6c",
    "ConnectionId": "y3tD6y1HK-8J2NW4c1kDqg",
    "DisconnectCodeId": null,
    "Discriminator": "GameSession",
    "EndTime": "9999-12-31T23:59:59.9999999",
    "StartTime": "2019-05-29T17:21:20.5722608Z",
    "id": "GameSession|e8c49f74-c125-4fcf-82c1-6b04836c4e6c",
    "__partitionKey": "0",
    "_rid": "YsIUAK8YAXoRAAAAAAAAAA==",
    "_self": "dbs/YsIUAA==/colls/YsIUAK8YAXo=/docs/YsIUAK8YAXoRAAAAAAAAAA==/",
    "_etag": "\"0000f604-0000-1100-0000-5ceebf920000\"",
    "_attachments": "attachments/",
    "_ts":<PHONE_NUMBER>
}

The GameSession object:
public class GameSession
{
    public Guid GameSessionId { get; set; }
    public string ConnectionId { get; set; } = string.Empty;
    public DateTime StartTime { get; set; } = DateTime.MinValue;
    public DateTime EndTime { get; set; } = DateTime.MaxValue;
    public DisconnectCode DisconnectCode { get; set; }
}

Further technical details
EF Core version: 3.0.0-preview5.19227.1
Database Provider: Microsoft.EntityFrameworkCore.Cosmos (3.0.0-preview5.19227.1)
Operating system: Windows 10 Pro Version 10.0.17134 Build 17134
IDE: Visual Studio 2017 15.9.9

this error occurred with me calling Update method of EF Core Cosmos 3.0 oficial release (version 3.0.100).
I think that exception occurred because EF Core Cosmos not load yours internal json properties of cosmos document, for example "id", "_rid", "_self", "_etag", etc.
resolved it getting entity of database with AsTracking method, change values and save.
my implementation
Repository Base or you specialized repository
        protected virtual async Task<bool> Update(TModel model)
        {
            bool updated = false;

            try
            {
                var existingModel = await this.FindAsync(model, true);
                // See more details below
                model.CopyTo(existingModel);

                base.Db.Set<TModel>().Update(existingModel);
                updated = true;
            }
            catch (Exception e)
            {
                var msg = string.Format("An error ocurred during update model: " + e.Message);
                await this.ServiceBus.PublishEvent(new SystemError(model, msg));
            }

            return updated;
        }

Find Method - In Repository Base or your specialized repository
public virtual Task<TModel> FindAsync(TModel entity, bool track = false) => this.SearchByIdAsync(entity.Id, track);
        public virtual async Task<TModel> SearchByIdAsync(Guid id, bool track = false)
        {
            TModel entity;

            try
            {
                if (track)
                    entity = this.Db.Set<TModel>().Find(id);
                else
                    entity = this.Db.Set<TModel>().AsNoTracking().Where(m => m.Id == id).FirstOrDefault();
            }
            catch (Exception e)
            {
                entity = default;
                var msg = string.Format("An error ocurred during search a model: " + e.Message);
                await this.ServiceBus.PublishEvent(new SystemError(new { id, track}, msg));
            }

            return entity;
        }

CopyTo Method - In Model class
        /// <summary>
        /// Copy property values of the current object to another object.
        /// </summary>
        /// <param name="model">The model of destination changes.</param>
        /// <param name="changeId">Set 'True' to change Id or 'False' to do not change.</param>
        public override void CopyTo(IModel model, bool changeId = false)
        {
            var city = model as CityBaseModel;

            if (city == null) return;

            if (changeId)
                city.Id = this.Id;

            city.Name = this.Name;
            city.SGIndex = this.SGIndex;
            city.IsCapital = this.IsCapital;
            city.State = this.State;
        }


Images served without a file extension are not displayed properly

Since content is determined with regex, static files that are not served with their file extension (either as a base64 image, or served with a given route) are not displayed properly.
Example
Featherlight could check the response headers through jQuery's XHR parameter instead of analyzing the URL structure to determine the content-type ; there could also be a way to force the expected content ignoring the regex matches. I may have misunderstood the docs but couldn't find a way to do this yet.

If you know for sure that you want to load images into Featherlight, you can set the attribute data-featherlight-type="image".

Sorry I never chimed back on this. I don't see anything special in the example

I probably pasted he wrong jsfiddle sorry ! I will try and replicate the issue and post a proper link.

If you know for sure that you want to load images into Featherlight, you can set the attribute data-featherlight-type="image".

I can't believe I missed this in the docs. It seems to work just fine, as per this JSFiddle.
It would still be possible to automatically enforce the data type based on content-type in response headers, if provided.


Added Black & White Ring Game Codechef Solution in CPP

Great work @arsenicftw01 please give this repo a star if it really helped you to complete your PR.
Also follow me for follow back if you want to get connected with me for any such open source contribution program. Thank you 😊


Event log message indicates that the Windows Installer reconfigured all installed applications

Hi.  I noticed the following.  On start up I noticed all of my applications reconfigured themselves.
https://support.microsoft.com/en-us/kb/974524
Microsoft released this:
Cause
This problem can happen if one of the following is true:
You have a group policy with a WMIFilter that queries Win32_Product class.
You have an application installed on the machine that queries Win32_Product class.
Note For more information about WmiFilters please refer to: KB555253 - HOWTO: Leverage Group Policies with WMI Filters
Resolution
If you are using a group policy with the WMIFilter that queries Win32_Product, modify the filter to use Win32reg_AddRemovePrograms.
If you have an application that makes use of the above class, please contact the vendor to get an updated version that does not use this class.
To narrow down the application that is causing the problem, you can follow the "Clean boot" troubleshooting method in the following article in the Microsoft Knowledge Base:
How to troubleshoot configuration errors by using the System Configuration utility in Windows XP (http://support.microsoft.com/kb/310560)

Looks like it's being used as part of the upgrade path from the old gpmdp to the new electron version: Google-Play-Music-Desktop-Player-UNOFFICIAL-/src/old_win32.js

My question here as per #543 , although it looks like you have done more research and actually found root cause (claps @jostrander )
Is this actually an issue?  The code (and the call to wmic is required in order to uninstall the old version.  I might be able to limit the number of calls a bit more (E.g. call once and never again)  but the call still needs to be there.

I would assume something in win10 is notifying users of this otherwise there wouldn't be bug reports, because I highly doubt two windows users just happened to be poking around reliability monitor. I'm not sure if it's actually causing any direct side effects, but to limit future reports maybe add a flag to make sure it's only ran once?


Test pipeline support for off heap disabled testing

With Off-Heap Technology for Large Arrays in GC (top-level off-heap issue), we need to add the test pipeline support for off-heap enabled/disabled testing. The two proposed options to enable(will be default)/disable off-heap are -XXgc:enableVirtualLargeObjectHeap / -XXgc:disableVirtualLargeObjectHeap (should be introduced in https://github.com/eclipse-openj9/openj9/pull/14667)
Off-heap is enabled by default, which means off-heap enabled will be tested in the existing test pipeline. We may add additional tests for off-heap enabled. We also need to cover off-heap disabled testing. The focus is JDK21 for now. And off-heap disabled testing should be run for a relatively short term (~6mths) until the feature is stable.
Proposal:
Create the offheap_disabled test builds similar to fips test builds (i.e., Test_openjdk21_j9_sanity.openjdk_s390x_linux_offheap_disabled)
Details:

create TEST_FLAG=OFFHEAP_DISABLED in TKG. Ideally, all tests should run with off-heap disabled.
in the build pipeline, trigger the test build with offheap_disabled suffix and TEST_FLAG=OFFHEAP_DISABLED
in the test pipeline, if TEST_FLAG=OFFHEAP_DISABLED, set EXTRA_OPTIONS=-XXgc:disableVirtualLargeObjectHeap for the test build

@AdamBrousseau this will double JDK21 testing. We may need to adjust the pipeline schedule so that it can run with the current farm capacity. Currently, the focus is JDK21. Ideally GC is the same for all Java versions, but we may need to run off-heap disabled testing with other Java versions.
FYI @dmitripivkine @pshipton @renfeiw

The testing should cover Balanced GC policy (option does nothing for other GC policies)

@amicic FYI

The testing should cover Balanced GC policy (option does nothing for other GC policies)

Thanks @dmitripivkine . In this case, we do not need a separate offheap_disabled pipeline. We can just add -XXgc:disableVirtualLargeObjectHeap in the tests that use the Balanced GC policy. I will update the description.


feat: get token through gh auth token if not provided by --token or GITHUB_TOKEN env var

Mergify-cli stack change-Id: I8c0abb0ba9bb162a7f46d09be4e396097ce9f24b:
Depends-On: #163

This pull request is part of a stack:

chore: rename git_push_stack to mergify_cli (#162)
fix: Make worktree compliant (#163)
feat: get token through gh auth token if not provided by --token or GITHUB_TOKEN env var (#164)
fix: convert coroutine to task (#165)
feat: Add control on draft or open status through a param (#166)
feat: Define trunk through param --trunk/-t or git config mergify-cli.stack-trunk (#167)
fix: Improve hook (#168)
chore: remove useless param --stack (#169)
docs: reademe (#170)


no valid server found at the url / selfsigned CA and Server-Certificates

Describe the bug

Rocketchat-server is running on raspberry pi
Created Selfsigned CA & Server Certificate
Imported CA-Certificate to Workstations and Smartphones to trust the server-certificates
Adding Server to Desktop-App with https://....  -> no valid server found at the url
Launching RocketChat via Browser: No problem
Launching RocketChat via App on Smartphones: no problem
Problem only occurs on (Windows) Desktop App

What operating system and which version?
Rocket.Chat (Server):  1.3.2 on Raspberry Pi
(Electron/Desktop): 3.5.7
Expected behavior
Beeing able to add server to windows desktop app

I believe server version is too old now.
Please try more recent version, at least 3.18.2 and tell if it helps.

raspberry pi bash:
snap list
rocketchat-server  1.3.2      1397   latest/stable
snap refresh
All snaps up to date.
Seems to be the latest version on raspberry pi

is there a way to import (selfsigned) ca-certificates into the desktop app? I just imported "system-wide" on windows

Please test if the problem persists with our latest release. I will be closing the issue but if there is any problem yet, we will open it again.


Add users directory synchronization for post-install and upgrade jobs

This adds the command line flags needed to leverage https://github.com/redpanda-data/redpanda-operator/pull/265 -- so it depends on that version of the operator being released. Copying that PR message here for context:

This adds a routine to merge users.txt into the bootstrap.yaml read when performing post install/update jobs. This is needed due to the bootstrap user additions added in this commit causing https://github.com/redpanda-data/helm-charts/issues/1566.
What is essentially occuring is this:

When creating a bootstrap user via the REDPANDA_BOOTSTRAP_USER environment variable said user isn't marked automatically as a superuser
If you want to use something like admin_api_require_auth then the only way you can actually run management operations on your installation is to pass in the above user in to the nodes' bootstrap.yaml so that the user is immediately marked as a superuser and all of the config-watcher scripts that manage other specified users can leverage it to create the rest of the users.
The bootstrap user changes that added the user to bootstrap.yaml uncovered that setting any sort of superusers values in the bootstrap.yaml is incompatible with the users created by a pre-existing users.txt secret. This is due to the superusers entry found in the bootstrap.yaml not containing them. When an upgrade finishes the jobs reset the configuration to only contain what is found in superusers without regard to anything managed by the pre-existing secret/config-watcher.
The above manifested in all users from a pre-existing secret getting unmarked as superusers when an upgrade occurred. Restarting any StatefulSet pod clears this up.

Since we must still set the bootstrap user in the superusers section of bootstrap.yaml in order for admin_api_require_auth to function correctly on installations, this makes the config synchronization code aware of our users.txt.
It needs to be coupled with a change in the helm-charts code to add in a --users-txt flag as needed and an additional secrets mount to mount the users.txt file into our job containers.

https://github.com/redpanda-data/helm-charts/actions/runs/11406017275/job/31739032071?pr=1567#step:5:617
    --- FAIL: TestTemplate/98-license-secret-values.yaml (0.61s)
        chart_template_test.go:131:
            	Error Trace:	/home/runner/work/helm-charts/helm-charts/pkg/testutil/testutil.go:131
            	            				/home/runner/work/helm-charts/helm-charts/pkg/testutil/testutil.go:209
            	            				/home/runner/work/helm-charts/helm-charts/charts/redpanda/chart_template_test.go:131
            	Error:
            	            	spec.template.spec.initContainers.bootstrap-yaml-envsubst.image  (apps/v1/StatefulSet/default/redpanda)
            	            	  ± value change
            	            	    - docker.redpanda.com/redpandadata/redpanda-operator:v2.2.4-24.2.5
            	            	    + docker.redpanda.com/redpandadata/redpanda-operator:v2.2.5-24.2.7

            	Test:       	TestTemplate/98-license-secret-values.yaml

Can you run
nix develop -c go test -update ./charts/redpanda/...


fast read option

too long don't read option

--tidy option to scan the whole lines

This feature has been pushed. Please see PR #4 for details.


fix(profiler): close connections/clients in shakesapp to fix memory leak

TESTED= enabled heap profiler and compared profiles w/ and w/o fix. Profiles w/ fix showed less memory being used.
Since this application is run only for short intervals in Profiler's How-To, I don't think we need to retake screenshots there.

@jqll -- Could you review this change?


Pasting from terminal window into entry box includes hidden formatting which breaks parsing

I was just trying to use https://mholt.github.io/json-to-go/ and found that it seemed to be unable to parse anything I was pasting into it, despite it being valid JSON.  Even more strange was that it seemed to be complaining about whitespace characters being present in the input (which should be fine), but then if I typed the same thing by hand (with the same whitespace) it liked it fine.  Even weirder was that if I started with a failing case, and edited out all of the whitespace, it liked it, but then if I manually inserted a space character, it broke again.
Eventually I noticed that when I cut-and-pasted JSON from a terminal window (on MacOS) into the web entry box, it was also copying over the formatting (coloring, text size, etc).  However, if I used Command-Option-Shift-V to paste just the text without any formatting, lo and behold it was able to parse it just fine.
I can only assume there's something about the rich-text entrybox being used and how it returns data to the underlying code that causes weird characters to be introduced only if there is formatting involved and there are whitespace characters present.
Perhaps this should be changed to use a non-rich-text entrybox of some sort, or to filter out whatever special characters the text box itself is introducing into things?

I noticed this too; probably some JS needed to prevent pasting of rich text (sigh): https://stackoverflow.com/questions/58980235/stop-pasting-html-style-in-a-contenteditable-div-only-paste-the-plain-text


Add integration tests for syscalls

The following syscalls need integration tests:

[ ] pipe
[ ] fcntl with F_GETFD, F_SETFD, F_GETFL, F_SETFL
[ ] ioctl with FIONBIO
[ ] getsockname
[ ] sendto
[ ] setsockopt
[ ] epoll_create1
[ ] epoll_ctl
[ ] epoll_wait
[ ] getpid

@haraldh I've folded these into our master list of syscalls that need testing: https://github.com/enarx/enarx-keepldr/issues/9. Is it okay to close this issue?


lettuce cluster performance is non-linear ?

my pc : 4core i7 8G
single redis one node , concurrent 200 ,total task 10001000, with ByteArrayCodec, QPS=45852
redis cluster four node, concurrent 200 ,total task 10001000, with ByteArrayCodec, QPS=30758
why cluster's performance is not so good as single redis?

Which commands did you use and which version? Can you post your test code?
You should not use more threads than you've got cores, since context switching will consume performance. Some 4 threads for 4 cores should be ok.
Connections to particular cluster nodes are opened on demand and impact your benchmark. You should make sure, that all connections are opened before you start measuring time.

single redis test
public static void main(String[] args) throws Exception {
	final RedisConnection<String, byte[]> sync1 = new RedisClient("<IP_ADDRESS>", 7000).connect(new BytesCodec());

// concurrent=200 ,total task 100*10000
StressTestUtils.testAndPrint(200, 100 * 10000, new StressTask() {
		@Override
		public Object doTask() throws Exception {
			byte[] bs = sync1.get("b");
			return null;
		}
	});
}

cluster test
public static void main(String[] args) throws InterruptedException {
	RedisClusterClient clusterClient = new RedisClusterClient(ImmutableList.of(RedisURI.Builder.redis("<IP_ADDRESS>", 7000).build()));
	final RedisClusterConnection< byte[], byte[]> connection = clusterClient.connectCluster(new ByteArrayCodec());
	String string = connection.clusterNodes();
	System.out.println(string);
	final AtomicLong i = new AtomicLong();
	final byte[] a = "a".getBytes();
	connection.get("a".getBytes());

// concurrent=200 ,total task 10010000
StressTestUtils.testAndPrint(200, 10010000, new StressTask() {
@Override
public Object doTask() throws Exception {
byte[] bytes = (""+i.incrementAndGet()).getBytes();
connection.get(bytes);
// async.
// System.out.println(set.get());
return null;
}
});
}

There are 3 reasons, why standalone is faster than cluster in the test you've measured:

The lookup of partitions within com.lambdaworks.redis.cluster.models.partitions.Partitions is in 3.0 unoptimized. Version 4.0 carries a better lookup (see comparison below)
The connection management and cluster determination carries a certain level of complexity. That code is not optimized and has some potential allow speed-up.
There is no warming in your test

In general, you should provide a percentile (95th) or a list of benchmark numbers. I executed the code below multiple times, and you can see the variance in the results.

I find   slotCache[] in Partitions.java now .

I would like to keep this ticket open.  I want to back port the improvement and improve internal connection management. There is about 30% of potential speed up.

I pushed a new branch (https://github.com/mp911de/lettuce/tree/improved-cluster-performance), you can give the code a try if you want. I noticed, that the Redis instances consume more CPU and the Java process itself less.


Multiple events ICS file

First of all, great work on khal.
I come up with an issue where I am trying to make khal read from a directory with multiple .ics files and each of these files has lots of events. In this situation, the name of each event is displayed with the last event of the file, but if I separate each event to a separate .ics file, everything works properly. Any idea why this happens?
Thanks.

Quoting @untitaker here on a similar issue for todoman:

Multiple events components in one file is a format that vdirsyncer can't support. In order to preserve interopability with other vdir implementations (such as vdirsyncer) [...]

There is a vdirsyncer backend that you can use to read from a single large file and spit that into individual files. That might help.

Thank you. Can you elaborate on which vdirsyncer backend you are talking about?

He means "singlefile". Do you need to have this format for import or storage?
On 30 January 2016 18:02:51 CET, Cedric Pimenta<EMAIL_ADDRESS>wrote:

Thank you. Can you elaborate on which vdirsyncer backend you are
talking about?

Reply to this email directly or view it on GitHub:
https://github.com/geier/khal/issues/329#issuecomment-177246059

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

https://vdirsyncer.readthedocs.org/en/stable/config.html#storage-singlefile

Right now, I created a script to go through my google calendars and build an .ics file with the events.
Now the only part missing is being able to that these .ics files and display them on khal. Maybe import is the way to go.

Right now, I created a script to go through my google calendars and build an .ics file with the events.

Doesn't google expose their events as CalDav anymore? Can't you use vdirsyncerfor that?

Now they have the whole OAuth authentication process so vdirsyncer no longer works for that sadly.
Thanks for the help, I think we can close :+1:

IIRC you can actually do read-only access without OAuth, see https://vdirsyncer.readthedocs.org/en/stable/supported.html#google
I certainly wouldn't rely on it. Your solution probably is more robust against Google's plans.


Added breadcrumb-file creation condition as attribute

I wasn't familiar with redis-sentinel so I was quite surprised that my configuration file was not updated by Chef. Please make this optional.

@chewi @shortdudey123 could you take a look to this PR. We have to use own fork due to such limitation.

@paulpalamarchuk can you rebase on master?
Otherwise i can carry this commit into another PR

@shortdudey123 rebased

small nit pick in spacing then will merge

@shortdudey123 done

thanks!

thank you all! 👍


Replace hyphens with n-dashes when joining two names.

E.g. 'L"owenheim-Skolem' becomes 'L"owenheim--Skolem'.
This comports with general typesetting practice in which names of distinct persons, date ranges etc. are joined by n-dashes rather than hyphens.
So e.g. 'Burali-Forti' remains as is, joined by a hyphen, since it is a single (albeit double-barrelled) name.

I have no objection, it's certainly logical. (Hart's agrees; Chicago doesn't FWIW). Do you know if that improves hyphenation? (I think LaTeX doesn't hyphenate hyphenated compounds at all, right?)


anytype: 0.42.3 -> 0.42.4

Includes minor fixes such as adding meta.mainProgram alongside removing an invalid move and set the pname properly.
Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Since you also included the update in this PR, I will close my update PR in #334893.

Sorry, I forgot to check if there are existing PRs. Thanks for being on the look out!


Team Acceptance Test

Given that I've finished up most of the things I'm working on, and have a decent amount of time, I thought I'd jump in and just write tests for a while. This may not be a PR for all of the needed acceptance tests, but hopefully I can get most of them done (except for the ones that are already taken/in progress)
refs #6039

Given that I've finished up most of the things I'm working on, and have a decent amount of time, I thought I'd jump in and just write tests for a while.
:heart_eyes:

@acburdine I was thinking of doing some of these tests too. I can take on some if you want or we can work together on this :)

@zinyando sounds good :smile: My plan at the moment is to do the team tests and finish up the setup tests. Let me know which ones you take on, or you could submit commits to this PR. Whatever works best.

@kevinansfield do you want me to put all the acceptance tests into one commit, or can I split them out into multiple commits? :wink:

@acburdine as discussed on the sprint call, splitting each test into a separate PR would be good as we can merge them in as they are ready.
:+1: on the team test, my only concern is the testing of all the validation text. Acceptance tests should only be a high-level overview, making sure the various areas are rendered and testing any core interaction with the API. We really want to push the testing of any detail down into integration and unit tests - there's a lot that needs to be filled out in that respect but they should eventually be what we rely on most, hopefully as our use of components increases it becomes easier to test at that instead of attempting to force everything into acceptance tests like we did in casper.

@kevinansfield fixed and rebased :smile: If the 404 PR gets merged before this I can add the 404 tests to /team/ as well.

@kevinansfield Added the 404 testing to teams and it seems to work :+1:

:+1: LGTM :smile:


Error message when using type = string instead of type = text is obscure

Hello, very happy Atlas user here :)
Our devs often mistakenly write the type of a given column in HCL. For example:
column "name" {
    null = true
    type = string
}

instead of:
column "name" {
    null = true
    type = text
}

This results in the following error message when running: atlas migrate diff
Error: schemahcl: failed reading spec as *postgres.doc: set field "type": converting cty.Value to *schemahcl.Type: incorrect type type

Given our project contains dozens of HCL files, it's very hard to understand where exactly the issue is based on the error message. Could you include the file and perhaps line from which the error originated in the error message?

Hey @torchsec-roni 👋
What editor do you use? Did you install the Atlas plugin there? You should get an error if you typed a non-valid value.
Also, what Atlas version do you use? Run atlas version.

Re editor: We are using vscode with the Atlas HCL extension

Re atlas version:
➜  /tmp atlas version
atlas version v0.27.1-fe19f37-canary
https://github.com/ariga/atlas/releases/latest

OK, let me check that and I'll update.

I just tried that and I got:

Can you try updating Atlas to the latest version and recheck this?

Interesting, when I cut down our Atlas configuration to only a single file, I get the same error as you:
➜  db git:(main) ✗ atlas migrate --env local diff
Error: failed parsing postgres schema files: schema/users.hcl
schema/users.hcl:37,5-18: cannot use hcl type as column type in assignment

But when I use the full configuration, I get this:
➜  db git:(main) ✗ atlas migrate --env local diff
Error: schemahcl: failed reading spec as *postgres.doc: set field "type": converting cty.Value to *schemahcl.Type: incorrect type type

Not sure at this point how to reproduce w a minimal config. Will look into it

Found the minimal reproduce:
// state.hcl
enum "state" {
  schema = schema.public
  values = ["NOT_STARTED", "IN_PROGRESS", "COMPLETED", "FAILED"]
}

// users.hcl
schema "public" {
  comment = "standard public schema"
}
table "users" {
  schema = schema.public
  column "name" {
    null = false
    type = string
  }
}

It appears that adding enums to the resolution logic screw things up somehow

I just tried this, and got the same error (like I pasted above):

Two questions:

Can you paste the full atlas.hcl configuration you use?
How do you install Atlas? The error you pasted above comes from the OSS (non-complete) version of Atlas.

Also, note that if you suffix your files with pg.hcl, the editor plugin will detect this file as PostgreSQL schema files.

Thanks for the tip on the file extensions! We actually solved this by just pushing out a shared settings.json but the file extension trick is much better.
Sorry for the bad reproduce tips. I set up the following repo with a full working reproduce of the issue: https://github.com/torchsec-roni/public-playground/tree/ab36817931ce3818d9971221dc4a3de0dcda93c6/atlasgo.
It appears that there is an issue with the file resolution order. Specifically, changing an empty file name from a.hcl to z.hcl reproduces this issue.

Thanks for spending the time to report this and compile an example. We identified the issue and fixed it in the latest build.
You're welcome to ping me on our Discord, and we'll send you a swag package <3


Enable compression for sketches now that the backend supports it

What does this PR do?
Turns on compression for sketch payloads sent by the agent.
Motivation
We're adding support for compression in the whole sketch pipeline and can save bandwidth from the agent as well.

Should this be mentioned in the changelog ? 🤔

Should this be mentioned in the changelog ? 🤔

I defer to the agent team on this; it mostly brings sketches in line with other submission types which were already being compressed by the agent (assuming either zlib or zstd are available). On the other hand sketch payloads are larger, so there's both the chance for a bit of CPU usage to compress them, as well as less bandwidth used sending them and improved latency on sending them.

Added a release note.


Azure devops integration

Hello,
thanks for this project. Is it foreseen to integration with "service connections" in Azure devops?
Our pipelines use kubectl tasks: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/kubernetes?view=azure-devops . These tasks use a "service connection". It would be nice if kubelogin becomes one of the "service connection" choices for kubernetes.

@TomGeske @palma21 WDYT?

@azooinmyluggage ^

@aelmanaa Could you please explain the scenario where you want to use kubelogin as a service connection? I think the three ways of creating a kubernetes service connection in Azure Devops already cover all types of clusters.

@shigupt202  we arleady use "service account option". however this option has the following disadvantages:

during the creation of a cluster, one has to foresee to manually create the service account and the role bindings (manual step)

service accounts are outside of AAD.  generally within an organization, we prefer to centrally manage an identity within AAD. There are already "service principals" or "managed identities" in AAD ; hence better to use these ones

Moreover, it seems that kubelogin was created as a response to this feature request https://feedback.azure.com/forums/914020-azure-kubernetes-service-aks/suggestions/35146387-support-non-interactive-login-for-aad-integrated-c?tracking_code=e9effc5721de49e6c717ca62967a20d2
hence it would be nice if in AZure devops, we could directly use a service principal or a managed identity as service connection. Behind the scenes, Azure devops uses kubelogin to authenticate (non-interactively) with the cluster

@aelmanaa Thanks for the feedback! I'll get in touch with the PM's to discuss this feature request.

Ack. We will work on adding this support post Ignite.
Adding @anraghun to take this forward.

thanks  @azooinmyluggage

In case someone else runs into this problem, this is how I ended up doing it.
It works but I agree with @aelmanaa and would prefer if there were a built-in ADO Task to handle this.
Summary

my AKS cluster setup

AAD integrated
local accounts are disabled
apps are namespaced

YAML

created separate pipeline template for kubelogin install and login
stored service principal credentials in Key Vault, which is integrated into ADO via Variable Group. These secrets must be injected into environment at step level. (You can just use regular variable group secrets if you want).

kubelogin

I used -l azurecli to avoid having to continue to set AAD_SERVICE_PRINCIPAL_CLIENT_ID for every subsequent step
kubeconfig is created in setup setup step. subsequent steps just need to know where it is.

sub-template just for kubelogin
For more about templates, see https://docs.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops
Note you will also need to set the following variables:
variables:
  appNamespace:            my-namespace
  aksClusterName:          my-aks-cluster
  aksClusterResourceGroup: cluster-rg-not-the-nodes-one

which are used below
# templates/kubelogin-setup.yaml
steps:
- bash: |

    # Download and install
    curl -LO "https://github.com/Azure/kubelogin/releases/download/$(kubeloginVersion)/kubelogin-linux-amd64.zip"
    sudo unzip -j "kubelogin-linux-amd64.zip" -d /usr/local/bin
    rm -f "kubelogin-linux-amd64.zip"
    kubelogin --version
  displayName: kubelogin - install

  # For details, see https://stackoverflow.com/questions/54004007/azure-devop-pipelines-authentication-to-aks-with-azure-ad-rbac-configured
- bash: |

    # First login to Azure
    az login \
      --service-principal \
      --username $AAD_SERVICE_PRINCIPAL_CLIENT_ID \
      --password $AAD_SERVICE_PRINCIPAL_CLIENT_SECRET \
      --tenant $AZ_TENANT_ID

    # Fails if it doesn't exist
    touch .kubeconfig-$(aksClusterName)
    chmod 600 .kubeconfig-$(aksClusterName)

    # Populate kubeconfig
    az aks get-credentials \
      --resource-group $(aksClusterResourceGroup) \
      --name $(aksClusterName) \
      --overwrite-existing \
      --file .kubeconfig-$(aksClusterName)

    # Pass kubeconfig to kubelogin to access k8s API
    kubelogin convert-kubeconfig -l azurecli

    # confirm it works
    kubectl get pods --namespace $(appNamespace)
  displayName: kubelogin - aks-credentials to kubecontext
  env:
    AZ_TENANT_ID:                        $(tenant-id)
    AAD_SERVICE_PRINCIPAL_CLIENT_ID:     $(aks-architect-ci-dev-sp-client-id)
    AAD_SERVICE_PRINCIPAL_CLIENT_SECRET: $(aks-architect-ci-dev-sp-client-secret)
    KUBECONFIG:                          $(Build.SourcesDirectory)/.kubeconfig-$(aksClusterName)

Pipeline excerpt
# …
jobs:
- job: Deploy
  displayName: Deploy
  variables:
  - group: mask-subscription-ids # hack to mask my subscription ids
  - group: aks-credentials-kv-group # the service principal credentials

  steps:
  - template: ../steps/setup-kubelogin.yaml # what we created above

  - bash: |

      # confirm it works
      kubectl get pods --namespace $(appNamespace)

    displayName: "kubectl apply"
    env:
      KUBECONFIG: $(Build.SourcesDirectory)/.kubeconfig-$(aksClusterName)

  - bash: |
      kubelogin remove-tokens
    displayName: kubelogin - clear cache

In this way I only need to repeat the env: KUBECONFIG… part for subsequent steps.
Special thanks to https://stackoverflow.com/questions/54004007/azure-devop-pipelines-authentication-to-aks-with-azure-ad-rbac-configured

@aelmanaa Could you please explain the scenario where you want to use kubelogin as a service connection? I think the three ways of creating a kubernetes service connection in Azure Devops already cover all types of clusters.

Use case - I disabled local accounts per https://docs.microsoft.com/en-us/azure/aks/managed-aad#disable-local-accounts-preview

To be honest this doesn't need a new service connection, there is already an Azure Resource Manager service connection that gets you most of the way, just need to configure kubelogin correctly.
As things stand right now I have no idea what the use case is for the Azure DevOps KubernetesV1 task configured with Azure Resource Manager connection type and useClusterAdmin set to false (the default). I don't understand how the Azure Resource Manager examples in the documentation could possibly work.

@julie-ng Thank you for the very useful write-up.
When I use an ARM Service Connection with MSI on an agent with kubelogin preinstalled the following just works:
      - task: AzureCLI@2
        displayName: aks non-admin login using kubelogin
        inputs:
          azureSubscription: ${{parameters.serviceConnection}}
          scriptType: bash
          scriptLocation: inlineScript
          inlineScript: |
            az aks get-credentials -n <aks cluster> -g <resource group>
            kubelogin convert-kubeconfig -l azurecli

      - task: AzureCLI@2
        displayName: test kubectl and helm from bash
        continueOnError: true
        inputs:
          azureSubscription: ${{parameters.serviceConnection}}
          scriptType: bash
          scriptLocation: inlineScript
          inlineScript: |
            kubectl get pods -n <some namespace>
            helm list --namespace <some namespace>

However, if I try to use the HelmDeploy@0 and Kubernetes@1 tasks that effectively do the same as second task with kubectl and helm commands, it does not work and I get (for the helm case):
2022-01-19T20:33:52.7909008Z ##[debug]set helmExitCode=1
2022-01-19T20:33:52.7910461Z Error: Kubernetes cluster unreachable: Get "https://<aks cluster>-a15872ed.hcp.westeurope.azmk8s.io:443/version?timeout=32s": getting credentials: exec: executable kubelogin failed with exit code 1
2022-01-19T20:33:52.7912489Z helm.go:88: [debug] Get "https://<aks cluster>-a15872ed.hcp.westeurope.azmk8s.io:443/version?timeout=32s": getting credentials: exec: executable kubelogin failed with exit code 1
2022-01-19T20:33:52.7915456Z ##[debug]Processed: ##vso[task.setvariable variable=helmExitCode;issecret=false;]1
2022-01-19T20:33:52.7925848Z ##[debug]publishPipelineMetadata=true
2022-01-19T20:33:52.7926838Z Kubernetes cluster unreachable
2022-01-19T20:33:52.7930444Z ##[debug]execResult: {"code":1,"stdout":"","stderr":"Error: failed to get token: expected an empty error but received: Azure CLI Credential: ERROR: Please run 'az login' to setup account.\n\nError: Kubernetes cluster unreachable: Get \"https://<aks cluster>-a15872ed.hcp.westeurope.azmk8s.io:***@v1.2.1/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.2.1/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.2.1/command.go:902\nmain.main\n\thelm.sh/helm/v3/cmd/helm/helm.go:87\nruntime.main\n\truntime/proc.go:225\nruntime.goexit\n\truntime/asm_amd64.s:1371\n"}
2022-01-19T20:33:52.7933131Z helm.sh/helm/v3/pkg/kube.(*Client).IsReachable
2022-01-19T20:33:52.7933638Z ##[debug]task result: Failed
2022-01-19T20:33:52.7934668Z 	helm.sh/helm/v3/pkg/kube/client.go:121
2022-01-19T20:33:52.7969750Z ##[error]Error: failed to get token: expected an empty error but received: Azure CLI Credential: ERROR: Please run 'az login' to setup account.

      - task: HelmDeploy@0
        displayName: test HelmDeploy task
        continueOnError: true
        inputs:
          connectionType: None
          namespace: <some namespace>
          command: list

      - task: Kubernetes@1
        displayName: test Kubernetes task
        continueOnError: true
        inputs:
          connectionType: None
          command: get
          arguments: pods -n <some namespace>

I created issue microsoft/azure-pipelines-tasks#15802 but I am not sure if they can solve this.
How does kubelogin retrieve the MSI credentials when using azurecli as login mode? Does it need access to certain environment variables or a token file that can explain the difference between the two scenario's?

@nlighten connectionType: None does not appear to be a valid connection type in the documentation https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/kubernetes?view=azure-devops
I dived into the code a few months back and the Kubernetes and Helm tasks appear to always create their own custom kubeconfig file based upon the defined service connection https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/KubernetesV1/src/clusterconnection.ts#L71
Browsing through the code again, what might work would be taking the kubeconfig that's generated by kubelogin and using that to create a Kubeconfig based service connection https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#kubeconfig-option with the caveat that you'll need to update the service connection any time you rotate the cluster certificates (can probably be automated if you're creating the service connections with terraform).

@sharebear I agree that None is not a documented connection type but it actually works for both tasks. The kubeconfig file is not created in that case (see https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/HelmDeployV0/src/helm.ts#L41) and a pre-existing kubeconfig file is used.
So if I do an kubectl aks get-credentials with --admin (no kubelogin) using connectionType: None works just fine. Additionally from the logs I can see that it actually uses the kubeconfig created by kubelogin convert-kubeconfig -l azurecli because it tries to use kubelogin for the authentication. It only fails on the authentication step. The puzzle is: why does it fail when helm/kubectl is invoked from the Task but not from the command line.

@nlighten

I agree that None is not a documented connection type but it actually works for both tasks. The kubeconfig file is not created in that case (see https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/HelmDeployV0/src/helm.ts#L41) and a pre-existing kubeconfig file is used.

TIL ... althought on a quick skim I don't see similar code in the Kubernetes task, the code isn't that easy to follow.

Additionally from the logs I can see that it actually uses the kubeconfig created by kubelogin convert-kubeconfig -l azurecli because it tries to use kubelogin for the authentication. It only fails on the authentication step.

In that case, what's your error message? I had a problem where I had forgotten to actually enable AzureRBAC on the AKS cluster, and it's not very visible in the portal, so might be worth verifying that.

@sharebear Thank you for thinking with me.

TIL ... althought on a quick skim I don't see similar code in the Kubernetes task, the code isn't that easy to follow.

In the Kubernetes task the switch is here: https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/KubernetesV1/src/clusterconnection.ts#L59

In that case, what's your error message?

The error I get is:
2022-01-19T20:33:52.7910461Z Error: Kubernetes cluster unreachable: Get "https://<aks cluster>-a15872ed.hcp.westeurope.azmk8s.io:443/version?timeout=32s": getting credentials: exec: executable kubelogin failed with exit code 1

So kubelogin apparently fails with exit code 1. Full task logs can be found at: https://github.com/microsoft/azure-pipelines-tasks/files/7900774/logs.zip
RBAC and AKS-managed AAD is definitely enabled on the cluster. In the pipeline steps listed above I can successfully use the managed identity credentials when using helm and kubectl directly from bash. As far as I know the DevOps Tasks are only a wrapper around the same binaries, but somehow when helm or kubectl are called from the Tasks kubelogin fails. My impression is that cannot get a hold of the managed identity credentials but logging is too limited to pinpoint.
Perhaps the helm and kubernetes tasks do not have access to the ~/.azure/ directory? I will do some testing to see if this directory is removed after the AzureCLI@2 is finished (which would make sense from a security perspective).

It was indeed caused by AzureCLI@2 doing az account clear at the end. If I use plain bash and an explicit login/logout everything works as expected in combination with a MSI:
      - task: Bash@3
        displayName: az and aks login
        inputs:
          targetType: 'inline'
          script: |
            az login --identity
            az aks get-credentials -n <aks cluster> -g <resource group>
            kubelogin convert-kubeconfig -l azurecli

      - task: HelmDeploy@0
        displayName: test HelmDeploy task
        inputs:
          connectionType: None
          namespace: <namespace>
          command: list

      - task: Kubernetes@1
        displayName: test Kubernetes task
        inputs:
          connectionType: None
          command: get
          arguments: pods -n <namespace>

      - task: Bash@3
        condition: always()
        displayName: az and aks logout
        inputs:
          targetType: 'inline'
          script: |
            az account clear
            rm -rf ~/.kube

Still, I would prefer explicit support for kubelogin as part an ARM Service Connection on the Kubernetes and HelmDeploy tasks.
@sharebear  Thanks for your time.

@nlighten Sorry I stopped replying there, got caught up in the day-to-day work.
Great that you were able to work it out, it's helped me develop my understanding too... I went with a "service account" based solution when I disabled the local admin account, but still keeping my eyes open for better solutions.
I do wonder though, as you are using msi, what's the motivation for using azurecli auth in kubelogin instead of using the msi support there? If you just ran kubelogin convert-kubeconfig -l msi would that also have solved the problem?

@sharebear I actually missed the -l msi option (rtfm). It works and results in a slightly more elegant:
      - task: AzureCLI@2
        displayName: aks non-admin login using kubelogin
        inputs:
          azureSubscription: ${{parameters.serviceConnection}}
          scriptType: bash
          scriptLocation: inlineScript
          inlineScript: |
            az aks get-credentials -n <aks cluster> -g <resource group>
            kubelogin convert-kubeconfig -l msi

      - task: HelmDeploy@0
        displayName: test HelmDeploy task
        inputs:
          connectionType: None
          namespace: <namespace>
          command: list

      - task: Kubernetes@1
        displayName: test Kubernetes task
        inputs:
          connectionType: None
          command: get
          arguments: pods -n <namespace>

@nlighten Great... now, as far as I understand it, the contents of the kubeconfig after you've run kubelogin convert-kubeconfig -l msi are pretty static and will only change if/when you rotate the cluster certificates.
Does it then make sense to move it into a service connection with kubeconfig so you don't need to generate it every pipeline run?

@sharebear

Does it then make sense to move it into a service connection with kubeconfig so you don't need to generate it every pipeline run?

I would prefer not. We rotate cluster certificates every 3 months and also rebuild clusters quite frequent. With moving it into the Kubernetes Service  Connection I would still need to update the Service Connection after each certificate rotation or rebuild.
In my mind something like kubelogin: msi option as part of the HelmDeploy and Kubernetes Task when using connectionType: Azure Resource Manager would be a better option.

@sharebear

Does it then make sense to move it into a service connection with kubeconfig so you don't need to generate it every pipeline run?

I would prefer not. We rotate cluster certificates every 3 months and also rebuild clusters quite frequent. With moving it into the Kubernetes Service Connection I would still need to update the Service Connection after each certificate rotation or rebuild.

Fair enough. We manage the service connections through terraform https://registry.terraform.io/providers/microsoft/azuredevops/latest/docs/resources/serviceendpoint_kubernetes so I think I could automate this.

In my mind something like kubelogin: msi option as part of the HelmDeploy and Kubernetes Task when using connectionType: Azure Resource Manager would be a better option.

I completely agree on this point, what we're discussing is still a workaround for something that should work out of the box. Especially when it's recommended to disable local accounts which the current implementations appear to depend upon being enabled.
Thanks for the good discussion!

@sharebear @nlighten have you considered using gitops? Because you're walking a slippery path and depending on your requirements, from a security perspective, it would be more straight-forward with less gaps to just have the cluster pull in the changes.
That being said, let me explain what concerns me about this discussion. Keep in mind I work for MSFT but am not part of the AKS product group.
kubelogin
In the future, you must use kubelogin to align with k8s upstream. Old methods will be deprecated soon starting with v1.23

re using msi, what's the motivation for using azurecli auth in kubelogin instead of using the msi support there? If you just ran kubelogin convert-kubeconfig -l msi

See AKS public roadmap item here

Azure/AKS#2728

TL;DR; is that it has to do with upstream Kubernetes moving cloud provider specific code out of core k8s announced in 2019. So simple azure cli will not work anymore. You will need an explicit kubeconfig.
Identity…AKS != Kubernetes
Well not exactly. AKS is an offering where Azure manages your Kubernetes control plane (some of which you do not see in your Azure account). The Kubernetes components that host your workload, e.g. VMs, Load Balancers, etc. are Kubernetes (not Azure). Azure will provide interfaces that make interaction easier, including identity integration.
Managed Identity is an Azure offering and designed for accessing Azure resources. See MSFT Blog - Demystifying Service Principals – Managed Identities.
Technically you're trying to access a Kubernetes resource, not an Azure resource. Because they both speak and understand OIDC/OAuth, it works.

Perhaps the helm and kubernetes tasks do not have access to the ~/.azure/ directory? I will do some testing to see if this directory is removed after the AzureCLI@2 is finished (which would make sense from a security perspective).

K8s and Helm are not Azure and should not have access to ~/.azure. Instead Azure must speak k8s and use kubeconfig. See roadmap link above.
Managed Identities - the slippery part
IMO Managed identities are often misunderstood and people think it's magic without understanding what's happening under the hood - and thus how Shared Responsibility Principle still applies. Keep this in mind

Managed Identities are still service principals under the hood. It saves you from managing secrets, but secrets still exist, in the form of tokens - although short lived and frequently rotated, they exist and are maybe "lying around" somewhere.
The pipeline tasks - and my KUBECONFIG environment variable set to $(Build.SourcesDirectory)/.kubeconfig-$(aksClusterName) are designed to ensure these files don't persist beyond their lifecycles

So… @sharebear

Does it then make sense to move it into a service connection with kubeconfig so you don't need to generate it every pipeline run? Or is kubelogin only installed when you run get-credentials?

No, please do not move the kubeconfig into a service connection because it shouldn't exist beyond lifecycles, e.g. pipeline runs. Even though OAuth tokens can be long lived, that's a slippery slope. IMO, in the spirit of OAuth, build agents should re-authenticate and fetch a new access token every time they start a new job.
Service Connections - are just wrappers
They're just wrappers around credentials that allow you to apply some authorization mechanisms, e.g. which pipeline can use this? Deployments should only happen during business hours, etc.
@nlighten

In my mind something like kubelogin: msi option as part of the HelmDeploy and Kubernetes Task when using connectionType: Azure Resource Manager would be a better option.

That's an interesting idea. I'd be curious to know why you use the tasks and not just the bash? Personally I find the tasks (except ARM deployment) limiting and abstractions not useful. If I'm using Terraform, kubectl, etc. I prefer scripting. Also lets me test/code locally and thus make pipelines faster.

@sharebear @nlighten have you considered using gitops? Because you're walking a slippery path and depending on your requirements, from a security perspective, it would be more straight-forward with less gaps to just have the cluster pull in the changes.

I've inherited a shared pipeline that is used by ~60 developers practicing continuous deployment, it's been stable and working well for them for a long while. While I do want to change this, as there's one big DX feature I can't implement on DevOps Pipelines, I've got lower hanging fruit to pick first. Such as removing local accounts from the clusters.
 How do you deploy the GitOps tooling? Sure AKS now provides Flux 2 as a plugin, but that's only preview stage and there are multiple stronger competitors.

That being said, let me explain what concerns me about this discussion. Keep in mind I work for MSFT but am not part of the AKS product group.
kubelogin
In the future, you must use kubelogin to align with k8s upstream. Old methods will be deprecated soon starting with v1.23

Exactly, this strengthens the argument that it should be supported out of the box by the DevOps Pipeline tasks provided for interacting with k8s.

re using msi, what's the motivation for using azurecli auth in kubelogin instead of using the msi support there? If you just ran kubelogin convert-kubeconfig -l msi

See AKS public roadmap item here

az login deprecation in kubelogin AKS#2728

TL;DR; is that it has to do with upstream Kubernetes moving cloud provider specific code out of core k8s announced in 2019. So simple azure cli will not work anymore. You will need an explicit kubeconfig.

Correct, an this is exactly what we are supporting with the proposed config here.

Identity…AKS != Kubernetes
Well not exactly. AKS is an offering where Azure manages your Kubernetes control plane (some of which you do not see in your Azure account). The Kubernetes components that host your workload, e.g. VMs, Load Balancers, etc. are Kubernetes (not Azure). Azure will provide interfaces that make interaction easier, including identity integration.
Managed Identity is an Azure offering and designed for accessing Azure resources. See MSFT Blog - Demystifying Service Principals – Managed Identities.
Technically you're trying to access a Kubernetes resource, not an Azure resource. Because they both speak and understand OIDC/OAuth, it works.

I really don't see how this is any different to using managed identities for data plane access for Azure Storage, Azure SQL, Azure Cache for Redis, etc. With the AzureRBAC support we can manage all access with one tool instead of constantly wondering whether access is granted in AzureRBAC or K8sRBAC, with potential confusion being a security risk.

Perhaps the helm and kubernetes tasks do not have access to the ~/.azure/ directory? I will do some testing to see if this directory is removed after the AzureCLI@2 is finished (which would make sense from a security perspective).

K8s and Helm are not Azure and should not have access to ~/.azure. Instead Azure must speak k8s and use kubeconfig. See roadmap link above.
Managed Identities - the slippery part
IMO Managed identities are often misunderstood and people think it's magic without understanding what's happening under the hood - and thus how Shared Responsibility Principle still applies. Keep this in mind

Managed Identities are still service principals under the hood. It saves you from managing secrets, but secrets still exist, in the form of tokens - although short lived and frequently rotated, they exist and are maybe "lying around" somewhere.
The pipeline tasks - and my KUBECONFIG environment variable set to $(Build.SourcesDirectory)/.kubeconfig-$(aksClusterName) are designed to ensure these files don't persist beyond their lifecycles

So… @sharebear

Does it then make sense to move it into a service connection with kubeconfig so you don't need to generate it every pipeline run? Or is kubelogin only installed when you run get-credentials?

No, please do not move the kubeconfig into a service connection because it shouldn't exist beyond lifecycles, e.g. pipeline runs. Even though OAuth tokens can be long lived, that's a slippery slope. IMO, in the spirit of OAuth, build agents should re-authenticate and fetch a new access token every time they start a new job.

Here I think you've misunderstood the contents of the kubeconfig file. There are no tokens stored in the file, just the public cluster root CA certificate and the necessary yaml to tell kubectl which arguments to pass. Nothing secret or sensitive here at all.

Service Connections - are just wrappers
They're just wrappers around credentials that allow you to apply some authorization mechanisms, e.g. which pipeline can use this? Deployments should only happen during business hours, etc.
@nlighten

In my mind something like kubelogin: msi option as part of the HelmDeploy and Kubernetes Task when using connectionType: Azure Resource Manager would be a better option.

That's an interesting idea. I'd be curious to know why you use the tasks and not just the bash? Personally I find the tasks (except ARM deployment) limiting and abstractions not useful. If I'm using Terraform, kubectl, etc. I prefer scripting. Also lets me test/code locally and thus make pipelines faster.

I can't anser for @nlighten but my answer would be;

It feels more idiomatic to use the provided tasks. All MS provided documentation I've seen points to this as being the recommended way to do things. This goes a long way to transferring skils from one workplace to another, these tasks work the same everywhere, you don't need to dig into the details of someone's home-grown bash script.
By using bash you lose all the advantages you just mentioned of providing additional controls per service connection.
While the current discussion has focussed on MSI, this requires self-hosted runners. Not everyone uses them, a solution that properly used the AzureRM service connection would allow usage from MS hosted runners too.

@julie-ng Thank you for sharing your thoughts. To add to @sharebear feedback:

have you considered using gitops?
Yes, but AFAIK this currently only covers K8S resources and not yet Azure resources (not considering Azure Service Operator  which does not seem to be production ready yet).

Perhaps to explain a little bit the background of my experiments with kubelogin: we are looking at providing individual subscriptions to product teams. We are considering using a dedicated agent pool per subscription for deployment (no build). This pool would have an MSI associated with it that would allow the the team to deploy to resources within that subscription. AKS is one of the targets, but they (for example) could also do database migrations (CosmosDB/SQL Server), access keyvault etc, all using the same MSI.

K8s and Helm are not Azure and should not have access to ~/.azure. Instead Azure must speak k8s and use kubeconfig. See roadmap link above.

Agreed. This was my mind trying to understand how this worked and doing it wrong ...

I'd be curious to know why you use the tasks and not just the bash?

Personally I prefer the command line, but there is a large developer community and existing pipelines that already use the Tasks and this is also how Microsoft shows them how to do it in the documentation.

In the future, you must use kubelogin to align with k8s upstream. Old methods will be deprecated soon starting with v1.23

Exactly, this strengthens the argument that it should be supported out of the box by the DevOps Pipeline tasks provided for interacting with k8s.

I agree with @sharebear . Perhaps we should open a feature request in the azure-pipelines-tasks project with this as motivation.

@sharebear - being pragmatic about your options is practical. I think that way too. But on this part I disagree:

Here I think you've misunderstood the contents of the kubeconfig file. There are no tokens stored in the file, just the public cluster root CA certificate and the necessary yaml to tell kubectl which arguments to pass to kubelogin. Nothing secret or sensitive here at all.

Are you sure? Check your ~/.kube/config and see also https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens
I haven't used all the different methods, for example certs for authentication. But my understanding and my personal kubeconfig is full of secrets.
Re: pipeline tasks - yep all the docs use it and the vast majority of my colleagues use them. I find more verbose than just using the command line. And often they're not stable or documentation of the tasks themselves are weird, i.e. break. Pipelines with bash are cleaner. I tend to hide them in tasks in sub templates. Personal preference. We'll probably just agree to disagree here… :)
@nlighten re: MSI

Subscription per product team is reasonable as it follows MSFT CAF Subscription Decision Guide
be careful with managed identities and build agents - because it becomes easy to mix up context of lifecycle, i.e. build jobs has its own lifecycle and VM has its own lifecycle. If build job overwrites the ~/.kube/config, well hello security challenge for future jobs. That's why the task and I create and send an explicit kubeconfig, where the lifecycle matches that of the build job. Maybe you're designing your node pools now to be product team specific - but when your company does an re-org, you'll need to update your tech concept.

I used to be an engineer and enterprise architect at Allianz Germany, a multibillion dollar company. Back in ~2019 (not sure what they use now), our team designed and created a Jenkins infrastructure that leveraged k8s pods for build agents.
Using a VM with a longer lifecycle? Be very careful. Housekeeping is not easy. And you then need to understand every toolchain you have on that machine and how it works, including kubectl
Anyhow, I'll probably sign off this thread now as I could rant about this forever. I don't expect or even want you to agree with me. But I'd love for everyone to dig deeper and understand how things work under the hood - esp. for people who have any architecture responsibility.
I'll probably make a YouTube video about this soon. Same discussion came up at work this week.
It's been a fun discussion :)

Are you sure? Check your ~/.kube/config and see also https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens

@sharebear is correct. The only thing that is in the kubeconfig is the cluster public cert and the config to delegate the token retrieval to kubelogin. For an MSI this looks like:
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ******
    server: https://mycluster-a15872ed.hcp.westeurope.azmk8s.io:443
  name: mycluster
contexts:
- context:
    cluster: mycluster
    user: clusterUser_myresourcegroup_mycluster
  name: mycluster
current-context: mycluster
kind: Config
preferences: {}
users:
- name: clusterUser_myresourcegroup_mycluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - get-token
      - --server-id
      - 6dae42f8-****-****-****-3960e28e3630
      - --login
      - msi
      command: kubelogin
      env: null
      provideClusterInfo: false

be careful with managed identities and build agents

Agreed. That is we are considering dedicated 'deployment' agents that cannot be used for builds. Deployment agents are restricted to a single team/subscription. It also gives us more control over the whole 'application supply chain'.

our team designed and created a Jenkins infrastructure that leveraged k8s pods for build agents

That is nowadays fairly simple to setup using AKS + KEDA which has an Azure Pipelines scaler. If you combine it with Pod Identity you can create very fine grained agents pools each with their own MSI (if you want). But in our scenario, an AKS cluster per subscription just for deployment agents is a little bit overkill.

 I don't expect or even want you to agree with me. But I'd love for everyone to dig deeper and understand how things work under the hood - esp. for people who have any architecture responsibility.

You know what happened when the Dwarves dug too greedily and too deep ;-). Good luck with your YouTube video.

@nlighten cool, thanks for sharing the example config. I've never used kubectl with msi.  Not really. You'd need a team to maintain it. A large insurance company has those resources, i.e. dedicated teams with dedicated people for maintaining patches, people's feature/software requests, etc. If not, it's overrated in my opinion.
It's a fun ops problem if you're just into ops. If you're like me and prioritize creating business value, I'd question whether if there's another solution that's simpler and just as secure.
Btw, because no one mentioned it… most people don't realize if you use managed identities, you have a ginormous single point of failure…AAD 😬 unlikely, but in my current non pre-sales role, I can tell customers that sorry, it's happened. So make an informed decision.

Adding @vijayma

I've been able to build out a proof of concept to this problem of the ADO kubernetes tasks not supporting kubelogin.  Going to put it here and describe how it works.   It's based on info gathered from earlier in this discussion.
Prereqs:

I'm using an AKS cluster with AAD integration and I’ve disabled local accounts.  This is per the best practice guidelines for enterprise aks deployments.
I've created a service principal and assigned the cluster role Azure Kubernetes Service RBAC Writer to it.
In Azure DevOps, I've created a Service Connection to hold the service principal objectid & key.  (In my example it is called myakstestsp)

And here's the pipeline.  I'll describe the components of it below.
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

#trigger:
#- master

pool:
  vmImage: ubuntu-latest

variables:
  azureResourceGroup: myrg
  kubernetesCluster: myakscluster
  useClusterAdmin: false

steps:
- task: AzureCLI@2
  displayName: azcli login and save creds
  inputs:
    azureSubscription: 'myakstestsp'
    scriptType: 'bash'
    scriptLocation: 'inlineScript'
    inlineScript: |
      #Unfortunately azurecli task does a logout at the end, so we need to grab creds and then login from bash
      echo "export servicePrincipalId=$servicePrincipalId" >.env
      echo "export servicePrincipalKey=$servicePrincipalKey" >>.env
      echo "export tenantId=$tenantId" >>.env
    addSpnToEnvironment: true

- bash: |
    source .env
    az login --service-principal -u $servicePrincipalId -p $servicePrincipalKey -t $tenantId
    mkdir -p .bin
    az aks install-cli --install-location .bin/kubectl  # install kubelogin
    az aks get-credentials -n $(kubernetesCluster) -g $(azureResourceGroup)
    kubelogin convert-kubeconfig -l azurecli
    kubectl get all -A

- task: Kubernetes@1
  inputs:
    # connectionType:None is not documented, but it's in the code https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/KubernetesV1/src/clusterconnection.ts#L59
    # with this setting, the task will just use the kubeconfig context
    connectionType: 'None'
    command: 'apply'
    useConfigurationFile: true
    configuration: 'myapp.yaml'
    secretType: 'generic'
    forceUpdate: false

- task: HelmDeploy@0
  displayName: Helm deploy
  inputs:
    connectionType: 'None'
    command: 'install'
    chartType: 'FilePath'
    chartPath: 'mydemo'
    waitForExecution: false

- task: KubernetesManifest@0
  inputs:
    action: 'deploy'
    kubernetesServiceConnection: 'myaksserviceconnection'
    namespace: 'default'
    manifests: 'demo2.yaml'

Let's look at each of the tasks in turn:
- task: AzureCLI@2
  displayName: azcli login and save creds
  inputs:
    azureSubscription: 'myakstestsp'
    scriptType: 'bash'
    scriptLocation: 'inlineScript'
    inlineScript: |
      # azurecli task does a logout at the end, so we need to grab creds and then login via az cli in bash in subsequent step
      echo "export servicePrincipalId=$servicePrincipalId" >.env
      echo "export servicePrincipalKey=$servicePrincipalKey" >>.env
      echo "export tenantId=$tenantId" >>.env
    addSpnToEnvironment: true

In the above task, I'm using the azurecli task to leverage the service connection to log into azure.  The nuance is that this task executes a logout when it completes, so it's not useful if you want to stay logged in for subsequent tasks.  So what I'm doing is grabbing the SP id & key and storing them into a file to be used later.  There are certainly other ways of achieving this; this was just a quick and dirty approach.
- bash: |
    source .env
    az login --service-principal -u $servicePrincipalId -p $servicePrincipalKey -t $tenantId
    mkdir -p .bin
    az aks install-cli --install-location .bin/kubectl  # install kubelogin
    az aks get-credentials -n $(kubernetesCluster) -g $(azureResourceGroup)
    kubelogin convert-kubeconfig -l azurecli
    kubectl get all -A

The next block does a number of things (and could have been split into multiple tasks.)

First it grabs the Id and Key from the prior step, and then uses these to log into Azure.
Next it runs az aks install-cli in order to install the kubelogin tool.
Next it downloads the kubeconfig using az aks get-credentials, and gets an AAD token using kubelogin
Lastly it runs a kubetctl get all -A simply to show that the login worked.  (obviously this would be removed in a real scenario)

Next we'll prove out the three Kubernetes-specific ADO tasks:
- task: Kubernetes@1
  inputs:
    # connectionType:None is not documented, but it's in the code https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/KubernetesV1/src/clusterconnection.ts#L59
    # with this setting, the task will just use the kubeconfig context
    connectionType: 'None'
    command: 'apply'
    useConfigurationFile: true
    configuration: 'myapp.yaml'
    secretType: 'generic'
    forceUpdate: false

This task Kubernetes@1 acts like a wrapper around kubectl.  Normally it wants to leverage either an Azure Service Connection or a Kubernetes Service Connection, but in this case, we make use of the undocumented option connectionType:None to force it to use the kubeconfig context.
- task: HelmDeploy@0
  displayName: Helm deploy
  inputs:
    connectionType: 'None'
    command: 'install'
    chartType: 'FilePath'
    chartPath: 'mydemo'
    waitForExecution: false

I take a similar approach for the HelmDeploy@0 task.
- task: KubernetesManifest@0
  inputs:
    action: 'deploy'
    kubernetesServiceConnection: 'myaksserviceconnection'
    namespace: 'default'
    manifests: 'demo2.yaml'

Last is the KubernetesManifest@0 task.  This was more of a challenge to get working, as there was no option for connectionType: 'None'.  Instead, to get this to work, I needed to create another Service Connection in Azure DevOps, but this new connection needs to be a Kubernetes Service Connection, and specifically, it needs to be of type Kubeconfig.  You will need to paste in the appropriate kubeconfig for this cluster that you previously pulled via az aks get-credentials.  (Note that this kubeconfig won't work without the associated kubelogin setup).  In the example above, you'll see my kuberentesServiceConnection is named myaksserviceconnection
I'm interested in any feedback on this solution.

Hi @larryclaman,
Essentially, there is no need to export SPN credentials as you can retrieve aks credentials and convert cubeconfig with kubelogin in one go:
- task: AzureCLI@2
  inputs:
    azureSubscription: '$(serviceConnection)'
    scriptType: 'bash'
    scriptLocation: 'inlineScript'
    inlineScript: |
      az aks get-credentials --resource-group $(resourceGroup) --name $(kubernetesCluster)
      kubelogin convert-kubeconfig -l azurecli
  displayName: 'Get cluster credentials'

Could you elaborate on the last task? Does it imply that cubeconfig has to be converted with kubelogin first and then to be pasted to the service connection?
Thanks.

@sergeidavydov As mentioned by several people in this thread, AzureCLI@2 runs az account clear at the end of the task, which invalidates the login session that was exported by kubelogin.

https://github.com/Azure/kubelogin/pull/142 has blocked some workarounds discussed here.
Following on from the existing AzureCLI@2 task logic mentioned above by @carlin-q-scott, you can no longer run a separate kubelogin task, followed by one or more another generic tasks using running kubectl commands or tasks like HelmDeploy@0 with connectionType: 'None'

AKS used to be easy to integrate into the pipeline, and easy to access. It is no longer so. reading all the heavy work-arounds and less than optimal implementations makes me wonder if anybody is really watching the ecosystem for such intrinsic changes like this that has cascaded all sorts of problems.
Thank you to the people working tirelessly that aren't microsoft and sharing their work-arounds, but still, All I see are work-arounds here.

For somebody who DOESN'T have Azure AAD enabled on his cluster and relied on service connections which used to work seamlessly with AKS for integration, what are my options? The service connection holds the spn / secret....

@darthmolen what's not working for you? az aks install-cli?

I have been playing around with this in Azure Pipelines.  Wanted to throw my 2 cents into the ring for some possible solutions.  The following solutions will work if your Azure DevOps Service Connection is a Service Principal
Solution 1 - If you don't mind writing your SPN's secret permanently into the the kubeconfig
- task: AzureCLI@2
  inputs:
    azureSubscription: $(serviceConnection)
    addSpnToEnvironment: true
    scriptType: bash
    scriptLocation: inlineScript
    inlineScript: |
      # download kubectl and kubelogin
      sudo az aks install-cli --client-version $(kubernetesVersion) --kubelogin-version latest --only-show-errors
      # download ~/.kube/config file
      az aks get-credentials --resource-group $(aksResourceGroup) --name $(aksClusterName) --only-show-errors
      # convert ~/.kube/config to a format compatible with kubelogin
      # this stores the SPN's secret directly in the kubeconfig file
      kubelogin convert-kubeconfig --login spn --client-id $servicePrincipalId --client-secret $servicePrincipalKey
- task: Bash@3
  inputs:
    targetType: 'inline'
    script: |
      kubectl get nodes

Solution 2 - Do not store secrets in the kubeconfig file
- task: AzureCLI@2
  inputs:
    azureSubscription: $(serviceConnection)
    addSpnToEnvironment: true
    scriptType: bash
    scriptLocation: inlineScript
    inlineScript: |
      # download kubectl and kubelogin
      sudo az aks install-cli --client-version $(kubernetesVersion) --kubelogin-version latest --only-show-errors
      # download ~/.kube/config file
      az aks get-credentials --resource-group $(aksResourceGroup) --name $(aksClusterName) --only-show-errors
      # convert ~/.kube/config to a format compatible with kubelogin
      kubelogin convert-kubeconfig --login spn
      # create secure azure devops variables for later steps
      echo "##vso[task.setvariable variable=spnId;isSecret=true]$servicePrincipalId"
      echo "##vso[task.setvariable variable=spnSecret;isSecret=true]$servicePrincipalKey"
# each subsequent task that uses kubelogin will need to set 2 environment variables and populate them with the secure variables we created in the previous task
- task: Bash@3
  env:
    AAD_SERVICE_PRINCIPAL_CLIENT_ID: $(spnId)
    AAD_SERVICE_PRINCIPAL_CLIENT_SECRET: $(spnSecret)
  inputs:
    targetType: 'inline'
    script: |
      kubectl get nodes

I am linking this to some issues filed in the azure-pipelines-tasks repo:
/microsoft/azure-pipelines-tasks#17486
/microsoft/azure-pipelines-tasks#15802
/microsoft/azure-pipelines-tasks#10022

We are attempting to disable local accounts in our private AKS clusters and have run into this same issue where we are unable to authenticate to our clusters using any of the built-in Azure DevOps pipeline tasks (e.g. HelmDeploy). I read this blog post:
https://devblogs.microsoft.com/devops/service-connection-guidance-for-aks-customers-using-kubernetes-tasks/
and it made it seem like this should be possible based on this:

AKS can be accessed even when local accounts are disabled.

however later in the Q&A it states:

A: Accessing Kubernetes when AAD RBAC is enabled is unrelated to token creation. To prevent an interactive prompt, we will support kubelogin in a future update and blog post in June.

So I think this is saying that it is not currently possible to authenticate to an AKS cluster with local accounts disabled but there will be an update in June that will support this. Is anybody able to confirm?

I can confirm the work is ongoing, however, I cannot confirm on June is the expected time frame. I am waiting on this feature myself :=)

We are attempting to disable local accounts in our private AKS clusters and have run into this same issue where we are unable to authenticate to our clusters using any of the built-in Azure DevOps pipeline tasks (e.g. HelmDeploy). I read this blog post:
https://devblogs.microsoft.com/devops/service-connection-guidance-for-aks-customers-using-kubernetes-tasks/
and it made it seem like this should be possible based on this:

AKS can be accessed even when local accounts are disabled.

however later in the Q&A it states:

A: Accessing Kubernetes when AAD RBAC is enabled is unrelated to token creation. To prevent an interactive prompt, we will support kubelogin in a future update and blog post in June.

So I think this is saying that it is not currently possible to authenticate to an AKS cluster with local accounts disabled but there will be an update in June that will support this. Is anybody able to confirm?

I can only confirm the work is ongoing, so lets wait together. I am looking forward to it.

Will it also be available for ADO environments of type Kubernetes? https://learn.microsoft.com/en-us/azure/devops/pipelines/process/environments-kubernetes?view=azure-devops
Or is there an existing way to integrate environments without using Kubernetes service accounts? We're looking for an integrated AAD AKS connection method to use in ADO environments, without the "static" service accounts.

Will it also be available for ADO environments of type Kubernetes? https://learn.microsoft.com/en-us/azure/devops/pipelines/process/environments-kubernetes?view=azure-devops
Or is there an existing way to integrate environments without using Kubernetes service accounts? We're looking for an integrated AAD AKS connection method to use in ADO environments, without the "static" service accounts.

I don't know, I am not from AKS team


Feat: top level transformations and remapping

Based on the discussion #1615
⚠ Only transformations to another object will be allowed
Todo:

handle output schema transformations in the documentation ✅

done by adding ZodObject::remap() to the distributed Zod plugin

more tests ✅
documentation ✅

✅ QA passed


[ja] Update container-runtimes.md and related links

Update container-runtimes.md and related links.
https://kubernetes.io/docs/setup/production-environment/container-runtimes/
This closes #39125.

/assign

@t-inu Thank you! I applied your suggestions.

/approve
Thank you~


Override doesn't work

I use the standard template from nuxt-starter and try to import scss to override variables. But it does not work:
@import '../node_modules/at-ui-style/src/variables/default.scss';
@import '../node_modules/at-ui-style/src/variables/timing-function.scss';

$link-color: #c03;
$btn-primary-bg: #c03;

@import '../node_modules/at-ui-style/src/mixins/index.scss';
@import '../node_modules/at-ui-style/src/core/index.scss';
@import '../node_modules/at-ui-style/src/components/index.scss';

Embeded in the Layout.vue
<template>
  <div>
    <Header/>
    <nuxt/>
  </div>
</template>

<script>
  import Vue from 'vue'
  import AtComponents from 'at-ui'
  import Header from '~/components/Header.vue'

  Vue.use(AtComponents)

  export default {
    components: {
      Header
    }
  }
</script>

<style lang="scss">
 @import '../assets/main.scss';

  img {
    image-rendering: auto;
    image-rendering: crisp-edges;
  }
</style>

I ran into the same issue. Filing a pull request to handle this.

Thank you, did you already got accepted?

The maintainer was a bit busy recently. So...

@Michael-Reich Unfortunately it's still open: https://github.com/AT-UI/at-ui-style/pull/18

How can we push this? The package is super awesome with this fix applied and basically useless without it.

@shredding Unfortunately we would need a maintainer, which appears to be @koppthe or @JChehe, to merge this in and they seem back up currently. There has been no response to this ticket or pull request.

There has been no response

At least they maintain the same spirit of support as in their mobile phone contracts.
Just kidding :-)

At least they maintain the same spirit as in their mobile phone contract support.

Thankfully this isn't associated with AT&T. 😆


Multiple external field names

I'm using serde to interact with Amazon's EC2 API.  The API uses url-encoded parameters on input and returns XML.  I am using serde for both serialisation and deserialisation.  However, the field names expected by EC2 on input are different from the field names returned in the XML.
I can see several ways of solving this:
a) Extend serde to allow renaming fileds differently on serialisation and on deserialisation, possibly via rename_ser and rename_de annotations.
b) Extend serde to allow Serialiser specific field encodings.
c) Create parallel data structures for input and output data.
Option c) obviously does not require any change in serde, but I'm reluctant to go this route, as it involves significant extra code (there are approx 650 structs used in the EC2 api), particularly in order to hide the duplication in the exposed api, and compile times are already awful (>5mins).
Option a) seems relatively simple to implement, but adds some complexity to serde.  Cognitive complexity can be limited by keeping the rename annotation for the case where the field names are identical for both serialisation and deserialisation.
Option b) seems the most flexible, but would probably add more complexity to serde, espescially for implementors of Serializer, and would likely incur extra runtime overhead.
Would either of a) or b) be considered for serde?

Of course EC2 would have inconsistent names. Why make things simple for us?
So (a) looks good in general, though it's a bit strange that we can't roundtrip the format using this scheme. Do you think if we had specialization it could work for this?
And yuck! 5 minutes? That could be a useful stress test to help optimize the type checker. Do you have it online somewhere?

I pushed the rust AWS lib.  Note this is still very much a work in progress.  It depends on #62 and #39.

@hugoduncan: #69 landed, so is this safe to close?

I think so.


feat: add generalized entity model for v2 search

Issue: https://github.com/moov-io/watchman/issues/527

Codecov Report

Merging #530 (5d38f46) into master (c092b54) will decrease coverage by 0.42%.
Report is 2 commits behind head on master.
The diff coverage is 0.00%.

:exclamation: Current head 5d38f46 differs from pull request most recent head ce83beb. Consider uploading reports for the commit ce83beb to get more accurate results

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff            @@
##           master    #530      +/-   ##
=========================================
- Coverage    9.67%   9.25%   -0.42%
=========================================
  Files          39      40       +1
  Lines        2874    3004     +130
=========================================
  Hits          278     278
- Misses       2573    2703     +130
  Partials       23      23


[NEW-FEATURE] Add support for FW3.1

closes #285

We will skip this update will move directly to FW4


Crash on startup

Is MCEdit-Unified supported on terminal servers?
minecraft@Volta:~/mcedit$ ./mcedit.sh
Starting MCEdit...
Running in fixed mode. Support files are in your Documents folder.
Splash load...
Traceback (most recent call last):
  File "/tmp/release-master/MCEdit-Unified/__main__.py", line 42, in <module>
  File "/tmp/release-master/MCEdit-Unified/mcedit.py", line 1046, in main
  File "/tmp/release-master/MCEdit-Unified/mcedit.py", line 772, in main
  File "/tmp/release-master/MCEdit-Unified/utilities/gl_display_context.py", line 25, in __init__
  File "/tmp/release-master/MCEdit-Unified/utilities/gl_display_context.py", line 51, in reset
error: OpenGL not available

==================================
                          MCEdit has crashed
==================================
Press the Enter key to close this window

Using https://github.com/mcedit/pymclevel


The process cannot access the file because it is being used by another process

Hi,
When shutting down or restarting my server, Amend downloads the latest purpur.jar but is unable to replace the existing jar due to it being used by another service (control panel?). I am using the Crafty Controller control panel.
[13:22:04 WARN]: [Amend] -------------------------------
[13:22:04 INFO]: [Amend] Amend
[13:22:04 INFO]: [Amend] Server-Type Selected: PURPUR
[13:22:04 INFO]: [Amend] Current Version: 2250 (MC: 1.21)
[13:22:04 WARN]: [Amend] Version is NOT up to date! Newest PURPUR version is 2251
[13:22:04 INFO]: [Amend] Downloading update and applying to purpur.jar...
[13:22:04 WARN]: java.nio.file.FileSystemException: purpur.jar: The process cannot access the file because it is being used by another process
[13:22:04 WARN]: 	at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
[13:22:04 WARN]: 	at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
[13:22:04 WARN]: 	at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
[13:22:04 WARN]: 	at java.base/sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:273)
[13:22:04 WARN]: 	at java.base/sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:109)
[13:22:04 WARN]: 	at java.base/java.nio.file.Files.deleteIfExists(Files.java:1191)
[13:22:04 WARN]: 	at java.base/java.nio.file.Files.copy(Files.java:3133)
[13:22:04 WARN]: 	at Amend-1.7.6--V.1.21.jar//com.tronmc.beta.amend.Amend.onDisable(Amend.java:199)
[13:22:04 WARN]: 	at org.bukkit.plugin.java.JavaPlugin.setEnabled(JavaPlugin.java:291)
[13:22:04 WARN]: 	at io.papermc.paper.plugin.manager.PaperPluginInstanceManager.disablePlugin(PaperPluginInstanceManager.java:237)
[13:22:04 WARN]: 	at io.papermc.paper.plugin.manager.PaperPluginInstanceManager.disablePlugins(PaperPluginInstanceManager.java:161)
[13:22:04 WARN]: 	at io.papermc.paper.plugin.manager.PaperPluginManagerImpl.disablePlugins(PaperPluginManagerImpl.java:97)
[13:22:04 WARN]: 	at org.bukkit.plugin.SimplePluginManager.disablePlugins(SimplePluginManager.java:541)
[13:22:04 WARN]: 	at org.bukkit.craftbukkit.CraftServer.disablePlugins(CraftServer.java:612)
[13:22:04 WARN]: 	at net.minecraft.server.MinecraftServer.stopServer(MinecraftServer.java:999)
[13:22:04 WARN]: 	at net.minecraft.server.dedicated.DedicatedServer.stopServer(DedicatedServer.java:881)
[13:22:04 WARN]: 	at net.minecraft.server.MinecraftServer.runServer(MinecraftServer.java:1330)
[13:22:04 WARN]: 	at net.minecraft.server.MinecraftServer.lambda$spin$0(MinecraftServer.java:332)
[13:22:04 WARN]: 	at java.base/java.lang.Thread.run(Thread.java:1570)

Hi! I apologize for the late reply, are you by chance using Windows?

Yes, Windows 10
On Sun, 7 Jul 2024, 16:32 will seidel, @.***> wrote:

Hi! I apologize for the late reply, are you by chance using Windows?
—
Reply to this email directly, view it on GitHub
https://github.com/MrTronMan/Amend/issues/12#issuecomment-2212486509,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AOKT35RH4FVAXT5EAQ2VK2TZLFNPJAVCNFSM6AAAAABKEB753WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJSGQ4DMNJQHE
.
You are receiving this because you authored the thread.Message ID:
@.***>

Ah, okay that makes more sense.  In reference to #7, Windows tends to the following.

Hi. Yes, I did some tests myself and such; since MCSS is its own application and not straight up started from the jar, Windows (more specifically Java honestly) will not let Amend access purpur.jar (since MCSS is reading and launching purpur.jar from its program and not just running the jar itself). Amend updates by saving all your worlds (to avoid any corruption) and while the server is running (technically), it will replace the jar upon shutdown and since the jar itself is what's running it will replace the file with no problem. However, with MCSS, Amend can't do anything with the jar or replace it since MCSS is the one opening it and accessing it 24/7. Unfortunately, I don't think there is a way around this. Unless Amend were to create a separate .jar and you would have to delete the old one every time it updated. I can do some more digging if you'd like?

To sum it up, Windows filesystems are different than Linux ones and Windows doesn't allow access to the jar as it's directly being used. I also have Crafty Controller (a fantastic piece of software) but I run it on Linux. I have tried to see if there was a way to make this work, but I am 95% sure I would have to create a separate exe. I will see what I can do though!

Hi,
Thanks for the information, its pretty much what I thought was happening!
I have managed to create a solution for my use case in the form of a Powershell script that grabs the latest release, downloads it and then stores it until my daily backup occurs (3am, server is offline for 15 minutes anyway) at which point the purpur.jar is replaced. With this in mind, please don't feel the need to investigate further for me but if you do find a solution and would like me to test it, feel free to let me know!
Thanks.

Hi,
Feel free to have a look at my repository (https://github.com/Git-Ross/PurpurMC-Auto-Updater-for-Windows). It should give you a insight off how I went about it.
I would love to have a look through your plugin (as a learning tool more then anything) when I get a chance and see if I can come up with a solution!
Let me know if your able to figure anything out.

Hello! As you can see I have added updates referencing this issue. I noticed Crafty had an Executable URL (being a Crafty user myself) hoping to allow more Windows users, I created in my API a way to get this.
I know you have your PowerShell script but this is my temporary fix for others with Crafty.
If you do want to test it there is the README or you are welcome to look through my quickly written-up docs @ api.tronmc.com
Let me know! 😄


Updating yarn lock for ML

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
5 unchanged lines in 3 files lost coverage.
Overall coverage increased (+0.003%) to 41.656%

Files with Coverage Reduction
New Missed Lines
%

extensions/notebook/src/dialog/configurePython/configurePathPage.ts
1
57.02%

extensions/sql-bindings/src/common/azureFunctionsUtils.ts
1
80.65%

extensions/machine-learning/src/common/processService.ts
3
77.78%

Totals

Change from base Build<PHONE_NUMBER>:
0.003%

Covered Lines:
30329

Relevant Lines:
67413

💛  - Coveralls


build(package.json): add jas and kieran as contributors

Description

Add Jas and Kieran as contributors

Type of change

[ ] 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
[x] 🏗 Build configuration (CI configuration, scaffolding etc.)
[ ] 🐛 Bug fix (non-breaking change which fixes an issue)
[ ] 📝 Documentation update(s)
[ ] 📦 Dependency update(s)
[ ] 👩🏽💻 Improve developer experience
[ ] ⚡ Improve performance
[ ] ✨ New feature (non-breaking change which adds functionality)
[ ] ♻ Refactor
[ ] ⏪ Revert changes
[ ] 🧪 New tests or updates to existing tests

:tada: This PR is included in version 1.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Merge branch 'main' of https://github.com/dlfrnaos19/course into main

merge brange pr

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

🥰


Monitor data is rewriten if I use twiss calculation during tracking

Dear developers,
I am writing for reporting a user feedback about ParticlesMonitor.
I calculated twiss parameters after tracked a few turns,  and then the first row of the monitor data (at_turn=0) was rewritten because of twiss calculation.
I inserted a particle monitor into the line at s=0 with the name 'monitor_start':
n_part =1
monitor_start = xt.ParticlesMonitor(_context=context,
                                start_at_turn = 0,
                                stop_at_turn = 1,
                                n_repetitions=10,
                                repetition_period=1,
                                num_particles =n_part)
line.insert_element(index=0, element=monitor_start, name='monitor_start')
tracker = line.build_tracker(_context=context)

Initial monitor data:

particle_id
s
x
px
y
py
zeta
delta
at_element
at_turn
state

0
0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0
0
0

1
0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0
0
0

2
0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0
0
0

3
0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0
0
0

4
0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0
0
0

Then I started tracking particles using following lines:
tracker.track(particles, num_turns=3, turn_by_turn_monitor=True) # track 3 turns

output:

particle_id
s
x
px
y
py
zeta
delta
at_element
at_turn
state

0
0
0.0
-0.000839
-0.000002
-0.001854
-0.000014
0.002173
0.000035
0
0
1

1
0
0.0
-0.000002
0.000839
-0.003779
0.001882
0.002171
0.000035
0
1
1

2
0
0.0
0.000839
0.000002
0.001853
0.000014
0.002165
0.000035
0
2
1

3
0
0.0
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0
0
0

4
0
0.0
0.000000
0.000000
0.000000
0.000000
0.000000
0.000000
0
0
0

Perform twiss calculation:
tracker.twiss(method='4d')  #twiss calculation

particle_id
s
x
px
y
py
zeta
delta
at_element
at_turn
state

0
0
0.0
1.000000e-07
0.000000
0.000000
0.000000
0.000000
-0.000100
0
0
1

1
0
0.0
-1.657675e-06
0.000839
-0.003779
0.001882
0.002171
0.000035
0
1
1

2
0
0.0
8.394636e-04
0.000002
0.001853
0.000014
0.002165
0.000035
0
2
1

3
0
0.0
0.000000e+00
0.000000
0.000000
0.000000
0.000000
0.000000
0
0
0

4
0
0.0
0.000000e+00
0.000000
0.000000
0.000000
0.000000
0.000000
0
0
0

The data at first row was rewritten, shown in the table above.
I still could get the particle coordinates by using the sampled initial coordinates from xpart.Particles object,
however I think  it would be better if the Monitor data is overwritten after twiss calculation.
Thank you!
Best regards,
Jiangyan.
@eltos

It should be easy to handle. Fix coming soon

Many thanks :)

Solved in xsuite v0.26.3


Fix errors and warnings building swift/reflection on Windows with MSVC

Remove improper template qualifiers for non-template methods
Fix WIN32 build by including <io.h> instead of POSIX's <unistd.h>
Unhandled control path warnings - we can't use llvm_unreachable as this is imported in the stdlib

Will update to use swift_unreachable when #6025 is merged

@hughbe #6205 is merged

@compnerd updated, thanks

@swift-ci please test and merge

Please retrigger the tests as they didn't get triggered, thanks!

@swift-ci please test and merge


feature please

before in GSE before the pre-patch mumbo jumbo when importing we were able to see the code what was going in and able to change it to a different name if we wanted
can we get a feature to at least allow us to give the import a different name if wen wanted before importing? the importing now just isn't working right new code is not getting merged or allowed to replace the string even after deleting the old stuff completely and /reload

This feature is there but not working correctly.  I am working on it.
See #447 for the not working part and #429 for where it was introduced.