ESENT database provider

Provide Entity Framework 7 provider for a ESENT NOSQL database.
This database is officially supported on UWP platform, but now there is only ISAM imperative access to ESENT DB in C#.
ESENT is proven as very fast, stable and reliable NOSQL database, and it is supported in UWP apps.
It's a good alternative to SQLite on UWP platform. But ESENT lacks any high-level API, it have only low-level imperative C++ style APIs for C#.

Could you link to the current C# api?

Current ESENT C# api:
https://managedesent.codeplex.com/

And here documentation of ESENT C# on MSDN:
https://msdn.microsoft.com/en-us/library/dn375980(v=exchg.10).aspx

ErikJS, could you answer if EF Core team planning to support ESENT DB in future? To be honest, I'm not satisfied with SQLite on UWP platform. I have used SQLite many times in a past and it certainliy not a best option. But today UWP platform have not any other options, only SQLite DB. It's very weird, because UWP platform (and Microsoft Windows as well) natively support ESENT embeddable database, known for decades as a good, stable, fast and reliable database engine. The only problem with ESENT it's out of date APIs, lack of support from any major data access frameworks. In the server or desktop worlds it's not a problem - there is many other supported database engines, SQL or NoSQL. But on the UWP platform officially supported database engines is scarce. SQLite is recommended by Microsoft as a primary solution. And it's very weird! Microsoft have it's own well tested and proven reliable embeddable database engine, ESENT. And Microsoft is totally silent about ESENT. It's officially supported on UWP, but to productively use ESENT in UWP you must be a die hard C++ developer, or at least have this hardocre experince in the past. Newbie developers have not any chances to use ESENT in their UWP applications, because APIs is very hardcore and aged, no sane C# developer would use ESENT when there is much easy to use SQLite option. But, I must write it again, SQLite is not a best option technically speaking. ESENT is a much more technically advanced database engine. For an embeddable database engine ESENT is certainly a piece of art.ESENT supports snapshot isolation (feature supported not by all server-scale full blown SQL databases!) for a transactions, multi-value index keys, pre-joined tables,  an much more... And same time ESENT have very small memory and CPU footprint, it's truly embeddable database, suited for use even on a smartphones (and internally Microsoft use ESENT DB in many parts of Windows OS itself, ESENT was an internal OS related project made publically available some time ago).

@Opiumtm I do work for the EF team, so I am not aware of any plans. One major issue I see with ESENT is that it does not have a query language and a query processor as in a RDBMs system. But I think it would be a good candidate for a useful NoSQL provider, and I would maybe be interested in taking a stab at it, but the main issue with that is, that there are no NoSQL EF providers implemented based on the current codebase.
The plans as I understand them are to revive the Azure Table Storage provider, which exposes a key/value storage quite similar to the PersistentDictionary.

@ErikEJ

major issue I see with ESENT is that it does not have a query language and a query processor as in a RDBMs system
It will not be a problem because ESENT does not have its own query language, but it anyway a relational DB and ESENT API supports querying, indexes, tables, joins via its own ISAM imperative style (open table, have an open handle, add filters and indexes enumerate filtered records via method calls and so on) API. It does not differ fundamentally from any SQL database and ESENT APIs can be mapped to a LINQ language without a SQL intermediate language. In fact, this direct mapping to LINQ queries will be much faster, because it will be mapped directly, without intermediate text based language, without query processing overhead.

So, ESENT is a relational NoSQL database (NoSQL in a sense that it does not support SQL queries, but support relational queries and operations directly via low level API). ESENT is well suited for a direct LINQ-to-"low level relational API" mapping!

@Opiumtm Sounds great!

@ErikEJ yes!
And think about ESENT is an embeddable database, designed to use in a resource constrained environments (but it as well used at an enterprise level too - just think about a fact that ESENT used as a data storage solution in such products as Miscrosoft Exchage Serever and Active Directory!).
Its embeddable nature matches well an opportunity to direct map its relational API to LINQ query language without any intermediate layers. It will be a perfect solution for an UWP universal platform, it will make anyway modest SQLite out of competition easy.

@ErikEJ
Also I personally have some experience in ESENT use on an UWP platform. Major negative factor for its use is not a technicall issues (it works good technically) but a out of date low level API and lack of support from any data access frameworks. It require a big amount of code to use in production. When you familiar with ESENT hardcore low level API it is not a problem to write workable code, but problem with a big amount of that boilerplate error prone code still persists.

@Opiumtm this is interesting because SQLite is very limited in what it supports, and this flows on to a bunch of limitations in EF functionality (see http://docs.efproject.net/en/latest/providers/sqlite/limitations.html). I have never looked at ESENT, do you think it would support a number of the things that SQLite does not?

@rowanmiller Not sure, what you mean as "schemas". If you mean an queryable database structure scheme - it's supported. If you mean "schema" as a SQL logical construct - it's not supported by ESENT because ESENT is not a SQL database, ESENT do not support access rights (it is supposed to do by an application layer). ESENT is a relationanal transactional (it supports snapshot isolation) database, but it does not have any text based universal query language, all queries, sorts and joins are performed via direct ESENT API calls, ISAM style. So, ESENT is perfectly suited for a direct mapping of relational API method calls into a LINQ query language without any query language intermediate layer. ESENT is fundamentally designed for this approach as a fast small footprint embeddable database engine - omit any intermediate logical layers. ESENT is a perfect candidate for a LINQ query language support.. LINQ by design is not a text based language but a direct mapping to a method calls represented in code or in a abstract query tree (in case of query expressions). To be honest - SQL is an useless excess entity here. SQL is not needed at all. LINQ fundamentally is a method mapping to a relational operators - and ESENT supports exactly this, relational operators without useless (for an embeddable database)  intermediate text based language.

@ErikEJ @rowanmiller guys, could you share more info about planning (or not planning...) a support for ESENT DB in EF Core?

@Opiumtm I do not work for the team, I am just working on the SQL Server Compact provider. I might consider having a look at this when the Azure Table Storage provider has been released, after RTM of 1,0.
I do however see limited scope in the Esent provider, as it is only available for desktop and UWP tablet apps, not Windows Phone (or any other platforms). One advantage of sqlite is that it is available on all platforms

@ErikEJ You are not correct. ESENT is officially supported on Windows 10 Mobile for ARM and I already use ESENT in one my UWP application which targets both desktop and mobile. So, Windows 10 for mobile support ESENT and it's tested.

@Opiumtm  That is great, did not realize that!

@ErikEJ I successfully deployed application that use ESENT on a Lumia 920 smartphone and it works nice. ESENT supported on smartphones as of Windows 10 version of Windows Phone (now Mobile).

@Opiumtm this probably isn't a provider that our team would develop, we typically work with database vendors or third parties to develop the vast majority of providers. This allows us to push forward with new features in the core of EF.

@rowanmiller And it's weird because ESENT is a native Microsoft embeddable database known for decades for its speed and reliability, it's an official part of classic native WinAPI (and part of UWP API as well). I do not understand why ESENT is not considered as first-class database provider by a Microsoft recommended API team. As I described above, ESENT is perfectly suited for LINQ. If EF Core does not suited for ESENT - it's a major issue. Maybe modern cross-platform minded developers does not care about this brilliant Windows classics, but for an aged veteran windows developer like me it looks like betrayal and spit directly in a soul. Sorry for an emotions, but it's exactly what I feel. And it certainly not a wise decision. You must not discard things that works perfectly and promises strong technical advantages. ESENT simply is far better than SQLite and its lack of SQL query language support is not a weakness for an embeddable database suited for even a smartphone use. SQL language is not needed at all when you can directly map ESENT relational API to LINQ queries. SQLite is a mediocre and limited database suffering from a performance issues when database size is slightly larger. ESENT on the other hand is very well known for its speed and scalability. And ESENT supports snapshot isolation for a transactions, have normal strong-typed data model. And it all we got in a embeddable database. It's certainly a CRIME to does not support ESENT for an embedded database scenario on UWP applications

@Opiumtm I am sure everyone would be happy for a open source contribution to enable this provider

@rowanmiller Well, and at EF 3-6 times I understands why ESENT is not supported by Entity Framework. In the past EF was a strictly SQL language based framework. As ESENT does not support SQL language queries it was not a subject for an EF support. But now it is claimed that Entity Framework Core does not tightly binded to a SQL language and it will be NoSQL providers for an EF. When I read this on a technical news feed I think - time have come. I think Microsoft would certainly support its own ESENT as a part of a first-class EF providers, especially when there is claimed that EF Core is suited for an UWP platform too. SQLite only support for an UWP platform is a lame. I understand that EF goal is primarily to support enterprise-scale server scenarios involving Microsoft SQL Server, PostgreSQL, Oracle and so on. But please, does not forget an embedded and client-oriented world! Until now there is very few options for a client-oriented .NET developers. SQLite is a well known cross-platform client oriented embeddable database, but certainly is not a better one. It will be nice for a UWP developer to have an alternatives to mediocre and not best at all SQLite database (which personally I consider a lame and mostly unprofessional database engine flourishing on the client-side because of lack of an adequate and well-known alternatives, and ESENT is certainly an adequate alternative for a Windows Platform and it here for a decades). And after all ESENT is a Microsoft native database and must be respected at least by a Microsoft staff. It certainly deserves more respect. As I see, knowledgeable 3-rd party developers have more respect for an ESEN DB than Microsoft.

@ErikEJ It's bad that ESENT DB have not much attention in a developer circles. Known not by all, even not well known by a many Microsoft teams. It was a shocking fact for me that ESENT is not well known even to a Entity Framework Core team! It looks like a sadomasochistic perversion when Microsoft officially recommends SQLite for a client UWP applications and totally silent about anyway officially supported ESENT. What? Microsoft staff are totally replaced by a iOS hipsters or what?

@Opiumtm ESENT is actually one among a few data stores that were discussed when we started prototyping non-relational providers at the very beginning of the development of EF Core. Personally I agree that an EF Core provider for it would be a very good addition and potentially a rich and unique source of challenges that could help us improve our designs.
Plans and priorities can change at any time, however as @rowanmiller explained we currently intend to only focus on building providers for a very small set of stores. For the rest we prefer to partner with people from the OSS community, from database vendors and other teams at Microsoft that already have the necessary expertise on the specific data store technologies and that are willing to go through the exercise of learning how to create an EF Core provider with us.
I have been following this thread with attention hoping that you would be interested in contributing to that effort. As @ErikEJ mentioned, after EF Core 1.0 RTM and after we have made more progress on some of the non-relational providers using the current codebase would be a much better time to start with an effort like this.

@divega

ESENT is actually one among a few data stores that were discussed when we started prototyping non-relational providers at the very beginning of the development of EF Core

Well, ESENT is a relational transactional ACID-compliant database provider. "SQL language" is not equal to a "relational", it's just a relational database ANSI standard for an text-mode API and a declarative programming languate. Just point to this again because there is some theoretical misunderstanding on this thread.

however as @rowanmiller explained we currently intend to only focus on building providers for a very small set of stores

For a server enterprise-level tier ESENT provider certainly is not a top priority. But it's vital for a mobile client-oriented UWP platform. And EF Core is designed for an UWP platform too. So, it's very different set of real life requirements and challenges on a different platforms. As Joel Spolsky said in his famous Law of a leaky abstractions article, it's an exact situation. Abstraction of "support of a most needed data providers" leaks when we see that on a UWP client-side platform there is only mediocre and arguably lame SQLite provider is supported and there is no clear plans for much better alternatives like ESENT (which itself is violently kicks a leaky "SQL language" abstraction, that certainly leaks on a mobile and embedded platforms and become such abomination as SQLite database engine is).

I have been following this thread with attention hoping that you would be interested in contributing to that effort
Personally I thinked about this. But as for now I am already writing a some open-source client oriented project (not on a GitHub, on a CodePlex). Maybe later I can do some contributions to a ESENT DB provider, and in near future I plan to upgrade one of my Windows Phone applications to a Windows 10 universal UWP platform and migrate it from a SQLite to ESENT (which is supported starting with UWP Windows 10 and was not supported on a Windows Phone 8.1 runtime). This certainly will give me a live measurements of an ESENT usage in a wild. As for SQLite, I already got a serious performance degradation issue with it when application is actively used and data in a database updated many times after a each-day usage basis and grows to a small but many indexed records. In my somewhat distant developement past for a native desktop Win32 platform I already got this SQLite issue when it used for a client-side database cache. It works well and fine first time but as time pass and cache grow (not to terabytes or even not to a gigabytes, I must say!)  Maybe I need more SQLite expertise and must invoke some magical SQL commands to optimize and defragment SQLite database... But it certainly a discussed above point to a law of leaky abstractions. SQLite abstraction of a "full featured SQL database on a smartphone and embedded system" is certainly leaks. SQL query processing on an embedded system is have its overhead, is not complete, is not optimal (SQLite certainly does not have such a powerful query optimizer as Microsoft SQL Server or PostgreSQL have) and come with many compromises and limitations. It's how abstractions leaks and it is not a SQLite technical issue, it's an issue with a fundamentally lame abstraction which violently leaks.

After all, I am not against some contributions (at least theoretical) to an embedded scenarios of EF Core including ESENT.

ESENT is a relational transactional ACID-compliant database provider

Thanks for clarifying that. I had read you mention that before, so sorry if I used the term inaccurately anyway :smile: BTW, when I say "non-relational" in the context of EF Core providers I am actually using it at a shortcut to refer to a category of providers that won't inherit from our base relational provider. The latter makes lots of assumptions about SQL being supported, an ADO.NET provider being used to execute commands, etc.

Maybe later I can do some contributions to a ESENT DB provider, and in near future I plan to upgrade one of my Windows Phone applications to a Windows 10 universal UWP platform and migrate it from a SQLite to ESENT ... This certainly will give me a live measurements of an ESENT usage in a wild.

Cool! We also need to do some groundwork to enable this kind of non-relational provider again on the EF Core 1.0 codebase. I hope the timing will work well and to hear back from you.

@divega if it's not too late before release consider rename your inside EF Core terminology from a "relational" to a more correct "SQL based" :) It may be a big benefit for a better theoretical understanding of a EF Core codebase and architecture. "SQL based" in practical terms means only that LINQ queries is translated into a SQL intermediate language and SQL representation of a query is a central point of a "SQL based" provider, it's all about generation of such a SQL representation for any raltional query and operation, even for a database tables construction at a run time. "Relational" is a more general term and does not require SQL language (or any standartized intermediate text based language at all). Tables, indexes, rows, primary keys, relational operators ("select" or "join" for example) - it's a "relational" and it's supported by many non-SQL databases, especially by any ISAM database engines (on top of which SQL database abstraction can be naturally built, as it did by MySQL database on top of MyISAM storage engine). So, any classical ISAM engine is relational but not SQL.And as ISAM engine can be upgraded into SQL engine naturally and historically most well known SQL databases (starting with IBM DB2) are indeed built on top of classical ISAM or some ISAM variation - it is correct to generalize "relational" term into a ISAM data source providers, which are anyway historically a foundation for an SQL databases, on top of which by most a SQL query processor and SQL query optimizer are built.

@divega So, "NoSQL" term is largely incorrectly used buzz-word and much abused in a modern developer circles. Non-relational databases are hierarchical databases (every day used OS file system is a perfect example of a hierarchical database and most FS is built on top of a typical database B+ tree data structure and typical database journaling for a data integrity - just think of it), document oriented databases, "key-value" databases and so on.

"ESENT Data Provider" and #329 "Migrations: Table rebuilds" are boring issue titles that mask the emergency situation that there is still no practical working EF on UWP over 7 months after Windows 10 was generally available.
Should we be making a specific new issue for that?
I have listed an 8 step plan in #329 which was written in a light-hearted style to encourage creative thinking, but one of them could perhaps be the spark for a solution.
Maybe I need to back up 1 step and confirm with anyone on the EF team that UWP is actually an important target platform for Microsoft and why 7 months into Windows 10 there is no sense of emergency priority about what appears to an outsider as an absurd situation.

@10Dev @Opiumtm (also addressing comments from https://github.com/aspnet/EntityFramework/issues/329#issuecomment-187429902)
It seems like there are two main concerns/questions that underpin your comments:

Why are you focusing on SQLite when it has the limitations discussed? Why not something else, like ESSENT.
Why did we not have EF support for UWP earlier (like at RTM of UWP) and with support for more database providers.

Here are a few thoughts that hopefully provide some insight into where our team is coming from.
EF Core is a long term strategy. There are lots of things (features, database providers, etc.) that we want to have, however we are still laying the foundations of this product to ensure we have the right core to build on for the coming years. To that end, we need to focus on a subset of the problem space to get to an initial v1 of the product. We are also very much aware that EF Core will not be the right choice for many folks even when we reach v1 RTM. We know that the missing features and/or database providers will mean many applications can’t use it. That said, the other option is to build all those things now and delay the v1 RTM for everybody.
I think it’s generally accepted these days that assigning more people to a software project doesn’t automatically result in a reduced timeline or more features in the same timeline. Sometimes it does, but when it comes to v1 products where you are building an architecture and components that will underpin future releases, adding more people usually just creates more chaos and churn – thus pushing out the date at which you have a stable core to build on. That was actually what drove our decision to shelve out Azure Table Storage and Redis provider work so that we could get to a stable v1 release.
The feedback we heard from the developer community before starting work on EF Core was overwhelmingly that SQLite is the database of choice for local data. There are obviously reasons that SQLite isn’t amazing, but it is widely used and available on the vast majority of devices. I totally hear that you would love to see ESSENT supported too (or with a higher priority than SQLite) – and that is 100% valid. I just want to get across that we didn’t make the SQLite decision in a vacuum – we decided to start there because that’s what our customers wanted. Obviously no decision will satisfy everyone, and if EF Core doesn’t have what you want at v1 then I am going to be the last person to tell you that you should be using it anyway.
Regarding the concern about EF being “Microsoft's recommended data access technology” but having limited providers available on UWP (raised by @10Dev in #329). We are not applying that statement to EF Core now, or when we reach v1 RTM. EF Core is very much a v1 and it needs time to mature before it should be the defacto choice over EF6.x – this was the primary driving force behind the change from EF7 to EF Core. Of course, EF6.x doesn’t work everywhere, so if you really want to use EF on UWP then you will have to use EF Core – but I am not going to push you to do that over another technology that is working for you today.
Regarding timing compared to the UWP RTM. We started EF Core a couple of years ago, and building a solid data access stack just takes time. UWP didn’t even exist when we started (we originally used wpa8.1 as a proving ground). I agree it would have been great to have EF Core ready to go for when UWP shipped… but we missed that train long before we even knew it existed.
Hopefully that provides some insight. I realize I’m unlikely to change your mind, but I wanted to at least give an honest answer to your questions.

@10Dev @Opiumtm
I want to be clear that the following comments are from me personally, not my team or company…
I understand that online interaction with someone you have never met lacks the personal connection you get from sitting face to face. That said, don’t forget that comments on social media, GitHub projects, blogs, etc. are still ultimately directed to a person (or set of people). I realize from your end that you may feel like you are directing a passionate objection toward Microsoft, but in reality we are a team of individual developers. Individuals who enjoy being torn to shreds about as much as anyone does. We care deeply about data access, we care about our product, and we want to build something that helps our customers get their jobs done. We try to make the right decisions, sometimes we get it wrong. Even when we get it right in the eyes of the majority, with over 16 million downloads of our NuGet package, at least some folks that own those projects are going to disagree with what we do.
 A different outcome should not be expected if that interaction is online.
Today at work I was in a good discussion about the need to assume the best in people – something I do not always do. I am going to apply that to our interaction and look forward to collaborating with you in the future.
Peace,
Rowan

Just :+1: to everything @rowanmiller said.

+1 to @rowanmiller's comments

Hey Rowan,
I didn't say anything I wouldn't say straight to your face in a meeting, but you would have seen my smile as I said it. I did point out that the intent of my "8 step plan" was a light hearted way to get some creative ideas started.

I don't know Opiumtm but I suspect he also was making some attempt to lighten things up but who knows.
The one thing that is certain and you might have fogotten here is that we wouldn't be making these somewhat passionate crys for help if we didn't deeply admire the design of EF and desparately want to be using it with UWP. Within that context there are absolutely zero alternative choices for UWP. When you say "it's ready when it's ready and we aren't forcing you to use it" you are forgetting that as our drug dealer you have already addicted us to the Magic Unicorn crack cocaine of EF Code First Data Migrations.
You already have my eternal respect.

Although surprsingly I run into developers all the time unfamiliar with Brook's Mythical Man Month, as I said previously I was hoping some sort of creative thought or brainstorming could uncover options to consider that would not derail the core effort.
No matter how it came to pass, the lack of a workable EF target for UWP this long after the Windows 10 launch is going to end up being seen as an absurd situation.  As an act of kindness, you could perhaps make an even larger callout in your documentation asking people to avoid using SQLite until #329 is completed.
And I suspect the effort to handle #329 will be even worse than it looks as SQlite edge cases will come out of the dark closet to haunt that approach but if no alternative presents itself then I hope it can be made to happen. And it's easy to see that if you resourced an additional team from somewhere the bandwith of interaction with the current team would be high.
But it seems that perhaps a project for a Data Provider might be able to be comparmentalized in a way that doesn't violate Brooks. Whether that might be ESENT or some other DB that runs on UWP, it might be the right approach for EF to take ownership on a UWP solution even it it's a bit out of band for the types of things the EF team might normally undertake?
Or maybe the SQL Server team or the Azure guys could get dragged into it?

I didn't say anything I wouldn't say straight to your face in a meeting, but you would have seen my smile as I said it.

Understood, the subtleties of humor, sarcasm, etc. do get lost in the online world.

And I suspect the effort to handle #329 will be even worse than it looks as SQlite edge cases will come out of the dark closet to haunt that approach.

Absolutely agree, that is one of the reasons we are hesitant to head down that path. We've actually been discussing it since the EF4.1 days... but it does have the makings of a Pandora's Box.

But it seems that perhaps a project for a Data Provider might be able to be comparmentalized in a way that doesn't violate Brooks.

This will be absolutely true once we reach RTM (or get closer to it). Each provider we add requires some changes/improvements in the core - which at the moment works against our efforts to stabilize for the providers we have. But yes, soon spinning up small teams to work on a provider (internal and external) will be productive.

@rowanmiller I do not mean any personal offence and when I pointed to a UWP problems and overall platform inconsistence (after all, ESENT and SQL CE are native Microsoft products and SQLite is a not Microsoft solution and it have first-class support instead of native Microsoft technology) I described just how it looks from an outside world. I do not feel personal hatred against any member of EF Team. I feel very disturbed with general direction of things and I think if no one points to a problem - it does not mean problem is not exists. When I say about "iOS hipsters" and "sadomasochistic" - it is exactly how it looks from an outside. I know it is not true, but it is how it looks and it leave bitter taste in a mouth. It may be somewhat rude, but it at least honest and personally I wrote it with good intentions and with love for Microsoft and its technology. If all people will be too polite and politically correct, no major problem in our society will be noticed (and personally I think, political correctness is already ruining society). Sometimes it is good to be straight (in all meanings of this word, he-he) and not be quiet when problem exists.
I can certainly tell why majority of people wanted SQLite support first place and why they voted for it. It is not because SQLite is best solution. It is because SQLite is a most well known solution in a mobile developing world. When you google for an "embeddable database" you will see a SQLite in the top and no links to ESENT at all. Did you think about WHY there is so much links to SQLite? It's because SQLite is heavily advertised, for now it is advertised even by a Microsoft itself on a MSDN. When you see it from this point of view, such a poetic metaphors as "sadomasochistic" and "iOS hipsters" are not looks  much rude, it looks like a bitter truth which must be spoken openly by a loyal Microsoft developer veteran.
Microsoft does not paid much attention to their own technologies in a past (ESENT itself is not well known at all, people who know about ESENT can be counted by a fingers on a hand metaphorically speaking), and there is perfectly expectable outcome - Microsoft already lose developer acknowledgement on most of its outright brilliant technology. Even when Microsoft deliver a better technology most of the modern developers are totally ignorant and do not want to play with Microsoft at all. I'm not joking. I work in a developer firm as a senior C# developer and architect and I feel myself as a (sorry for a rude speach again...) sexual minority in a modern developer environment. Everybody speaking about Linux and cross-platform. Everybody fap on a Apple devices, even a people who work as Windows Server system administrators. Everybody tell that Microsoft is utterly a legacy platform. Did you hear this? Did you understand this? As a senior Windows developer with also a heavy UNIX-like C/C++ developing background I do not agree with that. I do not love Linux and have many reasons to this. I do not love Java, Python and so on. But if things with Microsoft commitment to its own intellectual property is not changed (if a course to abandon many good things) remain - I think my next work will be again a C++ development for a NIX operating system and it's matter of some years before vast majority of developers will abandon Microsoft platform once and for all. If you (not you personally, I speak about Microsoft as whole) try to adopt "most adopted by developers cross platform technologies" such as SQLite as first priority thing for a flagship UWP universal platform - it's just a next stage of acceptance of failure. When you must fight bravely you accept yourself as loser and leave yourself on a mercy of victor. As history tell us, victory is never come with a mercy. It always come with death of a loser, fast and painless death if you are lucky.
So I told you some emotional background about my poetic metaphors on this thread and provided some practical knowledge about why people asked you to support SQLite on a UWP platform.  As I see, this ignorance about own intellectual property is poisoning even a Microsoft itself. Just open MSDN in a browser and read SQLite is a recommended data access solution for an UWP platform. Next step will be to abandon UWP at all and deliver an Android distributive as a recommended Microsoft Mobile Platform. Why not? It's the direction in which Microsoft is moved last years. Maybe customers will not care (they already are very ignorant about Microsoft technology and see Microsoft products as legacy at best), but for a veteran Windows developer to see that is like to see a slow death of own wife from a cancer and be unable to help.
Well, and when we cry about support of ESENT and anything else for a UWP we cry in desperation. There is basically no good local database solution for UWP. It's  our everyday reality and we must face it outright. There was SQL CE Server local database for a classic desktop .NET apps. On UWP platform there is no choice but already well known for a years as mediocre and despised by a hardline developers SQLite nonsense.
So, excuse me if I was too straight and not too politically correct on some points. I just wanted to speak truth as I personally feel it deeply in my heart. I mean no offence and do not want to attack anyone personally. I speak this not so beautiful looking truth in an attempt to make things better and with an admiration of a Microsoft developer platform.

If we can choose to ignore individual words here and there, Opiumtm is expressing  a heart felt desire for the success of EF.
When looking at the  entire Microsoft development stack, there appear the a few points where missteps in execution or communication have led to massive developer issues and abandonment of the platform. There are probably a huge number of developers sitting on the fence post right now to see if UWP platform will be another dissappointment or will spread some wings since the architecture provides a way for all organizations large  and small to move forward into the future in a flexible manner covering a huge range of target devices. One of the potential roadblocks is EF not showing up. For people like Opiumtm who care, it makes them fearful of how this will work out.
And Yoda says fear leads to anger and anger leads to the dark side of the Force.
I don't have a crystal ball to know if the trigger event of killing Silverlight which led to huge adbandonnment ended up producing a "Dead Man Walking" situation or if the recovery  is just around the corner with huge uptake of Windows 10 leading to universal UWP but the concerns are real.
If the EF team has been trying to figure out this stuff since 4.1, then I am doubting I can come up with any great ideas but can only think that clear communication  by the EF team who are both indviduals and also part of some nebulous concept called "Microsoft" will be important.
For your daily mental gymnastics, just like  electrons are both particles and waves at the same time, if you hear bad things direct that to the Microsoft "entity" and if you hear good things, well that's the individual team members...

As a minor idea, if it doesn't degrade your  workflow, it would be sensitive to your "customers" if you could re-tag this issue to "bug" (or nothing) since having a working EF on UWP just can't be seen as an "enhancement" to the outside world. In that context, "up-for-grabs" is like a slap in the face and "under review" might be a better choice?
And #329 is a "bug" which maybe  also needs "can't fix" but #329 is surely not an enhancement in the context of a fully working EF on UWP.

@10Dev I think there is never too late to recover. Yes, I am very sceptical about future of Microsoft developer platform. Not because I like this skepticism. I'm certainly not like it. It's because I can not deny it anymore. Yes, rapid abandonment of own technology is not help to make things clear. There is no point to invest in a new platform/technology when its future is utterly uncertain. No matter how brave and determined Microsoft looks at developer conferences when speak about some technology. As we have learned already, bravado on a scheduled developer conference essentially mean nothing. Last years Microsoft is too accustomed to abandon things even before it fully mature. And most terrifying of all Microsoft is too eager to accept "actual developer trends". Not dictate such trends, merely accept. For example, I see JSON as very weird format for a project config format in VS 2015 .NET Core projects.  Well, it looks good in a browser, but in a VS solution folder it looks like a homeless miserable on a aristocracy banquet. It's just a wrong place for him.
It's just one example, but it good in illustration of how Microsoft essentially shifted from a major influential force in a developer world to a merely "accepting the trend" position even within its own environment. It is utterly a passive position. Well, any extremes are harmful. But today we see a very passive Microsoft, passive even on its own platform. Microsoft is so passive it could abandon any technology any time because some girlishly looking hipsters say "this technology is out of trend pheeeee".

Opiumtm, I enjoy considering your thoughts and hopefully with the huge wealth of focused code level issues on the EF list, the team will be kind enough to tolerate a few tangental comments related to this rather key issue.
I always feel a bit weird when people refer to a large company with many thousands of people as if it is some sort of pseudo-person with its own thoughts and desires... I'd rather think of Microsoft as a kind of process flow diagram for a nuclear power plant.
Another point is that people in that "plant" have access to huge amounts of metric data about sales  and usage although I suspect some of the large blunders come from how such data is compressed and "sanitized" to upper management. But they have a lot of market data that nobody outside the organization has acess to and it is part of their often perplexing decision making.
It is possible that what you are seeing as "passive" is an attempt to be "inclusive" and there is also the horrible list of communications and marketing firms they have used over the last decade. Most of the Silverlight mess which did huge mindset damage was just horrible communication. (with a possible dash of typical large corporation absurdity - rumour has it that Balmer killed Silverlight as part of a secret deal in CEO to CEO meeting he had with Adobe becuase he saw the Silverlight effort as a "Flash Compete" and didn't realize that a large numer of enterprises were planning to adopt the technology for in-house application deployment via the OBE option) and then they disbanded the WPF team and it just looked like a complete slash and burn to the outside world.
But all that wonderful technology is still available, the WPF team is operational again and XAML has been embedded into the Windows API to yield both performance benefits and universal availability to other languages and the things you can get done in the Microsoft ecosystem is better than it ever has been (well, except the EF UWP thing) and somehow even now the positive steps are not being well communicated. It's not "pasive", just failure to communicate.
The EF team has developed an excellent world leading product that has also been under-promoted. At the least they should have a dedicated web site leading out with "As a developer is your time worth anything at all to you? The EF Team gives you SuperPowers in a Bottle"

@10Dev I agree with you on that now Microsoft doing a couple of truly wonderful things. For example, .NET Core project and .NET foundation activity as whole are things that make .NET what it intended to be from the start - truly cross platform not in a cross-OS meaning, cross platform in a meaning that it can run on a wide spectrum of different hardware platforms, existing and future.
But, all right and wonderful long-term movements does not answer  a sharp questions of present day. On present day there is very much amount of uncertainty, unfilled gaps, lack of public relation (for example, there is no understandable roadmap to Microsoft platform and UWP future), lack of integration and lack of backward compatibility with a lot of .NET classics and lack of feeling of that Microsoft is truly dedicated and put enough resource to make developers certain in their future. After all, our future is tightly coupled with a future of Microsoft development platform. It's our wages, work places, business opportunities, our sales.

I'm going to wrap this one up with a few comments and then step away from the conversation for a while.
We hear you on the fact that SQLite has some serious limitations that do surface heavily in the EF workflow. We are not ruling out the idea of table-rebuilds (#329) as a way to solve these - we're kicking around some ideas at the moment about how to achieve it without too much hidden magic.
We also hear you that SQLite is not a good fit for everyone - the concerns you raise are valid. We do want to have other providers available for UWP. It's just a matter of timing - I know you disagree with this, but our focus for 1.0.0 is SQLite.
Somewhere you had asked about calling out exactly what is supported in UWP rather than just saying "EF Core works on UWP" (I can't find it now though). This is a fair point and until we have other providers we can adjust the message to be more like "EF Core can be used in UWP with SQLite for local data access (note some limitations of SQLite)".

As a minor idea, if it doesn't degrade your workflow, it would be sensitive to your "customers" if you could re-tag this issue to "bug" (or nothing) since having a working EF on UWP just can't be seen as an "enhancement" to the outside world. In that context, "up-for-grabs" is like a slap in the face and "under review" might be a better choice?

Here is how we use the tags, and I think this is consistent with generally accepted use:

An enhancement is a feature that has not yet been worked on.
A bug is a piece of functionality that is already implemented but does not behave as expected.
up-for-grabs doesn't have any impact on priority or whether we will do the work on our team, it just means we are flagging the issue as something that is a good candidate for someone outside of the EF team to work on.

And for my last personal comment on this thread too…

So, excuse me if I was too straight and not too politically correct on some points.

I am all for direct speech and addressing things head on rather than skirting around issues. That said, being direct and to the point is different than name-calling and using minority/less-privileged people groups in a derogatory manner.

When you must fight bravely you accept yourself as loser and leave yourself on a mercy of victor. As history tell us, victory is never come with a mercy. It always come with death of a loser, fast and painless death if you are lucky.

Mercy may not be a common tactic to win a war… but it sure goes a long way to preventing one.

I would guess from the style of speech that English is not Opiumtm's native language. On the interenet, good idea to think about the information content and leave the rest of it to "lost in translation"
I  can't agree that taking something that is not working and making it work is a "feature" although in some abstract kind of Kafkaesque way I can see the intent.
And although it  feels like quibbling, from your previous description of how work on providers might easily lead to required changes in the core project (in the current state of affairs) then it only follows that "up-for-grabs" should not even be a practical option at this point.
In any case, I for one appreciate the feedback the EF has provided and to the extent that they are individuals completely appreciate their efforts on our behalf and to the extent that they are "oracles" of the entity known as MIcrosoft, there is the sense of some sort of management roadbloack in the flow chart of the organization that has impeded severely the delivery of an EF solution into the WInRT environment for 3 years now. Taken from a broader perspective this is clearly a dysfunctional situation that has caused some damage abstractly to the reputation of Microsoft and practically in the form of the cost of lost opportunity. Reading between the lines, this is completely beyond the control of the EF team to deal with and they are doing the best they can with the resources they have. They probably feel as much pain about the issue as we do on the outside.
Opiumtm: I think the EF team has provided as much guidence as they can and from that I would guess it will be about 6 months before we see something workable for production use and even that  feels optimistic. It is what it is. Everything else about the UWP ecosystem seems very positive as all the previous roadblocks such as needing "sideload keys"  to deploy apps have been removed. The new Composition API removes the previous DX interop roadblocks and other perf roadblocks that were previously hampering large complex enterprise WPF applications. Of course none of this has been communicated properly and that apears to be an eternal "feature" of Microsoft. Keep an eye on my  and I will try to document all of the data options for UWP but nothing can compare to EF.

@10Dev You are right, English is not my native language, I am from Russia.  After all, very popular, commercially successful and award winning modern writer Viktor Pelevin (successful not only in Russia, his works are translated in many languages and Germany creating a film based on his works "Buddha's Little Finger") write in a genre that can be described as intellectual, sarcastic, philosophical fiction with hard social criticism, which is very funny but extremely political incorrect (not sure how such a political incorrectness is translated to German language and how film is created based on such a sharp sarcastic material including drug abuse, mental instability, social exploitation, racial, national  stereotypes, violence, humor on nazi, communism, liberalism and capitalism ideologies alike).
And for UWP platform, it is very promising in long perspective, but not generally polished, lacks vital functionality and certainly is not suited for now as a substitution for a classic desktop.NET/WPF applications. WCF support is limited, database support is even worse, essentially non existent.

@rowanmiller as for war, I am not against open source movement and accept Microsoft determination to make many of its technology accessible as open source. It's great, because to look into reference code of Microsoft tools and frameworks is a good school and helps understand technology which I as a developer use. Microsoft is going into a right direction with its open source initiative. But when we look from a commercial perspective (I am not a big fan of a capitalist system, because capitalism is well known for its ability to kill brilliant technology things in a commercial wars, bankruptcies, cost reduce process and so on), there are Apple, IBM and Google IT giants. They are heavily invest to drive customers and developers from a Microsoft platform and it is a dire reality of our days, in many aspects they are already succeeded. It's what I mean when I said "war". They openly abuse open source movement, bribe IT journalists and gadget dealers, heavily invest in a post-PC and non-Microsoft platforms and so on. Spirit of a collaboration and peace is a good thing, but war for a survival is already here and Google is a worst technology abusing domination hungry monster I ever seen in my professional life, no matter how Google is painting itself as a Good Empire in a mass media (not unlike from a USA government strategy, USA government and Google corporation share too much similar traits, well it's just another political incorrect question). If Google will be a totally dominant winner in a platform wars we will see a much worse toxic IT world. So be brave to fight back.

@10Dev on your page

ESENT Managed Interop it is not clear if the .NET wrapper is UWP compatible
It is clear. ESENT Managed Interop is compatible with UWP. Not all nuget package are, but core Interop package on nuget is compatible with UWP, I tested it myself and it work.

Joseph Conrad was a native Polish speaker that somehow ended up writing some of the greatest works of English literature. Such things are rare and Optiumtm,  your English is very eloquent and humbles me in that I couldn't  write a paragraph of anything in another language.
Cultural differences can be quite large and form a communication gap perhaps larger than language. I would say this discussion has proceeded rather smoothly despite such gaps and is a testimony to the kindness and patience of all the participants.
I think the UWP platform is the next generation of both Silverlight and WPF and fixes many of the serious blocking issues in those previous platforms. It should be possible to harmonize the XAML and API a bit better between then and now that Microsoft has purchased Xamarin, there might be some motivation to create common platform that is more of a "superset" than the current "subset"
As it exists today UWP is mature, performant, reliable and very deployable in large applications within a few significant limitations. As we all know, the largest limitation is in the data area which is why we are here. In fact I finally got around to making a GitHub account for the sole purpose of determining the EF situation on UWP. And so I find out it is dismal. And that's frustrating and almost physically  painful actually. For now it hurts Microsoft more than us since we can make work-arounds but all the enterprise project starts that switch to other technologies are lost forever. That's opportunity cost and the EF team has to juggle that as best they can for reasons we may never know. But for me, I choose not to let this apparent absurdity of a situation affect my view of the overal platform. Like the internet, we just need to route around the damaged part of the network!

@10Dev well, not all thing with UWP is such good as you described. For now UWP have one maybe corner case but very fundamental problem. In Silverlight and desktop .NET frameworks underlying operational system calls was carefully wrapped and internally optimized (just study how elaborated platform Interop in a classical .NET is). On the UWP platform things are much worse. Windows Runtime essentially is a COM with fancy winmd metadata on top. And here we got significant performance penalty when we deal with Windows Runtime calls. Even .NET Native is not good with Windows Runtime Interop from a performance point of view. And on UWP such Interop is a fundamental nature of things because all Windows Runtime API surface is exposed via COM ABI. COM/.NET Interop was never good, but in a traditional .NET vast majority of operational system calls was done via plain win32 platform invoke, not COM. And for now we have something similar to Frankenstein monster full of Interop between .NET/COM boundary suffering from a terrible performance penalty. It's not matter if you avoid too many direct Windows Runtime calls, but File I/O and manual XAML UI manipulation in code (not rely on data templating engine and bindings, go more low level - which is unavoidable if you write very non trivial UI, custom markup rendering for a social app for example) can be painfully slow because it involve massive Interop between .NET managed code and native COM Windows Runtime API.
So, .NET Native toolchain must be improved to eliminate (or at least significantly reduce) .NET/COM Interop performance penalty. After all, Windows Runtime provide most basic and vital services, and in non trivial scenarios massive system calls are unavoidable. For now, .NET Native do not help much to reduce COM calls marshalling costs, on the other hand plain C++/CX code does not suffer from such an issue. Even on MSDN it is clearly suggested to avoid excessive marshalled calls to a Windows Runtime API. And C++/CX code is fundamentally COM based and its calls are marshalled only in a rare cross Thread Apartment scenario. So, now it is reasonable to write performance critical relatively low level code in C++ and split application into a C++ and C# parts. C# is not suited well for all scenarios.

@10Dev to make things more clear, on a x86 desktop .NET/COM calls marshalling cost is tolerable. But on a ARM based smartphones it become a complete disaster. ARM processors are very resource constrained if compared with powerful x86 bulldozers. On ARM .NET/COM Interop penalty become a major issue. Such a subtle and not self obvious issue (one must inderstand internal Windows Runtime architecture and be familiar with not well known to a typical .NET developer details) must be well known to all UWP library authors

Wow at this conversation! My respect for the EF team just went up tenfold.
To get back on topic, I would love to see ESENT as a provider but I'm not sure what the difficulty level would be to implement it. Is anyone currently planning on doing this or should I start poking around when my schedule clears a bit?

@jnm2 I think, SQL providers are more difficult to implement than NoSQL providers. With ESENT challenge would be to integrate all ESENT benefits and optimizations within Entity Framework. For example, ESENT support some specific concepts like "pre-joined tables" and "tuple indexes". Also, ESENT model is very different from SQL and document oriented NoSQL. Additional challenge is to develop ISAM compatible layer. ESENT is a very advanced transactional relational storage engine, which is decoupled from SQL query language (it's not a database product, it's a foundation for database engines and is used as it by RavenDB, for example). So, ESENT is a very special case - it gives Entity Framework unique opportunity to build object-oriented persistence directly over an advanced data storage layer without SQL language overhead and its shortcomings. ESENT is ideal foundation for first-class object-oriented data storage transactional engine without any intermediate abstractions.

@jnm2 : I am thinking about doing something, but as stated earlier, I think awaiting a NoSQL provider implemented by the EF Core team itself would in my opinion be a very good idea, to save a lot of effort.
@Opiumtm :

I think, SQL providers are more difficult to implement than NoSQL providers
I disagree, with all the work done in the core relational ef component, I believe it is actually quite simple to implement a ANSI SQL based provider - and there is currently no sample NoSQL provider implemented based on the current codebase.

SQL Query Languanges sucks, binary is the way

@ErikEJ  after all, Entity Framework Core design notes state that NoSQL providers would be supported as first-class citizens. But in reality, EF Core provide very limited NoSQL support and as I can assume, EF Core architecture and codebase was made with SQL in mind and lack proper infrastructure to support NoSQL.

@Opiumtm we prototyped providers for Azure Table Storage and Redis as we were building out EF Core, to make sure that main components weren't relational specific. Everything relational is split out into a separate package that each relational provider builds on. We also have the InMemory provider, which is somewhat simplistic, but not a relational provider.
That said, we fully expect that as providers for non-relational stores are built they will drive some changes in the central EF Core components to address things we didn't think  of.

I have executed some benchmarks of embedded DBs for UWP platform.
On Intel Core i7 Windows 10 Desktop PC, ESENT shows much better performance than SQLite.
These results are very important for resource constrained handheld devices or IoT.
Bulk DB inserts inside transaction:
SQLite: ~60 microseconds/insert
ESENT: ~15 microseconds/insert
Bulk random record searches by primary key:
SQLite: ~30 microseconds/search
ESENT: ~3 microseconds/search
These benchmarks are self-explanatory.

@opiumtm Looking forward to your 3rd party provider!

@ErikEJ I have tested DBs performance using low-lever APIs for each DB provider. So, no ORM was used. Just plain SQL commands for SQLite and raw ManagedEsent invokes for ESENT.
As far as I know, there is no EF provider exists for ESENT. Personally I have reasonable experience with raw ESENT APIs (using ManagedEsent lib) and have used ESENT this way in some of my projects, but have minimum knowledge about EF internals and how to develop DB providers for EF.

Back in the days I wrote some quick and dirty prototype for e LINQ QueryProvider and GenericRepository for esent. It's far from being production ready and making a good quality EF provider will be a much bigger job anyway. If anyone wants to tackle this I can share this code (I personnally would be more inclined to work on LiteDB)

Closing this as we do not plan to implement and maintain a ESENT provider for EF Core. However, we would be supportive of a community effort to create such a provider.


Small improvements (humidity, outdoor temp + readme)

Enabled Humidity Sensor
Readme: remind to check "Always use..."

Can this be approved?


Duplicate entityKeys

This is my Article structure
{
	Article: {
		id: 1,
		text: '...',
		AuthorId: 1 // -> User
	}
}

If I query my database, and include the Author relation, I'd end with something like this:
{
	Article: {
		id: 1,
		text: '...',
		AuthorId: 1,
		Author: {
			id: 1,
			name: '...'
		}
	}
}

All peachy, but normalizr would turn something like that into this:
{
	Article: {
		id: 1,
		text: '...',
		AuthorId: 1,
		Author: 1
	}
}

As my response objects grow, these extra fields strart to add up if you have allot of relations.
If you'd be running with a document database like mongo you'll probably not have an AuthorId defined in the first place, but this is different for lets say a MySQL database. It would be awesome if there was an option to disable this piece of code:
if (!bag[entityKey]) {
    bag[entityKey] = {};
}

What do you suggest the normalized output should look instead in your case?

Without the Author key i'd say. If you work with some kind of models library you'd have the AuthorId field defined in there and use that anyway.

Are you running normalizr on server? Trying to understand the use case better..

Ahh yes, I am. Forgot to mention that.

I'd accept a PR that lets you specify a third optional options argument to normalize call.
The only supported option would be assignEntity.
It default implementation is
function assignEntity(obj, key, val) {
  obj[key] = val;
}

It would be called from this line instead of the direct assignment a la
// before:
// normalized[prop] = visit(obj[prop], schema[prop], bag);

var entity = visit(obj[prop], schema[prop], bag);
assignEntity(normalized, prop], entity);

(and you'd need to drag assignEntity into that function from the main normalize call)
For your use case, you'd implement something like
function assignEntity(obj, key, val) {
  obj[key] = val;
  delete obj[key + 'Id'];
}

If you do decide to PR this, please include tests.
Thanks!

Im not quite sure about how you think it should be implemented. But this change on index.js:13 made it happen for me:
-      normalized[prop] = visit(obj[prop], schema[prop], bag);
+      var val = visit(obj[prop], schema[prop], bag);
+      if (! schema[prop] || ! schema._virtual[prop])
+        normalized[prop] = val;

The option is saved on the schema instead of passed along the normalize call. That made the most sense to me when defining a schema/relations. This also enables you to specify which relation-keys you'd like to omit instead of just all of them for the entire normalize call.
Option name on the schema is called virtual, but if you can think of something that makes more sense...
Im passing the virtual option along as a second argument on Schema.define. That's so you can use it to call define a second time with the virtual option to true to only have those keys in nestedSchema be set virtual.
I'll send a PR (with test) if it's all good.

Bandwidth is precious!

To be fair gzip helps a lot with repetitive JSON so make sure to measure the savings with gzip enabled.

The option is saved on the schema instead of passed along the normalize call. That made the most sense to me when defining a schema/relations. This also enables you to specify which relation-keys you'd like to omit instead of just all of them for the entire normalize call.

Does this distinction really help? It sounds to me, you'd either have one consistent format, or another consistent format..
I can't promise to accept a PR but please send one to keep the discussion going.

Im running with websockets though, which dont have support for that.
I had use for the option up until a few days ago when my project's database underwent some changes which made it kinda obsolete now. I'll change it to be passed along the normalize call.

This is available in 1.0 via #14.


Implement ability to do server-side copying

demo
$ tower-cli inventory copy 11 --new-name='localhost smart copy' -v
GET http://localhost:8013/api/v2/inventories/11/
Params: {}

POST http://localhost:8013/api/v2/inventories/11/copy/
Data: {'name': u'localhost smart copy'}

== ==================== ============
id         name         organization
== ==================== ============
12 localhost smart copy            1
== ==================== ============

Coverage decreased (-0.1%) to 90.526% when pulling 0d7cff579bdaa22b8580340e58e688a3d60bad72 on AlanCoding:server_copy into 4583d8b0b54a8dce90aa26d6f33023808afb2f55 on ansible:master.

Looks good, but we need to hold this one in tune with 3.3 IMO.


Server side telemetry does not log 403 response codes

Bug description
The wasm stackdriver plugin will not log requests server-side that 403. Requests with a 200 response code are logged.
This issue seems to occur after this istio/proxy sha update commit on istio: https://github.com/istio/istio/commit/b31e8bf5d3182b20f3e8e424e0142d883cd8a786
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[x] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Expected behavior
Both allowed and denied requests should be logged
Steps to reproduce the bug
Download the 1.7 pre-release of istio: https://github.com/istio/istio/releases/tag/1.7.0-rc.1.
Install istio with the stackdriver filter enabled.
Deploy bookinfo, and send a request to the productpage. All requests are logged.
Apply istio Authorization Policy to only allow requests to the productpage.
Only the 200 requests to the productpage pod will appear on the logs viewer, 403 requests to other pods will not appear.
This can also be reproduced in stackdriver integration tests, on a commit after the one specified above.
Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
istioctl version: 1.7.0-rc.1
kubectl client version: v1.18.3
kubectl server version: v1.16.11-gke.5
How was Istio installed?
istioctl install --set profile=demo --set values.telemetry.v2.stackdriver.enabled=true --set values.telemetry.v2.stackdriver.logging=true --set values.telemetry.v2.stackdriver.topology=true
Environment where bug was observed (cloud vendor, OS, etc)
Debian GNU/Linux rodete

Can you try this out with prometheus?

@JimmyCYJ @jwendell This is potentially release blocking if we can reproduce the problem with other log callbacks.

@gargnupur

Yes I can try it out

@davidraskin can you enable debug proxy logs and attach them here? Thanks.

tmp.txt -> debug logs

Potentially localReply is no longer resulting in the log callback. @jplevyak @PiotrSikora

There is a change in behaviour at the commit specified here.
ClearDeferred list does result in calling access log handlerS.

Here is what we found so far.
https://github.com/envoyproxy/envoy-wasm/commit/ede27304cf02bf243c4d403240b65d9e1ed22957
commit added a guard http_request_started_ which is set in wasm Context::decodeHeaders and it is checked in wasm Context::log. Context::log does an early return.

https://github.com/envoyproxy/envoy-wasm/pull/621 Is under review to fix this problem.

What is the ETA to fix ?

The base envoy change has merged, we will get it into istio build EOD.

PR for review, @istio/release-managers PTAL https://github.com/istio/envoy/pull/260

PR for review, @istio/release-managers PTAL https://github.com/istio/proxy/pull/2990

Proxy PR under review: https://github.com/istio/proxy/pull/2989

This is fixed in main and 1.7 is verified.


Google::APIClient - Please provide :application_name and :application_version when initializing the client

require 'pp'

task :list_labels => :environment do

	client = Google::APIClient.new

	client.authorization.access_token = Token.first.refresh_token
	service = client.discovered_api('gmail')

	result = client.execute(
		api_method: service.users.labels.list,
		parameters: {'userId' => 'me'},
		headers: {'Content-Type' => 'application/json'})

	pp.JSON.parse(result.body)

end

I am using google api to fetch the list of mails but I am getting the following error when I run my rake file:

Google::APIClient - Please provide :application_name and :application_version when initializing the client

It's not an error, just a warning. You're encouraged to set those when creating the client.
E.g.
client = Google::APIClient.new(:application_name => 'my app', :application_version => '1.0')

@sqrrrl where do you find the application name and version number?


Cluster provisioning failed, and so did the deletion attempt

I tried to create a cluster as described here and the cluster creation process failed with an error 'unauthorized'
(I'm the subscription owner). I tried 3 times, the result is the same. Now I can't even delete these clusters from my subscription. What should I do?
    "properties": {
        "statusMessage": "{\"status\":\"Failed\",\"error\":{\"code\":\"ResourceOperationFailure\",\"message\":\"The resource operation completed with terminal provisioning state 'Failed'.\",\"details\":[{\"code\":\"OperationFailed\",\"message\":\"Operation returned an invalid status code 'Unauthorized'\",\"details\":[]}]}}",
        "eventCategory": "Administrative",

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 3e0f1828-dfd2-8b0d-886e-0e7bf4741b99
Version Independent ID: a5ab364a-cb3a-bae0-b454-9125610e1dc8
Content: Deploy a Service Fabric managed cluster (preview) using Azure Resource Manager - Azure Service Fabric
Content Source: articles/service-fabric/quickstart-managed-cluster-template.md
Service: service-fabric
GitHub Login: @erikadoyle
Microsoft Alias: edoyle

Thanks for the question! We are taking a look into this and will get back to you soon.

Aplologize for the delay on this. I am working with the internal teams to help resolve your issue. I will close this issue and will answer/track both in https://github.com/MicrosoftDocs/azure-docs/issues/64695 since they are related.


Day before league tournament testing completed

All autos and tuning were reverted to the last version. V5 backstages have minor changes but work consistently. TeleOp is also updated to v4 with drone launcher controls. Perp odometry wheel direction was fixed.

wrong repository.


chore: Refactor sagemaker-templates to use pydantic for inputs

Describe your changes
Refactor sagemaker-templates to use pydantic for inputs
Checklist before requesting a review

[ ] I updated CHANGELOG.MD with a description of my changes
[ ] If the change was to a module, I ran the code validation script (scripts/validate.sh)
[ ] If the change was to a module, I have added thorough tests
[ ] If the change was to a module, I have added/updated the module's README.md
[ ] If a module was added, I added a reference to the module to the repository's README.md
[ ] I verified that my code deploys successfully using seedfarmer apply

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

AWS CodeBuild CI Report

CodeBuild project: AIOpsModuleTestsInfrastruct-doMLXEYsnmxr
Commit ID: 6e5cd4e8ebf63ef0a2db9af3f154b063b61ed509
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: AIOpsModuleTestsInfrastruct-doMLXEYsnmxr
Commit ID: 8180e9cb5bd9001027bc4e69158bf3b6df5ee0a5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: AIOpsModuleTestsInfrastruct-doMLXEYsnmxr
Commit ID: 3b08f828439d41f225ce13d122d84fe68c6e076e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository


OnConnectedAsync missing some of the connections

I'm creating 10 clients in a loop. Almost all the time, a few of these clients don't trigger the OnConnectedAsync method when they connect to the hub. These clients don't throw any exceptions when they start a connection using the connection.StartAsync() or when the connection.On handler is registered. Which makes it hard to diagnose any problems.
Fiddler traffic displays 10 client/negotiate, 10 client/?hub, 10 /message/negotiate requests. But I see 9 OnConnectedAsync calls in the console.
This is the fiddler traffic:

This is the Hub's OnConnectedAsync method:
public override async Task OnConnectedAsync()
{
	Console.WriteLine("@@@ OnConnectedAsync is executed");
	if (Context?.User?.Identity != null)
	{
		var playerId = ((System.Security.Claims.ClaimsIdentity)Context.User.Identity).Claims
			.First(p => p.Type == "PlayerId")
			.Value;

		//some logic
	}

	await base.OnConnectedAsync().ConfigureAwait(false);
}

This is the client code:
for (var i = 1; i <= 10; i++)
{
		var token = await program.GetToken(i);
		await program.CreateClient(i, token);
}

private async Task CreateClient(int playerId, string token)
{
	var connection = new HubConnectionBuilder()
		.WithUrl(_configurationRoot["SignalRHubUrl"],
			options => options.Headers = new Dictionary<string, string>
			{
					{
						"Authorization", $"bearer {token}"
					}
			})
		.Build();

	await connection.StartAsync();

	connection.On<string>("send", async (m) =>
	{
		Console.WriteLine($"Message received: {m}");
		await connection.StopAsync();
	});

	Console.WriteLine($"Player {playerId} connected.");
}

These are the console logs that display OnConnectedAsync's:
Now listening on: http://localhost:51234 Application started. Press Ctrl+C to shut down. info: Microsoft.AspNetCore.Http.Connections.Client.HttpConnection[3]       HttpConnection Started. info: Microsoft.AspNetCore.Http.Connections.Client.HttpConnection[3]       HttpConnection Started. info: Microsoft.AspNetCore.Http.Connections.Client.HttpConnection[3]       HttpConnection Started. info: Microsoft.AspNetCore.Http.Connections.Client.HttpConnection[3]       HttpConnection Started. info: Microsoft.AspNetCore.Http.Connections.Client.HttpConnection[3]       HttpConnection Started. info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]       Request starting HTTP/1.1 POST http://localhost:51234/message/negotiate  0 info: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[2]       Successfully validated the token. info: Microsoft.AspNetCore.Authorization.DefaultAuthorizationService[1]       Authorization was successful. info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]       Request finished in 264.7313ms 200 application/json info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]       Request starting HTTP/1.1 POST http://localhost:51234/message/negotiate  0 info: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[2]       Successfully validated the token. info: Microsoft.AspNetCore.Authorization.DefaultAuthorizationService[1]       Authorization was successful. info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]       Request finished in 8.4074ms 200 application/json info: Microsoft.AspNetCore.SignalR.HubConnectionContext[1]       Completed connection handshake. Using HubProtocol 'json'. info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]       Request starting HTTP/1.1 POST http://localhost:51234/message/negotiate  0 info: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[2]       Successfully validated the token. info: Microsoft.AspNetCore.Authorization.DefaultAuthorizationService[1]       Authorization was successful. info@@@ OnConnectedAsync is executed : Microsoft.AspNetCore.Hosting.Internal.WebHost[2]       Request finished in 7.6316ms 200 application/json info: Microsoft.AspNetCore.SignalR.HubConnectionContext[1]       Completed connection handshake. Using HubProtocol 'json'. @@@ OnConnectedAsync is executed info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]       Request starting HTTP/1.1 POST http://localhost:51234/message/negotiate  0 info: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[2]       Successfully validated the token. info: Microsoft.AspNetCore.Authorization.DefaultAuthorizationService[1]       Authorization was successful. info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]       Request finished in 13.7569ms 200 application/json info: Microsoft.AspNetCore.SignalR.HubConnectionContext[1]       Completed connection handshake. Using HubProtocol 'json'. @@@ OnConnectedAsync is executed info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]       Request starting HTTP/1.1 POST http://localhost:51234/message/negotiate  0 info: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[2]       Successfully validated the token. info: Microsoft.AspNetCore.Authorization.DefaultAuthorizationService[1]       Authorization was successful. info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]       Request finished in 8.4681ms 200 application/json info: Microsoft.AspNetCore.SignalR.HubConnectionContext[1]       Completed connection handshake. Using HubProtocol 'json'. @@@ OnConnectedAsync is executed info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]       Request starting HTTP/1.1 POST http://localhost:51234/message/negotiate  0 info: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[2]       Successfully validated the token. info: Microsoft.AspNetCore.Authorization.DefaultAuthorizationService[1]       Authorization was successful. info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]       Request finished in 6.1647ms 200 application/json info@@@ OnConnectedAsync is executed : Microsoft.AspNetCore.SignalR.HubConnectionContext[1]       Completed connection handshake. Using HubProtocol 'json'. info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]       Request starting HTTP/1.1 POST http://localhost:51234/message/negotiate  0 info: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[2]       Successfully validated the token. info: Microsoft.AspNetCore.Authorization.DefaultAuthorizationService[1]       Authorization was successful. info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]       Request finished in 6.8534ms 200 application/json info: Microsoft.AspNetCore.SignalR.HubConnectionContext[1]       Completed connection handshake. Using HubProtocol 'json'. @@@ OnConnectedAsync is executed info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]       Request starting HTTP/1.1 POST http://localhost:51234/message/negotiate  0 info: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[2]       Successfully validated the token. info: Microsoft.AspNetCore.Authorization.DefaultAuthorizationService[1]       Authorization was successful. info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]       Request finished in 6.5252ms 200 application/json info: Microsoft.AspNetCore.SignalR.HubConnectionContext[1]       Completed connection handshake. Using HubProtocol 'json'. @@@ OnConnectedAsync is executed info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]       Request starting HTTP/1.1 POST http://localhost:51234/message/negotiate  0 info: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[2]       Successfully validated the token. info: Microsoft.AspNetCore.Authorization.DefaultAuthorizationService[1]       Authorization was successful. info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]       Request finished in 7.1773ms 200 application/json info@@@ OnConnectedAsync is executed : Microsoft.AspNetCore.SignalR.HubConnectionContext[1]       Completed connection handshake. Using HubProtocol 'json'. info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]       Request starting HTTP/1.1 POST http://localhost:51234/message/negotiate  0 info: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[2]       Successfully validated the token. info: Microsoft.AspNetCore.Authorization.DefaultAuthorizationService[1]       Authorization was successful. info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]       Request finished in 6.7461ms 200 application/json info: Microsoft.AspNetCore.SignalR.HubConnectionContext[1]       Completed connection handshake. Using HubProtocol 'json'. @@@ OnConnectedAsync is executed
ConnectionId of player 2 is missing:

@yigitgursoy Could you give me the name of your instance? Also, which region is it in?
You can email the instance name to kevinzha at microsoft.com.

Unrelated:
You should be using the access token provider to set the bearer token.
var connection = new HubConnectionBuilder()
		.WithUrl(_configurationRoot["SignalRHubUrl"],
			options => options.AccessTokenProvider = () => Task.FromResult<string>(token))
		.Build();

You don't need to cast this:
var playerId = ((System.Security.Claims.ClaimsIdentity)Context.User.Identity).Claims
.First(p => p.Type == "PlayerId")
.Value;
It's already a ClaimsIdentity.

Thanks @xscript. I've sent you an e-mail with the details.

@yigitgursoy We found the root cause of this issue. It is an edge case and only happens on Free tier. We have fixed it and will update here once it is deployed.

@yigitgursoy  Fix has been deployed to all instances. Please try again to double confirm.


Asset management in Yii 2.1

This issue is about asset-management in general for Yii 2.1. While there's already a checked box for "Make fxp composer plugin optional. Describe how to use asset packagist instead." on  https://github.com/yiisoft/yii2/wiki/Plan-for-next-major-versions#client-side I still think we need to elaborate on this topic.
Current situation with bower
In the current state the Yii ecosystem mainly uses bower-assets for installing client side scripts (or CSS) packages. This can either be done using https://github.com/fxpio/composer-asset-plugin (cap) or https://github.com/hiqdev/asset-packagist (ap).
One of the main issues with this approach is that bower is deprecated https://github.com/yiisoft/yii2/issues/14297, and projects start dropping bower support (eg. https://news.ycombinator.com/item?id=10891324, https://github.com/lodash/lodash/issues/2206#issuecomment-205494973, https://github.com/chartjs/Chart.js/issues/3081 {they brought it back, but still...}).
While this currently does not affect "near-core" packages, like jquery or bootstrap, there should be a safe migration path to npm in planning.
Yii 2.1 should also deprecate bower usage, while still having the possiblity to use 2.0 extensions with bower via a separate ap-URL, see https://github.com/hiqdev/asset-packagist/issues/67
Moving to npm
While both tools mentioned above (cap, ap) are able to work with npm packages, there might (I'd say will) be huge issues, when switching to npm packages (ie. https://github.com/hiqdev/asset-packagist/issues/69, https://github.com/hiqdev/asset-packagist/issues/68) One problematic dependency is minimist - a lib with over 10M downloads ... a week(!). A dependency of over 8000 other projects; details are outlined here - it boils down to flat (composer, bower) vs. tree (npm) installation.
That's also one reason why foxy was created. The basic idea is to use native tooling for client side packages (npm/yarn/webpack - currently), along with package.json files, while still having a minimal connection for dependency resolution between server and client side packages, for ie. widgets or REST API connectors.
There are also discussions about smooth transitions for older packages, see https://github.com/foxypkg/foxy/issues/8
The main question here is: How will Yii2 core-extensions require assets? (should be best-practice)
Using npm-asset/package looks convenient in the first place, but as layed out above, this might fail even in smaller setups - IMHO we currently do not see issues popping up about this, because only few people use npm-asset. Another contra is that composer.json is "polluted" with a package-syntax which does not belong there.
[update] Another con of ap is that every not-registered package will throw an error unless manually registered on their website, which can be pretty cumbersome on projects with a lot of dependencies.
A con using foxy + packages.json requires node + npm or yarn to be installed.
[update] There's also a proposal to use a CDN for delivering assets by default, which is not a viable solution IMHO, see https://github.com/yiisoft/yii2/issues/8452#issuecomment-330141866
Conclusion
The ideal solution would be something where the developers can freely choose which solution they want to use.
Feedback highly appreciated. CC: @yiisoft/core-developers @mikehaertl @tonydspaniard
More related issues/PRs:

https://github.com/yiisoft/yii2/issues/8452
https://github.com/yiisoft/yii2/pull/14832
https://github.com/yiisoft/yii2-gii/pull/301
https://github.com/yiisoft/yii2-bootstrap/pull/212
https://github.com/yiisoft/yii2/pull/13839

JS is crazy, it is growing too fast and libs are getting deprecated/replaced too soon. I think you are right, 2.1 release is a good opportunity to Yii to get rid of as much JS dependencies as possible and figure out a flexible way to let developer implement what to use.

@schmunk42 please, do not cc to me anymore. Thanks buddy.

It might help to think in terms of what would work well for the demo apps.
IMO we want yii2-app-basic to have an easy quick-start but also allow scaling to realistic complex projects. If it would it be acceptable that it has a packages.json in root dir and requires the user to run an npm command then I think we can do it. The npm command runs npm-scripts that run sass, webpack or whatever we want.
This means that Yii takes a bit different direction in that it expects the user to take responsibility for managing asset dependencies and processing separately from Yii.
Npm and packages.json is not the right tooling except for rather simple projects and I think this works well for us. a) It means users have freedom to take asset management it in any direction. And b) it nudges them to take responsibility for this freedom sooner rather than later as their projects grow.

Btw, to explain myself a bit (or possibly to confuse y'all because I'm going meta:), I have given up hope that DRY is an appropriate goal in the configuration of a Yii project's PHP and non-PHP dependencies. My history in Yii 1.1 and 2.0 led me to conclude that, in the long run, the tooling needed for DRY leads to more, not fewer, maintenance headaches, such as those Tobias diagnosed here and elsewhere.
In practice this means that I now prefer to require developers to "manually" maintain the relationship between their PHP and non-PHP worlds. That is to say, they have to keep two configurations in a working alignment. I believe this is an exception to DRY, which "says that every piece of system knowledge should have one authoritative, unambiguous representation." Yii will serve its users well if it gives them a good start on this road and this is better than involving magic that obscures the inevitable problem.

Maybe 2.1 could remove bower library that it's only used in registering assets position. I think each engineer has to know which codes or register position depends on bower's library, so I hope it's an optional way, not required for installing.

Bower isn't used in registering asset position... would you please elaborate?

When programmer registered position using POS_READY, the View component will register jQuery asset bundle into page @see https://github.com/yiisoft/yii2/edit/master/framework/web/View.php#L469, but maybe i am wrong, because I didn't view all lines. ^^

Yes. That's correct. That has nothing to do with bower though. Ability to register scripts on DOM ready will be moved out of the core in 2.1 or removed altogether.

from https://github.com/yiisoft/yii2-jquery/pull/1#discussion_r171220301

How should I know?
It seems you and @schmunk42 have already decided to move towards new asset plugin. If you did, you should start to upgrade infrastructure, including the docs at the core and all asset-related extensions.

There's no decision at all made about any plugin - the only proposal I made is to remove the JS/CSS-
dependencies from composer.json. The goal is that you can choose the way how to install assets.

I would rather use CDN as default asset bundle source to aviod necessity of any plugin or other composer hack usage. This will allow particular extension to function out-of-the-box in most case, while leaving the developer choice in which way he want to install the assets.

CDN is a no-go IMHO, I explained that here: https://github.com/yiisoft/yii2/issues/8452#issuecomment-330141866 - Yii should not use this by default.

The core extensions yii2-gii and yii2-debug should be stripped from asset dependency first. Only after then this matter can be considered.

As said above, there are no-plugins involved or mandatory ones planned.

If asset dependencies are removed from composer.json it means the extension is not functioning after installation. It also means that version constraints checking will be unavailable.
This brings issue #4391 back.
So it seems you have decided that it should be fine, did not you?

CDN is a no-go IMHO

It may have a drawbacks, but in case CDN is absolutely unacceptable as you say, then no official CDN (like https://code.jquery.com/) would ever exist.
Fo me using CDN allows avioding of baby-crying issues like 'I have installed yii2-jquery but it does not work' (no assets manually installed) or 'my installation failed: there is no package npm/jquery' (no asset plugin configured).
One way or another each asset-dependent package should have an explicit instructions about possible asset setup. I have already added those:
https://github.com/yiisoft/yii2-jquery/blob/master/docs/guide/assets-setup.md
https://github.com/yiisoft/yii2-bootstrap/blob/2.2/docs/guide/assets-setup.md
They already include the instrutions, which allows to aviod of installation of assets via composer.
The final decision on this matter depends on the status of yii2-debug and yii2-gii. Those are crucial development tools, which almost always have to be installled. We must attempt to make these tools asset independent, so they will not require neither yii2-jquery nor yii2-bootstrap to function.
If it will become so, then the asset installation problem will be localized up to 2 repositories only and can be solved anytime with thier major releases without core to be involved.
In this case we may decide to strip asset dependencies from composer.json for those packages, forcing developer to maintain assets on his own.
However in case JQuery can not be removed from yii2-debug and yii2-gii, we should choose the option, which garantees asset functioning out-of-the-box. Otherwise setup of basic development tools will produce too much pain for the developer.
I do not consider this issue to be a top priority - the top priority is upgrading yii2-debug and yii2-gii stripping assets from them as the core itself is already free of assets. Only after it is finished with some resolution it would make sense to return to this matter.

I'm positive that it's possible to remove jQuery dependency from both debug and gii.

I will believe it when I see it.

If asset dependencies are removed from composer.json it means the extension is not functioning after installation. It also means that version constraints checking will be unavailable.

The big downside, if assets are in composer.json, is that the packages are "polluted" with them and you need some kind of other tooling to install them. Either AP or CAP - or hacks like asset-free repos.

This brings issue #4391 back.
So it seems you have decided that it should be fine, did not you?

Version checking could be handled with foxy, if you want that, but it must not be required for installation. In a nutshell: Install composer packages, install npm packages, in case of an error with npm, revert both.

As laid out in the inital posting, I see no way how an installation of npm packages will be maintainable with AP or CAP in the future, since multiple versions of dependent packages can be installed with npm, see for example https://github.com/hiqdev/asset-packagist/issues/69 - that problem did not exist with bower.
If we keep npm-asset/... in composer.json the design is broken right from the beginning.
I've researched this topic quite a lot, let me know if I have overlooked something.

@schmunk42, if you are sure what to do, then do it: open the pull requests with documentation adjustments, which shows how developer can install assets using 'foxy', using NPM client and so on.
Describe pros and cons of different approaches.
Write the UPGRADE notes.
Each asset-dependent package should have those documentation adjsustments.
Simple removal of the lines from composer.json is not a solution. If we pass asset istallation to developer, we need to provide the explicit instructions for him.
Pull requests like https://github.com/yiisoft/yii2-bootstrap/pull/212 can not be considered as a solution: removing package dependencies without providing inforamation about how it should be handled now is  actually a bug creation.

@klimov-paul I agree that there has to be documentation etc. but I did not want to step too far ahead ;)
The work in progress was not continued, because issues like https://github.com/yiisoft/yii2/pull/14832 are totally blocking development, since you can't install the core in 2.1 without CAP or AP. The same goes for gii and debug, since they require bootstrap and assets. And continues with application templates, it can not be done all at once.
I forked the repos to circumvent that for inital development, but it's pretty cumbersome to work with, although I could give it a try again.
All proposed changes should be done in 2.1 branches - IMO some smaller breaks here are OK in such an early devleopment stage. I'd also consider it as a bug to require non-mandatory stuff (AP or CAP) otherwise you can not even install.
About prios and PRs
core

Since the core will be asset free, I think this one can be merged; https://github.com/yiisoft/yii2/pull/14832 as far as I can see no doc updates are required here

jquery

since bootstrap depends on it (in 2.0 through BootstrapPluginAsset on core) and gii and debug depend on bootstrap, it should be tackled first

bootstrap

next one in the hiearchy

gii, debug

since they depend on bootstrap

App Templates

those may provide different ways of installing assets

We also need to differentiate the installation of assets for development/testing in the repo - and in an application.

Here's another idea, although it involves a plugin and maintenance of two files, I'd like to share it.
If you want automatic asset installation without npm, add the following lines to your composer.json:
    "require": {
        "wikimedia/composer-merge-plugin": "~1.4"
    },
    "repositories": [
       {
          "type": "composer",
          "url": "https://asset-packagist.org"
       }
    ],
    "extra": {
        "merge-plugin": {
            "include": [
                "vendor/*/composer.assets.json"
            ]
        }
    },

A package would have to maintain composer.assets.json with the same deps from packages.json in the current format.
Disclaimer: untested but I am pretty confident that it would work and might be a compromise we all can live with. Let me know if this is an option to check out.

because issues like #14832 are totally blocking development, since you can't install the core in 2.1 without CAP or AP

???
'2.1' branch is already free from asset dependency.
There is a living proof of it - yii2-captcha:
https://github.com/yiisoft/yii2-captcha/blob/master/composer.json#L31-L36
https://travis-ci.org/yiisoft/yii2-captcha/builds/346997276

Great! I didn't notice, even though it was on 2.1 in the repo for 19 hours already :)
btw: You missed the documentation part in the README from my PR to remove the asset plugin, see https://github.com/yiisoft/yii2/pull/14832/files#diff-8db66c46d0073891dfc96775fcf3682c - also the repositories section for AP can be removed.
Could you take care of it, I'll close my PR since it's mostly obsolete and can not be updated easily.

7cf9116fe6eb755b9cfa4c383f42b7ce401df864

One more general thing about asset-management in Yii 2.1.
We should remove all possiblities to add logic to asset-bundles, like calling init(), they have to be simple data structures only.

There could be situations that it could be useful apply some changes. Take specific version of js or css (for debug, eccc..)., isn't it ?

So do you suggest for development to use a different asset bundle? In this way, I should change also references inside source code, to switch from production asset bundle to development asset bundle.
I don't understard this part:

Espcially this can be a real pain when bundling assets, since the asset is bundled by its properties, but it's not initialized during bundling.

Can you make an example?

Take this asset-bundle for example https://github.com/beowulfenator/yii2-json-editor/blob/master/src/JsonEditorAsset.php
It uses different files when in YII_ENV_DEV (some others use YII_DEBUG btw.) - so you make changes to the non-minified files, even submit a patch to upstream (of the bower/npm-asset) - and when you run it in production you get the old errors, because it hasn't been minified (rebuilt) properly. Happend to us, extremly hard to debug.

So do you suggest for development to use a different asset bundle?

Actually the opposite. If you need to bundle an minify, this is your job, not the one of asset-bundles, which might work differently. An asset bundle should only be able to define files and dependencies.
Another example is this https://github.com/dmstr/yii2-adminlte-asset/blob/master/web/AdminLteAsset.php
If a property is set, include another CSS file. This does not happen during bundling, so you usually get an error, when using the bundled version.

It uses different files when in YII_ENV_DEV (some others use YII_DEBUG btw.) - so you make changes to the non-minified files, even submit a patch to upstream (of the bower/npm-asset) - and when you run it in production you get the old errors, because it hasn't been minified (rebuilt) properly. Happend to us, extremly hard to debug.

This could happen also using an asset bundle that defines only files and dependencies; because if you work on a non-minified version and forget to update the minified version, the result is the same.

(about AdminLTE) If a property is set, include another CSS file. This does not happen during bundling, so you usually get an error, when using the bundled version.

In that case, the "skin" property is never set out of AdminLteAsset. I don't understand how could be used. Is that a code that could be removed?


Add New Toggle to Disable/Enable Widgets found in Taskbar

Some Background
A while back when I was watching one of your live stream records on youtube, You pointed out that Widgets found on Taskbar gets shown for whatever reason after applying tweaks, so I wondered if there's a way to disable/enable this feature, one or two google searches and found this great blog post by top-password.com
After days of wanting to do it, I've found the free time to implement/understand how it all works, not knowing how to implement it was the hard part, noticed there isn't much on the documentation side for devs who want to contribute, may be something a more experienced person can help with this if they have the time to do so.. but nonetheless, any project can be understood with good old grep's recursive search 🤣 (worked great actually, not gonna lie)
What This Solution do "In a Nutshell"
Changing the registry key TaskbarDa found inside HKCU:\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced
1 to Enable it
0 to Disable it
The Feature in-action

Very nice, I hate those damn widgets in Win11 lol.


Make university logos on "about" all the same size or at least proportional.

@algeboy we have a few options to achieve this. I would recommend this solution:
If you provide logos that are relatively similar in aspect ratio, I will make them the same height. This will effectively give you your results.
I would suggest finding logos that are square as I think they'd look the best together.
Another option is to zoom in on the pictures I currently have. I wouldn't recommend this because it would be difficult to recognize the logos.

There should be multiple sizes to choose from here:

CSU https://brand.colostate.edu/logo/
Bucknell https://www.bucknell.edu/communications/guidelines-policies/bucknell-style-guide
Kent https://www.kent.edu/ucm/kent-state-university-logo
For Bielefled, use what is there as I don't understand german enough to know where to point.
Use this for NSA
Use the following for NSF https://www.nsf.gov/policies/logos.jsp
Use this for Simons Foundation

I have decent, roughly 1:1 logos for all the universities. The exception is Bielefled.
They have a logo, but I think it's difficult to tell which university it is: see their square logo
Currently, we're using this logo from them.
Also, would you like a logo for HIM Hausdorff institute for Mathematics? Currently, we have a logo, but it is a banner and not square. Where can I find a roughly 1:1 logo for them?
Or is the current logo ok? I can also photoshop just the "HIM" out of this logo to make it closer to a 1:1 ratio.

Here is a screenshot with the new logos. I imagine we want to change some images?