Deprecate HttpListener

System.Net.HttpListener does not support many modern protocols. It is compat-only (i.e. only critical fixes, no new improvements, enhancements). It's useful for low-volume basic server requests, but does not scale to modern requirements of HTTP servers (incl. performance).
Use KestrelHttpServer instead.

@karelz Wait a sec.  Someone points out how being based on http.sys provides very useful functionality that they're relying on, that Kestrel can't duplicate, and your response is "meh, we're probably going to remove that anyway"?  Seriously?!?
Also, how is it that you regard something that drags in a dependency on ASP.Net Core as a good alternative to a simple, lightweight server class?  Sorry, but no.  From the perspective of a developer actually using it, that's simply insane!

@masonwheeler we need to weight in usefulness vs. maintenance cost. Is that particular feature so much more important to EVERY developer vs. other investments. Maintaining something, keeping it up to latest standards, etc. does cost something - see the cost in Kestrel.
Using your arguments we would not be able to remove libcurl Http client ever, because it may have some useful corner-case features, we can't provide easily in SocketsHttpHandler.
If you want to help change things, I would recommend to focus on facts, and usage scenarios / commonality (e.g. how many customers truly depend on specific feature?)

@karelz

If you want to help change things, I would recommend to focus on facts, and usage scenarios / commonality

I chose HttpListener specifically because it does not drag dependencies on a massive, bloated ASP.Net framework into your project.  Saying now that it's deprecated in favor of something that does is no good.  Deprecation is supposed to mean "don't use this because we've got a better alternative."  Providing a worse alternative is a very bad use of the concept.  The Khronos Group is infamous for not understanding this; please don't be like them.

Also note, that the fact we do not evolve HttpListener further, that it is compat-only API is a fact since its introduction into .NET Core 2.0. The deprecation is just clear public statement of where we are and where we have been for last year. It sets expectations. If you want something modern, it is not HttpListener.

Yes, I understand that that's your position.  I'm saying that this is a bad position and needs to be reconsidered.  If there are new and better core HTTP server features available, what's stopping you from making them available to everyone even if they don't want to bloat up their project with unwanted frameworks and unnecessary IIS crud?

So, late to the party here.  Please consider the magic that is happening over at Ooui, which allows you to pipe the rendering of .NET elements straight into a webpage.  Additionally, you can perform this same magic in a native-hosted/store scenario (UWP as an example).  This is all supported at present through the use of HttpListener.  If I am understanding this correctly, this issue is saying that we would remove the use of System.Net and instead add a dependency to Microsoft.AspNetCore.Server.Kestrel in our native-hosted/store scenarios to get the same functionality?  It would be incredibly valuable to know that what "just works" with HttpListener presently with Ooui also "just works" with KestrelHttpServer, too.

First a little bit of history: Long time ago, HttpListener was found to be insufficient in the areas of modern feature richness, modern API surface and performance characteristics. ASP.NET team forked the code base and evolved it into WebListener, which then evolved into Kestrel.
The moment this evolution path started, HttpListener became maintenance-only (aka non-modern aka obsolete).
If current HttpListener feature set is sufficient for your needs, you can still use it. We are just warning users to not expect modern evolution, APIs and performance improvements in the space. If you need any of that, we suggest to consider Kestrel or other alternatives.

@masonwheeler I chose HttpListener specifically because it does not drag dependencies on a massive ... Providing a worse alternative is a very bad use of the concept.

HttpListener may have fewer dependencies, for which you pay by limited capabilities (e.g. feature set, perf). Overall we are convinced that Kestrel is the better alternative from many point of views, especially those which are most important to majority of our customers.

@masonwheeler I'm saying that this is a bad position and needs to be reconsidered. If there are new and better core HTTP server features available, what's stopping you from making them available to everyone even if they don't want to bloat up their project with unwanted frameworks and unnecessary IIS crud?

As I suggested above, if you want us to reconsider, we will need some hard data on usage, limitations of Kestrel, etc. So far I am aware of only 1 feature which is not better in Kestrel. Its usefulness seems to be fairly limited.
Maintaining multiple HTTP servers is cost, it does not come for free. The cost is high (you have to do everything twice at minimum), it also fractures ecosystem by providing multiple official ways of doing the same thing. Investing into Kestrel to fulfill all the needs of various HTTP server scenarios seems to be overall better investment.
BTW: AFAIK the IIS dependency in Kestrel is just one of the options, it is not tied to it.

@Mike-EEE this issue is saying that we would remove the use of System.Net and instead add a dependency to Microsoft.AspNetCore.Server.Kestrel in our native-hosted/store scenarios to get the same functionality?

If your projects are satisfied with current limited HttpListener capabilities "as is", you can keep using it. If you need more, you might want to consider Kestrel (although I don't think it is available in UWP).
HttpListener is not going to disappear, however, you should not expect it to evolve further. It has reached its full potential.

It would be incredibly valuable to know that what "just works" with HttpListener presently with Ooui also "just works" with KestrelHttpServer, too.

That sounds like a very good idea. I am sure quite a few developers would appreciate that. I hope some Kestrel experts will be able to provide such guidance.
FYI: For context, check some of the discussion here: https://github.com/dotnet/designs/issues/9 (just please do not reopen the discussion there - it does not belong on the uber-issue).
If there are additional real-world usage patterns of HttpListener, I'll be happy to hear details.

Cool... thanks @karelz ... I am starting to dig and do some analysis around this space, myself.  In my view it is incredibly important as at a glance it totally (and finally) satisfies the request for a Ubiquitous.NET.  That is, using this magic bridge/connection, you can enable a world where it's one .NET codebase and one .NET viewbase total in your solution, rendered through a WebView (as presented/demonstrated above).  That is a big deal and I do not want to jeopardize it. 😄
So it does sound like at the very least we will still be able to use HttpListener, but it will be deprecated?  I am sure that will feature all the expected warnings in code as such.

If you need more, you might want to consider Kestrel (although I don't think it is available in UWP).

So I thought that UWP supported .NET Standard 2.0?  That is the TargetFramework of Microsoft.AspNetCore.Server.Kestrel so it should "just work" right?  Or am I missing something (again 😆)?

So it does sound like at the very least we will still be able to use HttpListener, but it will be deprecated? I am sure that will feature all the expected warnings in code as such.

Yes, I am not aware of any instance when we deprecate things and they immediately stop working :). The warnings are soft, easy to disable and opt-in currently - read more in this repo about the platform compat tooL; https://github.com/dotnet/platform-compat

So I thought that UWP supported .NET Standard 2.0? That is the TargetFramework of Microsoft.AspNetCore.Server.Kestrel so it should "just work" right?

I'd ask Kestrel folks where and how they are supported. IMO Kestrel needs to listen on ports which is not supported in UWP. Not sure if it can work without that (you have reached the limits of my knowledge).
BTW: Also note, that there are currently limitations on various platforms:

HttpListener does not support HTTPS in UWP and on Linux - https://github.com/dotnet/corefx/issues/14691 (because it is C# implementation, not based on http.sys which is not available on the platforms).
WebSocket not supported on Win7 - https://github.com/dotnet/corefx/issues/13663#issuecomment-313783837.

Kestrel needs to listen on ports which is not supported in UWP

So to add a little more context here, the HttpListener in Ooui is listening on a port and works in UWP.  This is because it isn't intended for any external connections, and it's only listening for the process in which it is running.  This is a supported scenario in UWP, so it would be interesting to see if Kestrel works as well when listening within its own process.

and it's only listening for the process in which it is running

I wonder how this is being checked?

@springy76  It's probably better stated that it doesn't accept any connections from external requests, so as a result it only accepts requests/connections running from within the process from which the application is running.  That is my understanding, at least.  Also, while I have gotten this to work locally on UWP, it hasn't been officially verified as a deployed product.
As such, I do have an outstanding question on Xamarin's forums, but if I do not get an answer there, I will have to brogrammer-up and get some HelloWorld's deployed to get a definitive answer.  Unfortunately, Droid development hasn't exactly been friendly to me, and I haven't even touched iOS.
If someone has some valuable knowledge around these scenarios it would be greatly appreciated if you could share. 👍

@masonwheeler
See discussion at: https://github.com/dotnet/designs/issues/9

@valeri2 , it seems to me that Kerberos (e.g. Active Directory) is one of the few reasons a developer would want to run an application on Windows vs. Linux. The Linux community by and large doesn't want to deal with Kerberos, whereas Windows has been pretty good at handling it for a long time. If Kerberos is a requirement (and tbh I would use if I could for an internal or enterprise app), you should stick with Windows.

I would also like it if you could keep the HttpListener as it is and maybe evolve it as a no-frills HTTP Api/Server alongside the other ones. Seeing how many times it has been ported by other people already there seems to be plenty of demand for it.
As for my own use: I like that in 3 lines of code I can listen for a connection and do almost anything with it without needing a lot of dependencies or learning a new paradigm.
There are some things that can easily be made better, like the certificate format it expects. And some other things I am sure you are already aware of.
In other words, I think you'd make a lot of people happy, including me, if you keep HttpListener around and Microsoft would still support it.

Hello @karelz , your recommendation is not to use HttpClient and use Kestrel that has been moved into ASP.NET Core. However, I don't fully understand how to go from one to the other. I have a desktop application that listens on port 443 and replies web requests, and not full blown ASP.NET Core project with all the configuration and setup that a web server requires, frameworks, paradigms and all its baggage. Is there a way to make a console application to respond to HTTP requests? or are we adding the constraint that only ASP.NET Core projects can do that?

@pablocar80 AFAIK you can write minimal Kestrel server with little overhead, baggage, etc. Check out their docs, I am not expert on the topic.

The docs are at https://docs.microsoft.com/en-us/aspnet/core/fundamentals/servers/kestrel?view=aspnetcore-2.2 . The minimal sample linked from the docs is not a full blown ASP.NET project, but it still carries some of the baggage with it: https://github.com/aspnet/Docs/blob/master/aspnetcore/fundamentals/servers/kestrel/samples/2.x/KestrelSample/Startup.cs#L4
cc @davidfowl

Would it make sense for the DotNetCore team to offer the HttpClient class as a thin wrapper that under the hood uses Kestrel? This would help with migration A LOT.

@jkotas thank you for the links. I checked out the documentation and the references needed are specific to NET Core. Ideally we should have HttpClient or its replacement compiling under NET Standard. I would write a very long post explaining why NET Standard is awesome but you guys work at Microsoft and know already.

Nothing in the docs show a minimal example. There are different levels of minimal:

Using the webhost/generic host

With or without a startup class (the generic host has better startup performance as part of 3.0)

Wire up the server instance yourself without a host.

The latter isn’t pretty but we could spend some cycles making a nicer API to get to the pre request callback if that was a concern.

Hello @davidfowl ! If I understood correctly then option 2 is what HttpListener is for.
Right now, if you have HttpListener, the Portability Analysis tool tells you that you're good to go.
Then you realize that not really -- the .NET Core implementation doesn't support HTTPS which is a reasonable requirement to put a system in production.
Then you look up the issue on why HTTPS isn't supported, and it's because there's this thread that says that HttpListener should be abandoned altogether.
So there's really a disconnect between the happy land of the portability analyzer that tells you you're ready to jump, and the reality that HttpListener in NET Core compiles yet isn't really implemented.

The latter isn’t pretty but we could spend some cycles making a nicer API to get to the pre request callback if that was a concern.

Suggestion:
Spend those cycles on an API that mimics the .NET HttpListner API.
The API does not have to serve as a foundational API for upper ASP.NET layers, but rather be considered an optional slim "Adapter" class aimed for developers who feel comfortable with that API definition.
How does that sound?

Exposing the HttpListener API from Kestrel IMO doesn’t make any sense. It would make more sense to just “fix” HttpListener but to make it production ready would be a wasted effort (wasting cycles fixing hard networking and security issues I’m not places etc). Now you could argue that Kestrel and HttpListener could share the same underlying core but there’s tons of code in the upper layers that actually handle the protocol control flow that wouldn’t be shared. HttpListener is also a pull model and ASP.NET Core exposes a push model for handling requests.
There are other layering concerns with replatting one on the other (like namespace and assembly dependencies). Even the core of kestrel is pluggable and things like TLS are plugged in via this extensibility (also multiple transports) and I’m not sure there’s an appetite to push this down into the lower levels.
Layering the other way (building Kestrel on top of HttpListener) would be wasteful layering wise and it’s already possible plug server implementations into ASP.NET Core wholesale so this would be pointless IMO.
All that said I’m not sure what the best call is here because we don’t have true replacement that’s built into the “core” shared framework that is a “production ready” (whatever that means here) server. That’s a bigger strategic decision that needs to be discussed.

cc @DamianEdwards

Exposing the HttpListener API from Kestrel IMO doesn’t make any sense.

Why?

It would make more sense to just “fix” HttpListener

Why?

Now you could argue that Kestrel and HttpListener could share the same underlying core but there’s tons of code in the upper layers that actually handle the protocol control flow that wouldn’t be shared.

The .NET HttpListener class does not implement any protocol control flow. Thats handeled by lower level Windows components which HttpListener “wraps”

HttpListener is also a pull model and ASP.NET Core exposes a push model for handling requests.

What is Kestrel “Push model” ?

There are other layering concerns with replatting one on the other (like namespace and assembly dependencies). Even the core of kestrel is pluggable and things like TLS are plugged in via this extensibility (also multiple transports) and I’m not sure there’s an appetite to push this down into the lower levels.

How do these concerns impact a user facing HttpListener API? What dependecies exist?

Why

It would be a performance regression to use another API which has a different header and HttpContext structure.

Why

If this is about API porting then HttpListener is compatible with what was there on .NET Framework.

The .NET HttpListener class does not implement any protocol control flow. Thats handeled by lower level Windows components which HttpListener “wraps”

Sure it is. On Linux the entire protocol is implemented. It’s uninteresting to discuss the windows http.sys implementation as we would never base our implementation on that since it is tied to the OS and requires a new OS to get fixes and features.  It’s also inconsistent with the Linux behavior (see prefix registration and HTTPS). There’s a fork of HttpListener in ASP.NET Core called HttpSysServer that doesn’t pretend to be an abstraction. We also don’t pay for another abstraction as we gutted it and bind directly to the native http abstractions to avoid the indirection.
So yes the entire control flow for both HTTP/1 and HTTP/2 needs to be implemented in managed code.

What is Kestrel “Push model” ?

With kestrel and ASP.NET Core in general you register a callback (usually a piece of middleware) that gets invoked per request. Unlike HttpListener the caller is responsible for implementating the accept loop (PS accepting on connection at a time may not be optimal).

How do these concerns impact a user facing HttpListener API? What dependecies exist?

Not sure what you mean. The entire discussion is about supporting the existing API, not about a replacement right?

The entire discussion is about supporting the existing API, not about a replacement right?

I personally would be open to change my code as long as we get to a new standard that works everywhere. The class doesn't need to be HttpListener, it could even be a higher abstraction class that already receives requests and dispatches async calls to request controllers.
But in any case, it needs to (1) compile in NET Standard, (2) implement the HTTP protocol, (3) allow to read the header and content of requests, (4) allow to override and send custom response headers and write the content of the response.
What I would NOT want is a web server that reads the URL and expects to find files on the disk or programs with a matching name, or that requires the user to install IIS/Apache, or that I need to change the project type to an ASP.NET flavor, etc.

Does it need to be included in netstandard? Does it need to run on .NET Framework? Is the goal compatibility or performance or “edge readiness”? What platforms does it need to run on? Does it need to an extensibility model (like middleware) so that static file support can be added) what about logging?

Does it need to be included in netstandard? Does it need to run on .NET Framework?

NET Standard has been critical to me to work migrating. This is because I have been able to convert libraries to NET Standard and keep compiling the current version in the NET Framework. If it wasn't for NET Standard you would have a "NET Core migration" project in parallel, that takes a lot of time, while the codebase evolves in two fronts. So NET Standard is an amazing thing to keep everything together.

Is the goal compatibility or performance or “edge readiness”?

I don't know what "edge readiness" is.
For compatibility, it's really frustrating when you move to another platform and something doesn't work, and you have to debug and research specifically for that platform. Other languages that are cross platform give you a reasonable confidence that if your program works in one operating system then it will work in another (Java, Kotlin, Golang, Rust, Python).
For performance, here's an open source web server developed in C# on top of TcpListener, and the performance is very good, complies easily with C10K: uhttpsharp

What platforms does it need to run on?

For me, the answer is Windows, Linux and Mac computers.

Does it need to an extensibility model (like middleware) so that static file support can be added) what about logging?

Like HttpListener, it should have none of that built-in. The programmer handles the requests and decides what to reply. If they want to put a static file in the response that's up to them. If they want to log somewhere they add a line of code to log with library of choice.

Sounds like you’re looking for a compatibility story and I think HttpListener is that but it’s not our web server story for .NET Core.
We have a cross platform sever and it’s called kestrel + asp.net core and it scales from very small libraries to very large applications with a consistent application model.
HttpListener is a lower level API with 0 extensibility originally designed for expose http.sys to managed code. This issue IMO is about compatibility between .NET Framework and .NET Core. Maybe it’s best to do the bare minimum to expose the right set of APIs to fix whatever is broken and then open another isssue to pinpoint the need for a non asp.net based  webserver built into some future version of netstandard

It would be a performance regression to use another API which has a different header and HttpContext structure.

Maybe, but how can you be so sure that such “hit” is unacceptable to a point where it doesn’t make sense - for users? Nuget shows 33,003 downloads for “NETStandard.HttpListener” so its pretty clear that demand exist for the API. Moreover, given that HttpListener Request/Response API exposes standard Http Structures (e.g query string params, methods, cookies) I’m quite confident you can avoid most reparsing/copy penalties by passing those structures from kestrel as is so the hit should be negligible.

If this is about API porting then HttpListener is compatible with what was there on .NET Framework.

This is about having one underlying server implementation, consistent across platforms, activity developed, exposing an alternate API. One is simpler than Two.

Sure it is. On Linux the entire protocol is implemented. It’s uninteresting to discuss the windows http.sys implementation as we would never base our implementation on that since it is tied to the OS and requires a new OS to get fixes and features. It’s also inconsistent with the Linux behavior (see prefix registration and HTTPS). There’s a fork of HttpListener in ASP.NET Core called HttpSysServer that doesn’t pretend to be an abstraction. We also don’t pay for another abstraction as we gutted it and bind directly to the native http abstractions to avoid the indirection. So yes the entire control flow for both HTTP/1 and HTTP/2 needs to be implemented in managed code.

That’s all beside the point – HttpListener API surface stands orthogonal to who handles control flow and that separation is clearly evident in the Windows implementation. Point being, having an Adapter class convert “Kestrel API” to “HttpListener API” would be equivalent  to how currently the .NET HttpListener class serves as an Adapater class which converts the “Windows HttpServer Win32 API” to a .NET OO API. From .NET HttpListener standpoint “Control Flow Handling” is “Uninteresting” and I see no reason why it can’t stay that way if you move the underlying Http Server implementation from a “Windows” implementation to a “Kestrel” implementation.

With kestrel and ASP.NET Core in general you register a callback (usually a piece of middleware) that gets invoked per request. Unlike HttpListener the caller is responsible for implementating the accept loop (PS accepting on connection at a time may not be optimal).

Wait. Let’s discuss “Kestrel” in separate from “ASP.NET” (because they are separate).  From what I have come to learn there is a tradition where network servers are architectured around the Socket model which is essentially a blocking request queue model providing callback registration as a user abstraction. Why isn’t kestrel following this proven model, especially if it was not designed to be used by users directly but rather through ASP.NET abstractions?

Not sure what you mean. The entire discussion is about supporting the existing API, not about a replacement right?

I don’t quite understand your arguments in this context. You wrote about: “namespace and assembly dependencies”, “extensibility”, “TLS”. How do these hinder implementing an alternate user-facing API?

@davidfowl thank you for taking the time to review this request, even though your profile says you're on vacation. It seems that you have reached a decision on this and we won't change it. For me, HttpListener is extensible because you can do pretty much everything the HTTP protocol allows you to do. Do you think there is any chance we will see a low level HTTP library like HttpListener in .NET Standard? I think of if as the equivalent of TcpListener but with HTTP built on top of it. It's what I will be doing on my own since I won't get it from the platform.

@clrjunkie

Moreover, given that HttpListener Request/Response API exposes standard Http Structures (e.g query string params, methods, cookies) I’m quite confident you can avoid most reparsing/copy penalties by passing those structures from kestrel as is so the hit should be negligible.

I'm not sure how you would pass those structures as is when the ASP.NET Core HTTP abstractions are different. Can you clarify? It would also be possible for you to test this easily by just plugging the existing HTTP Listener implementation to measure the performance difference. Having worked on webserver performance over the past 5 years, I disagree with the assessment that it'll be negligible but happy to be proven wrong 😄.

That’s all beside the point – HttpListener API surface stands orthogonal to who handles control flow and that separation is clearly evident in the Windows implementation. Point being, having an Adapter class convert “Kestrel API” to “HttpListener API” would be equivalent to how currently the .NET HttpListener class serves as an Adapater class which converts the “Windows HttpServer Win32 API” to a .NET OO API. From .NET HttpListener standpoint “Control Flow Handling” is “Uninteresting” and I see no reason why it can’t stay that way if you move the underlying Http Server implementation from a “Windows” implementation to a “Kestrel” implementation.

What's not clear? We're discussing implementation details.

Wait. Let’s discuss “Kestrel” in separate from “ASP.NET” (because they are separate). From what I have come to learn there is a tradition where network servers are architectured around the Socket model which is essentially a blocking request queue model providing callback registration as a user abstraction. Why isn’t kestrel following this proven model, especially if it was not designed to be used by users directly but rather through ASP.NET abstractions?

I'm not sure if we have a deep dive on how Kestrel an ASP.NET Core's abstractions are designed but I don't think this is the right place to discuss it. Maybe this will help clarify things in the implementation (https://github.com/aspnet/AspNetCore/issues/4772). There is no request queuing model (other than using async/await).

I don’t quite understand your arguments in this context. You wrote about: “namespace and assembly dependencies”, “extensibility”, “TLS”. How do these hinder implementing an alternate user-facing API?

namespace and assemblies have to do with the current ASP.NET Core types that exist at the connection/socket layer. Kestrel's HTTP implementation is built on top of "connection middleware" (https://github.com/aspnet/AspNetCore/blob/562d119ca4a4275359f6fae359120a2459cd39e9/src/Servers/Kestrel/Core/src/Internal/HttpConnectionMiddleware.cs#L15) which goes back to what I mentioned above (the bedrock issue) above how the stack comes together. What that boils down to is that there's no split of layering where you have a set of System.* namespaces only. Source would need to be copied. You could imagine an alternative world where the ConnectionContext and friends would live in System.Networking or something and the HTTP stack was a piece of middleware that was built on top.
As for the extensibility and TLS, historically, supporting HTTPS with HttpListener meant using admin privileged tools on windows to register prefixes and certs (e.g. http://blog.boxofbolts.com/ssl/windows/owin/guide/2015/06/29/https-self-hosted-windows/) AFAIK there's no user facing API to give HttpListener a certificate. That's the user facing API that I assume needs to be added.
@pablocar80

Do you think there is any chance we will see a low level HTTP library like HttpListener in .NET Standard

I don't know. If you're asking if there's a plan to expose an API like this then there isn't currently and if there was it would be in a future netstandard version that wasn't compatible with .NET Framework.

@davidfowl The underlying Architecture behind Windows Http API is beautiful! The idea of having separate request queues identified by Url and Port independent from consumer threads represents a clean separation of concerns. Where one would choose to handle the request be it “callback”, “handler”, “controller” or what not is matter of style not architecture. I honestly don’t understand why you guys decided not to architect kestrel the same even though the goal is not to provide an O/S level API.

@davidfowl The underlying Architecture behind Windows Http API is beautiful! The idea of having separate request queues identified by Url and Port independent from consumer threads represents a clean separation of concerns. Where one would choose to handle the request be it “callback”, “handler”, “controller” or what not is matter of style not architecture.

I agree it's great but I don't think it's relevant to the discussion. Adding a request queue wouldn't affect the public API in anyway. Callbacks, handlers and controllers are all push based callbacks. You register a method that the server "calls back" so I'm not sure what that has to do with anything.
It's also why it's possible to plug HTTP.Sys into ASP.NET Core and the application logic doesn't change. Have you used ASP.NET Core at all?

I agree it's great but I don't think it's relevant to the discussion. Adding a request queue wouldn't affect the public API in anyway. Callbacks, handlers and controllers are all push based callbacks. You register a method that the server "calls back" so I'm not sure what that has to do with anything.

Everything! It appears your "Push Model" is an integral part of Kestrel as well as ASP.NET Core meaning there is an architectural dependency among the two. IMO that's probably the single main reason you find exposing an HttpListener API to be difficult - it does not "fit" your model - its more general. If the upper level ASP.NET Core abstractions where designed to consume Http Requests from Kestrel you could still provide a callback/controller API to ASP.NET users and it would have been trivial to develop an HttpListener interface for those users who like doing the consuming part themselves.

Everything! It appears your "Push Model" is an integral part of Kestrel as well as ASP.NET Core meaning there is an architectural dependency among the two. IMO that's probably the single main reason you find exposing an HttpListener API to be difficult - it does not "fit" your model - its more general. If the upper level ASP.NET Core abstractions where designed to consume Http Requests from Kestrel you could still provide a callback/controller API to ASP.NET users and it would have been trivial to develop an HttpListener interface for those users who like doing the consuming part themselves.

Honestly it's not insurmountable, it's just code and there are existing patterns for doing this. I was merely explaining the implementation differences that exist today and why I wouldn't build one of top of the other (including the fact that the performance would be worse). Again, I'm not sure what this adds to the discussion about deprecating HttpListener...

Again, I'm not sure what this adds to the discussion about deprecating HttpListener...

If you change Kestrel to support a "Pull Model" there would be no need to Deprecate HttpListener but rather have it co-exit, just evolved...

Maybe the amount of people that seem to like a low level HTTP server that let's you do anything you want with incoming requests should tell you that there are valid use cases for it.
I don't want kestrel in my micro-services. I just had a use case for HTTP requests in the Unity engine, I won't be running Giraffe in it, but a HTTP listener with some async workflows works a treat and does mess around with the main game loop.

I don’t think they don’t recognize valid use-cases, but rather they don’t have a cost metric for the learning effort and keeping up-to date with ASP.NET abstractions compared to HttpListener that is regardless if you can solve the problem using a subset of the ASP.NET API. From my experience the cost is very high when all you need is a familiar interface for doing local/remote IPC.

This thread, IMHO, does not mention the alternative to HttpListener. "Use Kestrel instead" is a bit broad and is not really an alternative:

The Kestrel main page says that it is http server for ASP.Net, which some people don't want to depend on;
The closest thing you can find throughout the tutorials by Microsoft is the creation of an ASP.Net application with no built-in middleware and a single custom middleware layer that provides the single handler. Which is also not an alternative because, on one hand, it raises the abstraction to the level of the whole application and, on the other hand, is a pain to implement (for example, it's a chore to utilize DependencyInjection extension when creating middleware, it works though);

If the class is deprecated, may we have either the clear alternative description or the clear statement that the class is deprecated without the alternative?
I can understand why .Net Core team can be (is) focused on web application scenarios, but the "Use KestrelHttpServer instead" is the same as saying "We can sell you a car" to a person that comes to buy a single wheel.
My two cents: my team does not develop "web applications", the "API surface" term has no meaning for our system, for us any connection is valid, so we need (as some people who commented above) a simple "TcpListener-style" abstraction with HTTP support that can be used locally 1) without raising the functionality to the "application level", 2) without imposing any external "coding doctrines" and 3) without any other features we don't need like model validation, routing, authorization, serialization helpers, and so on.

Something like this?:
https://github.com/davidfowl/BasicKestrel/tree/master/BasicKestrel

@davidfowl the code you've kindly provided shows one approach to the problem which, unfortunately, falls under my point 2.
For those who's interested but didn't do it, the other one is the creation of a single middleware class with InvokeAsync calling a user-provided asynchronous delegate; both are described here.
Your example code depicts the issues (well, not issues per se, but the pain points for the people who need a listener-only kind of abstraction):

Even barebones example introduced the dependencies on ASP.Net which bring a bunch of application level features (for example, the generation of the error pages). Yes, we can easily opt-out of this behavior, but it is still there in our dependency graph;
I have a feeling that in general ASP.Net Core abstractions are meant to be used as dependency roots, not terminal dependencies, while HttlListener naturally can be injected as a terminal dependency.
For example, to provide custom logic for the Run asynchronous callback, the most easy and obvious (IMHO) way to go is to inject application logic into the listener class. When using HttpListener the easy way to go is to inject the listener into the logic. This difference can be hidden under the additional user code, but it's there. It's hard to explain clearly but I hope you understand. This is what I meant when I wrote "it raises the abstraction to the level of the whole application".

Even barebones example introduced the dependencies on ASP.Net Core which bring a bunch of application level features (for example, the generation of the error pages). Yes, we can easily opt-out of this behavior, but it is still there in our dependency graph;

Can you clarify what "still there in the dependency graph" means? I might have an idea but I'd like to understand what added "cost" you are describing.

I have a feeling that in general ASP.Net Core abstractions are meant to be used as dependency roots, not terminal dependencies, while HttlListener naturally can be injected as a terminal dependency.

I don't know what that means, perhaps a code sample would clear things up for me.

For example, to provide custom logic for the Run asynchronous callback, the most easy and obvious (IMHO) way to go is to inject application logic into the listener class. When using HttpListener the easy way to go is to inject the listener into the logic. This difference can be hidden under the additional user code, but it's there. It's hard to explain clearly but I hope you understand. This is what I meant when I wrote "it raises the abstraction to the level of the whole application".

Same as above, show me some sample code and I'll try to provide you what what I think is the alternative. I don't think it raises the level of abstraction to the level of the "whole application", that's just the impression people have because of the history of ASP.NET (tied to IIS etc).

Hello! for anyone new coming to this thread -- I can confirm that I was able to migrate everything with HttpListener to Net Standard using Kestrel some months ago. It required lots of work refactoring but it was possible.

Hello! for anyone new coming to this thread

Yeah since we're updating everyone, I ended up dumping UWP/native altogether and am now on Blazor, which has that WebSocket magic built-in.  This doesn't account for native hosted scenarios just yet but we have that identified and discussed in the Blazor repo so that's good enough for me ATM. 😁
That aside, I have to give everyone on the .NET team so much credit and respect for how .NET has shaped up as of late.  The generic hosting model, EF, Blazor, Asp.NET Core, Razor Components, EVERYTHING!!!  Really, truly, it's like developing back in the days of Silverlight again.  You should unexpectedly and suddenly announce its unexplained deprecation for old times's sake.
(No don't actually do that please).
Bad jokes aside, kudos to everyone there.  I keep meaning to make a blog post about it but random GitHub issue praises is all I can muster BECAUSE I AM DEVELOPING SO HARD THESE DAYS. 😅

@davidfowl

It forces us to write a lot of non essential code that we do not need (see the snippet below).
It also brings a bunch of .dll-s along, and loads some of them in runtime presumably without any real need. For our project this is of no importance though.

I can't provide the real code, but the snippet we had looked more or less like this:

/* somewhere deep inside the calculations */
if (notEnoughData)
{
    try
    {
       var listener = new HttpListener();
       SetupPrefixes(listener);
       listener.Start();
       await OrderNDataPackagesAsync(dataPackagesToOrder);
       foreach (var i in Enumerable.Range(dataPackagesToOrder))
       {
            var context = await listener.GetContextAsync();
            /* some processing and returning 200 OK at the end of the iteration */
        }
    }
    finally
    {
        listener.Close();
    }
}

The code sometimes needs the additional data and it orders it via http call, then receives the required number of packages through the listener, closes the listener and proceeds. It can't get the data as responses to the requests because it's being requested from the system that is out of our control (and it can only push data to the endpoint).
To remove HttpListener I had to (at least I didn't find any better/easier way) host WebApi with a single controller with a single post method, then inject our whole service into the controller and write some safeguards, so we'd skip the data if none was requested. All of it was done only to receive a few data dumps when they are needed (an they are not needed often). The number of lines of code (with all configuration, code in the controller and safeguards) skyrocketed, and now the service looks like some kind of an insane web abomination that processes some requests and ignores the others. That what I was talking about: we don't have a server, nor do we need one, but we're kinda forced to have one.
IMHO, bringing the web server infrastructure (and code to configure it) on board is a bit of an overkill for such a simple task.
Bottom line: we ported the code, but I would not call Kestrel/ASP.Net viable alternative for the reasons I've tried to explain above.

forces us to write a lot of non essential code that we do not need (see the snippet below).
It also brings a bunch of .dll-s along, and loads some of them in runtime presumably without any real need. For our project this is of no importance though.

I think the default template absolutely does that but not the framework itself. There are minimal ways to use the framework that aren't documented as mainstream but exist as public API.

To remove HttpListener I had to (at least I didn't find any better/easier way) host WebApi with a single controller with a single post method, then inject our whole service into the controller and write some safeguards, so we'd skip the data if none was requested.

Right, like I said above, the docs maybe pointed you in that direction but that isn't the case.

All of it was done only to receive a few data dumps when they are needed (an they are not needed often). The number of lines of code (with all configuration, code in the controller and safeguards) skyrocketed, and now the service looks like some kind of an insane web abomination that processes some requests and ignores the others. That what I was talking about: we don't have a server, nor do we need one, but we're kinda forced to have one.

I believe you, you could have written something in about ~20 LOC without services and dependency injection.
The biggest pattern change is the push vs pull API which I agree can be hard if you were doing something like handing 4 requests to then shut the server down.
Here are 2 different implementations:
Push:
using System.Threading;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

class Program
{
    static async Task Main(string[] args)
    {
        var dataPackagesToOrder = 5;
        var processingTask = new TaskCompletionSource<object>(TaskCreationOptions.RunContinuationsAsynchronously);

        var host = new HostBuilder()
                .ConfigureWebHost(webBuilder =>
                {
                    webBuilder.UseKestrel(options => options.ListenLocalhost(8081))
                                .Configure(app =>
                                {
                                    app.Run(async context =>
                                    {
                                        var remaining = Interlocked.Decrement(ref dataPackagesToOrder);

                                        if (remaining < 0)
                                        {
                                            // Nothing to do here
                                            return;
                                        }

                                        // Handle the request here

                                        if (remaining == 0)
                                        {
                                            // Done handling the last package to order, we can shut the server down
                                            processingTask.TrySetResult(null);
                                        }
                                    });
                                });
                })
            .Build();

        await host.StartAsync();

        await processingTask.Task;

        await host.StopAsync();
    }
}

Pull:
```C#
using System;
using System.Threading.Channels;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

class Program
{
    static async Task Main(string[] args)
    {
        var dataPackagesToOrder = 5;
        var requests = Channel.CreateBounded<HttpContext>(dataPackagesToOrder);

        using var host = new HostBuilder()
                .ConfigureWebHost(webBuilder =>
                {
                    webBuilder.UseKestrel()
                              .UseUrls("http://localhost:8081")
                              .Configure(app =>
                              {
                                  app.Run(async context =>
                                  {
                                      // Handle the request here
                                      var tcs = new TaskCompletionSource<object>();

                                      context.SetCompletion(tcs);

                                      await requests.Writer.WriteAsync(context);

                                      await tcs.Task;
                                  });
                              });
                })
            .Build();

        await host.StartAsync();

        for (int i = 0; i < dataPackagesToOrder; i++)
        {
            var context = await requests.Reader.ReadAsync();

            // process request
            // ...

            // complete the request
            context.CompleteRequest();

        }

        // Don't want to handle any more requests, don't do graceful shutdown
        try
        {
            await host.StopAsync(TimeSpan.Zero);
        }
        catch (OperationCanceledException) { }

        // Shut down the requests channel
        requests.Writer.TryComplete();
    }
}

public static class HttpContextExtensions
{
    private static readonly object _completionKey = new object();

    public static void SetCompletion(this HttpContext httpContext, TaskCompletionSource<object> tcs)
    {
        httpContext.Items[_completionKey] = tcs;
    }

    public static void CompleteRequest(this HttpContext httpContext)
    {
        if (httpContext.Items[_completionKey] is TaskCompletionSource<object> tcs)
        {
            tcs.TrySetResult(null);
        }
    }
}

@davidfowl wow, looks great. I was trying to implement something similar using CancellationTokenSource and RunAsync(token) instead of StartAsync/StopAsync pair and Channel<(HttpContext Context, TaskCompletionSource<bool> Handle)> to offload the context from Run, but dropped it halfway =( The idea of using Items to pass completion callbacks is great (never thought about it).
Thank you very much, this looks like a viable alternative to me (a little bit more code than with HttpListener, but much less than with full fledged WebApi controller, so, IMHO, this does not matter much).
Do you think it would be possible in the future to see more examples like this in the official documentation? I guess I'm not the only one who had the one-sided impression of the concept.

I read through this entire thread yesterday, and it has deviated quite a lot from it's original topic.
I'd like to return to the original post for a moment:

System.Net.HttpListener does not support many modern protocols. It is compat-only (i.e. only critical fixes, no new improvements, enhancements). It's useful for low-volume basic server requests, but does not scale to modern requirements of HTTP servers (incl. performance).
Use KestrelHttpServer instead.

Your point is well taken - System.Net.HttpListener does not support modern protocols or requirements for HTTP servers.
I don't think performance is an issue for most of those who protested this issue, who just want to embed a lightweight HTTP server in their applications - that's likely not motivation for them to switch. That's just extra work for something that makes no practical difference to them, and it's obviously not the use-case that Kestrel was designed to satisfy.
Here's what I would propose: refactor and extract a simple, low-level HTTP kernel that the ASP framework can depend on.
It just needs to do what you said: support modern protocols, scale to high performance and modern requirements, and of course it needs to be maintained.
This is the approach taken by most other platforms and languages, including at least Dart, Node, PHP (via PSR-7/15/17 standards) and Ruby - for example, the Rails framework is essentially middleware that can run on different servers.
From an HTTP kernel, all I'd expect is request/response abstractions, a simple abstraction of header names/values, a URI abstraction, and a simple middleware delegate. Anything beyond that can be layered into middleware.
That gives users of HttpListener a more reasonable path of recourse.
And it would probably make the .NET platform more attractive to framework authors - which seems to be an area where the .NET community isn't especially prolific, perhaps because there's a "guilty feeling" associated with abstracting away the entire ASP stack to achieve something perceptibly simpler, or because the barrier of entry is too high.
There's currently no practical way around ASP if you want to build for the web with .NET, and I think this can hamper creativity. I realize there are a few micro-frameworks out there, but for the most part, these depend on the ASP stack, and rather than providing a simpler place to start for people who aren't ready to invest their time in the entire stack, these effectively become just another thing you have to learn.
There's a sense of ASP being almost synonymous with .NET for web development, as there was for Rails and Ruby in it's early days - and this is not to knock on ASP or anything, but lots of people out here have a more minimalist mindset, or just different opinions about what a web framework should be like, and there is currently very little incentive for these people to get onboard and experiment with the .NET platform.
Providing a lightweight HTTP kernel where multiple frameworks can coexist would elmost definitely elevate adoption among those who stick with Node, PHP or Ruby for the simple reason that it gives them more choices; even if it's just the sense of having more choices, that's a very real decision making factor for people who aren't totally sold on ASP.
Just my 2¢ 😊

I agree in theory but not in practice. ASP is such a bad name it’s been an uphill battle to change the feel of that legacy but the fact is frameworks should be embedding kestrel, and building on the asp.net core primitives. It was built with that purpose in mind.
I’m not opposed to having something built in that’s extremely low level, but it would need to be low level for kestrel to be built on top of it. It comes down to effort de-duplication of efforts. Kestrel supports HTTP/1/2/3 currently and doesn’t pay the cost of translating between abstractions (header copying anyone?). That low level ness goes against having nice high level abstractions that library authors can use.
Another option would be to ship 2 APIs, the HTTP kernel and the abstractions or just another higher level set of APIs that ASP.NET Core doesn’t use.

Another option would be to expose a .NET interface to MsQuic along with a .NET HTTP3 parsing lib and finally end this issue.
https://techcommunity.microsoft.com/t5/networking-blog/msquic-is-open-source/ba-p/1345441

Another option would be to expose a .NET interface to MsQuic along with a .NET HTTP3 parsing lib and finally end this issue.

I don't think that would solve the issue for the masses, you need parsers for all the  HTTPs. Also HTTP/2 is sufficiently complex that hard to do well, HTTP/3 is less hard but still non-trivial compared to HTTP/1 (which also has lots of quirks). All of that to say I think that wouldn't solve what's being asked.
Likely in .NET 6, it's internal in 5.0:

https://github.com/dotnet/runtime/tree/master/src/libraries/Common/src/System/Net/Http/aspnetcore/Quic.
https://github.com/dotnet/runtime/tree/master/src/libraries/Common/src/System/Net/Http/aspnetcore/Http3

@davidfowl I think maybe you misunderstand me.
Of course, yes, we would want the transport independence between HTTP 1/2/3 - other platforms afaik have successfully upgraded their kernels to HTTP 3 without breaking anything. What all HTTP versions have in common is headers, methods, status codes, URIs, request and response, etc. - that's as "low level" as I'm talking about. Extracting that to a separate layer/package ought to be feasible, as it is in other languages, why not?
As soon as you get to the layer where, for example, all the known header types are abstracted as objects, that's where things cross over into opinionated framework territory. It's not essential to abstracting the transport layer, or to interoperability between frameworks, for example - it's just ASP's opinions about how headers should be done.
Building that on top of kernel layer where headers are just name and values represented as strings ought to be completely feasible, since ultimately they have to be strings to get trabsported, right? HTTP kernels in other languages to my knowledge all represent headers as strings.
Just one example, but I don't think there's any technical reason why there can't be a lower level of abstraction that has only an essential degree of abstraction, e.g. request/response and a simple request->response delegate. (or context object, since likely you'd have to stick with the idea of mutable request/response instances for the sake of performance and compatibility with the high-level middleware abstraction in ASP.)
I don't think a separate, incompatible package is the way to go. That'd almost be worse than the status quo. Might as well just leave HttpListener in place then, since it already seems to meet the humble requirements of those who commented on this thread.

I don't misunderstand you.

Of course, yes, we would want the transport independence between HTTP 1/2/3 - other platforms afaik have successfully upgraded their kernels to HTTP 3 without breaking anything. What all HTTP versions have in common is headers, methods, status codes, URIs, request and response, etc. - that's as "low level" as I'm talking about. Extracting that to a separate layer/package ought to be feasible, as it is in other languages, why not?

Sure, that's not low level enough to control the types of performance optimizations that happen today in ASP.NET Core. Trust me we've done this before with OWIN which built further abstractions on top of HttpListener. In fact we still do this today with a ASP.Core. We gutting the HttpListener implementation on windows to directly integrate with the ASP.NET Core abstractions (https://github.com/dotnet/aspnetcore/tree/master/src/Servers/HttpSys/src).
All that said, it's completely reasonable to build this high level API. There would just need be something lower that ASP.NET Core could build on top of.

As soon as you get to the layer where, for example, all the known header types are abstracted as objects, that's where things cross over into opinionated framework territory. It's not essential to abstracting the transport layer, or to interoperability between frameworks, for example - it's just ASP's opinions about how headers should be done.

So HttpListener is already too high level? https://docs.microsoft.com/en-us/dotnet/api/system.net.httplistenerresponse?view=netcore-3.1#properties. Even the cookies are represented as a CookieCollection. I don't buy your definition of what makes an opinioned framework.

Building that on top of kernel layer where headers are just name and values represented as strings ought to be completely feasible, since ultimately they have to be strings to get trabsported, right? HTTP kernels in other languages to my knowledge all represent headers as strings.

Should be bytes not strings, that the kernel I need.

I don't think a separate, incompatible package is the way to go. That'd almost be worse than the status quo. Might as well just leave HttpListener in place then, since it already seems to meet the humble requirements of those who commented on this thread.

I'd be fine not deprecating it, but it's not going to get any new features.

So HttpListener is already too high level?

No, that's totally fine - of course we should have a collection of cookies that makes them manageable.
Where it gets opinionated is where ASP also abstracts individual types of cookies - something that can easily be built on top of a simple cookie collection.

All that said, it's completely reasonable to build this high level API. There would just need be something lower that ASP.NET Core could build on top of.

This makes me think we're not really in disagreement here. 🙂
I didn't mean for that short list to be exhaustive - I only omitted cookies because I forgot. Things like sessions, authentication, and so on are just cookies being exchanged though, and all that doesn't need to exist in an HTTP kernel layer - you could comfortably and efficiently build those things on top of a lightweight kernel with a simple cookie collection, right?

Where it gets opinionated is where ASP also abstracts individual types of cookies - something that can easily be built on top of a simple cookie collection.

It does? Clue me in please? Which ones? Is it on the abstraction?

I didn't mean for that short list to be exhaustive - I only omitted cookies because I forgot. Things like sessions, authentication, and so on are just cookies being exchanged though, and all that doesn't need to exist in an HTTP kernel layer - you could comfortably and efficiently build those things on top of a lightweight kernel with a simple cookie collection, right?

The kernel I want wouldn’t have headers as strings or cookies exposed. Nor would it represent the body as a stream. That’s the core of why making another abstraction is a no go. If you can reasonably use the guts of the kernel somehow then I’m all ears. Having spent the last 5 years of my life working on an HTTP server I have some strong opinions on how to build them 😬

Just having the same functionality as the current HTTP listener does would be great. Scratch the stuff that is outdated ( like the cert handling etc. etc. ) and I would be more than happy. :)

Where it gets opinionated is where ASP also abstracts individual types of cookies - something that can easily be built on top of a simple cookie collection.

It does? Clue me in please? Which ones? Is it on the abstraction?

There's a whole bunch of classes to deal with many common header types:
https://github.com/dotnet/aspnetcore/tree/master/src/Http/Headers/src

There's a whole bunch of classes to deal with many common header types:

Those aren't even part of the abstractions. They're optional and were made intentionally so as a convenience to other framework authors (and higher level applications).

@davidfowl okay, sorry about the bad examples - tbh I'm returning to .NET after 7 years away, and I don't know the current state of the platform very well yet. I'm only noticing, as somebody who's worked with several other languages/platforms, that the API surface area of ASP is much bigger and much more complex than basically anything else I've worked with. I wish there was a much smaller and simpler kernel layer that was easier to learn and more suitable for smaller/simpler tasks, embedding, etc. - but I guess I can't say what that would entail in concrete terms.
You would definitely have a much better idea than me as to what would be possible and meaningful. The problem is real enough though, so I do hope this is something that improves in the future. I want to be able to tell a friend, "try this", rather than "read this book and spend the next couple of weeks learning this", you know? I think the main reason Node is so popular, for example, is because it's easy to get on board. If you know HTTP, you'll figure out middleware in a couple of hours. It should be possible to design something that gives new devs a more immediate sense of security - "this works and I understand why". I don't think most people get that with ASP right now.

Point your friends here https://github.com/featherhttp/tutorial

@davidfowl I think I'll point myself there! 😀 ... this looks nice and clean. I'm also aware of Nancy in this space. They are of course both abstracting away a large, complex framework - but I guess, in the same sense, PSR standards abstract away PHP's outmoded CGI API, and I was quite happy with that, so this might be the direction I'm looking for. Thanks! 🙂


[FEATURE REQUEST] Add possibility to specify SSH port in Epiphany configuration

Is your feature request related to a problem? Please describe.
In some cases we want to use different than standard 22 port for SSH. E.g. in some environments SSH is exposed via NAT on non-standard port.
Describe the solution you'd like
We want to have ability to specify port in machine configuration and expose SSH on non-standard port.
Describe alternatives you've considered
We have workaround for this. We can specify ip in specification section of kind: infrastructure/machine like:
ip: <IP_ADDRESS> ansible_port=47901
Additional context
None.

DoD checklist

Changelog

[ ] updated
[ ] not needed

COMPONENTS.md

[ ] updated
[ ] not needed

Schema

[ ] updated
[ ] not needed

Backport tasks

[ ] created
[ ] not needed

Documentation

[ ] added
[ ] updated
[ ] not needed

[ ] Feature has automated tests
[ ] Automated tests passed (QA pipelines)

[ ] apply
[ ] upgrade
[ ] backup/restore

[ ] Idempotency tested
[ ] All conversations in PR resolved
[ ] Solution meets requirements and is done according to design doc
[ ] Usage compliant with license

@erzetpe @rafzei can you describe the reason we want this? :)

@erzetpe @rafzei can you describe the reason we want this? :)

Because we had users that were requiring non-standard configuration because of using NAT connection to their environment with SSH ports exposed on different ports.

Because we had users that were requiring non-standard configuration because of using NAT connection to their environment with SSH ports exposed on different ports.

Not going todo anymore feature requests for Epiphany.


Notify developers when devcontainer is changed to rebuild workspace

We want to add notifications into our product. We should wait until this is complete before announcing v1.0 and leverage it to notify users when the devcontainer changed and prompt them to rebuild their workspace.

This may need an RFC to flesh out exactly how we want to handle the envbuilder -> coder integration, given that we want to avoid direct integrations when possible.

yeah, let's pause on this until we have generic notifications support


Importer crashes: linking with cc failed

I'm trying to import a new city and run ./import.sh --oneshot=
The importer crashes on "Compiling importer".
macOS 10.14.6
Rust 1.44.0
   Compiling importer v0.1.0 (/Users/jerry/Downloads/abstreet-master/importer)
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-m64" "-L" "/usr/local/Cellar/rust/1.44.0/lib/rustlib/x86_64-apple-darwin/lib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/importer-09d22dcce49f6b04.importer.5x3omg2d-cgu.0.rcgu.o" "/Users/jerry/Downloads/abstreet-master/target/release/deps/importer-09d22dcce49f6b04.importer.5x3omg2d-cgu.1.rcgu.o" "/Users/jerry/Downloads/abstreet-master/target/release/deps/importer-09d22dcce49f6b04.importer.5x3omg2d-cgu.10.rcgu.o" "/Users/jerry/Downloads/abstreet-master/target/release/deps/importer-09d22dcce49f6b04.importer.5x3omg2d-cgu.11.rcgu.o" "/Users/jerry/Downloads/abstreet-master/target/release/deps/importer-09d22dcce49f6b04.importer.5x3omg2d-cgu.12.rcgu.o" "/Users/jerry/Downloads/abstreet-master/target/release/deps/importer-09d22dcce49f6b04.importer.5x3omg2d-cgu.13.rcgu.o" "/Users/jerry/Downloads/abstreet-master/target/release/deps/importer-09d22dcce49f6b04.importer.5x3omg2d-cgu.14.rcgu.o" "/Users/jerry/Downloads/abstreet-master/target/release/deps/importer-09d22dcce49f6b04.importer.5x3omg2d-cgu.15.rcgu.o" "/Users/jerry/Downloads/abstreet-master/target/release/deps/importer-09d22dcce49f6b04.importer.5x3omg2d-cgu.2.rcgu.o" "/Users/jerry/Downloads/abstreet-master/target/release/deps/importer-09d22dcce49f6b04.importer.5x3omg2d-cgu.3.rcgu.o" "/Users/jerry/Downloads/abstreet-master/target/release/deps/importer-09d22dcce49f6b04.importer.5x3omg2d-cgu.4.rcgu.o" "/Users/jerry/Downloads/abstreet-master/target/release/deps/importer-09d22dcce49f6b04.importer.5x3omg2d-cgu.5.rcgu.o" "/Users/jerry/Downloads/abstreet-master/target/release/deps/importer-09d22dcce49f6b04.importer.5x3omg2d-cgu.6.rcgu.o" "/Users/jerry/Downloads/abstreet-master/target/release/deps/importer-09d22dcce49f6b04.importer.5x3omg2d-cgu.7.rcgu.o" "/Users/jerry/Downloads/abstreet-master/target/release/deps/importer-09d22dcce49f6b04.importer.5x3omg2d-cgu.8.rcgu.o" "/Users/jerry/Downloads/abstreet-master/target/release/deps/importer-09d22dcce49f6b04.importer.5x3omg2d-cgu.9.rcgu.o" "-o" "/Users/jerry/Downloads/abstreet-master/target/release/deps/importer-09d22dcce49f6b04" "/Users/jerry/Downloads/abstreet-master/target/release/deps/importer-09d22dcce49f6b04.316v6e5bry5st743.rcgu.o" "-Wl,-dead_strip" "-nodefaultlibs" "-L" "/Users/jerry/Downloads/abstreet-master/target/release/deps" "-L" "/Users/jerry/Downloads/abstreet-master/target/release/build/backtrace-sys-3854c3a9dd99d460/out" "-L" "/usr/local/Cellar/rust/1.44.0/lib/rustlib/x86_64-apple-darwin/lib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libgdal-8ad036ccd0b82075.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libgeo_types-680ed845e1fd374c.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libgdal_sys-a60a4052a6fbd169.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libfailure-04f94679277c563d.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libbacktrace-6fd4181b0d87c9e5.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libbacktrace_sys-1aa0b0903f552039.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/librustc_demangle-2c8ac167b7c8e698.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libconvert_osm-6792e461292da6b5.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libosm_xml-ec14f9e5856eb7ac.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libxml-6218d255b4c1eecd.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libbitflags-5ffbffbe2f5aec26.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libkml-50bdfff515795d67.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libquick_xml-4563d72bfe5b897a.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libsim-ec398a8d340eddb5.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libdowncast_rs-333660cdac089e92.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/librand_distr-10bd5df56a8d68e9.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libmap_model-b6b1549fbde2dba7.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libseattle_traffic_signals-9a08748ef8113da8.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libinclude_dir-2da57b61f61e82a1.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libglob-51752d0e452a101f.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libthread_local-d4db6aa80aedc62c.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libfast_paths-2a33482fff7b4f2b.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libpriority_queue-e5187a0964baff5d.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libindexmap-bf227ceba561c83b.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/liblog-1f75497f6752c3e8.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libnbez-23f99bffd57b1673.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libnum_traits-6c05195de7346553.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libgtfs-4f8c0bb7fe04b06a.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libtransitfeed-f7fa4873166c9969.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libtempfile-b708c651ec277e3e.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libremove_dir_all-f8a152f32ed69f6d.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libzip-22e6c568fc7095ab.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libcrc32fast-95ddcef450d0846e.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libpodio-96e15d408c451a85.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libcsv-01f949b7a2ac34ae.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libcsv_core-7594c7808b89b466.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libbstr-704e100f42155f83.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libregex_automata-afe07969cfb1bd80.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libmemchr-e1b5595a57f9cd6b.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/liblazy_static-cd1c7ba47b13de29.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libchrono-61b055b5600cf2c3.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libnum_integer-9fb17e4c50aeff9c.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libtime-f090054f32151bc1.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libpetgraph-8e1a64a73718db14.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libindexmap-3c7a656a99b4e402.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libfixedbitset-ab39d371cc74336c.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libgeom-08963bd9c4cec570.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libhistogram-8c09abe56669739b.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libpolylabel-807953a0100dc69a.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libthiserror-8ba786a44fcffea9.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libgeo_booleanop-62ecb3df628bdf05.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/librobust-e5b83f8ca579d463.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libfloat_next_after-661784340df95b78.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libordered_float-da54dc55c704d082.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libgeo-cb8d4de9e60c22e1.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libgeo_types-403930e97f542927.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/librstar-9d631eb10d0b7794.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libpdqselect-2e6254f1ae017b23.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libnum_traits-4b5f679b7719be40.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libabstutil-48f6ae4fb41414a7.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libtermion-a11e74bbcc499189.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libnumtoa-85060ac9e102699e.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libnum_cpus-23c10b6f28a495b0.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libscoped_threadpool-bc0cd311cde6ec55.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/librand_xorshift-84c3cbb7fea26b9e.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/librand-f8952740ffbbba35.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/librand_chacha-578aceff9adc4c55.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libppv_lite86-b22321c8404a84fe.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/librand_core-80a163aedc210aa2.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libgetrandom-41921966cd0c7028.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/liblibc-e6919b089a507cc9.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libcfg_if-2b894764c7634612.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libserde_json-eb619fd1320d127b.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libryu-4cc52f39a5601932.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libitoa-41cc1e68a1ad6bb6.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libinstant-966b1afa4dcfed25.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libbincode-2568fbf5f139a3c9.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libbyteorder-608ce6d3d88950f7.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libitertools-4a61895d1dcc70fb.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libeither-12db6fc10ab1b05a.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libaabb_quadtree-87a46e44cab3ee4a.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libfnv-f163f5a8e10ef210.rlib" "/Users/jerry/Downloads/abstreet-master/target/release/deps/libserde-8df9b1b67da14074.rlib" "/usr/local/Cellar/rust/1.44.0/lib/rustlib/x86_64-apple-darwin/lib/libstd-3db50325635fcb92.rlib" "/usr/local/Cellar/rust/1.44.0/lib/rustlib/x86_64-apple-darwin/lib/libpanic_unwind-fc8a847d3c2a5f7c.rlib" "/usr/local/Cellar/rust/1.44.0/lib/rustlib/x86_64-apple-darwin/lib/libhashbrown-2820d85ebe274c06.rlib" "/usr/local/Cellar/rust/1.44.0/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_alloc-5f9b1fe1aaaddf02.rlib" "/usr/local/Cellar/rust/1.44.0/lib/rustlib/x86_64-apple-darwin/lib/libbacktrace-7b47a519c4e1797e.rlib" "/usr/local/Cellar/rust/1.44.0/lib/rustlib/x86_64-apple-darwin/lib/libbacktrace_sys-4dc6a6ade3375394.rlib" "/usr/local/Cellar/rust/1.44.0/lib/rustlib/x86_64-apple-darwin/lib/librustc_demangle-bf0c42eb5925bf4f.rlib" "/usr/local/Cellar/rust/1.44.0/lib/rustlib/x86_64-apple-darwin/lib/libunwind-399f646292278309.rlib" "/usr/local/Cellar/rust/1.44.0/lib/rustlib/x86_64-apple-darwin/lib/libcfg_if-3f375bac4d418cc0.rlib" "/usr/local/Cellar/rust/1.44.0/lib/rustlib/x86_64-apple-darwin/lib/liblibc-7b5393163b4ce1c2.rlib" "/usr/local/Cellar/rust/1.44.0/lib/rustlib/x86_64-apple-darwin/lib/liballoc-9df9d56d9f312d5b.rlib" "/usr/local/Cellar/rust/1.44.0/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_core-45cc4e944387083e.rlib" "/usr/local/Cellar/rust/1.44.0/lib/rustlib/x86_64-apple-darwin/lib/libcore-7b640d8d174d1215.rlib" "/usr/local/Cellar/rust/1.44.0/lib/rustlib/x86_64-apple-darwin/lib/libcompiler_builtins-1dccb424fc93c3f4.rlib" "-lSystem" "-lresolv" "-lc" "-lm"
  = note: Undefined symbols for architecture x86_64:
            "_CPLGetLastErrorMsg", referenced from:
                gdal::utils::_last_cpl_err::h6df09bb5bc5814b2 in libgdal-8ad036ccd0b82075.rlib(gdal-8ad036ccd0b82075.gdal.yksoymak-cgu.13.rcgu.o)
                gdal::utils::_last_null_pointer_err::h64266ebe9d459bd5 in libgdal-8ad036ccd0b82075.rlib(gdal-8ad036ccd0b82075.gdal.yksoymak-cgu.13.rcgu.o)
            "_OSRGetAuthorityCode", referenced from:
                gdal::spatial_ref::srs::SpatialRef::authority::h153d89d86b62882e in libgdal-8ad036ccd0b82075.rlib(gdal-8ad036ccd0b82075.gdal.yksoymak-cgu.2.rcgu.o)
            "_OCTNewCoordinateTransformation", referenced from:
                gdal::spatial_ref::srs::CoordTransform::new::h6589c9dab2749d88 in libgdal-8ad036ccd0b82075.rlib(gdal-8ad036ccd0b82075.gdal.yksoymak-cgu.2.rcgu.o)
            "_OSRGetAuthorityName", referenced from:
                gdal::spatial_ref::srs::SpatialRef::authority::h153d89d86b62882e in libgdal-8ad036ccd0b82075.rlib(gdal-8ad036ccd0b82075.gdal.yksoymak-cgu.2.rcgu.o)
            "_OSRImportFromEPSG", referenced from:
                gdal::spatial_ref::srs::SpatialRef::from_epsg::h98cfedea6d7a2f89 in libgdal-8ad036ccd0b82075.rlib(gdal-8ad036ccd0b82075.gdal.yksoymak-cgu.2.rcgu.o)
            "_OSRRelease", referenced from:
                _$LT$gdal..spatial_ref..srs..SpatialRef$u20$as$u20$core..ops..drop..Drop$GT$::drop::h995b186f837728f8 in libgdal-8ad036ccd0b82075.rlib(gdal-8ad036ccd0b82075.gdal.yksoymak-cgu.2.rcgu.o)
            "_OCTTransform", referenced from:
                gdal::spatial_ref::srs::CoordTransform::transform_coords::h74b97bb82328eb28 in libgdal-8ad036ccd0b82075.rlib(gdal-8ad036ccd0b82075.gdal.yksoymak-cgu.2.rcgu.o)
            "_CPLErrorReset", referenced from:
                gdal::utils::_last_cpl_err::h6df09bb5bc5814b2 in libgdal-8ad036ccd0b82075.rlib(gdal-8ad036ccd0b82075.gdal.yksoymak-cgu.13.rcgu.o)
                gdal::utils::_last_null_pointer_err::h64266ebe9d459bd5 in libgdal-8ad036ccd0b82075.rlib(gdal-8ad036ccd0b82075.gdal.yksoymak-cgu.13.rcgu.o)
            "_OSRNewSpatialReference", referenced from:
                gdal::spatial_ref::srs::SpatialRef::from_epsg::h98cfedea6d7a2f89 in libgdal-8ad036ccd0b82075.rlib(gdal-8ad036ccd0b82075.gdal.yksoymak-cgu.2.rcgu.o)
                gdal::spatial_ref::srs::SpatialRef::from_proj4::hcd8948a16c938dea in libgdal-8ad036ccd0b82075.rlib(gdal-8ad036ccd0b82075.gdal.yksoymak-cgu.2.rcgu.o)
            "_CPLGetLastErrorNo", referenced from:
                gdal::utils::_last_cpl_err::h6df09bb5bc5814b2 in libgdal-8ad036ccd0b82075.rlib(gdal-8ad036ccd0b82075.gdal.yksoymak-cgu.13.rcgu.o)
            "_OSRImportFromProj4", referenced from:
                gdal::spatial_ref::srs::SpatialRef::from_proj4::hcd8948a16c938dea in libgdal-8ad036ccd0b82075.rlib(gdal-8ad036ccd0b82075.gdal.yksoymak-cgu.2.rcgu.o)
            "_OSRExportToProj4", referenced from:
                gdal::spatial_ref::srs::CoordTransform::new::h6589c9dab2749d88 in libgdal-8ad036ccd0b82075.rlib(gdal-8ad036ccd0b82075.gdal.yksoymak-cgu.2.rcgu.o)
            "_OCTDestroyCoordinateTransformation", referenced from:
                _$LT$gdal..spatial_ref..srs..CoordTransform$u20$as$u20$core..ops..drop..Drop$GT$::drop::h2abe32dc209b6bd5 in libgdal-8ad036ccd0b82075.rlib(gdal-8ad036ccd0b82075.gdal.yksoymak-cgu.2.rcgu.o)
          ld: symbol(s) not found for architecture x86_64
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

error: aborting due to previous error

error: could not compile `importer`.

To learn more, run the command again with --verbose.

https://github.com/dabreegster/abstreet/blob/master/docs/dev.md#building-map-data
Do you have the gdal dev library installed? https://trac.osgeo.org/gdal/wiki/BuildingOnMac says it's in brew

Do you have the gdal dev library installed?

Not before you've mentioned it. But after installing gdal it's still the same issue.
Do I need osmconvert? I can't find a macOS version.

Not before you've mentioned it. But after installing gdal it's still the same issue.

I'll look through the issues on the rust gdal bindings. I'm not really familiar with how it tries to find the library to link. In the meantime, you could remove --features scenarios from import.sh to get rid of this dependency. GDAL is only needed to import some Seattle-specific scenario data, which is unfortunately expressed in some weird coordinate system that's hard to translate to lat,lon without an extra library. I'll update the new_city docs with this tip.

Do I need osmconvert? I can't find a macOS version.

Not necessarily; you could change the osmconvert method in importer/src/utils.rs to use osmosis or some other tool that'll clip OSM files. osmconvert is the fastest tool I've found, though. Indeed weird that the wiki doesn't list a Mac binary. It's maybe in brew; does the last command here work for you?

It's maybe in brew; does the last command here work for you?

Yes, thanks. It installed.

you could remove --features scenarios from import.sh

That worked. I'm now stuck at https://github.com/dabreegster/abstreet/issues/119
Thanks.


i18n(zh-cn): Update ui.ts & nav.ts

Description (required)
Update ui.ts & nav.ts
Related issues & labels (optional)

#9182
Suggested label: i18n

Lunaria Status Overview
🌕 This pull request will trigger status changes.

Learn more
By default, every PR changing files present in the Lunaria configuration's files property will be considered and trigger status changes accordingly.
You can change this by adding one of the keywords present in the ignoreKeywords property in your Lunaria configuration file in the PR's title (ignoring all files) or by including a tracker directive in the merged commit's description.

Tracked Files

Locale
File
Note

zh-cn
nav.ts
Localization changed, will be marked as complete.

zh-cn
ui.ts
Localization changed, will be marked as complete.

Warnings reference

Icon
Description

🔄
The source for this localization has been updated since the creation of this pull request, make sure all changes in the source have been applied.


HOLD: Another sharing approach

Short version: this removes replaces most of the logic around "shared" content, and instead adds a new app for migrating content between sites in a single wagtail instance.
getting started
To try it: copy your homepage (probably called 'CFGOV'), and call the new one something like 'www'. Don't check the boxes.

configure your sites to use localhost:8000 and content.localhost:8000, and configure 'localhost' to point to the new root page.

In settings, select "Workflow Destination Settings"

Configure content.localhost:8000 (select the site from the right corner") and set the destination to 'localhost:8000)

Using it
If you've set things up correctly, http://content.localhost:8000/cfpb-ombudsman/ should work, while http://localhost:8000/cfpb-ombudsman/ 404's. Let's fix that.
Go to http://content.localhost:8000/cfpb-ombudsman/  and you should see the wagtail 'user bar' (the little bird in the bottom right corner). If you don't,  go log in at http://content.localhost:8000/admin/ and then come back. Click the bird, and you'll see a new link for "Migrate to localhost:8000 [default]". Click it.
This page tells you what's going to happen.

Click "Do it!" to copy the page. This will leave you looking at the wagtail explorer, a level above the page you just created or updated

Confirm that http://localhost:8000/cfpb-ombudsman/ now works
What works
You can use this to publish or update pages from the "content" site to the "www" site, as long as the parent page exists. In other words, you would need to push '/foo'/ before '/foo/bar/'. If you want to play around with blog posts, you'll need to manually copy the 'About Us' page using wagtail admin, since it's a sheer powered page and doesn't expose the userbar.
What's missing

Akamai integration is assumed broken here
RIght now, we refuse to update a page if the source and destination are different page types
there's no bulk operations, or strategy for migrating the whole site to the new approach.
haven't investigated how this interacts with scheduled publishing
Everything is based on paths right now-- it you published /foo/bar, changed it to /foo/baz, and pushed again, both /foo/bar and /foo/baz would exist, which probably isn't what we actually want.

I don't think any of those things would be hard to fix, they just didn't seem necessary for proving the concept.
(I have more to say, but have to leave, so I'm creating the PR)

closing this!


Ays #89 Documentation for blueowncloud

Initial documentation describes what should it do and the schema required.

@xmonader can you also add:

where to find the g8os-test-1604 image
what will happen if you don't specify any VDC
clarification on the domain ("site name for generating configurations for nginx"), since as far as I understand no DNS record will be created automatically... What exactly does the actor do with this info, and what steps are still required in order to have access owncloud using the specified domain name
point to the disk.ovc actor template when talking about datadisks
an example blueprint

Also see the exemplary documentation written by @muhamadazmy : https://github.com/Jumpscale/ays_jumpscale8/blob/master/app/s3/REAME.md


fix: possible race condition leading to 404s

This PR resolves a "unknown webhook" error message appearing due possible to a race condition. This error has received multiple complaints over at the DDevs server, though they are not using this library. The current speculation is that the error occurs due to a race condition, where the PATCH is processed by Discord before the initial interaction reply.
I have received hundreds of 404 errors over a year and so far have received none after implementing this fix in my own fork for a week. Only thing I noticed was that average ping times have gone up by around 200ms (servers in Germany), but at least the implementation is correct.
Relevent Node.js documentation: https://nodejs.org/api/http.html#responseenddata-encoding-callback
Event: 'close'
Indicates that the response is completed, or its underlying connection was terminated prematurely (before the response completion).
Event: 'finish'
Emitted when the response has been sent. More specifically, this event is emitted when the last segment of the response headers and body have been handed off to the operating system for transmission over the network. It does not imply that the client has received anything yet.
response.end([data[, encoding]][, callback])
If callback is specified, it will be called when the response stream is finished.

This requires @kyranet's review

Released<EMAIL_ADDRESS>


[Event Request] Codeunit 10753 “SII Job Upload Pending Docs."

There are four events in the Codeunit 10753 “SII Job Upload Pending Docs." that we need to change the first parameter of type record as VAR:
Change:
LOCAL [IntegrationEvent] OnBeforeCreateSIIRequestForVendorLedgEntry(VendorLedgerEntry : Record "Vendor Ledger Entry";VAR IsHandled : Boolean)
LOCAL [IntegrationEvent] OnBeforeCreateSIIRequestForCustLedgEntry(CustLedgerEntry : Record "Cust. Ledger Entry";VAR IsHandled : Boolean)
LOCAL [IntegrationEvent] OnBeforeCreateSIIRequestForDtldVendLedgEntry(DetailedVendorLedgEntry : Record "Detailed Vendor Ledg. Entry";VAR IsHandled : Boolean)
LOCAL [IntegrationEvent] OnBeforeCreateSIIRequestForDtldCustLedgEntry(DetailedCustLedgEntry : Record "Detailed Cust. Ledg. Entry";VAR IsHandled : Boolean)
As:
LOCAL [IntegrationEvent] OnBeforeCreateSIIRequestForVendorLedgEntry(VAR VendorLedgerEntry : Record "Vendor Ledger Entry";VAR IsHandled : Boolean)
LOCAL [IntegrationEvent] OnBeforeCreateSIIRequestForCustLedgEntry(VAR CustLedgerEntry : Record "Cust. Ledger Entry";VAR IsHandled : Boolean)
LOCAL [IntegrationEvent] OnBeforeCreateSIIRequestForDtldVendLedgEntry(VAR DetailedVendorLedgEntry : Record "Detailed Vendor Ledg. Entry";VAR IsHandled : Boolean)
LOCAL [IntegrationEvent] OnBeforeCreateSIIRequestForDtldCustLedgEntry(VAR DetailedCustLedgEntry : Record "Detailed Cust. Ledg. Entry";VAR IsHandled : Boolean)

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


HTTP Server return error

I test http server with knative.
ubuntu@ubuntu:~$ time curl -H "Host: http-wasm.default.<IP_ADDRESS>.nip.io" -d "name=HelloWorld" http://localhost:32234
Error(TrackableError { kind: UnexpectedEos, cause: Some(Cause("assertion failed: `remaining >= required`; remaining=0, required=15")), history: History([Location { module_path: "bytecodec::combinator", file: "/home/hydai/.cargo/registry/src/github.com-1ecc6299db9ec823/bytecodec-0.4.15/src/combinator.rs", line: 729, message: "" }, Location { module_path: "httpcodec::body", file: "/home/hydai/.cargo/registry/src/github.com-1ecc6299db9ec823/httpcodec-0.2.3/src/body.rs", line: 237, message: "" }, Location { module_path: "httpcodec::message", file: "/home/hydai/.cargo/registry/src/github.com-1ecc6299db9ec823/httpcodec-0.2.3/src/message.rs", line: 67, message: "" }, Location { module_path: "httpcodec::request", file: "/home/hydai/.cargo/registry/src/github.com-1ecc6299db9ec823/httpcodec-0.2.3/src/request.rs", line: 153, message: "" }, Location { module_path: "bytecodec::decode", file: "/home/hydai/.cargo/registry/src/github.com-1ecc6299db9ec823/bytecodec-0.4.15/src/decode.rs", line: 448, message: "" }]) })
real	0m0.024s
user	0m0.008s
sys	0m0.004s

Image id is docker.io/wasmedge/example-wasi-http@sha256:14e3454b240ff1104222f349eaddab26b07f2ae086d80bb0f84e5b0d03dc8242

Hi,
Have you tried the wasmedge CLI? Or is it only failing at knative?
In the knative environment, how do you run the Wasm file? In a Linux container? In crun-wasm? In containerd? Thanks!
Also, I think perhaps a better HTTP server example to try is this one:
https://github.com/WasmEdge/wasmedge_hyper_demo/tree/main/server
Thanks.


The program gets stuck at acceptedSocket.Wait(); (Telegraph.gen.cs#L63)

Hello,
I tested your API by running the TestMonocle source code, but it doesn't work.
The program gets stuck at acceptedSocket.Wait(); (https://github.com/egomeh/anno1800-scripting-api/blob/e3e5ab90a63dcb8abac8d51b8c37deab519e31c1/Monocle/Telegraph.gen.cs#L63).
I also got something that looks strange to me, the address returned by IPAddress "ipAddress = ipHostInfo.AddressList[0];" (https://github.com/egomeh/anno1800-scripting-api/blob/e3e5ab90a63dcb8abac8d51b8c37deab519e31c1/Monocle/Telegraph.gen.cs#L30) is ":::1", if I replace the line with "ipAddress = ipHostInfo. AddressList[1]", I get my local ip address as "<IP_ADDRESS>"
Thanks in advance,
Have a good day

Hello,
I have news, the program no longer blocks at "acceptedSocket.Wait();" (https://github.com/egomeh/anno1800-scripting-api/blob/e3e5ab90a63dcb8abac8d51b8c37deab519e31c1/Monocle/Telegraph.gen.cs#L63). Randomly, I didn't make any changes except changing the ip from 0 to 1, but it didn't work yesterday with this change.
However, it gets stuck on telegraph.GetAllAreas(out areas); (https://github.com/egomeh/anno1800-scripting-api/blob/e3e5ab90a63dcb8abac8d51b8c37deab519e31c1/TestMonocle/Program.cs#L10). As if he was getting no response.
Thanks in advance,
Have a nice day.

Thank you for opening this issue :)
I think there are a few things at play here that I'll try to address.
The first issue, I don't know. I have not seen that myself but I can think of a few gotchas in the setup that might cause some unexpected halts. I'll try to fix those as well.
The problem of being stuck on GetAllAreas is simply because the game was updated two days ago and I have not yet had the time to patch up the code.
When I've addressed both issues, I'll let you know. The latter should be a matter of a few days.

Hello,
Thank you for your responsive response.
I am really sorry about this long text, As we say in French : "Oh ! Pavé César !"
Unscheduled stops
What's surprising is that the stops (before "GetAllAreas") don't happen anymore, I don't know if it's a matter of probability or if there's a reason behind it. However, before modifying the local IP, I made other modifications and tests which may have also caused deadlocks again at this level, once a new project was set up and only the IP was corrected, this seemed to work, the reason is surely there.
Local IP problem
To solve the IP problem, can I suggest you to use a code that analyzes all available local IPs (in the table) and matches them with a regex to take the first one that seems correct
"from a point of view" syntactic ? (:::1 -> incorrect, <IP_ADDRESS> -> correct), if you wish, I volunteer to help you with this and to do this.
Blocking at GetAllAreas
It seemed to me that the update could be the cause of the blocking at GetAllAreas.
On this subject, I would like to know, if possible, how do you do to determine the functions of the game so efficiently (through the assembly code I imagine?) as well as the memory addresses.
For example, a line in assembler can easily make one think of a "return" in programming, which signifies the end of a function, but how to find the beginning of a function?
How do you determine which variables come into play, what exactly does this function do, etc?
I also noticed that the memory addresses of the game variables change regularly, especially after restarting it, does this not do the same thing with certain functions?
Collaboration
Finally, I would like to know if you would allow me to provide you with my assistance, to help you with about this update of the game and find a new solutions ? If so, do not hesitate to ask me anything and especially if you would agree to let me join your team in order to help you develop your project.
I'm not an expert but I have very varied knowledge that can always be useful (memory, pointers, functions, C#, assembler, c/c++, python, java, ...)
Thanks in advance,
Have a good day.

Hello again,
Thank you for your reactive and exhaustive answer which allowed me to see much more clearly.
Unscheduled stops
It's true that the code may seem a bit "hard"' at first but you understand very quickly especially when you are used to reading code. What slowed me down a bit is that my areas of expertise are mainly concentrated around Java and C/C++, although I have a rudimentary knowledge of C# (finally, a lot of languages are alike, it's not so difficult to understand new ones), I don't know the specific ways of doing things, libraries, conventions etc, but with a little time, reading and Console.WriteLine(... ) can be understood very quickly.
However, I don't think I fell into a "trap", I just had modified the IP and it seemed to work after recreating a clean project with this modification.
Blocking at GetAllAreas
No problem, take your time to understand where the problem comes from, thank you for your work.
How do you do to determine the functions
Thank you for all your answers which enlightens me a lot. It's true that I hadn't thought of it, but between one return and another, there is most likely a function hiding underneath.
Nevertheless I have one last question, what tools do you use to "analyze" the source code in assembler and the memory of the game ? (I did it with cheat engine, but if there is better or more adapted...)
Collaboration
I will start by doing the IP analysis code, see how to do it best, I will offer you an overview once finished.
Next, I'll attempt to document the code against the ways I understood it, so you can save some time on this.
I could also write a wiki later if you agree.
I will also let you know if I manage to find functions or some sort of redundancy in the game's memory (I have already started to find something in relation to the various global variables such as money, influence, etc.)
I have some other ideas to slightly improve the code (some small features to help develop or help the user), I'll let you know as soon as I'm done.
If you ever want to discuss somewhere other than on github or by email, don't hesitate to ask me (typically Discord...)
Have a good day !


Docker release job: Intermittent failure

When attempting to release the Docker images for Graylog 4.3.5 (https://github.com/Graylog2/graylog-docker/commit/b326f09c00d9b3f246acddffe56e73f73e8fe189), the following error occurred for the 4.3.5-1 tag release job:
12:29:38  #45 0.523 standard_init_linux.go:228: exec user process caused: exec format error
12:29:38  #45 ERROR: process "/bin/sh -c install -d -o \"${GRAYLOG_UID}\" -g \"${GRAYLOG_GID}\" -m 0755 ${GRAYLOG_HOME}/plugin" did not complete successfully: exit code: 1
12:29:38  ------
12:29:38   > [linux/arm64 graylog-downloader 15/16] RUN install -d -o "1100" -g "1100" -m 0755 /usr/share/graylog/plugin:
12:29:38  #45 0.523 standard_init_linux.go:228: exec user process caused: exec format error

To retry, I bumped the image release version to 4.3.5-2 in https://github.com/Graylog2/graylog-docker/commit/adb69e039358288af5a017b779c3404855fa4156 and re-ran the release job. This time, it ran successfully.
From doing some research, it seems that the exec format error might be related to a platform/target mismatch, but that seems to not be the case here, since the retry worked.
@bernd Capturing this as an issue, to see if we can find the cause and avoid a recurrence.

This happened on the arm64 part of the build.
Could be that the cross compilation is having an issue. @bernd could we update the docker version on the jenkins node?
What is particularly weird is that the other install (#44)  command worked just fine:
19:29:37  #44 [linux/arm64 graylog-downloader 14/16] RUN install -d -o "1100" -g "1100" -m 0755 /usr/share/graylog/plugins-merged
19:29:37  #44 DONE 0.7s
19:29:37
19:29:37  #45 [linux/arm64 graylog-downloader 15/16] RUN install -d -o "1100" -g "1100" -m 0755 /usr/share/graylog/plugin
19:29:38  #45 0.523 standard_init_linux.go:228: exec user process caused: exec format error
19:29:38  #45 ERROR: process "/bin/sh -c install -d -o \"${GRAYLOG_UID}\" -g \"${GRAYLOG_GID}\" -m 0755 ${GRAYLOG_HOME}/plugin" did not complete successfully: exit code: 1
19:29:38  ------
19:29:38   > [linux/arm64 graylog-downloader 15/16] RUN install -d -o "1100" -g "1100" -m 0755 /usr/share/graylog/plugin:
19:29:38  #45 0.523 standard_init_linux.go:228: exec user process caused: exec format error
19:29:38  ------
19:29:38  Dockerfile:70
19:29:38  --------------------
19:29:38    68 |     RUN mv ${GRAYLOG_HOME}/plugin ${GRAYLOG_HOME}/plugins-default
19:29:38    69 |     RUN install -d -o "${GRAYLOG_UID}" -g "${GRAYLOG_GID}" -m 0755 ${GRAYLOG_HOME}/plugins-merged
19:29:38    70 | >>> RUN install -d -o "${GRAYLOG_UID}" -g "${GRAYLOG_GID}" -m 0755 ${GRAYLOG_HOME}/plugin
19:29:38    71 |
19:29:38    72 |     COPY config ${GRAYLOG_HOME}/data/config

@mpfz0r The build workers run Docker version 20.10.17 which is the latest version, AFAIK.

From a chat with @bernd, since the error is intermittent, we can ignore it for now. I am OK with closing this if we don't plan to take any action on this now. We can always open it again if the issue happens again.