set_column_by_name/ set_column?

Thanks for the work that you're doing here!
I would like to request the addition of DataFrame.set_item_by_name to the API.
From the demonstration at euroscipy last week, the ugliest part of the usage was adding columns by:
for col in cols:
    df.insert(len(df.get_column_names()), col)

This seems like an obvious omission, but I haven't found any documentation saying why it hasn't been included. Seems like it was discussed in a call (https://github.com/data-apis/dataframe-api/pull/138#issue-1670699315)?

thanks @ivirshup !
I've got a PR open to add DataFrame.update_column and DataFrame.update_columns - I think this is what you're after? https://github.com/data-apis/dataframe-api/pull/232
my general issue with set_column was that it tried to do two things:

add a column if not present (but where? not clear)
update a column

From the demonstration at euroscipy last week, the ugliest part of the usage was

😄 yeah I realised it was pretty "ugly" whilst preparing and presenting the presentation - would

insert_column(loc, column)
update_column(column)

be less ugly in your opinion?

my general issue with set_column was that it tried to do two things:

add a column if not present (but where? not clear)
update a column

I guess I want the behavior you are describing 😅. But I also want there to be a unique solution to DataFrame.get_column_by_name.
Personally, I think of dataframes as Mapping[str, Array] where every array has the same length. This model, which works well in across the R and python libraries I've used, would say "you are overwriting or inserting the column", and I don't think this has been confusing. Of course, many of these implementation have strange ways you can construct dataframes with non-unique column names which leads to extremely confusing behavior:

 e.g. confusing definition of union
In [1]: import pandas as pd

In [2]: pd.Index(["a", "a", "b"]).intersection(pd.Index(["a", "a", "b"]))
Out[2]: Index(['a', 'b'], dtype='object')

In [3]: pd.Index(["a", "a", "b"]).union(pd.Index(["a", "b", "a"]))
Out[3]: Index(['a', 'a', 'b'], dtype='object')

add a column if not present (but where? not clear)

I think most libraries do "key insertion order", but I don't really care since I'll be accessing columns by name anyways.

I sort of see an argument for saying overwrite is different than insert and these should be distinct. I don't really see an argument for set by position, get by name. I'm also fine with writing:
if col.name not in df.get_column_names():
    df = df.set_column(col)  # or whatever other than position
else:
    raise SomeError(f"{col.name} was taken, be more original")

From the demonstration at euroscipy last week, the ugliest part of the usage was

😄 yeah I realised it was pretty "ugly"

I would actually like to rephrase from "the ugliest" to "the only ugly part".

 Well.... maybe the names could be shorter

Thanks for your thoughts, much appreciated
@kkraus14 has brought up the idea of unordered columns before - maybe we should do that? So then we'd have:

insert_column
update_column
get_column
drop_column

and corresponding plural versions.
Then there'd be no need for by_name because that's all they could be
And get_column_names would return a set.
The name get_columns_by_name has been a source of annoyance for me to be honest, I'd be glad if we could move away from it

@kkraus14 has brought up the idea of unordered columns before - maybe we should do that?

I think I brought up a question about whether we had explicitly decided that columns were ordered or not. From my perspective, it's a common pattern to iterate over the columns in a dataframe and returning column names as a set doesn't guarantee you a consistent iteration order as far as I'm aware, which is undesired behavior.
My stance here is that columns should be both ordered and named. We could update the behavior of df.insert to default to inserting at the end of the dataframe?

@MarcoGorelli your proposal sounds right, but I agree that set's unstable iteration order is undesirable.
A case where order really matters: visual inspection of the table. It's quite annoying when:
df.head()
# a | b | c
# ...
df.write(store)
df = store.read()
df.head()
# c | a | b
# ...

But you don't need that much order awareness to accomplish this, basically just insertion order and stable iteration order.

We could update the behavior of df.insert to default to inserting at the end of the dataframe?

For insert_column to have a default location the argument order would have to change as well, no? Does position become key word?

I would also like to make an argument for a set_column that overwrites or inserts, but maybe I'll leave that to a separate issue?

Ssure, let's keep columns ordered then, and default to inserting columns at the end. people can always reorder them with get_columns_by_name and get_column_names() anyway
for set_column - can't you do:
if col.name in df.get_column_names():
    df = df.set_column(col)
else:
    df = df.insert_column(col)

?
If this is meant to be a developer-focused API, I'd suggest forcing such explicitness
But I also don't really care too much about this particular topic, I'm much more concerned Ibis and polars-lazy not being compatible with some column-related parts of the API - if you had any thoughts / inputs on https://github.com/data-apis/dataframe-api/issues/229, that would be appreciated
Just for my understanding, what's your use case for the dataframe standard?

Context
I work on anndata. This library provides the AnnData object, which most tools in the scverse ecosystem (focussed on bioinformatics, in particular single-cell data analysis) use to interchange data. It also has hdf5 and zarr based storage formats. This object contains arrays and dataframe objects and keeps them aligned along two dimensions: observations and variables.
It's sort of like a 2-D xarray.Dataset specialized for exploratory analysis of high dimensional data, where your data looks like something that could be passed into a scikit-learn transformer. It's not actually an xarray.Dataset because we need support for sparse arrays and dataframes (for 1d annotations along the observations or variables, e.g. "cell type"). We're currently expanding the types of arrays we can work, sometimes with help from the array-api, but not as often as we'd like since we have sparse data.
However, we currently only support pandas dataframes, and I'd like to change that. This initiative seems like a great way to do that.
What I'd like to be able to do with arbitrary dataframe types
The big operations we need are:

Creation, e.g.

df = DataFrame()
store: h5py.Group | zarr.Group = ...

df = Dataframe.from_dict(
    {k: v.decode() for k, v in store.items()}
)

Row based indexing
Concatenation

And if we can do these without having to specialize for each case, as we've been doing to have sparse array support.

Eventually I would also like to support more dataframe types in our analysis libraries, but being able to handle them in our data container would need to happen first.

Thanks @ivirshup

Row based indexing

Could you expand on what you need please? Row labels are disallowed by the standard, would you be able to manage without?

Could you expand on what you need please? Row labels are disallowed by the standard, would you be able to manage without?

Yes. We just need positional indexing of rows.

ok that should be fine then, thanks!
How about
df = Dataframe.from_dict(
    {k: v.decode() for k, v in store.items()}
)

, when you do v.decode(), what object is that?
The standard doesn't have a dataframe.from_dict method, where you could create a dataframe from a dict of columns. So you could do something like:
df_standard = df.__dataframe_consortium_standard__()
namespace = df_standard.__dataframe_namespace__()
df = namespace.dataframe_from_dict(
    {k: v.decode().__column_consortium_standard__() for k, v in store.items()}
)

Hi @ivirshup - if you're still interested in this, you may be interested in Narwhals
I think it covers your needs (creation, indexing, concatenation) - I also noticed that someone has already built a library annsel on top of your AnnData library using Narwhals


Links not working. Regression?

In the Download Section (Javadoc, Github/Source)

Duplicate to #25


Bug in iOS 8 when run CLImageEditor from landscape mode.

Done button beyond the screen and imageview off-center

+1

+1

+1

+1

If you embed in a UINavigation controller before before presenting CLImageEditor it will control the rotation for you and fix this issue you just have to change shouldAutorotate to YES like below
 - (BOOL)shouldAutorotate
{
    return YES;
}

In CLImageEditor.m

please try #183 changes.

For this we need to make the presenting view controller portrait before presenting editor.
    let value = UIInterfaceOrientation.portrait.rawValue
    UIDevice.current.setValue(value, forKey: "orientation")


Option for multi-line hash comments

Native apache has support for multi-line hash comments, such as:
# here is a \
   multi-line hash comment

where the entire body is interpreted to be a part of the hash comment.
Perl's General::Config doesn't seem to support this, but maybe we can provide an option for it.

+1 from me! Looks neat from usability standpoint.

Done in #65


General Questions

Hi! I hope you're doing well. I had a couple of questions I was hoping you could answer.

Is the Dogfight scene the same scene you used to train your agents? I was wondering why you don't have borders if the ships fly out of the asteroid field.
Why doesn't the episode reset if an agent collides with something?
How long did you run the training?
In your YouTube video you had teams but don't seem to have them in the demo, why?
Why is the step amount 8000? Seems like a lot for training
Do the agents spawn in random positions and orientations on reset?

Apologies if you've already answered these somewhere. I'm curious as to how you accomplished this :)
Best,
James

Hi James,

Is the Dogfight scene the same scene you used to train your agents? I was wondering why you don't have borders if the ships fly out of the asteroid field.

Yes, there's a "brake zone" - if ships fly outwards beyond a given radius, the code applies some counter velocity to stop them.
https://github.com/mbaske/grid-sensor/blob/master/Assets/Examples/Dogfight/Scripts/Spaceship.cs#L111
When that happens, agents get less rewards because they're slower, so they should learn to avoid that region. The NormPosition field indicates a ship's distance from the center and is observed by its controlling agent.

Why doesn't the episode reset if an agent collides with something?

I think it'll work either way. One thing to consider though is that if episodes end and collision penalties are too low, the agent might want to exploit that by crashing into asteroids in order to evade enemies.

How long did you run the training?

I haven't kept the original logs or checkpoints, but I usually try to set the max_steps in the config yaml file to a value that roughly matches what I was doing. In this case, training should run for about 5 million steps.

In your YouTube video you had teams but don't seem to have them in the demo, why?

That was an older version of the project. When I updated it for using grid sensors, I just wanted to simplify the code a bit and removed the teams.

Why is the step amount 8000? Seems like a lot for training

The idea was to give the agents some time for figuring out flight paths, but a lower value like 5000 should also be fine.

Do the agents spawn in random positions and orientations on reset?

The initial asteroid and agent spawn positions sit on a 3D grid. For the agents, I randomized the specific grid pos per episode. See https://github.com/mbaske/grid-sensor/blob/master/Assets/Examples/Dogfight/Scripts/AsteroidField.cs

Thank you so much! I was able to update my model to reflect your explanation. I'll post back here to show you the results!

Have you ever gotten the "unity Resource ID out of range in SetResource: 1803109 (max is 1048575)" error? Been training for a few hours and it just popped up. Most threads say it's related to Post Processing but I'm not using any...


Add site generator

I've been using Peggy for a while now for some cool parsing stuff, and it's a really great tool. I've browsed around the GitHub issues for a while and noticed that there are many issues related to the website (which is essentially a copy of the PEG.js website). It seems like many of these issues stem from the difficulty around updating the website and maintaining it.
Proposal
In order to make maintaining and extending the website easier: I propose that a static site generator should generate the Peggy website. I don't have a personal favorite, but I think any static site generator would be easier than manually modifying the HTML files directly.
Static site generators
Like I said, I don't have a personal bias towards any of these tools particularly, but I picked a few notable ones:

11ty

Used by several notable developer websites: https://eslint.org, https://web.dev, https://netlify.com, https://v8.dev, etc.

Jekyll

Used by Ruby on Rails, Sketch, Spotify for Developers, etc.

Next.js

Used by too many sites to count

Gatsby

Used by lots of sites

Suggestion
I have used Gatsby, Next.js, Jekyll, and Hugo for static site generation before. I have not used 11ty at all. However, based on what I have seen about 11ty, it may present the easiest migration path for the existing Peggy site, because it accepts HTML files as-is and does not rely on React/Vite/Svelte or other frameworks. This might make it easy to replace the existing site implementation without adding lots of new code.
I'd be happy to contribute a PR that adds static site generation (or at least looks into it) if that would be a helpful contribution.

The direction sounds right.  Can you pick one of the above and do a quick prototype PR, just so we can see how it would feel to use?  Note that our current workflow is:

Github Pages is served from the docs branch
On release, main gets merged onto docs
In the meantime, some things get cherry-picked one way or the other

The direction sounds right. Can you pick one of the above and do a quick prototype PR, just so we can see how it would feel to use? Note that our current workflow is:

Github Pages is served from the docs branch
On release, main gets merged onto docs
In the meantime, some things get cherry-picked one way or the other

Yes, agreed! I'm going to put together a prototype solely focused on adding a static site generator (no refactoring). I'll try to see if I can show how the GitHub pages workflow would work with a static site generator as well.

Oh nice! Leave some space for /online there, please :)

@camchenry BTW, from your Github profile it became apparent you might also be interested in LPeg. It's a eDSL, so the syntax is not as clean, but it's one of the most mature PEG libraries technology-wise.

I opened a PR which I think will resolve this issue: https://github.com/peggyjs/peggy/pull/311. It doesn't address any issues other than adding a static site generator. But, I think this should make it easier to maintain the Peggy website and start making other changes without worrying about breaking things.


Default spherical light volumes are not being extended far enough

See the following image:

The hard edges of the sphere can be seen as they intersect the wall. However, if the sphere was large enough, then the light contribution would have faded to nothing and the edges would not be visible.

This is actually due to the use of very low polygon meshes for sphere lights. See the following:

Because of the straight edges on the mesh, they are not covering the area that a real sphere would cover. Using higher polygon meshes will reduce but never fully solve this problem.

For a given circle with radius r, the area of the enclosed octagon is given by 8 * t, where t is the area of a triangle with two sides of length r and the included angle between the two sides is (2 * π) / 8. The area of a circle with radius r is given by 2 * π * r, and therefore the scale factor s by which the enclosed octagon must be scaled to cover a greater than or equal area than the circle would is given by: (2 * π * r) / (8 * t).
Presumably this scales to any n-sided regular polygon.


[wip] feat: incremental resolver in lsp

1. Does this PR affect any open issues?(Y/N) and add issue references (e.g. "fix #123", "re #123".):

[ ] N
[ ] Y

2. What is the scope of this PR (e.g. component or file name):

3. Provide a description of the PR(e.g. more details, effects, motivations or doc link):

[ ] Affects user behaviors
[ ] Contains syntax changes
[ ] Contains variable changes
[ ] Contains experimental features
[ ] Performance regression: Consumes more CPU
[ ] Performance regression: Consumes more Memory
[ ] Other

4. Are there any breaking changes?(Y/N) and describe the breaking changes(e.g. more details, motivations or doc link):

[ ] N
[ ] Y

5. Are there test cases for these changes?(Y/N) select and add more details, references or doc links:

[ ] Unit test
[ ] Integration test
[ ] Benchmark (add benchmark stats below)
[ ] Manual test (add detailed scripts or steps below)
[ ] Other

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

24 of 27 (88.89%)  changed or added relevant lines in 5 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.1%) to 71.985%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

kclvm/tools/src/LSP/src/request.rs
3
6
50.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.1%

Covered Lines:
44632

Relevant Lines:
62002

💛  - Coveralls


Improve documentation for grouping by time

This is for InfluxDB 1.x.  I was surprised recently to discover that when I do a query using a time range as a where criteria and group by 10 seconds, InfluxDB locks the time windows to 00,10,20,30,40,50 seconds on the clock.  What I expected to happen was that it would use 10 second windows starting from the timestamp of the beginning timestamp in the where clause.
In other words, if I did a query with a where and group like: where time >= '2020-01-02T01:02:01.000000Z' and time < '2020-01-02T01:07:01.000000Z' GROUP BY time(10000ms), "host" FILL(0) the time windows will NOT be 01:02:01-01:02:11, 01:02:11-01:02:21, etc but instead will be 01:02:00-01:02:10, 01:02:10-01:02:20, etc.
This caused a strange behavior in an export script that exported data in chunks of 5 minute windows due to memory constraints, but the last data set it retrieved from InfluxDB would consistently be all 0s (due to the fill) because the window it was looking over would be very small and there would be no data recorded in it.
The documentation about group by time is pretty good, but all of the examples use 00 seconds as the start time, so it would be good to add some extra documentation around this behavior.

@Supagoat thanks for opening this! I think it is unlikely we will make changes to the way group by works, but we can definitely update the docs to be more clear.


Query

In the #3 domain Query is mostly about selecting releases, interface components and documentation sections, so it is mostly about selecting everything.

To make this work I guess I need a query language. I don't want to force the usage of a specific query language, but it is hard to make this work if I want to describe the filters in a general way.
Another option to move the code to QueryServices, and implement it in adapters. So the query descriptor would not be general, but QueryService specific. I'd like to do that because I don't want to design a general Query descriptor vocab. The only disadvantage, that I have to write QueryService code by every single Query, which would be nice to avoid.
If I want to do neither, then I have to use something else and forget Queries. Queries are about binding components together with the usage of a base collection and a filter. It make maintenance easier, because by adding a new Component to the collection I don't have to manually bind Components together, because Queries do this automatically. (I think pipes and filters is about something like this. Hmm I could use this in Dataflower.) If I want to keep this feature, the only option to add Components manually to Query outputs. So I would be able to give meaningful names to the Queries, and add the output manually. Actually that would be a nice 1.0 feature, and I could implement the Query vocab later to make the output automatically. I don't have to call them Query. I mean we are talking about Collection-s here, which we assemble manually. Ofc. later they could be Query results, but the code has to be backward compatible. So if I want to remove this feature from the Queries later, I won't be able to do so or I'll have to increase the major version number. So I'll remove the Query from the domain, and I'll use Collection instead. Later, after I made a Query domain/vocab, I'll reevaluate this feature.

I will force the usage of specific set of query templates. So I won't need general queries and the Query project. This way I can start the work immediately.


webpack3 node.js api - multi-compiler callback doesn't work

Do you want to request a feature or report a bug?
bug

What is the current behavior?
Using webpack3 node.js api and passing multiple config files, callback will not be triggered.
If the current behavior is a bug, please provide the steps to reproduce.
var webpack = require("webpack");
var config1 = require("webpack.config1.js");
var config2 = require("webpack.config2.js");
var compilers = webpack([config1, config2]);
compilers.run(function(err, stats) {
console.log("Yeahhhhh");
});

What is the expected behavior?
Passing multiple config files, callback should be triggered.
If this is a feature request, what is motivation or use case for changing the behavior?
N/A
Please mention other relevant information such as the browser version, Node.js version, webpack version, and Operating System.
Node.js v6.12.0
webpack v3.11.0
OS windows 7

I found the root cause. It is caused by stats-webpack-plugin. The param which I passed in is in wrong format.
@ibeeger Thanks for your solution.


Data: project addition by Killer2OP

Add a new project to the list
I want to add a Slack UI clone using flutter.
Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSoC'23 contributor
[X] I want to work on this issue

you don't need to create multiple issues for project addition. You can add multiple projects in a single issue


Replace file editor with openhands-aci

End-user friendly description of the problem this fixes or functionality that this introduces

[ ] Include this change in the Release Notes. If checked, you must provide an end-user friendly description for your change below

Give a summary of what the PR does, explaining any non-trivial design decisions
This PR is to:

[x] Verify feature parity of openhands-aci with the existing file editor
[ ] Integrate openhands-aci into OpenHands

Link of any specific issues this addresses

@xingyaoww I have tried to test via the UI and it looks good to me. Your existing unit tests also passed, with minor changes in error messages. I think it's ready for eval now

(1) We got the same success rate with CodeAct v2.2:


Will it leave “scrcpy-server.jar” on the phone?

It looks like it pushes “scrcpy-server.jar” to the phone every time it connects, so if I use it frequently, will there be a lot of “scrcpy-server.jar” in my phone?

It always pushes it to the same location, and it removes it immediately:
https://blog.rom1v.com/2018/03/introducing-scrcpy/#clean-up-the-device


Use IronJacamar's WorkManager and BootstrapContext implementation

This gets rid of the original implementation I had for jakarta.resource.spi.BootstrapContext and jakarta.resource.spi.work.WorkManager and uses the implementations
provided by IronJacamar
Fixes #43
Initial work on #42

@vsevel can you give it a try and let me know if this (still) works for you? 😉


[UX] Don't display the reset message, if the conversation is already reset.

Here is a scenario that exemplify the behavior:

If you type !reset
and then a ask a question
the bot answer that as it is a long time that I haven't discuss with him, and that he reset my conversation. it's confusing as I just reset the conversation already.

This should do the trick: https://github.com/etalab-ia/albert-tchap/pull/67


SVG Pins: performance issues

I suspect that the memory handling for SVG Pins is not quite optimal, and might have some issues.
Let me illustrate this with the following scenario:

I have a map based on Mapsui.UI.Forms.MapView with a potentially larger number of pins.
There are a number of different types of pins, where each type has its own image (e.g. given by an svg or png file). Let's say we have something like 10-20 types, and each image file has a few kB.
The map will contain many pins of each type (say: hundreds or thousands).

Since there are many pins, I'd like to avoid duplicating the memory allocated for their images, for reasons of performance and memory footprint. I can see some potential issues in this direction (I've been looking at the Pin class and Pin.CreateFeature in particular):

For svg pins, the svg data is stored as a string (Pin.Svg). In Pin.CreateFeature this is then converted (copied) into a byte array via Encoding.UTF8.GetBytes. This means that for every Pin object I create, I make a separate copy of my svg data, right? For small svgs this might not be a problem, for for more complex ones that have several kB in size, it adds up. And it seems to be avoidable by making Pin.Svg a byte array (just like Pin.Icon) instead of a string. Correct?

IIUC the BitmapRegistry class could be used to avoid duplication of image data, but it seems to me that for every new pin a new entry in the BitmapRegistry is generated (since Pin.CreateFeature alwas seems to call BitmapRegistry.Instance.Register). Is that right, or have I simply not looked deep enough into the code? For png pins it might not be a problem: We might have multiple registry entries, but ultimately they all point to the same data in memory. But for svg (since we make a copy along the way, see first point above), we have multiple entries and they all point to their own private copy of the data, right?

@pauldendulk @charlenni
I'd be grateful for any kinds of comments on this. Am I missing something here?

No, you are correct. BitmapRegistry doesn't check, if a "Bitmap" is already registered. And the Forms implementation is very dump in this case too. I didn't implement a check, if the SVG string is already registered. As always: MapView is a convenience wrapper around MapControl for Forms. If you want to do more than the basics of Mapsui, then you should use MapControl. There you could check, if your SVG is registered, and if yes, then use the ID again.
The conversion to a byte array was done, because BitmapRegistry once upon a time only accepted this format.

No, you are correct. BitmapRegistry doesn't check, if a "Bitmap" is already registered. And the Forms implementation is very dump in this case too. I didn't implement a check, if the SVG string is already registered. As always: MapView is a convenience wrapper around MapControl for Forms. If you want to do more than the basics of Mapsui, then you should use MapControl. There you could check, if your SVG is registered, and if yes, then use the ID again.

Thanks for the comments, @charlenni.
I just stumbled upon ...
https://github.com/Mapsui/Mapsui/blob/8c527ea79d8153f0e448dcf0064d5647ae01bef9/Samples/Mapsui.Samples.Common/Maps/SvgSample.cs#L73
... which seems to implement something like this, right? However, I'm not sure if this approach is feasible with a Pin, since the _bitmapId is private there, and not accessible from the outside, right?
Could you be a bit more specific about how to check if the SVG is registered already?

The conversion to a byte array was done, because BitmapRegistry once upon a time only accepted this format.

So, are you saying that the conversion is not necessary any more? If so, what's the best way to get rid of it of?
From my point of view, it would be ideal to set the svg data via a stream, not a string. After all, a stream is what is put into the bitmap registry, and I also get the SVG data from the assembly as a stream (via Assembly.GetManifestResourceStream).
Instead of reading a string from it, which is then converted into a byte array, from which we again create a MemoryStream, couldn't we directly pass the original stream?

Btw, when looking through the list of samples, there are two that are closely related to the issue raised here:

Mapsui.Samples.Forms.ManyPinsSample
Mapsui.Samples.Common.Maps.SvgSample

The first one is based on Mapsui.UI.Forms, while the second is not. Both present basically a static map with a large number of pins/symbols (1000 and 2000, respectively). In the first case, the class Mapsui.UI.Forms.Pin is used (of type PinType.Pin), in the second case the class Mapsui.Providers.Feature, with a SymbolStyle based on an SVG image.
Interestingly the ManyPinsSample seems to be much more responsive and performant than the SvgSample, even when using the same number of points. I observe this when running them via Mapsui.Samples.Forms.iOS on an iPhone 8. When zooming out as much as possible and then dragging the map with one finger, the ManyPinsSample follows very fluently, while the SvgSample has large gaps and reacts quite slow.
If anything, I'd have expected the opposite (due to SvgSample appearing to be more low-level, and due to the optimization mentioned above (https://github.com/Mapsui/Mapsui/issues/1084#issuecomment-786655954). Can anyone tell me why the ManyPinsSample is so much more performant? (And can anyone even reproduce this behavior?)

If anything, I'd have expected the opposite (due to SvgSample appearing to be more low-level, and due to the optimization mentioned above (#1084 (comment)). Can anyone tell me why the ManyPinsSample is so much more performant? (And can anyone even reproduce this behavior?)

Btw, the imageCache optimization was apparently implemented by @inforithmics in 5034bc6d551eaf573e78b42982510ade9e97fcae. @inforithmics, do you have any further insights here?

I tried some modifications of Mapsui.Samples.Forms.ManyPinsSample:

Using PinType.Icon with
Icon = assembly.GetManifestResourceStream("Mapsui.Samples.Common.Images.loc.png").ToBytes();
results in a map that is just as responsive as the original one (the png file is 3kB). Check.

Using PinType.Svg with the following modifications ...

-- Samples/Mapsui.Samples.Forms/Mapsui.Samples.Forms.Shared/ManyPinsSample.cs --
index 376a8aa8..66f10799 100644
@@ -129,12 +129,18 @@ namespace Mapsui.Samples.Forms
         {
             var position = new Position(0 + rnd.Next(-85000, +85000) / 1000.0, 0 + rnd.Next(-180000, +180000) / 1000.0);

+            var assembly = typeof(AllSamples).GetTypeInfo().Assembly;
+            var stream = assembly.GetManifestResourceStream("Mapsui.Samples.Common.Images.Pin.svg");
+            StreamReader reader = new StreamReader(stream);
+            string svgString = reader.ReadToEnd();
+
             var pin = new Pin()
             {
-                Label = $"PinType.Pin {num++}",
+                Label = $"PinType.Svg {num++}",
                 Address = position.ToString(),
                 Position = position,
-                Type = PinType.Pin,
+                Type = PinType.Svg,
+                Svg = svgString,
                 Color = new Xamarin.Forms.Color(rnd.Next(0, 256) / 256.0, rnd.Next(0, 256) / 256.0, rnd.Next(0, 256) / 256.0),
                 Transparency = 0.5f,
                 Scale = rnd.Next(50, 130) / 100f,

results in the map becoming very sluggish when zooming out completely. Note that the svg file is only about 300 bytes, i.e. a factor of ten smaller than the png used in step one. Note further that this is the same svg as used internally for PinType.pin in the original version of ManyPinsSample.
I guess this is a good test case for the problem reported in this issue. It shows clearly that there is an issue with PinType.Svg.

If you look at the code from Pin (see here), then you could see, that, if you use Pin as PinType, the SVG drawing is converted to a bitmap. This is done, because you could then change the color. I didn't find a possibility to do this for SVGs. That's the reason, why these pins are not so clear as the pins in the SVG sample.
So, if you use PintType.SVG, you get an SVG pin. If you use a PinType.Pin, you get a bitmap pin. The later one is drawn much faster, because only a bitmap has to be copied to a place on the canvas, while by the first an SVG image has to be interpreted and then be drawn on the canvas. That's it.

If you look at the code from Pin (see here), then you could see, that, if you use Pin as PinType, the SVG drawing is converted to a bitmap.

Yeah, already saw that.

So, if you use PintType.SVG, you get an SVG pin. If you use a PinType.Pin, you get a bitmap pin. The later one is drawn much faster, because only a bitmap has to be copied to a place on the canvas, while by the first an SVG image has to be interpreted and then be drawn on the canvas. That's it.

Ok, so the difference in behavior of the samples is not related to memory issues at all, but just due to the fact that an svg renders much slower than a bitmap. I guess that means if I need a responsive map with many pins, I better avoid SVG completely. Thanks for clearing that up.
So then, with this in mind, wouldn't it be a good idea to add a new pin type (maybe called PinType.CachedSvg), which would take SVG data as input, but cache it into a bitmap for performance reasons (as for PinType.Pin, but with a freely chosen svg as input)? If you agree, I'd volunteer to implement that.
Apart from this, I'd argue that Pin.Svg should be byte[] instead of string (analogous to Pin.Icon). This makes it easier to use for typical cases, is less confusing and avoids extra conversion. It's a breaking change to the API, though, but since Mapsui 3 is in alpha state, I hope that's fine. If there are no objections, I'll prepare a patch for this as well.

Instead of PinType.CachedSVG you could use PinType.Icon for this. If I would see a PinType.CachedSVG, I would expect, that each time an SVG is drawn. But this isn't the case. So, create your own bitmap from an SVG and use this as icon.
Where do you see these conversions from string to byte array to stream?
And be careful changing anything in this part. You have to remember, that you have to change the same in the Xaml renderer.

Instead of PinType.CachedSVG you could use PinType.Icon for this. If I would see a PinType.CachedSVG, I would expect, that each time an SVG is drawn. But this isn't the case. So, create your own bitmap from an SVG and use this as icon.

Well, of course I can do the conversion myself and use PinType.Icon. But I guess it would be a convenient option for people, if one could simply use an svg as input, and cache it into a bitmap internally to get better performance (in particular since the necessary code is basically already there and could be shared with PinType.Pin, which is also nothing but a convenience option).
Expectations can vary of course. So, if you see PinType.CachedSVG, there will be documentation that will tell you what it does. This documentation should also mention the performance improvement over PinType.SVG and the slight loss of quality. This will give users the choice of performance vs quality, and keep naive guys like me from falling into this performance trap ;)

Where do you see these conversions from string to byte array to stream?

Here:
https://github.com/Mapsui/Mapsui/blob/b4df2b9388cc94d61f54a09fea7f99967a392e54/Mapsui.UI.Forms/Objects/Pin.cs#L454
and in my user-side code I have the inverse (stream -> string). I opened #1085 now for using a byte array. Any comments to this?

And be careful changing anything in this part. You have to remember, that you have to change the same in the Xaml renderer.

I don't quite understand this comment. You mean regarding the new PinType?

@pauldendulk, what do you think about my idea with PinType.CachedSVG? Maybe it just needs a better name?

Wouldn't it be better to create a helper function, that converts an SVG to an Icon and then use this? Or create a new PinType like PinType.Own, where you could provide your own BitmapID? With the last you could register your few bitmaps and use it for more than one pin.

Wouldn't it be better to create a helper function, that converts an SVG to an Icon and then use this?

In which way is that better?

Or create a new PinType like PinType.Own, where you could provide your own BitmapID? With the last you could register your few bitmaps and use it for more than one pin.

Actually I don't fully understand what's the point of using this BitmapRegistry at all, if it does not even help to deduplicate the bitmap references. Why not use references to the bitmap data directly instead of generating integer IDs for them?
Moreover it seems that the BitmapRegistry is somewhat misnamed, since you put not only bitmap data into it, but also SVG vector data.

A quick reaction without fully reading this thread.
This is a very complicated topic. Some smart form of caching may seem the best solution, but there are many different scenarios in which Mapsui is used, which need different caching strategies. What I would like to avoid is some smart solution with a 'we know what is best for you' mentality. I would prefer a solution where it is clear to the developers what is happening so they can use this understanding to develop a solution that is suited for their use case.

If anything, I'd have expected the opposite (due to SvgSample appearing to be more low-level, and due to the optimization mentioned above (#1084 (comment)). Can anyone tell me why the ManyPinsSample is so much more performant? (And can anyone even reproduce this behavior?)

Btw, the imageCache optimization was apparently implemented by @inforithmics in 5034bc6. @inforithmics, do you have any further insights here?

When I migrated to skia.svg i discovered that the ManyPinsSample was very slow, the main reason was that every pin got its own BitmapId even if it was the same Image. So 1000 pins meant 1000 MemoryStreams where loaded. So to only have one Memory Stream I implemented caching so the same svg image gets the same Id. Saving a lot of memory and improving the Performance considerable. Because the MemoryStream was later used for caching having the same memory Stream for the Same made the caching work.

Skia.Svg has two parts for painting an image.
First it Parses the svg into SKSvg (which is much slower than drawing the Image).
Second it Draws it using SKSvg which is very fast only Vector Draw lines.
So this means that when I assign the same BitmapId for the Same Svg it is caching the parsed SkSvg and draws it very quickly.

Important parts to lock at it
BitmapHelper (Maybe not the best name, would be better image helper) stores the parse SKSvg Image
if (bitmapStream is Stream stream)
            {
                if (stream.IsSvg())
                {
                    var svg = new SKSvg();
                    svg.Load(stream);

                    return new BitmapInfo {Svg = svg};
                }

Then there is the SymbolCache where each Bitmap Id is Cached with the corresponding BitmapInfo
public BitmapInfo GetOrCreate(int bitmapId)
        {
            if (_cache.Keys.Contains(bitmapId)) return _cache[bitmapId];
            return _cache[bitmapId] = BitmapHelper.LoadBitmap(BitmapRegistry.Instance.Get(bitmapId));
        }

So in conculsion when you do it right. only register one bitmap id per svg pin Kind. Your Svg is cached. if you assign a bitmap id per new pin it isn't cached or you get 1000 svg instances for 1000 pins. Which destroys Performance.
I will have a look at the SvgSample and report back why it is slower, but I guess that it assigns for every Image a new BitmapId instead a BitmapId per PinType.

I found the relevant checkins.
In this checkin I had put the SkSvg Image into the cache so that it is only created once. Which improved the Performance a lot
https://github.com/Mapsui/Mapsui/pull/981/commits/5034bc6d551eaf573e78b42982510ade9e97fcae
But in this checkin I had to revert it, because it interfered with the Xaml Rendering Backend which doesn't now anything about SkSvg Or SkImage.
https://github.com/Mapsui/Mapsui/pull/981/commits/5c87e81acb7079622a8e1eb9d16cd0fe9005390e
So To work with the Xaml Rendering Backend I had to cache the MemoryStream (still better than before but slower).

I found the relevant checkins.
In this checkin I had put the SkSvg Image into the cache so that it is only created once. Which improved the Performance a lot
5034bc6
But in this checkin I had to revert it, because it interfered with the Xaml Rendering Backend which doesn't now anything about SkSvg Or SkImage.
5c87e81
So To work with the Xaml Rendering Backend I had to cache the MemoryStream (still better than before but slower).

Thanks for the comments. That sounds a lot like we need a smarter BitmapRegistry, whose caching mechanism should depend on the Rendering Backend ...?

For me, it is a two part problem: one part is in Mapsui implementation, the other is in Mapsui.Forms.
In Mapsui we have the BitmapRegistry, which should register any type of bitmap, images, what ever you want, before drawing. It should check, if it has a valid format, so that this is correct when drawing. It is a lightweight and easy part. There is also the place for improvements. One would be to convert the string with drawing commands from an SVG to an easy drawable format. This is done by the SKSVG class as you could see here. It is converted in a SKPicture, a very fast format. So the suggestion from @inforithmics is correct: if you add many independent SVGs, then the time for converting them is high, but the drawing is fast. If you create one or hundreds of the same SVG is no problem for drawing time. So I would say, BitmapRegistry is correct, and we could not make things faster.
What could be made better in BitmapRegistry is a check, if an image already exists. But do we want to make this? We have to search to the already registered images and look, if we found the new one. It is time-consuming and not very effective. But could be made (check for length and only compare those images, which having the same length).
The other part is Mapsui.Forms. There is a Pin object, which could have an SVG image. Because this should have changeable colors, I converted this SVG to a bitmap, so that I could use the function to replace colors. It would be nicer (in terms of drawing quality), to use a SKPicture for that, but a long time ago I didn't find a function for this. And because each pin could have another color, I didn't implement a caching algorithm. Wasting time for the normal case. And to be sure, the normal case is not to have more than thousands of pins ;-)
So where should we make changes to support normal cases for everyone? Perhaps to make BitmapId of Pin public would be the best. With this, you could handle images by your own. The setter of BitmapId could check, if the PinType is PinType.Own and if yes, allows to set the intern _bitmapId.
And as you could read in the last comment of inforithmics, there was always a problem with the Xaml renderer. In an ideal world (only one renderer system), you could make improvements, that are not possible with two or more.

Very strange writing. English isn't my preferred language ;-)
To be clear: BitmapRegistry is renderer independent and doesn't know anything about the registered image formats. There exists no SKPicture or SKImages. Forget all SkiaSharp things in this part of code.

I just was just recently interested in such performance concerns with the Mapsui.Forms mapview(I eventually found out most of my performance issues where coming from another piece of my application but I can still share my thoughts about it),
One thing I noticed (but didn't investigate much further) is that the pins in the MapView use an private readonly ObservableRangeCollection _pins and is exposed as an public IList<Pin> => _pins. Wouldn't calling Add() on the Pins property call the CollectionChanged event handler (and thus the Refresh() itself calling the RefreshData() method) each time we add a pin ? (this is a real question, I did not tested it...)

Thanks for the comments. That sounds a lot like we need a smarter BitmapRegistry, whose caching mechanism should depend on the Rendering Backend ...?

@janusw just like @inforithmics said the SymbolCache class in the SkiaRenderer already does exactly that.

Actually I don't fully understand what's the point of using this BitmapRegistry at all, if it does not even help to deduplicate the bitmap references. Why not use references to the bitmap data directly instead of generating integer IDs for them?

Generating and integer ID for them is probably the sole purpose of this class, deduplication is the responsibility of the caller,
the problem actually lies in the MapView Pins implementation which we could probably expects to do that since it is a rather high level component. That integer ID is used in the renderer specific cache SymbolCache (at least in the skia renderer, I did nto read about the xaml renderer) to identify the resource.

Why not use references to the bitmap data directly

Technically it could probably be possible to store opaque object in the BitmapRegistry that actually are SKImage, SKPicture or the Xaml Renderer counterparts, created in a renderer specific factory, ... but i do not think it has any advantage over the current implementation other than saving a little of memory (maybe ?), and it may probably make the code more complicated to think about.
I figured I could just use the mapsui api directly as i didn't required much of the Forms.MapView functionnality from the beginning.
Here is the code i ended up writing for my use case. Here I rasterized the SVGs, which may not be what you want if you need variable size SVGs, the end result is not perfect either from an aliasing point of view, I probably have missed something here.
using Mapsui.Geometries;
using Mapsui.Layers;
using Mapsui.Providers;
using Mapsui.Styles;
using Mapsui.UI.Forms;
using Mapsui.UI.Objects;
using MyApp.Helpers;
using SkiaSharp;
using Svg.Skia;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;

namespace MyApp.Controls
{
    public class PinBitmapHelper
    {
        private int CreateBitmap(Stream data, double scale)
        {
            var svg = new SKSvg();
            svg.Load(data);

            var info = new SKImageInfo(
                (int)(svg.Picture.CullRect.Width * scale),
                (int)(svg.Picture.CullRect.Height * scale)) {
                AlphaType = SKAlphaType.Premul
            };

            var bitmap = new SKBitmap(info);
            var canvas = new SKCanvas(bitmap);
            canvas.Clear();
            canvas.Scale((float)scale);

            // paint.ColorFilter = SKColorFilter.CreateBlendMode(
            //     Color.ToSKColor(),
            //     SKBlendMode.SrcIn
            // ); // use the source color

            using var paint = new SKPaint() { IsAntialias = true };
            canvas.DrawPicture(svg.Picture, paint);

            using var image = SKImage.FromBitmap(bitmap);
            using var bitmapData = image.Encode(SKEncodedImageFormat.Png, 100);
            var bitmapDataArray = bitmapData.ToArray();
            var stream = new MemoryStream(bitmapDataArray);
            var bitmapId = BitmapRegistry.Instance.Register(stream);
            return bitmapId;
        }

        private readonly Dictionary<string,int> resToBitmapId = new Dictionary<string,int>();

        public int GetOrCreateBitmapFromSvgData(string name, string svgData, double scale)
        {
            if (!name.StartsWith("data://"))
            {
                name = "data://" + name;
            }
            if (resToBitmapId.TryGetValue(name, out int bitmapId))
            {
                return bitmapId;
            }
            var data = new MemoryStream(Encoding.UTF8.GetBytes(svgData));
            bitmapId = CreateBitmap(data, scale);
            resToBitmapId[name] = bitmapId;
            return bitmapId;
        }

        public int GetOrCreateBitmapFromSvgResource(string resName, double scale)
        {
            if (resToBitmapId.TryGetValue(resName, out int bitmapId))
            {
                return bitmapId;
            }
            string shortName = resName.Replace("resource://MyApp.Assets.","");
            string svgData = AssetFileLoader.LoadFile(shortName);
            var data = new MemoryStream(Encoding.UTF8.GetBytes(svgData));
            bitmapId = CreateBitmap(data, scale);
            resToBitmapId[resName] = bitmapId;
            return bitmapId;
        }
    }

    public class MyPin : IFeatureProvider
    {
        private static readonly PinBitmapHelper pinHelper = new PinBitmapHelper();
        public Feature Feature { get; private set; }
        public Position Position { get; }

        public MyPin(double x, double y,
                     double offsetX=0.0, double offsetY=0.0,
                     double scale=1.0,
                     string color="#FFFFFF")
        {
            Position = new Position(y, x);
            Feature = new Feature {
                Geometry = Position.ToMapsui(),
                Styles = new List<IStyle> {
                    new SymbolStyle {
                        SymbolType = SymbolType.Ellipse,
                        Fill = new Brush { Color=Color.FromString(color) },
                        Outline = new Pen { Color = Color.Transparent },
                        SymbolOffset = new Offset(offsetX, offsetY),
                        SymbolScale=scale,
                    }
                }
            };
        }

        public MyPin(string resName,
                     double x, double y,
                     double offsetX=0.0, double offsetY=0.0,
                     double scale=1.0, string svgData=null)
        {
            Position = new Position(y, x);

            int bitmapId;
            if (svgData != null)
                bitmapId = pinHelper.GetOrCreateBitmapFromSvgData(resName, svgData, scale);
            else
                bitmapId = pinHelper.GetOrCreateBitmapFromSvgResource(resName, scale);

            Feature = new Feature {
                    Geometry = Position.ToMapsui(),
                    Styles = new List<IStyle> {
                    new ImageStyle {
                        BitmapId = bitmapId,
                        SymbolScale = 1.0,
                        SymbolOffset = new Offset(offsetX, offsetY),
                    }
                }
            };
        }
    }

    public class MapView : MapControl
    {
        private readonly MemoryLayer _mapPinLayer;
        private readonly MemoryLayer _locationLayer;
        private List<MyPin> _myPins;
        public event EventHandler<MyPin> PinClicked;
        private static int bitmapStillId = -1;

        private MemoryLayer CreateLocationLayer()
        {
            if (bitmapStillId == -1)
            {
                var bitmapStill = Mapsui
                    .Utilities
                    .EmbeddedResourceLoader
                    .Load($"Images.MyLocationStill.svg", typeof(MyLocationLayer));

                if (bitmapStill != null)
                {
                    // Register bitmap
                    bitmapStillId = BitmapRegistry.Instance.Register(bitmapStill);
                }
            }
            var defaultPos = new Position(GpsPosition.DefaultPosition.Latitude,
                                          GpsPosition.DefaultPosition.Longitude);
            var layer = new MemoryLayer("locationLayer") {

                Style = null,
                DataSource = new MemoryProvider(new List<IFeature> {
                    new Feature {
                        Geometry = defaultPos.ToMapsui(),
                        Styles = new List<IStyle> {
                            new ImageStyle {
                                BitmapId = bitmapStillId,
                            }
                        }
                    }
                }),
                Enabled = false,
            };
            return layer;
        }

        public void SetMyLocation(Position myPosition)
        {
            var provider = _locationLayer.DataSource as MemoryProvider;
            provider.Features.First().Geometry = myPosition.ToMapsui();
            if (MyLocationFollow)
                MoveToMyLocation();
            Refresh();
        }

        public void MoveToMyLocation()
        {
            var provider = _locationLayer.DataSource as MemoryProvider;
            var point = provider.Features.First().Geometry as Point;
            Navigator.CenterOn(point);
        }

        public MapView()
        {
            _mapPinLayer = new MemoryLayer("pinLayer") {
                Style = null,
                DataSource = new MemoryProvider(new List<IFeature>()),
                IsMapInfoLayer = true,
            };
            _locationLayer = CreateLocationLayer();
            TouchMove += (s, e) => { MyLocationFollow = false; };

            SetPins(new List<MyPin>(), refresh: false);
            Map.Layers.Add(_mapPinLayer);
            Map.Layers.Add(_locationLayer);
            Info += (s,e) => {
                if (_myPins == null)
                    return;
                if (e.MapInfo.Layer != _mapPinLayer)
                    return;
                foreach (var pin in _myPins)
                {
                    if (pin.Feature.Equals(e.MapInfo.Feature))
                    {
                        PinClicked?.Invoke(this, pin);
                        break;
                    }
                }
            };
        }

        public bool MyLocationEnabled
        {
            get => _locationLayer.Enabled;
            set => _locationLayer.Enabled = value;
        }

        private bool _myLocationFollow;
        public bool MyLocationFollow
        {
            get => _myLocationFollow;
            set
            {
                _myLocationFollow = value;
                if (value && MyLocationEnabled)
                {
                    MoveToMyLocation();
                }
            }
        }

        public void SetPins(List<MyPin> myPins, bool refresh=true)
        {
            if (myPins == null)
                myPins = new List<MyPin>();
            _myPins = myPins;

            var demPins = myPins.Select(p => p.Feature).ToList();
            _mapPinLayer.DataSource = new MemoryProvider(demPins);
            if (refresh)
            {
                Refresh(); // this call InvalidateSurface();
            }
        }
    }
}

So then, with this in mind, wouldn't it be a good idea to add a new pin type (maybe called PinType.CachedSvg), which would take SVG data as input, but cache it into a bitmap for performance reasons (as for PinType.Pin, but with a freely chosen svg as input)? If you agree, I'd volunteer to implement that.

For performance the best solution would be to store bitmaps in the BitmapRegistry. Does the BitmapRegistry need to know if that bitmap originates from an SVG? Why not convert the SVG to bitmap early on and treat it like a bitmap after that? Perhaps render quality? Also an SVG could be rendered with different colors, but this could be the responsibility of a separate layer which generates different images and imageIds for the same SVG with different colors.
A general thing. As charlenni mentioned, the Forms Pins are a convenience layer. If the focus is on performance perhaps Forms should be avoided altogether. Also, in November this year MAIU is planned to be released. So better to focus on platform independent rendering.

So then, with this in mind, wouldn't it be a good idea to add a new pin type (maybe called PinType.CachedSvg), which would take SVG data as input, but cache it into a bitmap for performance reasons (as for PinType.Pin, but with a freely chosen svg as input)? If you agree, I'd volunteer to implement that.

For performance the best solution would be to store bitmaps in the BitmapRegistry.

Yes, this currently looks like the best option to me.

Does the BitmapRegistry need to know if that bitmap originates from an SVG?

I don't think it does.

Why not convert the SVG to bitmap early on and treat it like a bitmap after that? Perhaps render quality?

Sure, one can do the conversion on the user side (and I currently do that as a workaround). But I still think it would be nice to have an option in Mapsui to do it automatically. It would really only need a new PinType, the conversion code is already there and with a bit of documentation it will save many users a lot of time to find out: Whoops, svg pins are creepily slow.
The new PinType would also function as an implementation suggestion that basically says:

Dear user, you can use both svg pins and bitmap pins in Mapsui, but if you choose svg, please be aware that it might be very slow if you have many pins. In case you do have svg pins, we can conveniently buffer them into a bitmap for better performance. The drawing quality might be slightly worse, though. Choose whatever you prefer.

I have svg icons that I also use in non-map parts of my app (where the performance is no problem), so I'd like to keep them in my assembly without adding a png version for each one.

A general thing. As charlenni mentioned, the Forms Pins are a convenience layer. If the focus is on performance perhaps Forms should be avoided altogether.

Nope, strongly disagree here for a couple of reasons:

All of Mapsui is a convenience layer. In principle any user can re-implement all of it on his own. No one wants to do that. People want a nice API for displaying maps without going into all the gory details.
I'll not be moving away from Mapsui.Forms. I need a cross-platform implementation, and I don't want to implement things for each platform separately.
Using a MapControl instead of a MapView might be slightly more feasible, but even here: I think it's a useful high-level abstraction. I don't want to rewrite all the boiler-plate code (and I assume most other users don't want that either).
It's a fallancy to think that everything will be automatically more performant if one uses the lower-level APIs or platform specific-code. See my comment above about how SvgSample (which is not using Mapsui.Forms) is much slower than ManyPinsSample (which is using Mapsui.Forms and the Pins class). The latter is blazingly fast on any hardware I tested, even with a large number of pins. The big difference is whether you use svg icons or bitmap icons.
Synergy effects: If every user implements their own version of a MapView, everyone will have their own bugs in there (and possibly bad design decisions). We as a Mapsui community should provide a MapView that is as performant and convenient as possible, with all performance bottlenecks identified and removed.

If it weren't for Mapsui.Forms and high-level components like a MapView with Pins and Callouts, I wouldn't be using Mapsui at all.

If the focus is on performance perhaps Forms should be avoided altogether.

I did not mean to say Mapsui.Forms should be avoided altogether, but that it should be avoided with respect to  SVG Pins if you need high performance. And a Forms independent alternative can still be user friendly.

If the focus is on performance perhaps Forms should be avoided altogether.

I did not mean to say Mapsui.Forms should be avoided altogether, but that it should be avoided with respect to SVG Pins if you need high performance.

Well, the lesson that I take from the discussion in this issue so far is that SVG pins are to be avoided, if you need performance. Doesn't matter if you use them through Mapsui.Forms or not.
Again:

ManyPinsSample is fast (based on Mapsui.Forms, but without SVGs).
SvgSample is slow (not based on Forms, but with SVGs).

And a Forms independent alternative can still be user friendly.

Might be. But it requires you to write platform-specific code, right?
(Btw, we're not talking "end-user friendly" here, but "developer friendly", right?)
Platform-specific code is not what I call developer friendly. It's one of the great achievements of the Xamarin.Forms framework that you can define all your UI and logic in a platform-independent way, which saves a lot of development and maintenance time. As long as you're developing for one specific platform, that may not matter to you, but if you develop a large app framework that targets multiple platforms, it's a crucial advantage.

(Btw, we're not talking "end-user friendly" here, but "developer friendly", right?)

Yes, developer friendly.
Note, there might be some confusion in terminology because I see Xamarin.Forms as just one platform within Mapsui but Xamarin.Forms itself supports multiple platforms.
In general I would like to see more functionality in Mapsui core that is now still only in Mapsui.Forms. Putting a pin on a map or showing a dialog does not have to depend on Forms.

Note, there might be some confusion in terminology because I see Xamarin.Forms as just one platform within Mapsui but Xamarin.Forms itself supports multiple platforms.

Well, ok. When I used the word 'platform' above, I was basically referring to the OS you develop for. Xamarin.Forms supports things like Android, iOS, MacOS, Windows (via UWP or WPF), Linux (via GTK#). To XF itself I would rather refer as a 'framework' than a 'platform', but let's not start splitting hairs here (as long as we can agree on a common language).

In general I would like to see more functionality in Mapsui core that is now still only in Mapsui.Forms. Putting a pin on a map or showing a dialog does not have to depend on Forms.

I see this from a somewhat different angle: As far as I am concerned, Mapsui.Forms is the only relevant 'platform' ('backend'?) for Mapsui. Doesn't it sort of supersede all the others, since it supports all the OS platforms that Mapsui also supports 'natively' (Android, iOS and Windows via UWP and WPF) and even more (MacOS and possibly Linux/GTK#)?
Why would I ever use Mapsui.UI.Android, if I can instead use Mapsui.UI.Forms and get all other platforms "for free" instantly?

@janusw That's a developer decision. There are reasons, why someone wants to develop only for Android or only for WPF.
We were, before the discussion goes in a platform direction, at the point, why SVG pins slow, while bitmap pins are fast. There could be two reasons:

Conversion from SVG to drawing commands is slow
Drawing the commands to the canvas is slow
To check this, we have to make some performance checks.

One problem I had, was, that Mapsui.Forms relays on SkiaSharp, but the underlying Mapsui doesn't. The used renderer than use SkiaSharp again. So I had to bring special SkiaSharp objects through the Mapsui part, so that they are usable. I made this by changing the Register(Stream data) function to Register(object data). With this, it is possible to bring all types of SkiaSharp objects into the renderer.
Bad is, that for an easy way, we have to change the renderer to handle this object.
Ok, that are the circumstances.
Now to the pin problem. As I mentioned before, I didn't find a way to change the color of an SVG pin. If you have better knowledge, perhaps you have a way to do this. My solution was to convert the SVG pin to a bitmap pin and change the color for this bitmap. That was the reason, why I didn't implement a caching, because the user could change the color of the pin, and we need a new bitmap for each color. An Advantage of this is, that the pin draws faster, because copying bitmaps is really fast. The Disadvantage is, that the bitmap isn't as sharp as the SVG.
So you have to decide, what you want:

Accept lesser quality and use bitmaps. Works already.
Want to have better quality, than there are two ways:
a. Add a new PinType for this and extend Mapsui.Rendering.Skia to render this correct.
b.

Sorry, that I could not say, that this works out of the box.

BTW. is it possible to see, what your pins look like?

BTW. is it possible to see, what your pins look like?

I'm afraid I cannot share them here. Is it relevant how they look like? They're not even horribly large or complicated. Typically around 1kB in size.
Mapsui's SvgSample demonstrates nicely that already rather small SVGs like the Pin.svg in the Mapsui repo generate a considerable performance hit (if you have a large number of them).

It is relevant in the case of the complexity of the pin. If you have to draw this pin each time, it could be a problem.

Ok, when I look into the code, then there some "special" problems.
The conversion from string to stream is only for the old version of BitmapRegistry. The new one, which use object doesn't need such conversion. As you could see, the stream is consumed by module BitmapHelper in Skia renderer (see here) or in Xaml renderer (see here). But we need no stream. We could provide a string. That's enough.
So the best way would be to provide a string instead of a stream. Or in the case of the Skia renderer, we could provide a ready drawn SKPicture, which is much faster than a non converted string.
But we can't change the Forms Pin, because of the color. Or do you have any idea?

I think the SvgSample re-use always the same 1 "bitmapId" (here). During rendering the Svg is parsed the first time but the second time it is already parsed as a SKPicture and stored in the SymbolSache (here (skia)or here (xaml)) so the problem is not with the Svg parsing since it is already done once,(on xaml there is a call to the Freeze() method which according to the comment if i understand correctly it even rasterize it ?)
I dont think it will be "much faster" to do it on the BitmapRegistry.
Also questioning the "quality" problem, It seems weird to me that we can't achieve the same level of quality we rasterizing an svg than when we draw the svg using the parse SKPicture, is it not possible that the problem is just with the SKBitmap dimension are not correct to match the screen density and is scaled up afterwards ?

Here additional Information:
If you create your Svgs with Adobe Illustrator they are where slow, when you clean up your Svg. They render parse much faster afterwards.
Program to clean up Svg.
https://github.com/RazrFalcon/svgcleaner-gui
Gitthub issue:
https://github.com/svg-net/SVG/issues/784

While working on #1091 I tested the Many Pin Sample. I don't know why (any HTTP error), but I had no map in the background. With this, the sample is "very" fast. No hanging. So I assume, that drawing the map is the problem and not the Pins.

While working on #1091 I tested the Many Pin Sample. I don't know why (any HTTP error), but I had no map in the background.

That's strange. The OSM background map works well for me in this sample (both on iOS and Android).

With this, the sample is "very" fast. No hanging.

Well, yes. The ManyPinsSample is fast also for me. The SvgSample is the one that is slow (in particular on iOS / iPhone8).

So I assume, that drawing the map is the problem and not the Pins.

I don't think this is the right conclusion.

While experimenting with my demo project I encountered a problem, that is perhaps the reason for the performance problem of Mapsui.
In the demo project I refresh the screen each 1/60 s if needed. This is done by InvalidateSurface(). Up to now, I called this always on the main thread. If I look, how long this takes, I get the following results:

Last draw needs 0.160 ms
Last draw needs 0.168 ms
Last draw needs 0.133 ms
Last draw needs 0.216 ms
Last draw needs 0.200 ms
Last draw needs 0.198 ms

The main thread is blocked for around 200 ms while drawing.
If I call InvalidateSurface() from the timer thread (non main thread), then I get the following results for the blocking of the main thread:

Last draw needs 0.005 ms
Last draw needs 0.005 ms
Last draw needs 0.005 ms
Last draw needs 0.005 ms
Last draw needs 0.004 ms
Last draw needs 0.005 ms

No differences in the result, but is 40x faster back in the main thread.
With these results, I would say, SkiaSharp could draw in the background, if there is no access to main thread parts while drawing. This isn't used for Mapsui. Mapsui is much slower than it should be. The control isn't very reactive.

No differences in the result, but is 40x faster back in the main thread.

Wow, that's an interesting observation. It might also explain a possible platform dependence of this phenomenon (due to different main-thread handling?). Generally I observe Mapsui-based apps to have rather good performance on Android, but in some cases very bad performance on iOS.

With these results, I would say, SkiaSharp could draw in the background, if there is no access to main thread parts while drawing. This isn't used for Mapsui.

Would be great if this worked. Could you push your InvalidateSurface change to a branch, so that one can try it?

Mapsui is much slower than it should be. The control isn't very reactive.

Yes, the control being unresponsive to user interaction is really the main problem I'm seeing ...

@janusw The problem is, that it at the moment isn't working with Mapsui. Perhaps there is some part, that access a part, which only can be access by the main thread. It seems much more work to do this.
Meanwhile, I added some vector tiles to my demo. It is a MBTiles file of Monaco. There isn't any work on performance improvement up to now, so I think, the performance could be made even better.

if (redraw)
{
    dt = _stopWatch.Elapsed.TotalMilliseconds;
    Invalidate();
    Debug.WriteLine($"Last draw needs {(_stopWatch.Elapsed.TotalMilliseconds-dt).ToString("N3", CultureInfo.InvariantCulture)} ms");
}

The actual drawing is not necessarily done synchronously during the InvalidateSurface() call, the actual draw may not even be done on the UI thread, if you want to measure the duration of a draw you must time at the beginning and the ending of the PaintSurface event handler. Look at this test project I made, the actual number of draws is not equal to the number of call of InvalidateSurface() and (at least on my phone) the PaintSurface is not called on the main thread at all, and the PaintSurface method may be called after InvalidateSurface returns ( the conditions when this happens in that particular project may be biased because my PaintSurface is doing almost nothing except clearing the screen)
also the documentation clearly states that invalidateSurface must be called on UI thread
https://docs.microsoft.com/en-us/dotnet/api/skiasharp.views.forms.skcanvasview.invalidatesurface?view=skiasharp-views-forms-2.80.2
This needs to be called from the main thread.

Above you mentioned the documentation from Microsoft regarding SKCanvasView. That is correct. If you look into SKGLView, you won't find such a comment. So SKCanvasView and SKGLView are different.
So the measurement above will show, how long the Invalidate() uses on this (main?) thread.
I would say, SKGLView does the work in the background and needs only a small-time amount on the main thread for copying the screen. SKCanvasView needs much more time on the main thread. That's what I see on my vector tile demo, which never calls InvalidateSurface() on the main thread.
And yes, a benchmarking version would be nice.

Above you mentioned the documentation from Microsoft regarding SKCanvasView. That is correct. If you look into SKGLView, you won't find such a comment. So SKCanvasView and SKGLView are different.

this may also be just something that was forgotten , there is a lot of "To be added" across the documentation of Xamarin.Forms / Skiasharp so I would not rely on the quality of the documentation to take that conclusion.

I would say, SKGLView does the work in the background and needs only a small-time amount on the main thread for copying the screen. SKCanvasView needs much more time on the main thread. That's what I see on my vector tile demo, which never calls InvalidateSurface() on the main thread.

if you replace the SKGLView by a SKCanvasView in my example, you can see that most of the conclusion remains the same,
like you said the drawing is done on the main thread, with SKCanvasView but even if you use the button that calls InvalidateSurface() 300 times, you can see that paintsurface is only called once after the ButtonCLick handler returns
so the implementation of InvalidateSurface is just most probably setting a boolean to indicate a draw need to be done, and probably like i said there is maybe a timer that will trigger the draw immediately during the InvalidateSurface() if the time that has passed since last draw is greater than a certain offset. I suspect this is what is happening in your case since your draw is very long, the timer is always exhausted when you call InvalidateSurface(), but that does not change the fact that you should measure in the PaintSurface event handler to get the timing reliably

So the measurement above will show, how long the Invalidate() uses on this (main?) thread.

just exactly that, nothing less, nothing more, the Invalidate call MAY OR MAY NOT  call the drawing synchronously

I assume, that this wasn’t forgotten. It doesn‘t make sense to draw things for a special GPU on the main thread. So all things for the SKGLView should be drawn on an extra thread. For SKCanvasView this is a totally different thing. It is CPU base and there are things, that have to run on the main thread.

I don‘t want to measure the time the drawing needs on another thread. I‘m interested in the time, I use on the main thread. This keeps Mapsui reactive.

It doesn‘t make sense to draw things for a special GPU on the main thread

what do you mean exactly ? you can control gpus from any thread/core.
To have mapsui responsive, you need to draw 1 frame every 16 ms (to achieve 60fps), so the thread it is drawn on does not matter.
Not doing things on the main thread does not automagically make things responsive, the opposite only is true only : doing blocking operations on the main thread may make things unresponsive.
If that SKGLView thread is producing 1 frame every 200ms you will see only one frame every 200ms.
The main thread can have additional worker threads to perform the drawing because it needs to render multiple things and doing them in parallel may improve performance if you have multiple cores, but the framework still cant allow user code to modify the UI data while drawing is going on so there will probably be synchronisation to wait for all drawing threads to finish their tasks before handling back control to user code. (i.e. if you modify the styles while that SKGLView thread is drawing, that would still be a bug, i assume during the time SKGLView is drawing the main thread is free to draw other UI(for example other xamarin forms) components, but not user code)
There is a very easy experience we can make to verify this : is to add a blocking 5seconds sleep in PaintSurface (on the SKGLView thread) and see if the rest of the interface is still responsive...
The result of my experience is, ... well the main thread does not wait for the SKGLView paint to complete and if you update the ui during the SKGLView is drawing, then you can have race conditions in the drawing thread, it seems very much like a bug to me
see that issue other people are asking about this: https://github.com/mono/SkiaSharp/issues/1465#issuecomment-799277533
(seems the extra thread to draw SKGLView is only on android)

So you will say, that you like to wait, say e.g. 200 ms, to draw the map on the main thread, while nothing else could be done. Correct?

that is the whole point of the UI thread , (which is "nothing else can be done on the UI data hierarchy while you draw, the ui thread mechanism purpose is to serialize all access to ui data, so that there is no race conditions between update and drawing). What i would prefer is obviously for the map to draw very fast.
https://github.com/tmijieux/TestForMapsui/blob/skglview-thread-android/TestMapsui/TestMapsui/MainPage.xaml.cs
in my update example for the SKGLView i added a thread.sleep() in the SKGLView (PaintSurface call) to simulate for a long drawing time, you can see if you test it that if you press any button the program will crash because the ui is updated during the lifetime of the PaintSurface call, that would not happen if the drawing was done, The sleep is exagerated to trigger the problem everytime but this could technically happen without the Thread.Sleep and Task.Delay, the solution i can think of to avoid any concurrent access problem is to have the whole data hierarchy accessed by the SKGLView PaintSurface be hold in an immutable data hierarchy, and an update to the map would replace the whole hierarchy, that way we would have no race problems

to give an example more closely related to what happen in mapsui imagine the user is updating (on the main ui thread) the collection that is backing the GetFeatures() return value of some layer provider(for example, the user is adding or removing a feature from the collection), at the same time on the SKGLView drawing thread the renderer is iterating over that same collection in the VisibleFeatureIterator , what do you think will happen ?

You are right. That leads to problems. But for such a scenario Mapsui isn't made. In most cases Mapsui is fast enough to draw all in a very short time (bitmap tiles with few features). But if you start to add many features, you see a big loose in performance. With this in mind, Mapsui isn't ready for many Pins or vector tiles. If you know this, all is ok.

This can be closed. Most of the performance issues described above were basically fixed for me with #1192.


Broken link

404 for https://github.com/ulixee/scraper-report

Thanks for the catch!

Still broken. Was curious to see the results


Add filter/conversion function that can be applied to editable cells

Version: 2.4.2
beforeSaveCell or onCellEdit should be allowed to return a new value that is then used for newVal.

@bontibon, could you please describe it more detail?

I have an editable text column
When the user clicks the column, the user can edit the value in an input box
When the input box blurs

beforeSaveCell is called
onCellEdit is called (I want to return value to be used as the cellValue)
afterSaveCell is called

the value is saved back to the data object

The change would happen around here:
Replace
    if (onCellEdit) {
      onCellEdit(this.state.data[rowIndex], fieldName, newVal);
    }

With
    if (onCellEdit) {
      newVal = onCellEdit(this.state.data[rowIndex], fieldName, newVal);
    }

@bontibon, sorry for lately reply. I'll think about your requirement, it seems to be ok. :)

Released on v2.4.3


add routeRoles for strictJobs

https://github.com/Shippable/admiral/issues/1364

These are missing justUser permissions, lets add those


Following upgrade to 0.4, can't see the remaining level of the filter

This could be a consequence of a botched upgrade + an updated translation file, however after the upgrade to 0.4, the template sensor indicating the 'age' of the filter is not available anymore. As a consequence it's also not visible anymore in the graphical view.

Are you using another prefix than "Dantherm" when adding the integration? This prefix is now part of the unique ids, not only the friendly names. This means the helper you have created to covert the "level" attribute on filter remain to a sensor, may need to be changed.
If this is not the case I have some other things I need you to check.

The dantherm bit will be whatever prefix you use (with _ instead of spaces):
{{ state_attr('sensor.dantherm_filter_remain', 'level') }}

Sorry for the delay in replying and thank you for the help.
I used the Dantherm prefix by default, and I didn't see the filter remain value (I think). Then I renamed the device to dantherm (all lower case) and chose to rename the entities too. No change.
Currently if I filter the Entity list by "sensor.dantherm", I do see the Filter Remain Level

but it's value is unknown:

Any idea?

This means the helper you have created to covert the "level" attribute on filter remain to a sensor, may need to be changed.

Why do you say 'the helper you have created'? I thought it was created by the integration?

Because creating that sensor is part of the detailed description of how to make the Dantherm app look-alike picture-elements card in the integration description.
The level is an attribute on the filter_remain sensor. If click on the filter_remain sensor you should see the level at the bottom. Sorry for the danish pictures, I hope you get the sense of it.

The picture-elements card cannot access an attribute directly, therefore the need to create the helper to convert the attribute to a sensor.
What I need you to check is, if the helper still points to the filter_remain sensor to grab the level attribute. If the unique id names has changed since you created the helper, this would be the reason for the helper not being available anymore.
Click on the filter remain level helper and click the cog icon.

If the template is not correct anymore, you need to change it.
Let me know if this is the reason.
Tvalley71

This is of course the reason :) I completely forgot that I had created this template... Thanks for the reminder.
And indeed the 'issue' was that I had changed something in the translation for this value, and I didn't take the change into account.
Thanks again, this is fixed now.

Nice to hear.


fix scoping issue for serviceaccount. Fixes #6142

fixes #6142 - scoping issue for the serviceaccount

This change is

/assign @lachie83

/ok-to-test

/assign

/lgtm


[Python] Support using keyless auth for Azure OpenAI

We try to use keyless auth for all our samples, so if I were to use TypeChat in them, it'd need to support keyless auth, aka this constructor:
    openai_client = AsyncAzureOpenAI(
        api_version=api_version,
        azure_endpoint=endpoint,
        azure_ad_token_provider=token_provider,
    )

I'd be happy to just pass in my whole client to typechat as I typically already have it configured correctly for other uses in my app. Trying to match environment variables will be a pain since we have existing environment variable names that arent the same as yours.

Keyless auth is now a strict requirement for all our samples, so this a blocker for us.


Dune builtin libraries are not taken into account

A number of libraries are defined as builtin in dune itself: https://github.com/ocaml/dune/blob/cd7ea706bc3c56f8bf3bcd979515ddabb84e45e0/src/dune_rules/findlib/meta.ml#L149
Test case:
$ cat dune
(library
  (public_name test)
  (libraries threads))
$ cat test.opam
opam-version: "2.0"
depends: [
]
$ touch test.ml
$ dune-opam-lint
Done: 0/0 (jobs: 0)test.opam:directorieses
  "ocamlfind" {>= 1.8.1}
Wrote "./test.opam"

In this case ocamlfind is not needed as dune is gonna use the builtin threads module anyway.

Ah, OK. It's because the lib/threads/META file is created by ocamlfind, but it does indeed seem that we can build without it. I'll remove threads from the dependencies list.

The dune man-page says:

The output of dune external-lib-deps @install should be included in what is written in your <package>.opam file.

Perhaps dune should be updated to filter threads out itself?

there are a number of builtin libraries, not just threads. I just used threads as an example. Could this issue be reopened?

Shouldn't it be a bug in dune? Dune is telling us this needs to be in the opam file.

Anyway, I've enabled cram tests now. Feel free to add any examples of it not working to the tests directory.

I've added all the ones I could find in #11. If you know of more, please add them :-)


Change default branch to main

Update to the current github standard.

Related to https://github.com/rubyjs/libv8-node/issues/47

This was asked in https://github.com/rubyjs/libv8-node/pull/41#issuecomment-1411411344


update readme: add discord badge and link

Signed-off-by: xutweety<EMAIL_ADDRESS>

Codecov Report

Merging #287 (c397681) into main (cf1ae87) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #287   +/-   ##
=======================================
  Coverage   54.86%   54.86%
=======================================
  Files          41       41
  Lines        3820     3820
=======================================
  Hits         2096     2096
  Misses       1724     1724

Flag
Coverage Δ

rust
54.86% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


[Feature] Add support for persistent_workers in PyTorch >= 1.7.0

Codecov Report

Merging #71 (a6925c8) into master (d6ef7a6) will increase coverage by 0.00%.
The diff coverage is 50.00%.

:exclamation: Current head a6925c8 differs from pull request most recent head 0ff328b. Consider uploading reports for the commit 0ff328b to get more accurate results

@@           Coverage Diff           @@
##           master      #71   +/-   ##
=======================================
  Coverage   75.80%   75.80%
=======================================
  Files         117      117
  Lines        7715     7718    +3
  Branches     1422     1423    +1
=======================================
+ Hits         5848     5851    +3
  Misses       1494     1494
  Partials      373      373

Flag
Coverage Δ

unittests
75.80% <50.00%> (+<0.01%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

mmgen/apis/train.py
20.00% <ø> (ø)

mmgen/datasets/builder.py
51.85% <50.00%> (-1.09%)
:arrow_down:

mmgen/models/architectures/stylegan/mspie.py
91.58% <0.00%> (+0.99%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d6ef7a6...0ff328b. Read the comment docs.


ghcide fails with linker error due to missing libc symbols

When trying to use ghcide in my codebase I run into a weird linker issue where Haskell symbols can't be loaded because libc symbols they refer to are missing, as per the following error:
ghcide: /home/mverstra/.cabal/store/ghc-8.6.5/persistent-sqlite-<IP_ADDRESS>-7f73d67da3642575ea84da1a97de2359947f3009597095d9e4a57c670f242398/lib/libHSpersistent-sqlite-<IP_ADDRESS>-7f73d67da3642575ea84da1a97de2359947f3009597095d9e4a57c670f242398.a: unknown symbol `stat64'
ghcide: /home/mverstra/.cabal/store/ghc-8.6.5/persistent-sqlite-<IP_ADDRESS>-7f73d67da3642575ea84da1a97de2359947f3009597095d9e4a57c670f242398/lib/libHSpersistent-sqlite-<IP_ADDRESS>-7f73d67da3642575ea84da1a97de2359947f3009597095d9e4a57c670f242398.a: unknown symbol `sqlite3_errmsg'
ghcide: ^^ Could not load 'persistentzmsqlitezm2zi10zi6zi2zm7f73d67da3642575ea84da1a97de2359947f3009597095d9e4a57c670f242398_DatabaseziSqlite_zdfShowError_closure', dependency unresolved. See top entry above.

File:     /home/mverstra/gpu/benchmark-analysis/src/Schema/Implementation.hs
Hidden:   no
Range:    1:0-100001:0
Source:   typecheck
Severity: DsError
Message:
  Program error:
  ByteCodeLink.lookupCE
  During interactive linking, GHCi couldn't find the following symbol:
  persistentzmsqlitezm2zi10zi6zi2zm7f73d67da3642575ea84da1a97de2359947f3009597095d9e4a57c670f242398_DatabaseziSqlite_zdfShowError_closure
  This may be due to you not asking GHCi to load extra object files,
  archives or DLLs needed by your current session. Restart GHCi, specifying
  the missing library using the -L/path/to/object/dir and -lmissinglibname
  flags, or simply by naming the relevant files on the GHCi command line.
  Alternatively, this link failure might indicate a bug in GHCi.
  If you suspect the latter, please send a bug report to:
<EMAIL_ADDRESS>
I've tried to debug this further, but I keep getting stuck. The codebase where it goes wrong can be found here: https://github.com/merijn/Belewitte
The C++ parts of the repo aren't relevant to Haskell code or error, running make in the benchmark-analysis directory is sufficient to build the Haskell bits of the repo. The code/makefile assumes GHC 8.6.5 and cabal-install >2.4. It can be build with GHC 8.10.1 too, if the Config.mk is edited to have PROJECTFILE:=cabal.project.ghc-8.10 (You can also point it at the right GHC and cabal-install binary if the wrong one is on PATH).
Running ghcide in the benchmark-analysis or any of the source files in it invariably triggers this error.
/CC @mpickering

is ghcide built by the same version of ghc being used to build the codebase?

I'm using 8.6.5 for both. @mpickering was able to reproduce it before too. I suppose I could try what happens if I use 8.10 for both, but since the problem seems to be in the linking of libc I'm not sure whether it'd really affect anything.

How do you build ghcide? If you are downloading a pre-built ghcide binary, it could have been linked to a different libc version.

If you are embedding sqlite and using a statically linked build of ghc, you are probably running into this bug:
https://gitlab.haskell.org/ghc/ghc/issues/7072
Does ghci work?

I'm building ghcide myself from digital-asset master and from mpickering's fork. Copying ghcide's flags and running ghci manually works fine.

Have you tried dynamically linking ghcide?

Not yet, is it as simple as dynamic: True in cabal.project or do I need to do something more involved?

You can add executable-dynamic: True to the cabal.project or pass --enable-execute-dynamic on the command line.

Ok, executable-dynamic: True seems fo fix this issue, so this appears to be the same as the ghc issue @pepeiborra linked, I'm not sure if this can be/should be fixed in GHC, ghcide or it's just a matter of "always link ghcide dynamically", though.

I suspect the advice should indeed be "always link ghcide dynamically", just like ghc itself is built in most platforms.

Ok, maybe we should add the always link ghcide dynamically if ghc is to troubleshooting cause there are several issues here in hls about.


Privileged mode requirement

Hi,
I've noticed that when running this container on Google Cloud it's required to use --privileged mode as it's  giving EACCES error otherwise. Not sure if is there a workaround to avoid privileged mode.
I guess it would be good to update documentation. Took me days to find out how to make it work
See the gist -> https://gist.github.com/Maxim-Filimonov/36862bc1c4535f9ab036#file-docker-log-log

Can you please document your solution with --privileged?
This needs access to the docker socket, as it is documented in the README,
in order to call the Docker API to access the logs and stats.
Il giorno dom 1 mar 2015 alle 13:25 tpaktop<EMAIL_ADDRESS>ha
scritto:

Hi,
I've noticed that when running this container on Google Cloud it's
required to use --privileged mode as it's giving EACCES error otherwise.
Not sure if is there a workaround to avoid privileged mode.
I guess it would be good to update documentation. Took me days to find out
how to make it work
See the gist ->
https://gist.github.com/Maxim-Filimonov/36862bc1c4535f9ab036#file-docker-log-log
—
Reply to this email directly or view it on GitHub
https://github.com/nearform/docker-logentries/issues/9.

I presume you are asking me to update the docs. See https://github.com/nearform/docker-logentries/pull/10

Thanks, that was perfect! Cheers!


Update toolz version requirement (>=0.7.2 to >=0.7.3)

With toolz 0.7.2, import dask.bag fails.
$ pip freeze | grep toolz
toolz==0.7.2
$ python -c 'from toolz import peek'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name 'peek'
$ pip install toolz==0.8.2
...
$ pip freeze | grep toolz
toolz==0.8.2
$ python -c 'from toolz import peek'

Was peek first introduced in 0.8.2?  cc @eriknw

Also thank you very much for identifying the error, and for providing a fix.

For sure! Was pretty light on the investigation here, so there may be a lower compatible version.
We're ramping up our production usage of Dask here at Opendoor and are looking forward to contributing more going forward.

Ah, I appreciated the shoutout in the recent article about your stack that made it to Hacker News.
I hope to see more of you all around here :)

Thank you @brianmartin .  Merging


refactor: Generalise archimedean lemmas to semirings

A bunch of lemmas are true in ℕ but previously did not apply there.
Co-authored-by: Sébastien Gouëzel<EMAIL_ADDRESS>

Thanks! :tada:
bors merge


No package 'wayland-egl' found

When building in master for the raspberrypi3-64 machine (bitbake core-image-minimal),
I now encounter the error:
"No package 'wayland-egl' found" during the gtk+3_3.22.29.bb:do_configure task.
This appears to be related to the addition of do_install_append_rpi() in wayland_%.bbappend from commit de8912cb.

Hi @billj-korg , can you please post the layers you are using and the relevant parts of your conf/local.conf file for debugging? Thanks

Layers are:
BBLAYERS ?= "
/data/development/lfs/yocto/poky/meta
/data/development/lfs/yocto/poky/meta-poky
/data/development/lfs/yocto/poky/meta-yocto-bsp
/data/development/lfs/yocto/poky/meta-openembedded/meta-oe
/data/development/lfs/yocto/poky/meta-openembedded/meta-python
/data/development/lfs/yocto/poky/meta-openembedded/meta-multimedia
/data/development/lfs/yocto/poky/meta-raspberrypi
/data/development/lfs/yocto/poky/meta-intel
/data/development/lfs/yocto/poky/meta-korg-spark
"
and local.conf relevant parts are:
MACHINE ??= "raspberrypi3-64"
IMAGE_INSTALL_append = " apt gdbserver openssh-sftp-server mdns-dev perfcnt-enable libasound alsa-utils-alsactl alsa-utils-alsamixer ${VIRTUAL-RUNTIME_alsa-state} libjack jack-server jack-utils"
CORE_IMAGE_EXTRA_INSTALL += " avahi-daemon"
TOOLCHAIN_HOST_TASK_append = " nativesdk-cmake"
EXTRA_IMAGE_FEATURES ?= "debug-tweaks x11-base tools-debug splash ssh-server-dropbear package-management"
Regarding our meta-korg-spark custom layer, it adds a linux recipe that just adds an entry to the cmdline.txt boot file.

thanks, can you try the recent PR #257 please?
we addressed now to only remove libwayland-egl from wayland only if userland is used at the same time.

This is interesting because I don't see any vc4graphics setup there. @billj-korg please take a look at the PR @hhromic referenced above and tell us if it fixes your error.

I tried the change in PR #257 and it fixes the error.

good stuff @billj-korg !  let us know if you find something else.


Unable to open private Git repositories hosted on AWS CodeCommit

As per title, I receive the following error message:

我也想知道这个怎么解决，在vscode 是正常的 在codeapp 拉去就报错


Automated cherry pick of #5196: fix: gcp win os not support change config when vm running

Cherry pick of #5196 on release/3.1.
#5196: fix: gcp win os not support change config when vm running

/lgtm

/approve


No more GUI interface

What is your Quests version / build number (do not say "latest")?
3.6.1
Describe the problem and how to replicate it. Any console errors to post using Pastebin.com?
When clicking on NPC instead of GUI it prints a list in chat window.
If applicable, can you provide an example quest from quests.yml using Pastebin.com?

Thanks for your report. I'm afraid I need more information than this. Can you provide an example quest using Pastebin.com? Or, provide steps to replicate the issue?

OK I just tried upgrading a second time from 3.3.4 to 3.6.1 The gui seems to work now.  However only 1 log type allowed per stage with both old and latest version 3.6.1.  I try to put multiple log types in by LOG LOG:1 LOG:2 etc it says not a valid block type.  Someone said use durability option from menu  but then I have to enter LOG LOG LOG and it says duplicate names not allowed.  So all quests have to have just one log type per stage forcing them to do it in order which is not desirable at all.


[fix] Allow user to freely move the viewpoint while using the map

Allow use to drag and browse the map freely.