Performance issue with > 1000 markers and cluster

Hi, thanks a lot for this awesome component ! Great work !
I have few questions:

First thing is that the "click_lat_lng" on Marker seems to be absent of 0.0.17. You mention this addition in multiple issues, referencing 0.0.4. Have those 2 tags diverged somewhere ?
Since click_lat_lng is present on Circle, we used this instead of Marker. So that part works... but we have huge performance problems with > 1000 Circles... clicking on one item takes few seconds before the callback is called and update a "log" with clicked marker id... did you experience this as well ?

I don't have exeperience with react so I may not be able to help a lot here.
Anyway keep up the good work on this !

I have started working on events for the marker cluster component. One of the properties i had in mind was click_marker_id, which holds the id of the latest marker clicked. I just tried the previously posted example with the new syntax, i.e.
import json
import random
import dash
import dash_html_components as html
import dash_leaflet as dl
from dash.dependencies import Output, Input

n = 10000
lats = [random.randrange(44000, 48000) / 1000.0 for i in range(n)]
lons = [random.randrange(0, 4000) / 1000.0 for i in range(n)]
markers = [dl.Marker(id=str(i), position=(lats[i], lons[i])) for i in range(n)]
cluster = dl.MarkerClusterGroup(children=markers, id="cluster")

app = dash.Dash(external_stylesheets=['https://codepen.io/chriddyp/pen/bWLwgP.css'], prevent_initial_callbacks=True)
app.layout = html.Div([
    dl.Map(children=[dl.TileLayer(url="https://a.tile.openstreetmap.org/{z}/{x}/{y}.png"), cluster],
           style={'width': "100%", 'height': "400px"}, center=[46, 2], zoom=7, id="map", preferCanvas=True), html.P(id="log")],
    style={"position": "relative", 'width': '1000px', 'height': '300px'})

@app.callback(Output("log", "children"), [Input("cluster", "click_marker_id")])
def marker_click(marker_id):
    return marker_id

if __name__ == '__main__':
    app.run_server(debug=False)

and the performance it somewhat better, though still not blazing fast (for 10.000 markers, the previous code took 1300 ms on my desktop vs 300 ms for the new implementation). If you want to try it out, you can get the code like this,
pip install dash-leaflet==0.0.18rc2
Please note that the syntax (e.g. the name; should it be click_marker_id? or marker_click_id or something else?) is still subject to change.

Hi, thanks for this update !
I will try this as soon as I can.
On our side we started to implement a "simpler" solution with a very simple wrapper around leaflet without wrapping all items into react components and integrate prunecluster.
We reach ~100ms reaction on click on a marker with 400.000 clustered points.
The pros of your proposed component is that everything is customizable... in our solution we still have a lot of work to bind props... so we may switch back to this one if the performance are enough.
Thanks again for that project. I come back to comment and provide feedback as soon as I did my tests.

Wow that is impressive! Could you link to the code? I am curious why the performance difference is so big!

OK. I don't think that clustering is the problem, is it ? I think I tested dash-leaflet without clustering and there was still a perceivable delay on marker click with 10.000 markers...

@jbjuin - Yes and no. Without clustering, all the markers must be created, which slows down the map. With clusters, only a few markers (the cluster "bubbles") are drawn on the map, which is much faster. Hence, from this point of view, the clustering actually speeds up the map. However, the clustering also introduces some additional calculations, which might slow down the map. For the markercluster package that i am using, this seems to become a problem for more than a few 1000 markers. The prunecluster on the other hand seems to be able to handle a lot more markers, i guess due to a more efficient clustering algorithm.

Glad to find this thread - I am having similar performance issues when clustering 26,000+ points. Except, I'm not sure if I should continue in this thread or start a new one so just let me know if I should start a new one.
I am using the dl.MarkerClusterGroup.
I noticed that there is a chunkedLoading option in the original leaflet.js so tried enabling that option as follows:

Sadly that didn't seem to translate into any performance improvements. I did a little search/CTRL Fing through the dash-leaflet source code to see of any mention of a chunk option but, alas, did not see any. So I either missed it or it's not currently available. Did this option make it into your port?
I also tried setting the preferCanvas option as recommended above but that didn't seem to speed anything up :/ Either way it takes about 25 seconds to load... and also appears to drop about 10,000 points. Can see it drag here

Man, I really enjoy the UI of leaflets cluster map in your Dash port - it's so nice for example how it consolidates/arranges many pins associated with the same point

if there is anyway to improve the performance to get 10s of thousands of these in cluster, I am all ears. Glad to help as possible too :)

Note - there is activity on this topic posted on the Dash forum here

I haven't made any efforts to explicitly support chunkedLoading (which is why you cannot find it in the source code), but since the options are passed directly to the JavaScript layer, i don't see why it shouldn't work.
That being said, i don't think that chunkedLoading is the solution. To me it seems to just postpone the issue slightly. In order to really improve the handling of many markers, i think the proper solution would be to change the underlying clustering algorithm to use supercluster (like mapbox) or the algorithm used by PruneCluster.

@emilhe , thank you for your response! :)
On chunkedLoading
Yea, I'm not sure if this option is getting engaged or, like you said - if it is, it's not the solution so not having a significant impact. That's my perception after running network profiles under these conditions:
15ish seconds before significant visible dom rendering

"chunkedLoading": True,
preferCanvas=True,

10ish seconds before significant visible dom rendering

"chunkedLoading": True,
#preferCanvas=True,  ##(so no canvas)

15ish seconds before significant visible dom rendering

#"chunkedLoading": True, ##(so no chunked loading)
preferCanvas=True,

Judging by those numbers - it's a wash.
Support of slow clustering algorithm as culprit
In every profile I observed a 10 second gap between initial loading of resources and when leaflet loads its map tiles:

... to me this seems to support your idea that the clustering algorithm could be responsible for the delay we're seeing.
I tried to look into what the CPU was doing during this time. Not sure I used the right tool for this but here's what Firefox had to report on the matter:

there were about 18 seconds of rendering activity taking place, much of which seemed a play between dash renderer and dash_leaflet.
there was 11 seconds of idle time which I don't understand.

JSON payload optimization
As seen above, the JSON payload the browser received is over 6MB.
Missing markers / json payload optimization
None of this is able to explain why over 10k markers are missing. Will have to dig into this more. ### JSON payload optimization
The JSON payload the browser gets is over 6MB.

Circling back on these missing markers
Turns out I'm actually missing well over 20,000 markers. At nearly 27k markers in my cluster:

only about 2.7k get mapped:

I doesn't appear that this is because my lat-lon pairs are out of bounds... but perhaps I am wrong?

It would definitely be nice to reduce the JSON payload size. However, as i understand, the kind of optimizations that you are suggesting (e.g. renaming children) would need to be implemented in Dash itself. From a dash component (as Dash Leaflet is), you can only change the object structure - not how the structure is serialized to JSON.
That being said, it would be possible to redesign the cluster component in such a way that the JSON payload is reduced. As an example, say rather that than a list of children that the cluster component would take as arguments list of coordinates along with a single marker (which would be placed at all positions). This structure would reduce the JSON payload a lot (all the overhead that you point out would essentially be gone), but at the cost of flexibility.
If it turns out that the JSON payload is the main issue, i will look further into how it can be reduced without reducing flexibility too much. However, for now i still think that there are other issues (the clustering algorithm, and possible also the dash rendering) that might be more prominent. You benchmarks look really cool - please let me know if you come up with more definitive answers on what the actual performance bottlenecks are.

Hey! Thanks for that message :)
Yes, as you indicated, Dash rendering seems to be a bottleneck.. or at least, it is being held up by the 7 second average load time of async-plotlyjs.js as articulated here.

Perchance any thoughts on the dropped markers?

Little update here - and maybe I should file this into a new issue:
When zoomed out, the leaflet cluster indicates that there are no more than 3k markers in the main cluster. But when zoomed in it shows several clusters that have over 4.5k markers.

So it looks like the data is actually in the map, it's just not getting shown correctly. Any ideas why that may be?

I have never seen anything like the dropped markers, but my guess would be that you are seeing a bug in the underlying JS library. Are you able to share the data for testing?

Hey @emilhe, thank you for your message :) Here is a link to a self contained dash app example that includes the data (note that running the app via debug_mode=True will cause it to take a really long time to load which is why I have it running in debug_mode=False.
JSON payload

If it turns out that the JSON payload is the main issue, i will look further into how it can be reduced without reducing flexibility too much. However, for now i still think that there are other issues (the clustering algorithm, and possible also the dash rendering) that might be more prominent.

Having tinkered quite a bit over the last two days, I am thinking you are right about dash_renderer being an issue, except maybe with a twist - it's the combination of  dash_renderer having to parse the large JSON payload structure that dash_leaflet is sending.
As seen below, there is an average 1.5 seconds the client is waiting to receive the Dash Layout payload (which includes the Leaflet payload of 20k Markers). I imagine much of this wait is on Dash server bundling the Markers into a Dash Layout and then compressing this for delivery client side.

Once client side, the 20k Markers need to get added to the DOM so I think that's why we see a pause in network activity between seconds 4 and 8.5 above. Part of this time - I'm assuming - is decompressing the payload but most of it is indeed spent adding Marker elements to the DOM, as seen from the Chrome Dev Tools Performance tab below (which has a different timeline only because Chrome Dev Tools don't allow you to monitor Network and CPU performance at the same time and I had to run this benchmark against a separate page reload).

The big yellow block in the image above is the time tied up with javascript adding markers to DOM. ... Looking at memory allocation confirms this.
So if you think there is a way to reformat the Marker's Payload to have a smaller memory footprint, I am all ears and hands on deck to help with that. I don't have a dash components development setup just yet but if I can be of assistance by digging into the code that way and/or testing on my side - I'm glad to do that :) Let me know how I can help

I just did some quick testing. I think that the markers are actually not missing. In e.g. the image below, it might look like there are only "2553" markers in the blob,

But if you look closely, a "5" is placed below the bubble, i.e. the real number is probably "25535", but with the last digit printed on the next line. I guess you can avoid the splitting of numbers across lines by adjusting some CSS.
I just tried out the performance optimization i suggested in the previous post, and it speeds up your example a lot. Hence i think that you are right that the large JSON payload is responsible for a significant part of the performance issues. I will dig more into it tomorrow.

This is so cool! Will stand by excitedly :)
As for the carriage return on the fifth significant figure, I'm embarrassed I didn't catch that. Looking into the CSS to fix that. Thank you so much for your help :) Also, does this project by any chance have a way of receiving donations? I am interested in supporting this project :)

Changing the structure of the cluster component to accept a list of coordinates (rather than a list of markers), i was able to reduce the rendering time of your example by around 50% (from ~3s to ~1.5s on my laptop). The improvement seems to be mainly related to speed up of the dash renderer.
After doing a little digging into the function calls, it seems that most of the remaining time is spent constructing the Marker objects, first in React and then in Leaflet. It also explains why you see a speed up when omitting Tooltip object; when they are present, event more objects must be constructed.
Hence, for now, i would say that the key to improve (loading) performance would be to (somehow) delay the construction of markers until they are actually needed. Implementing this kind of behavior from scratch would require some work though. Hence i am not sure it would be worth it compared to just porting e.g. Mapbox GL, which has these features built in.

I have just made a quick and dirty implementation of a SuperCluster  component to get an idea about potential performance improvements. And it is fast. Like really fast. Here is your example with ~ 30k points,

On my laptop, the app renders in about 200 ms. In comparison a blank Dash app takes around 50 ms to load, i.e. the map load in about 150 ms!

Waaaaat!? This is amazing! How is that even possible :)

I am running the example locally on my laptop, so the JSON file is available on disk. If you would have to download it, the data transfer (in the current form, around 3 MB) would of course take some time. But unless you connection is very slow, download 3 MBs shouldn't take that long.

Only was only referring to the data loading taking place in the JS layer. The component is created from Dash, i.e. it looks like this,
import dash
import dash_html_components as html
import dash_leaflet as dl

app = dash.Dash(prevent_initial_callbacks=True)
app.layout = html.Div([
    dl.Map([dl.TileLayer(), dl.SuperCluster(data='assets/myfile.geojson')],  center=(42.54, -78.05), zoom=5,
           style={'width': '100%', 'height': '50vh', 'margin': "auto", "display": "block"}),
])

if __name__ == '__main__':
    app.run_server(debug=False)

I also tried passing the data directly. It adds a slight delay (seem to be around ~ 200 ms), but it is not that much.

Oh wow!! Very very cool :)
Like I've said before, I'm glad to help test, measure and debug however needed. Just let me know. This is so cool!

I just pushed a dev version (0.0.21rc1) with the new component. If you install it, you should be able to run the following example,
import dash
import dash_html_components as html
import dash_leaflet as dl
import json
from dash.dependencies import Output, Input

app = dash.Dash(prevent_initial_callbacks=True)
app.layout = html.Div([
    dl.Map([dl.TileLayer(), dl.SuperCluster(data='assets/myfile.geojson', id="supercluster")],  center=(42.54, -78.05), zoom=5,
           style={'width': '100%', 'height': '50vh', 'margin': "auto", "display": "block"}),
           html.Div(id="n_clicks"),
           html.Div(id="marker_click")
])

@app.callback(Output("n_clicks", "children"), [Input("supercluster", "n_clicks")])
def func(n_clicks):
    return n_clicks

@app.callback(Output("marker_click", "children"), [Input("supercluster", "marker_click")])
def func(marker_click):
    return json.dumps(marker_click)

if __name__ == '__main__':
    app.run_server()

Note that the data must now be supplied as GeoJSON. I have attached your data converted to geojson af an example (just rename it to .geojson).
myfile.txt

This is soo cool! Will play with this this evening :)

Okay, I was able to install dev version 0.0.21rc2 and ran it. Got the following error:
TypeError: The `dash_leaflet.SuperCluster` component (version 0.0.21rc1) received an unexpected keyword
argument: `url`
Allowed arguments: data, id, marker_click, n_clicks, options, zoomToBoundsOnClick

so I changed url to data (even though I knew it was still going to throw an error) and then got the following error
TypeError: The `dash_leaflet.SuperCluster` component (version 0.0.21rc1) received an unexpected keyword
argument: `format`
Allowed arguments: data, id, marker_click, n_clicks, options, zoomToBoundsOnClick

Thus it really doesn't like the line in dl.SuperCluster(url='assets/leaflet_50k.pbf', format="geobuf", options={"radius": 100})], in:
dl.Map([dl.TileLayer(),
	dl.SuperCluster(url='assets/leaflet_50k.pbf', format="geobuf", options={"radius": 100})],
	center=(-37.79, 175.27),
	zoom=13,
	style={'width': '100%', 'height': '50vh', 'margin': "auto", "display": "block"})
),

Can I feed a geopandas dataframe (as geojson) to this data argument?

From you error it seems you installed the wrong version?

Oh wow! I just upgraded to 0.0.21rc2. The difference is night and day :) Very exciting! Am now playing with injecting my data from Dash (rather than file).

Great! Ideally, how would you inject the data? Lists? Or maybe a data frame?

Ideally as a dataframe since that's the form the data is originally in. I
suppose that geopandas could be used as well but it looks like one
could achieve
the same effect by only importing geojson
https://gis.stackexchange.com/a/221040/139332 which I presume is lighter
than geopandas. But if it absolutely had to be converted to a list, then
I could always do dataframe.to_list().. That just adds a little overhead
for the conversion but is what I was doing earlier to generate markers for
the prior/non-supercluster version of dash-leaflet.
On Tue, Jul 14, 2020 at 6:43 AM Emil Haldrup Eriksen <
<EMAIL_ADDRESS>wrote:

Great! Ideally, how would you inject the data? Lists? Or maybe a data
frame?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/thedirtyfew/dash-leaflet/issues/24#issuecomment-658133017,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACAKN3M4FQCTXM5I2PWC6L3R3RAE5ANCNFSM4N2KEB7Q
.

By a list, i was thinking along the lines of a list of records. Hence, the syntax wold be something like,
markers = [
    dict(lat=55, lon=10, tooltip="I am a tooltip"),  # marker with tooltip
    dict(lat=56, lon=10, popup="I am a popup"),  # marker with popup
    dict(lat=55, lon=11, markerOptions=dict(opacity=0.5)),  # marker with marker options, in this case a change in opacity
    dict(lat=56, lon=12, markerOptions=dict(icon=dict(iconUrl="assets/icon_plane.png"))),  # marker custom icons options, in this case a plane as icon
]

If you had a dataframe with the data,
data = [
    [55, 10, "I am a tooltip", None, None, None],
    [56, 10, None, "I am a popup", None, None],
    [55, 11, None, None, "0.5", None],
    [56, 11, None, None, None, "assets/icon_plane.png"]
]
df = pd.DataFrame(data=data, columns=["lat", "lon", "tooltip", "popup", "opacity", "icon"])

the conversion would be something like,
def row_to_marker(row):
    marker = {key: row[key] for key in df.columns if row[key]}
    # Handle non-direct mappings.
    if "opacity" in marker:
        marker["markerOptions"] = dict(opacity=0.5)
    if "icon" in marker:
        marker["markerOptions"] =dict(icon=dict(iconUrl="assets/icon_plane.png"))
    return marker

markers = df.apply(row_to_marker, axis=1)

Since a lot of marker customization options are dictionaries, i don't see any other options that delegating the dataframe-to-maker conversion to the user. If you have suggestions for a better syntax, let me know.

That makes sense! Although I'm curious if that format brings us back to the original issue of the payload getting large (~6.6MB). Then again in this case I'm not sure how much leaner it could be made aside from minifying key names (i.e. from opacity to op)

The format that we that discussing now (e.g. the markers structure above) is only for use as a wrapper in Python; the actual data will be transferred as GeoJSON. For the structure above, the conversion would look something like this,

Gotcha!! That makes perfect sense - thank you for explaining this :)

I have now (0.0.21rc3) implemented some util methods to ease the construction of markers from python. I have also implemented tooltips, popups, marker options and custom icons. Here is an updated example,
import dash
import dash_html_components as html
import dash_leaflet as dl
import dash_leaflet.express as dlx
import geobuf
import base64
from numpy import random

# Create some simple markers.
markers = [dict(lat=56 + random.rand(), lon=10 + random.rand()) for i in range(10000)]
# Create a few customized markers.
markers += [
    dict(lat=55, lon=9, tooltip="I am a tooltip"),  # marker with tooltip
    dict(lat=55, lon=10, popup="I am a popup"),  # marker with popup
    dict(lat=55, lon=11, markerOptions=dict(opacity=0.5)),  # marker with options
    dict(lat=55, lon=12, markerOptions=dict(icon=dict(iconUrl="assets/icon_plane.png"))),  # marker with custom icon
]
# Convert markers to geobuf (smaller payload than raw geojson).
geojson = dlx.markers_to_geojson(markers)
pbf = base64.b64encode(geobuf.encode(geojson)).decode()
# Create example app.
app = dash.Dash()
app.layout = html.Div([
    dl.Map([dl.TileLayer(), dl.SuperCluster(data=pbf, format="geobuf", options={"radius": 100})],
    center=(56, 10), zoom=7, style={'width': '100%', 'height': '50vh', 'margin': "auto", "display": "block"})
])

if __name__ == '__main__':
    app.run_server(debug=False)

There are still a lot of things that could be added,

Customization of the cluster icon (currently not possible)
Fancy animations (e.g. spiderfy)
Support of more formats, e.g. gzip
Support of children in marker popup/tooltips
....

but i feel that the component is getting close to be "good enough" for release in a first version.

This is so amazing @emilhe ! I will be testing this tonight!
What are you predicting it would take to implement spiderfy? I ask because when zoomed in I've currently got 1000 markers that look like they're only one marker because they're all mapped onto the same spot :/ I think this is already the best cluster mapping package for dash but an addition of 'spiderfy' pushes that up by far. Like I said before, if there is any way I can help - including through donation support - I'd love to do that :) Will get back to you with test results tonight :)

Holy cow @emilhe this is sooo amazing!!! Sorry I'm just getting back to you now- finally got time to implement. This clustering improvement is blazing fast! Have it hosted at dash.scious.io and it's a night and day difference from the original loading speeds.
By the way, that was very clever what you did with geobuf :) I didn't even know that was a thing!
Are you sure there's no way I can support this project with a donation? I would love to help out however I can :)

Great to hear that! You pages looks cool - and the map does indeed load fast ;). I have now finally found the time to wrap of the first version of the component. Compared to the previous version, i have fixed a few bugs and added the following features

Customization of the cluster classes (which css classes to apply and when)
Spiderfy functionality (!)
Events (both click and hoover)

I have also added an example in the documentation. And last, but not least, i have added a donate button in the readme :)

Hey guys I know it is not the place to ask this question but it is kind of related and I have been trying some patters without success.
I am struggling to get the marker id from a LayerGroup by clicking on it, the thing is that the markers are not in a global variable to be accessed using:
@app.callback(Output("log", "children"), [Input(marker, "click_lat_lng") for marker in markers])
def marker_click(*args):
if not any(list(args)):
return ""
marker_id = dash.callback_context.triggered[0]['prop_id'].split(".")[0]
return "You clicked marker {}".format(marker_id)
but they are generated dynamically via callback output in a list using the info from some dropdowns something like this:
dl.Map([dl.TileLayer(), dl.LayerGroup(id="layer")],zoom=12, center=(-33.4569397,-70.6482697),
id="map", style={'width': '100%', 'height': '100vh', 'margin': "auto",})
@app.callback([Output("layer", "children"),Output('download-link', 'href')],
[Input('area_name', 'value'),
Input('score_level', 'value'),
Input('column-selector','value'),])
def update_figure(chosen_area,chosen_level,column):
return my_tab3.update_figure(chosen_area,chosen_level,column,)
Does anyone know the correct pattern to access the marker id by clicking on it if the marker list is generated dynamically via callback?
Thank you in advance.

Hey @emilhe  I just donated!! :) Just wanted to say thank you again for this super cool library!
One question I did have, having just upgraded to the latest release 0.0.21, is there an option I am missing to keep the spiderfied lines drawn while panning/zooming in? The lines currently disappear at max zoom and make it seem like each marker is associated with different places.

@zeluspudding Thanks!! This issue(s) you are seeing probably related to three different point,

There is a bug in (0.0.21) causing the markers to stay in the spiderfied positions. It should be fixed in (0.0.22).

In my demo, the maxZoom of the SuperCluster (defaults to 16) is not the same as the maxZoom of the TileLayer (which is 18 for the default one), which is why the clustering stops at the highest zoom levels. In most cases, you would probably want these two zoom levels to be the same, so i have implemented some auto detection logic (in 0.0.22), which should work if you add the  TileLayer before the SuperCluster.

I implemented the spiderfy logic with the intention to spiderfy a cluster, when you click it, if its children cannot be resolved at the max zoom level. When you change the view port, the spider is "collapsed" back to the cluster state. Would you prefer a different behavior?

@ELGUMOS1 For this kind of use case, you would probably need to use Pattern-Matching Callbacks. It quite a bit off topic though, so i won't post an example here. The question is better suited for the Plotly Dash forum or Stackoverflow.

Of course, you're very welcome!! Thank you! :)

In my demo, the maxZoom of the SuperCluster (defaults to 16) is not the same as the maxZoom of the TileLayer (which is 18 for the default one), which is why the clustering stops at the highest zoom levels. In most cases, you would probably want these two zoom levels to be the same, so i have implemented some auto detection logic (in 0.0.22), which should work if you add the TileLayer before the SuperCluster.

I agree - in most cases folks will probably want the maxZoom of TileLayer to be the same as SuperCluster's

I implemented the spiderfy logic with the intention to spiderfy a cluster, when you click it, if its children cannot be resolved at the max zoom level. When you change the view port, the spider is "collapsed" back to the cluster state. Would you prefer a different behavior?

I think that's exact the desired behavior :) Although perhaps with one tiny modification - rather than "collapse" the spider back to the cluster state on viewport change maybe it could be collapsed on zoom change (which in practice could only be a zoom out since, at this state, the map would be at full zoom). That would match the behavior of your standard leaflet dash component. But this seems like a minor thing. This component is already very useful!! :)

I have started working on events for the marker cluster component. One of the properties i had in mind was click_marker_id, which holds the id of the latest marker clicked. I just tried the previously posted example with the new syntax, i.e.
import json
import random
import dash
import dash_html_components as html
import dash_leaflet as dl
from dash.dependencies import Output, Input

n = 10000
lats = [random.randrange(44000, 48000) / 1000.0 for i in range(n)]
lons = [random.randrange(0, 4000) / 1000.0 for i in range(n)]
markers = [dl.Marker(id=str(i), position=(lats[i], lons[i])) for i in range(n)]
cluster = dl.MarkerClusterGroup(children=markers, id="cluster")

app = dash.Dash(external_stylesheets=['https://codepen.io/chriddyp/pen/bWLwgP.css'], prevent_initial_callbacks=True)
app.layout = html.Div([
    dl.Map(children=[dl.TileLayer(url="https://a.tile.openstreetmap.org/{z}/{x}/{y}.png"), cluster],
           style={'width': "100%", 'height': "400px"}, center=[46, 2], zoom=7, id="map", preferCanvas=True), html.P(id="log")],
    style={"position": "relative", 'width': '1000px', 'height': '300px'})

@app.callback(Output("log", "children"), [Input("cluster", "click_marker_id")])
def marker_click(marker_id):
    return marker_id

if __name__ == '__main__':
    app.run_server(debug=False)

and the performance it somewhat better, though still not blazing fast (for 10.000 markers, the previous code took 1300 ms on my desktop vs 300 ms for the new implementation). If you want to try it out, you can get the code like this,
pip install dash-leaflet==0.0.18rc2
Please note that the syntax (e.g. the name; should it be click_marker_id? or marker_click_id or something else?) is still subject to change.

D

I have started working on events for the marker cluster component. One of the properties i had in mind was click_marker_id, which holds the id of the latest marker clicked. I just tried the previously posted example with the new syntax, i.e.
import json
import random
import dash
import dash_html_components as html
import dash_leaflet as dl
from dash.dependencies import Output, Input

n = 10000
lats = [random.randrange(44000, 48000) / 1000.0 for i in range(n)]
lons = [random.randrange(0, 4000) / 1000.0 for i in range(n)]
markers = [dl.Marker(id=str(i), position=(lats[i], lons[i])) for i in range(n)]
cluster = dl.MarkerClusterGroup(children=markers, id="cluster")

app = dash.Dash(external_stylesheets=['https://codepen.io/chriddyp/pen/bWLwgP.css'], prevent_initial_callbacks=True)
app.layout = html.Div([
    dl.Map(children=[dl.TileLayer(url="https://a.tile.openstreetmap.org/{z}/{x}/{y}.png"), cluster],
           style={'width': "100%", 'height': "400px"}, center=[46, 2], zoom=7, id="map", preferCanvas=True), html.P(id="log")],
    style={"position": "relative", 'width': '1000px', 'height': '300px'})

@app.callback(Output("log", "children"), [Input("cluster", "click_marker_id")])
def marker_click(marker_id):
    return marker_id

if __name__ == '__main__':
    app.run_server(debug=False)

and the performance it somewhat better, though still not blazing fast (for 10.000 markers, the previous code took 1300 ms on my desktop vs 300 ms for the new implementation). If you want to try it out, you can get the code like this,
pip install dash-leaflet==0.0.18rc2
Please note that the syntax (e.g. the name; should it be click_marker_id? or marker_click_id or something else?) is still subject to change.

I tried this and I did not get the marker id in the log or printed?
Any update, is kinda what I am looking for.

@ELGUMOS1 In the new example in the documentation, you can see how to capture both click and hover events,
http://dash-leaflet.herokuapp.com/#super_cluster

@ELGUMOS1 In the new example in the documentation, you can see how to capture both click and hover events,
http://dash-leaflet.herokuapp.com/#super_cluster

Thank you, but one more question. Which version is needed for the supercluster function?, I am getting:
AttributeError: module 'dash_leaflet' has no attribute 'SuperCluster'

I just implemented it, so you should use 0.0.22

It works perflectly fine thanks @emilhe
But is there a way to inyect the markers to the super_cluster via callback?
I got into the same situation as I told you in the previous days, this works pretty well when the markers are previously defined but if they are dynamically generated I still can not find the way to make them appear in the map and then be able to get their ID or any other info. This is the kind of pattern I am using.
`# Create example app.
app = dash.Dash()
app.layout = html.Div([
dl.Map([
dl.TileLayer(),
dl.SuperCluster(data=[],id="sc"),
], center=(-37.8, 175.3), zoom=11, style={'width': '100%', 'height': '50vh', 'margin': "auto", "display": "block"}),
# Logging element.
html.Button('Submit', id='submit-val', n_clicks=0)
])
@app.callback(Output("sc", "children"), [Input("submit-val", "n_clicks")])
def generate_markers(value):
custom_markers = [
dict(lat=-37.8, lon=174.95, tooltip="I am a tooltip"),  # marker with tooltip
dict(lat=-37.8, lon=175.00, popup="I am a popup"),  # marker with popup
dict(lat=-37.8, lon=175.05, markerOptions=dict(opacity=0.5)),  # marker with options
dict(lat=-37.8, lon=175.10, ),  # custom icon
]
return dlx.markers_to_geojson(custom_markers)
@app.callback(Output("log", "children"), [Input("sc", "featureClick"), Input("sc", "featureHover")])
def read_markers_info(click, hover):
return json.dumps([click, hover])
if name == 'main':
app.run_server()`
With one callback generate the markers based on the info from some dropdowns or buttons, then in another callback(if it is possible the same callback)  be able to read the markers id by clicking on them.  I have not found the correct pattern to achieve this behavior, any advice?
Thanks in advance, this proyect is amazing.

@ELGUMOS1 Thanks! The data property of SuperCluster is static (at least for now), i.e. you can't modify it after a SuperCluster has been created. Instead, you should just create a new one, i.e. something along the lines of
import dash
import dash_html_components as html
import dash_leaflet as dl

from dash.dependencies import Input, Output

app = dash.Dash(prevent_initial_callbacks=True)
app.layout = html.Div([
    dl.Map([dl.TileLayer(), dl.LayerGroup(id="container", children=[])], id="map", center=(56, 10), zoom=10,
           style={'width': '100%', 'height': '50vh', 'margin': "auto", "display": "block"}),
    html.Div(id="log")
])

@app.callback(Output("container", "children"), ....)
def update_super_cluster(...):
   data = # generate the data here
    return [dl.SuperCluster(data=data,id="sc")]

Thanks a lot @emilhe I was very lost!
Whoever I tried that pattern several times and the markers in the map are still static, the data in the function changes dynamically based on the values from the dropdowns and buttons but the markers remain the same.
Any thoughts?

@ELGUMOS1 Could you share a MWE? Otherwise it's that to guess what goes wrong. One common pitfall to mention though is that you should change the id of the layer (in this case the SuperCluster) to trigger the map to update. Here is a small example,
import dash
import dash_html_components as html
import dash_leaflet as dl
import dash_leaflet.express as dlx
import random

from dash.dependencies import Input, Output

app = dash.Dash(prevent_initial_callbacks=True)
app.layout = html.Div([
    dl.Map([dl.TileLayer(), dl.LayerGroup(id="container", children=[])], id="map", center=(56, 10), zoom=10,
           style={'width': '100%', 'height': '50vh', 'margin': "auto", "display": "block"}),
    html.Button("Generate some markers", id="btn")
])

@app.callback(Output("container", "children"), [Input("btn", "n_clicks")])
def update_super_cluster(n_clicks):
    random_markers = [dict(lat=56 + 0.1 * random.random(), lon=10 + 0.1 * random.random()) for i in range(100)]
    return [dl.SuperCluster(data=dlx.markers_to_geojson(random_markers), id=str(n_clicks))]

if __name__ == '__main__':
    app.run_server()

@emilhe  You are right! Now it works changing the id of the super cluster.
But now the issue is that I am not able to capture the featureClick event in a Callback because the super cluster id is not fixed and at the end my objective is to get the id of the marker by clicking on it.
Is there any way to achieve this behavior? I Just donated by the way, keep it up guys!

Thanks! I guess you could achieve the desired behavior using pattern matching callbacks,
https://dash.plotly.com/pattern-matching-callbacks
I’ll make an example when I get to a PC. With your usecase I see that it would be simpler if the data property could be changed. I’ll look into that soon (:

I have just created a (0.0.23rc1) release in which the data and url properties have been changed to dynamic. Hence, the following example should now work,
import json
import dash
import dash_html_components as html
import dash_leaflet as dl
import dash_leaflet.express as dlx
import random

from dash.dependencies import Input, Output

app = dash.Dash(prevent_initial_callbacks=True)
app.layout = html.Div([
    dl.Map([dl.TileLayer(), dl.SuperCluster(id="sc")], center=(56, 10), zoom=10,
           style={'width': '100%', 'height': '50vh', 'margin': "auto", "display": "block"}),
    html.Button("Generate some markers", id="btn"),
    html.Div(id="log")
])

@app.callback(Output("sc", "data"), [Input("btn", "n_clicks")])
def update_super_cluster(n_clicks):
    random_markers = [dict(lat=56 + 0.1 * random.random(), lon=10 + 0.1 * random.random()) for i in range(100)]
    return dlx.markers_to_geojson(random_markers)

@app.callback(Output("log", "children"), [Input("sc", "featureClick")])
def super_cluster_click(feature):
    return json.dumps(feature)

if __name__ == '__main__':
    app.run_server()

Thank so much @emilhe  It is exactly what I was looking for.
I finally developed the missing features.  Did you try with > 100k markers ?
On our side we went for a very low level & specific leaflet integration using PruneCluster (https://github.com/SINTEF-9012/PruneCluster) which flawlessly handle 300k markers.
Great work ! Congrats !

Thanks! Yes, i tried with 1 million markers. It took ~ 2s to load, and after that the map was smooth :)

Hi, Guys ,
I have two arrays which consists of lat and long and I want to created circles for each location same as below:
for i in range (len(wellPositions)):
lat = wellPositions[i][0]
long = wellPositions[i][1]

markers = [dl.Marker(id=str(i), position=(lat,long) )]
cluster = dl.MarkerClusterGroup(children=markers, id="cluster")
app = dash.Dash(external_stylesheets=['https://codepen.io/chriddyp/pen/bWLwgP.css'], prevent_initial_callbacks=True)
app.layout = html.Div([
dl.Map(children=[dl.TileLayer(url="https://a.tile.openstreetmap.org/{z}/{x}/{y}.png"), cluster],
style={'width': "100%", 'height': "400px"}, center=[46, 2], zoom=7, id="map", preferCanvas=True), html.P(id="log")],
style={"position": "relative", 'width': '1000px', 'height': '300px'})
@app.callback(Output("log", "children"), [Input("cluster", "click_marker_id")])
def marker_click(marker_id):
return marker_id
I got no error but also I could not see any circle or point on my map.
Can anyone help me please?

Man, I really enjoy the UI of leaflets cluster map in your Dash wrapper- it's so nice for example how it consolidates/arranges many pins associated with the same point

if there is anyway to improve the performance to get 10s of thousands of these in cluster, I am all ears. Glad to help as possible too :)

brother if you didi this please tell me how can i do this i don't know much but i want to make like this round marker integration in map

Man, I really enjoy the UI of leaflets cluster map in your Dash wrapper- it's so nice for example how it consolidates/arranges many pins associated with the same point

if there is anyway to improve the performance to get 10s of thousands of these in cluster, I am all ears. Glad to help as possible too :)

brother if you didi this please tell me how can i do this i don't know much but i want to make like this round marker integration in map

Could you post an MWE of you current code?


Updating the Openverse logo nav component

Problem
In staging, Openverse logo and symbol used in Header and Footer components have inconsistent sizes and focus styles. I think this happens because the root component is not the same as VBrand and VLogoLoader do not relate to each other. See the following screenshots from staging environment.
Header

Resting
Focus

Header internal

Resting
Focus

Footer

Resting
Focus

Now that we started working on the homepage redesign (#1433) it seems worth improving this key nav element.
Description
Upon the homepage redesign, Logo nav component was updated in Figma to simplify the versions used across the site. Including the homepage and the full-screen menu.

The component has three sizes:

Small for the header's small breakpoints where Openverse symbol lives inside the search bar.
Medium for the header's large breakpoints where Openverse symbol lives outside the search bar.
Large for the header and footer versions where Openverse logo is displayed completely.

And three states: Resting, Hover, Focus, and Loading. The last one is for Small and Medium sizes as those sizes are used in the search flow.
Additional context
This request impacts the subsequent work coming from #1433 as

#2051
#2060

And past work coming from #1669 as

#2015

Linking the PR #2067 here for focus styles.

The initial ticket mentioned updating the Footer component (done in #2015), but it was a mistake. The request was updated.


Force hostnames lowercase

Resolves https://github.com/OSU-Net/cyder/issues/570

Is this ready?

No, because although the domain name and the system name are forced to be lowercase, the front label is not... unless that was changed in a different pull request.

So when that change is made the pull request will be ready?

As far as I can tell, yeah. The label cleaning should probably occur in CydnsRecord or LabelDomainMixin or some other centralized location, so that all fqdns are forced to lowercase.

LabelDomainUtilsMixin sounds like the best place to me. It already has a clean method and it's inherited from by every model with a label field.

The specs changed... we now want to retain case (as ugly as it is)


Max[A, B] type class.

I feel bad about basically just copying the Min[A, B] code with minimal changes and pasting my name over it, but here it is :)

Don't feel bad, it's very useful that missing bits like these are added.
You forgot to add yourself to the contributors file though.

Nice work! Thanks :-)
Merging this might have to wait until after 2.2.0, but there might be an RC5, in which case this is sufficiently low risk that it can probably go in.

There is going to be an RC5 so merging now. Thanks again :-)


🧞 Remove the hash from the grammar (if mathematically possible)

Bad:
tup # [ x .= 1 ]

Good:
tup [ x .= 1 ]

Just don't worship LR(1).


[FEATURE]: Add clickhouse SSL\TLS connection

sorry to bother you after PR is completed, but where is any documentation of how to use it? @dmitrybugakov

Hello @paf91
Unfortunately, there is a gap in our documentation regarding SSL connections.
However, you can use this test as an example.


LSP is not starting

I was trying to switch to distant.nvim v0.2, but I noticed that LSP is not started (distant.nvim -> v0.2, distant -> 0.20.0). I haven't had any problems in previous version distant.nvim 0.1.1 and distant 0.15.1. Is it known bug or maybe I have some problem with configuration. I haven't changed anything in configuration in comparison to v0.1.1.
My configuration:
require('distant').setup {
    -- Applies Chip's personal settings to every machine you connect to
    --
    -- 1. Ensures that distant servers terminate with no connections
    -- 2. Provides navigation bindings for remote directories
    -- 3. Provides keybinding to jump into a remote file's parent directory
    ['*'] = require('distant.settings').chip_default(),

    ['localhost'] = {
        distant = {
            bin = '/distant',
        },
        lsp = {
            ['cpp'] = {
                cmd = { 'clangd', '--background-index', "--compile-commands-dir=build" },
                filetypes = { 'c', 'cpp' },
                root_dir = '/mnt',
                on_attach = function()
                    nnoremap('gD', '<CMD>lua vim.lsp.buf.declaration()<CR>')
                    nnoremap('gd', '<CMD>lua vim.lsp.buf.definition()<CR>')
                    nnoremap('gh', '<CMD>lua vim.lsp.buf.hover()<CR>')
                    nnoremap('gi', '<CMD>lua vim.lsp.buf.implementation()<CR>')
                    nnoremap('gr', '<CMD>lua vim.lsp.buf.references()<CR>')
                end,
            },
        },
    },
}

I'm not aware of LSP being broken on the branch as it still passes the tests we have to validate it.
Can you verify that the LSP client is not attached to your buffer? Also, what do the logs say regarding LSP usage?
You can set the environment variables DISTANT_LOG_LEVEL to trace and DISTANT_LOG_FILE to something like /tmp/distant.nvim.log to get full log output.
I'd expect to see log output like
lsp.connect(4)
File /mnt/to/file is within /mnt of cpp
File /mnt/to/file of type ft applies to cpp
Starting LSP cpp
Attaching to buf 4 for LSP cpp

Trying to narrow down if this is an issue with the LSP not starting, the LSP client not attaching to the buffer, or something else.
You can also set neovim's general LSP log level and output using stuff like vim.lsp.log.set_level() and find the file via vim.lsp.log.get_filename() from vim.lsp.log.
If this all looks okay, then we'd want to get logs from the distant server, manager, and client to make sure that they are properly forwarding traffic between your computer and the LSP process.

Unfortunatelly there isn't any try a connection to lsp:
[DEBUG pią, 17 lut 2023, 20:56:56] ...ite/pack/packer/start/distant.nvim/lua/distant/state.lua:74: Manager not listening, so starting process
[DEBUG pią, 17 lut 2023, 20:56:56] ...ck/packer/start/distant.nvim/lua/distant/cli/manager.lua:399: Connecting with options: {
  destination = "localhost:36711"
}
[DEBUG pią, 17 lut 2023, 20:56:56] ...ck/packer/start/distant.nvim/lua/distant/cli/manager.lua:426: Connect cmd: { "/home/jedyny/.local/share/nvim/distant.nvim/bin/distant", "client", "connect", "--unix-socket", "/home/jedyny/.cache/nvim/distant.nvim/nvim-8034.sock", "--windows-pipe", "nvim-2717", "--format", "json", "localhost:36711" }
[DEBUG pią, 17 lut 2023, 20:57:14] ...ck/packer/start/distant.nvim/lua/distant/editor/open.lua:431: retrieved path info for "/mnt"
[DEBUG pią, 17 lut 2023, 20:57:14] ...ck/packer/start/distant.nvim/lua/distant/editor/open.lua:435: does buf "distant:///mnt" exist? "no"
[DEBUG pią, 17 lut 2023, 20:57:14] ...ck/packer/start/distant.nvim/lua/distant/editor/open.lua:474: loaded "/mnt" into buf 2
[DEBUG pią, 17 lut 2023, 20:57:14] ...e/pack/packer/start/distant.nvim/lua/distant/autocmd.lua:17: buf 2 is not initialized, so triggering BufReadCmd
[DEBUG pią, 17 lut 2023, 20:57:14] ...e/pack/packer/start/distant.nvim/lua/distant/autocmd.lua:42: Reading "/mnt" into buf 2
[DEBUG pią, 17 lut 2023, 20:57:14] ...ck/packer/start/distant.nvim/lua/distant/editor/open.lua:431: retrieved path info for "/mnt"
[DEBUG pią, 17 lut 2023, 20:57:14] ...ck/packer/start/distant.nvim/lua/distant/editor/open.lua:435: does buf "distant:///mnt" exist? "yes"
[DEBUG pią, 17 lut 2023, 20:57:15] ...ck/packer/start/distant.nvim/lua/distant/editor/open.lua:474: loaded "/mnt" into buf 2
[DEBUG pią, 17 lut 2023, 20:57:39] ...ck/packer/start/distant.nvim/lua/distant/editor/open.lua:431: retrieved path info for "/mnt/src"
[DEBUG pią, 17 lut 2023, 20:57:39] ...ck/packer/start/distant.nvim/lua/distant/editor/open.lua:435: does buf "distant:///mnt/src" exist? "no"
[DEBUG pią, 17 lut 2023, 20:57:39] ...ck/packer/start/distant.nvim/lua/distant/editor/open.lua:474: loaded "/mnt/src" into buf 3
[DEBUG pią, 17 lut 2023, 20:57:49] ...ck/packer/start/distant.nvim/lua/distant/editor/open.lua:431: retrieved path info for "/mnt/src/simple.cpp"
[DEBUG pią, 17 lut 2023, 20:57:49] ...ck/packer/start/distant.nvim/lua/distant/editor/open.lua:435: does buf "distant:///mnt/src/simple.cpp" exist? "no"
[DEBUG pią, 17 lut 2023, 20:57:49] ...ck/packer/start/distant.nvim/lua/distant/editor/open.lua:474: loaded "/mnt/src/simple.cpp" into buf 4

In lsp log there doesn't exist any entry, so it denotes that distant haven't been trying to establish connection with lsp. Do you have any idea what can be wrong ?
Also in client.log and manager.log I cannot see any things regarding lsp.
I want to highlight that with version 0.1.1 everything was working correctly.

Ok It looks that in newer version it doesn't get my confguration for localhost, when I've configured everything under '*' then it has started correctly lsp server.

Good catch. I'll look through the commits to see what changed and get a fix out for you to try.

@pjedynak out of curiosity, did you try setting the server to be <IP_ADDRESS> instead of localhost to see if it worked?

Okay, identified the bug, so need to get a fix out.
If I do something like DistantConnect distant://:{password}@{host}:{port}, the entire destination is passed as the label instead of just the host for matching purposes. We don't use any kind of partial matching (only exact match), so unless we just pass in the host, this will cause all of the host-specific configurations to not match.
Marking this issue as a bug to resolve.

Should be resolved via https://github.com/chipsenkbeil/distant.nvim/commit/6647d4e0f1b1e2c62327356c4c128936a4b464ff.
Tested it myself and it now parses the destination to extract the host and use that for the settings.


Fix android library loading

bumps up the JNI_VERSION that is returned (1_1 -> 1_2)
adds native library loding using System#loadLibrary if detected OS is Android

This essentially closes #37

Oh, if you want, I can also add instructions to README for including in android project

Oh, if you want, I can also add instructions to README for including in android project

Good idea, I will create a new release after

I'm sorry, I messed up a bit the gradle part:
  workingDir = file("libs/java-llama.cpp/")

should be
  workingDir = file("java-llama.cpp/")


Various modules crash the program upon insertion.

Describe the bug
During development of the automated tests for the module insert combobox, the following modules were found to be detrimental to the health of the program:

module_boundary  # still crashes the program.
module_chain  # still crashes the program.
module_graph  # still crashes the program.
module_motor  # still crashes the program.

Application: Brain Simulator
Module/Network: see above modules
To Reproduce
Steps to reproduce the behavior:

Create one of these modules.
See the program crash.

Expected behavior
Program should not crash but may issue a warning about needing another module.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS:
Version [v0.9a]

Add any other context about the problem here.

In the case of Boundary, apparently an imagefile module is needed first.
In the case of Chain, the module can not be found yet in the neuron array, since it is in the process of being created.
In the case of Graph, something obscure happen in the Initialize that crashes things.
In the case of Motor, the 2x2 size is insufficient for the labeled neurons.

Don’t know about graph…
Chain needs a line if (na==null) return; to its onresize method
Boundary and Motor don’t respect their module’s size.
I’ll look into graph module. DO you want to look at these others or wait for me to get to it?

Motor module now works too.

I already fixed Motor, I guess Boundary is similar. I'll check Chain too

Well, it already crashes because there is no ImageFile module. I'll sleep on it and attempt some improvements tomorrow.

All of these issues have now been fixed.


Implement EETypePtr.EETypePtrOf intrinsic

Resolves #852.

I tested this by replacing String.Empty.EETypePtr in String.cs with EETypePtr.EETypePtrOf<String>(), but I'm not sure if it's a win (we still call into the LdTokenHelper, which might be expensive), so I'm not including that part.
@jkotas PTAL

I'm not sure if it's a win (we still call into the LdTokenHelper, which might be expensive),

We need to make this inlinable. Not sure why the JIT is not inlining it ... you should be able to find out by building debug ryujit.dll at commit where @kyulee1 snapped it last time. Mark it as forceinline, inline the field fetch so that local is not needed, change the to EEType helper to static method so that the address taken local is not needed ... one of these should help.

Part of the plan to make this inlinable may be to change the following line in CorInfoImpl.Intrinsics.cs to map to the raw value helper:
 // table.Add(CorInfoIntrinsics.CORINFO_INTRINSIC_RTH_GetValueInternal, "GetValueInternal", "System", "RuntimeTypeHandle"); // not in .NET Core

The updated iteration inlines/expands everything as expected. EETypePtrOf is now a single load of a constant!

LGTM

@dotnet-bot test this please


[Bug] : Image is getting repeated in Image Utility (49)

What happened?
While using the Image Utility (49) App, I found that as soon as I upload an image that is not an square. The image is getting repeted within the workspace
How can we reproduce this bug?
This bug can be produced just by uploading an image. PS. The image should be a rectangular one (not a square one).

Desktop Information (Optional)
Device: Laptop
OS: Windows 11 Home
Browser: Chrome
Urgency (Optional)
Medium priority
Record

[x] I agree to follow this project's Code of Conduct
[x] I'm a GSSOC'24 contributor
[X] I want to work on this issue

@mondalsurojit All the best 👍🏻


GH-43506: [Java] Fix TestFragmentScanOptions result not match

Rationale for this change
JNI test was not tested in CI. So the test failed but passed the CI.
The parseChar function should return char but return bool, a typo error.

What changes are included in this PR?

Are these changes tested?
Yes

Are there any user-facing changes?
No

GitHub Issue: #43506

Rename to ->  GH-43506: [Java] Fix TestFragmentScanOptions result not match

@github-actions crossbow submit -g java

@jinchengchenghh I think still there is an inaccuracy in the test case. One CI is failing.

Still the CI is failing...

Still failing right...

I will take a look today.

Only gandiva failed now. @vibhatha

@jinchengchenghh it seems like there is a symbol not found issue again; https://github.com/apache/arrow/actions/runs/10363135011/job/28686234425?pr=43639#step:7:20512

Do I need to disable this test in this PR? Or you will create a new PR to disable it?

@kou shall we disable this like the last time but note this as an issue and comment in the code?

Yes but why is this happen again? Was #43503 not enough?

Yes but why is this happen again? Was #43503 not enough?

The tests related to Gandiva passed when we merged that PR, but it failed due to the exact reason in this PR. We might have missed something as it seems.

Yes, this test ProjectorTest.testMakeProjectorParallel is not disabled while ProjectorTest.testMakeProjector is disabled.
https://github.com/apache/arrow/blob/main/java/gandiva/src/test/java/org/apache/arrow/gandiva/evaluator/ProjectorTest.java#L152
Does it not occur before?

We should check that trace. We may have missed it.

@jinchengchenghh let's disable this too.

Yes, I think we could disable the gandiva test in a separate PR. Can we merge this one first? @vibhatha

@github-actions crossbow submit -g java

Can you help merge this one? @vibhatha Thanks!

@jinchengchenghh I don't have merge rights. @lidavidm will definitely merge this as it is already approved. LGTM as well.

Ok. Thanks for your kindly review. @vibhatha

@jinchengchenghh did we file a new issue for the Gandiva failure?

Maybe we can use the exists issue https://github.com/apache/arrow/issues/43576

Sure let's do that.

@vibhatha do we need to ping any Gandiva maintainers to figure out the failures?

@lidavidm I think so. And I think @kou also tried to fix this earlier.

ok, let me dig

Thanks @lidavidm, I am not well versed in this part of the project.


Bad usage of ${image_config_dir} in /usr/libexec/s2i/run

The envvar image_config_dir isn't populated yet at its reference here produces always an empty string thus file /jenkins.model.JenkinsLocationConfiguration.xml.tpl can't be found.
Function create_jenkins_location_configuration_xml is called immediately after its definition but the envvar image_config_dir is declared and default value populated far later here.

I ran into the same problem. Leads to this error:
/usr/libexec/s2i/run: line 24: /jenkins.model.JenkinsLocationConfiguration.xml.tpl: No such file or directory

as image_config_dir is not yet defined.

/remove-lifecycle stale


pythonPackages.pydiscourse: init at 1.4.0

Description of changes
Add the  pydiscourse Python module for Python 3.8, 3.9 and 3.10
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 252145 run on x86_64-linux 1

4 packages built:

python310Packages.pydiscourse
python310Packages.pydiscourse.dist
python311Packages.pydiscourse
python311Packages.pydiscourse.dist

Thanks a lot for the review @fabaff! I truly helped me get some details more clearly in my mind regarding Python packaging for nixpkgs in general.

Result of nixpkgs-review pr 252145 run on x86_64-linux 1

4 packages built:

python310Packages.pydiscourse
python310Packages.pydiscourse.dist
python311Packages.pydiscourse
python311Packages.pydiscourse.dist

Result of nixpkgs-review pr 252145 run on x86_64-linux 1

4 packages built:

python310Packages.pydiscourse
python310Packages.pydiscourse.dist
python311Packages.pydiscourse
python311Packages.pydiscourse.dist


Shibboleth OpenLDAP TLS Setting Not Set from API

What kind of request is this (question/bug/enhancement/feature request):
bug
Steps to reproduce (least amount of steps as possible):
add shibboleth config
check tls setting (located in OpenLDAP near Port and Servers)
save shibboleth config
edit shibboleth and see tls will not be checked
check /v3/authconfig/shibboleth API and you will see value is true
Expect behavior would be to see the TLS box checked, this can break your openldap connection config completely if the user does not notice this

Rancher version: 2.4 master-head (03/05/2020) commit id: f6872eb2c
I was able to validate that saving or configuring OpenLDAP in Shibboleth enabling the TLS checkbox in the UI now sends the tls: true configuration to the BE.
The configuration response also had this tls: true value in the config.
Validated that the TLS communication was stablished with the OpenLDAP server by monitoring the OpenLDAP server logs:
5e610a38 conn=1020 fd=30 ACCEPT from IP=x.x.x.x:51672 (IP=<IP_ADDRESS>:636)
5e610a38 conn=1020 fd=30 TLS established tls_ssf=128 ssf=128
5e610a38 conn=1020 op=0 BIND dn="cn=admin,dc=example,dc=org" method=128
5e610a38 conn=1020 op=0 BIND dn="cn=admin,dc=example,dc=org" mech=SIMPLE ssf=0
5e610a38 conn=1020 op=0 RESULT tag=97 err=0 text=
5e610a38 conn=1020 op=1 SRCH base="ou=people,dc=example,dc=org" scope=2 deref=0 filter="(&(objectClass=inetOrgPerson)(|(cn=test*)(givenName=test*)))"
5e610a38 conn=1020 op=1 SRCH attr=dn memberOf objectClass inetOrgPerson cn cn 1.1

Searching for users and groups and assigning roles for site access, cluster and project worked.

Opened https://github.com/rancher/rancher/issues/25799 for an UI enhancement.


Embeds/transclusions

With the plug-in enabled, this is the syntax I get when I want to transclude a block into another note:
[2024-07-15, Mon, wk29](<./2024-07-15, Mon, wk29.md#^7upkr6>)
but when I subsequently hover over that link, the original text does not show up in the pop-up. Of course it does show up in the pop-up with the original embed syntax:
[2024-07-15, Mon, wk29](2024-07-15,%20Mon,%20wk29.md#^7upkr6).
Am I doing something wrong?

Works for me

Please attach your sample vault to reproduce the issue

As per #2.

What is the meaning of ../ in this block link — 2 dots before the slash instead of 1 dot:
[embrace the blows so you can learn from them](<../Daily-notes-2024/2024-07-15, Mon, wk29.md#^8crwae>)

@DutchPete ../ means the standard thing in the file system: go to the parent directory

So, why are some prepended with a single dot and some with a double dot?

Imagine you have three notes: dir1/note1.md, dir1/note2.md, dir2/note3.md
And you insert links/transclusions into dir1/note1.md
[note2](<./note2.md>)
[note3](<../dir2/note3.md>)

Here we have single dot before note2.md, because note1.md and note2.md belong to the same directory dir1.
But we have double dot before note3.md, because note1.md and note3.md don't belong to the same directory, so we need to go to the parent directory, then to dir2 directory to reach note3.md.
That's how relative path work. It is not any specific to my plugin or to Obsidian, that's the common thing in file systems

OK, thanks very much for that explanation.


Packit verify fas

TODO:

[ ] Update documentation in packit/packit.dev.

Fixes #1476

RELEASE NOTES BEGIN
TODO
RELEASE NOTES END

Nice work, btw!


GoAccess Invalid deb signatures EXPKEYSIG - deb repo

The following command from https://goaccess.io/download#official-repo is failing:
sudo apt-get update
Hit:1 http://za.archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://za.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 http://za.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:4 http://ppa.launchpad.net/git-core/ppa/ubuntu focal InRelease
Hit:5 http://nginx.org/packages/ubuntu focal InRelease
Hit:6 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:7 http://ppa.launchpad.net/ondrej/php/ubuntu focal InRelease
Hit:8 https://esm.ubuntu.com/infra/ubuntu focal-infra-security InRelease
Get:9 https://deb.goaccess.io focal InRelease [2,345 B]
Hit:10 https://esm.ubuntu.com/infra/ubuntu focal-infra-updates InRelease
Err:9 https://deb.goaccess.io focal InRelease
  The following signatures were invalid: EXPKEYSIG 97BD1A0133449C3D Gerardo Orellana<EMAIL_ADDRESS>Fetched 2,345 B in 4s (532 B/s)
Reading package lists... Done
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://deb.goaccess.io focal InRelease: The following signatures were invalid: EXPKEYSIG 97BD1A0133449C3D Gerardo Orellana<EMAIL_ADDRESS>W: Failed to fetch https://deb.goaccess.io/dists/focal/InRelease  The following signatures were invalid: EXPKEYSIG 97BD1A0133449C3D Gerardo Orellana<EMAIL_ADDRESS>W: Some index files failed to download. They have been ignored, or old ones used instead.

I downloaded the key again but the same error occurred.

Thank you for creating this issue. I have been trying to find my way around the same problem for the last 30 minutes or so with no success. It looks like this issue comes up for various projects when the repo signing key changes.

I can confirm this problem on Debian as well. The repository key needs to be renewed or replaced.
Adding trusted=yes to the repo definition is a very much not recommended workaround, if you really absolutely have to install goaccess via the Debian / Ubuntu repositories now.

The GPG key has indeed expired:
/etc/apt/trusted.gpg.d/goaccess.gpg
-----------------------------------
pub   rsa3072 2020-05-17 [SC] [expired: 2022-05-17]
      C03B 4888 7D5E 56B0 4671  5D32 97BD 1A01 3344 9C3D
uid           [ expired] Gerardo Orellana<EMAIL_ADDRESS>

Thanks everyone for reporting this. The key has been renewed, please download it again, e.g.,
# wget -O - https://deb.goaccess.io/gnugpg.key | apt-key add -

The key fingerprint is the same: C03B 4888 7D5E 56B0 4671 5D32 97BD 1A01 3344 9C3D
Please let me know if that solves the issue on your end.

Resolved!
Thank you!

Thanks, however, in addition to the above instructions to avoid the "The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 97BD1A0133449C3D" I had to also do this:
cd /usr/share/keyrings;
sudo apt-key add goaccess.gpg

That didn't quite work for me on an Ubuntu 20.04 LTS x86_64 system, some other steps may help,

I wanted to verify the signature of the new key (look for signature ...9C3D showing up in output): wget -O - https://deb.goaccess.io/gnugpg.key | gpg --show-keys
Then I was happy to download it as suggested to /usr/share/keyrings/goaccess.gpg
apt-get update was failing with "N: Skipping acquisition of configured file 'main/binary-i386/Packages', as repository 'https://deb.goaccess.io focal InRelease' doesn't support architecture 'i386'"
To fix this I edited the contents of /etc/apt/sources.list.d/goaccess.list to (adding the arch setting),
deb [signed-by=/usr/share/keyrings/goaccess.gpg arch=amd64] https://deb.goaccess.io/ focal main
This way the key doesn't have to be on my apt keyring, but it could be (apt-key add /usr/share/keyrings/goaccess.gpg), in which case I think that signed-by option can be removed.

On Debian Bullseye, I had to place the gpg key in /etc/apt/trusted.gpg.d instead of /usr/share/keyrings

@ahayes just for reference, here are some great answers on apt-key and the path of keyrings:
https://askubuntu.com/a/1307181
https://unix.stackexchange.com/a/582853

@ahayes just for reference, here are some great answers on apt-key and the path of keyrings:

Thanks, but I think @Nardol and @olstjos are the ones who needs to read those. That's why I pointed out that your instructions used signed-by with a path to the key. You had good reasons for doing it that way rather than asking people to have apt trust your signing key for any repo.
As an aside, I need to read up on this a bit more though because I don't see what stops a malicious repo admin who gets their repo added to a system with a repo-specific signing key from putting their own versions of other well-known packages up and waiting for someone to install them.

I would have thought that when using the deb signed-by setting as documented you could put it anywhere. Is that not the case?

It is, GoAccess was installed by a script which did not follow the same procedure.
So I modified it now and won't use trusted.gpg.d.
Thanks @allinurl for the links.

@allinurl Please consider changing your repo to match Debian's.

publish your GPG key unarmored
make signed-by and arch unnecessary

Thank you! ❤

@szepeviktor I can take a look. Do you have some docs that reference how they're doing it? Thanks

publish your GPG key unarmored

Just upload a binary file e.g. /goaccess.gpg.key.

make signed-by and arch unnecessary

It depends on what software you use to generate your repo. I use reprepro.

I'm using reprepro as well. Did I miss a step from that guide?

@szepeviktor I see. So when you say match debian, I'm guessing you're referring to the guide you posted, correct? shouldn't users avoid putting them under trusted.gpg.d/ as described on the links below?

just for reference, here are some great answers on apt-key and the path of keyrings:
https://askubuntu.com/a/1307181
https://unix.stackexchange.com/a/582853

IMHO the official Debian wiki also does not recommend trusted.gpg.d: https://wiki.debian.org/DebianRepository/UseThirdParty
But a non-armored key and a package which should be updated when the signing key could also be two good points which would make things perfect :slightly_smiling_face:

shouldn't users avoid putting them under trusted.gpg.d/ as described on the links below?

I take the risk of malicous packages being injected into the official Ubuntu mirrors network by Gerardo 🙃

Thanks for the feedback!
Unarmored key was added https://deb.goaccess.io/goaccess.gpg.key
@szepeviktor you should be able to already run it without signed-by or without arch, e.g.,
$ echo "deb http://deb.goaccess.io $(lsb_release -cs) main" | sudo tee -a /etc/apt/sources.list
$ wget -O - http://deb.goaccess.io/gnugpg.key | sudo apt-key add -
$ sudo apt-get update
$ sudo apt-get install goaccess

Are you getting some sort of error?

Unarmored key was added

Thank you!

Are you getting some sort of error?

No!
Only people on multiarch will get one.

@szepeviktor I think that would work for both 64 and 32-bit. But I'm not sure it will handle arm64 if I don't specify the arch in reprepro. Do you know how that would work?

No, I've never been in the 32 bit world.

I installed goaccess via this guide when setting up my server: https://www.howtoforge.com/tutorial/perfect-server-ubuntu-20.04-with-apache-php-myqsl-pureftpd-bind-postfix-doveot-and-ispconfig/
In that guide it installs via these commands:
echo "deb https://deb.goaccess.io/ $(lsb_release -cs) main" | sudo tee -a /etc/apt/sources.list.d/goaccess.list wget -O - https://deb.goaccess.io/gnugpg.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/goaccess.gpg add - sudo apt-get update sudo apt-get install goaccess
Therefore to correct this issue, I modified the recommendation above slightly to:
sudo apt-key del 97BD1A0133449C3D sudo rm -rf **/etc/apt/trusted.gpg.d/goaccess.gpg** wget -O - https://deb.goaccess.io/gnugpg.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/goaccess.gpg add - sudo apt-get update
Hope this helps anyone with the key in trusted.pgp.d

I am having when running the command described here https://goaccess.io/download
wget -O - https://deb.goaccess.io/gnugpg.key | gpg --dearmor | sudo tee /usr/share/keyrings/goaccess.gpg >/dev/null
--2022-12-04 06:21:01--  https://deb.goaccess.io/gnugpg.key
Resolving deb.goaccess.io (deb.goaccess.io)... <IP_ADDRESS>, 2604:a880:400:d0::213c:1
Connecting to deb.goaccess.io (deb.goaccess.io)|<IP_ADDRESS>|:443... connected.
ERROR: The certificate of ‘deb.goaccess.io’ is not trusted.
ERROR: The certificate of ‘deb.goaccess.io’ has expired.
gpg: no valid OpenPGP data found.

Am I doing something wrong, or the certificate is truly expired?

@ypujante Certificate is fine. Have you run apt update and apt upgrade on your machine? I wonder if you have old cert packages... Then please try:
$ echo "deb http://deb.goaccess.io $(lsb_release -cs) main" | sudo tee -a /etc/apt/sources.list
$ wget -O - http://deb.goaccess.io/gnugpg.key | sudo apt-key add -
$ sudo apt-get update
$ sudo apt-get install goaccess

@allinurl I tried your suggestion and I am getting an error.
If I look at my /etc/apt/sources.list there is this:
deb http://deb.goaccess.io stretch main

When I run apt-get update I get this:
Ign:11 http://deb.goaccess.io stretch InRelease
Err:12 http://deb.goaccess.io stretch Release
  404  Not Found [IP: <IP_ADDRESS> 80]
Reading package lists... Done
E: The repository 'http://deb.goaccess.io stretch Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

I am not too sure why it lists "Release" and not "main". Not sure what to do to fix this.

Looks like you are still running stretch which is no longer supported. It appears debian ended LTS on June 2022. You could try changing the name under /etc/apt/sources.list to buster to see if that does the job. Let me know.

I missed that :(. Thank you for pointing it out.

With ansible on debian 12 i use this
- name: Add GoAccess's APT signing key
  ansible.builtin.get_url:
    url: "{{ goaccess_repo_key_url }}"
    dest: /etc/apt/trusted.gpg.d/goaccess.asc
    mode: '0644'
    force: true

- name: Add GoAccess's APT repository to system repository list and update cache.
  ansible.builtin.apt_repository:
    repo: "deb https://deb.goaccess.io/ {{ ansible_distribution_release }} main"
    filename: goaccess
    state: present
    update_cache: true

Work fine without errors


Enable all linters on this repo.

@geeknoid: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

buildmodern_cni
8b178e228ee39973ce64de480e1afb02977be585
link
/test buildmodern_cni

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


Limit on get requests has fix value 50.

/** set default value **/
queryParameters['limit'] = 50;
-overwrite limit value every time

Fixed in https://github.com/bart-sk/swagger-js-codegen/commit/6714158675f790eb42cada10e2f75b329c84d1da


Deleting a 1:1 contact, does not show up in Recycle Bin, so can't be undeleted.

Issue by kbromer from Friday Oct 12, 2012 at 20:37 GMT
Originally opened as https://github.com/SalesforceFoundation/Contacts_and_Organizations/issues/8

http://code.google.com/p/npsp/issues/detail?id=342
and
https://groups.google.com/forum/?fromgroups=#!topic/npsf/5pcBKQExlmc

https://powerofus.force.com/_ui/core/userprofile/UserProfilePage?u=00580000006R7l2&tab=sfdc.ProfilePlatformFeed&fId=0D580000027GDF2&emtm=1431698490202&s1nid=0DB800000004Fa7&OpenCommentForEdit=1&fromEmail=1&emkind=chatterPostNotification&s1ext=0&s1oid=00D00000000hWLM&s1uid=00580000003egB0

Included in beta release version 3.48 (Beta 5) and higher

Included in production release version 3.48

Great to see this fix, any chance this will be retrofitted to V2?

Hi @Filikin, we're not updating any of the NPSP 2 packages, and we strongly recommend upgrading to 3.

thanks for the reply Nicolas, but with 300 projects all on V2 each with their own customisations, we can't afford the risk that upgrading to V3 might break something.
regards,
Eamon
Eamon Kelly
ENCLUDE - Technology Donations and Solutions for Irish Charities
Drumcondra Business Centre
120 Drumcondra Road Upper
Dublin 9
Main Phone:      (01) 6535099
Support Phone: (01) 4853435
Mobile:<PHONE_NUMBER>
Web: http://www.enclude.iehttps://dbxprd0411.outlook.com/owa/redir.aspx?C=LQUfhFY4-kGcKgsvZ25WphfbGLfIKdAIm_Tcv_dDBy9ufPET69psI2tY2rOaFHkGW1M636IUrUI.&URL=http%3A%2F%2Fwww.enclude.ie
ENCLUDEit - technology donation website http://www.encludeit.orghttps://dbxprd0411.outlook.com/owa/redir.aspx?C=LQUfhFY4-kGcKgsvZ25WphfbGLfIKdAIm_Tcv_dDBy9ufPET69psI2tY2rOaFHkGW1M636IUrUI.&URL=http%3A%2F%2Fwww.encludeit.org%2F

From: Nicolas Campbell<EMAIL_ADDRESS>Sent: 01 September 2015 16:32
To: SalesforceFoundation/Cumulus
Cc: Eamon Kelly
Subject: Re: [Cumulus] Deleting a 1:1 contact, does not show up in Recycle Bin, so can't be undeleted. (#332)
Hi @Filikinhttps://github.com/Filikin, we're not updating any of the NPSP 2 packages, and we strongly recommend upgrading to 3.

Reply to this email directly or view it on GitHubhttps://github.com/SalesforceFoundation/Cumulus/issues/332#issuecomment-136762348.

Understood, although I'd say it's still worth a try to upgrade. You can test customizations in a sandbox first.
The actual fix for this issue is a simple controller class and visualforce page that overrides the contact delete button. You could fairly easily self implement this feature in an unmanaged context.

thanks Nicolas,
I suspect we will used a mixed approach, upgrading to V3 where we can justify it and adding the fix manually where we can't
regards,
Eamon
Eamon Kelly
ENCLUDE - Technology Donations and Solutions for Irish Charities
Drumcondra Business Centre
120 Drumcondra Road Upper
Dublin 9
Main Phone:      (01) 6535099
Support Phone: (01) 4853435
Mobile:<PHONE_NUMBER>
Web: http://www.enclude.iehttps://dbxprd0411.outlook.com/owa/redir.aspx?C=LQUfhFY4-kGcKgsvZ25WphfbGLfIKdAIm_Tcv_dDBy9ufPET69psI2tY2rOaFHkGW1M636IUrUI.&URL=http%3A%2F%2Fwww.enclude.ie
ENCLUDEit - technology donation website http://www.encludeit.orghttps://dbxprd0411.outlook.com/owa/redir.aspx?C=LQUfhFY4-kGcKgsvZ25WphfbGLfIKdAIm_Tcv_dDBy9ufPET69psI2tY2rOaFHkGW1M636IUrUI.&URL=http%3A%2F%2Fwww.encludeit.org%2F

From: Nicolas Campbell<EMAIL_ADDRESS>Sent: 02 September 2015 18:12
To: SalesforceFoundation/Cumulus
Cc: Eamon Kelly
Subject: Re: [Cumulus] Deleting a 1:1 contact, does not show up in Recycle Bin, so can't be undeleted. (#332)
Understood, although I'd say it's still worth a try to upgrade. You can test customizations in a sandbox first.
The actual fix for this issue is a simple controller class and visualforce page that overrides the contact delete button. You could fairly easily self implement this feature in an unmanaged context.

Reply to this email directly or view it on GitHubhttps://github.com/SalesforceFoundation/Cumulus/issues/332#issuecomment-137175113.


setCustomValidityRangeUnderflow value not showing

Please verify the version of auro-input you have installed
2.15.0
Please describe the bug

create the following auro-input

<auro-input
   id="birth-date"
   max="12/12/2023"
   min="12/13/2021"
   type="month-day-year"
   setCustomValidityValueMissing="birth date is required"
   setCustomValidityRangeUnderflow="baby cannot be over 2"
   required
   bordered
>
   Birth date
   Infant's birthdate is required.
</auro-input>

input a date less than the min value
observe the value you set for setCustomValidityRangeUnderflow is not shown under the input box

https://github.com/AlaskaAirlines/auro-input/assets/116121736/68770ad0-5e78-418b-9efa-93111a38ec68
Reproducing the error on the docsite
This issue cannot be reproduced on the Auro docsite
Expected behavior
If a user inputs a date less than the min value, the setCustomValidityRangeUnderflow value should be shown under the input box.
What browsers are you seeing the problem on?
Firefox, Chrome, Safari
Additional context
No response

@cynthiaricomendoza-alaskaair please schedule some time with @jason-capsule42 and/or @jordanjones243 to review this issue. I am unable to reproduce locally and we need more information about installed context.

@blackfalcon after doing some testing with auro-input and the different versions in our repo, we resolved the issue and are no longer able to reproduce this issue. i also canceled my meeting with @jason-capsule42 and @jordanjones243 with a note letting them know this has been resolved. i think this issue can be closed. thank you!


ComeBack family update

name: ComeBack
key: CB
color: Blue
emblem: Bolt_03
sort: trophy
info:
logo:
description: >
ComeBack is a French clan family. The first clan, ComeBack was created the 03/21/2018 by big ouf.
It started out as a clan of friends, then we quickly revealed great potential across the world rankings as well as in competition.
After the success of the clan, other clans were created later ComeBack 2, ComeBack 3 and ComeBack 4, ComeBack 5 et Comebackademie.
social:

discord: https://discord.gg/dB9pKDVVYS
twitter: https://twitter.com/comeback__cr/

clans:

name: ComeBack
tag: 989VGGV8
name: ComeBack 2
tag: 9LY2RCUQ
name: ComeBack 3
tag: Y0RYYL8J
name: ComeBack 4
tag: Y2LPVVGG
name: ComeBack 5
tag: LVG9QCP9
name: ComeBackacademie
tag: QGPU2GCP

Oh yes excuse me.🙏
name: ComeBack
key: CB
color: blue
emblem: Bolt_03
sort: trophy
info:
logo: /static/img/brands/logo/come-back.png
description: >
ComeBack is a French clan family.The first clan was created the 03/21/2018 by big ouf.
It started out as a clan of friends, then we quickly revealed great potential
across the world rankings as well as in competition.
After the success of the clan, other clans were created later ComeBack 2, ComeBack
3 and ComeBack 4, ComeBack5 and Comebackademie.
social:

discord: https://discord.gg/dB9pKDVVYS
twitter: https://twitter.com/comeback__cr/

clans:

name: ComeBack
tag: 989VGGV8
name: ComeBack 2
tag: 9LY2RCUQ
name: ComeBack 3
tag: Y0RYYL8J
name: ComeBack 4
tag: Y2LPVVGG
name: ComeBack 5
tag: LVG9QCP9
name: ComeBackacademie
tag: QGPU2GCP


Update PlMeetupConsensus2024.md

@WinsCodes added event host and logo

Update to the event completed using the Pull request - https://github.com/memser-spaceport/pln-events/pull/471

Update to the event completed using the Pull request - #471


cand[-1] in get_filtered_cands of attack_manager.py may cause error

https://github.com/llm-attacks/llm-attacks/blob/355b2ad245f8723b0f78e04d421eb8d13c90def8/llm_attacks/base/attack_manager.py#L622-L638
This means that if control_cand.shape[0] is 1 and not decoded_str != curr_control and len(worker.tokenizer(decoded_str, add_special_tokens=False).input_ids) == len(control_cand[i]),
count+=1 wiil be execute. However cands will be [] and below will occur:
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[47], line 1
----> 1 attack.run(
      2     n_steps=params.n_steps,
      3     batch_size=params.batch_size,
      4     topk=params.topk,
      5     temp=params.temp,
      6     target_weight=params.target_weight,
      7     control_weight=params.control_weight,
      8     test_steps=getattr(params, 'test_steps', 1),
      9     anneal=params.anneal,
     10     incr_control=params.incr_control,
     11     stop_on_success=params.stop_on_success,
     12     verbose=params.verbose,
     13     filter_cand=params.filter_cand,
     14     allow_non_ascii=params.allow_non_ascii,
     15 )

File /kewei-ai/36 Universal and Transferable Adversarial Attacks on Aligned Language Models/kewei-attack/attack_manager.py:1610, in IndividualPromptAttack.run(self, n_steps, batch_size, topk, temp, allow_non_ascii, target_weight, control_weight, anneal, test_steps, incr_control, stop_on_success, verbose, filter_cand)
   1595     attack = self.managers['MPA'](
   1596         self.goals[i:i+1],
   1597         self.targets[i:i+1],
   (...)
   1606         **self.mpa_kewargs
   1607     )
   1609     # personal prompt attack
-> 1610     attack.run(
   1611         n_steps=n_steps,
   1612         batch_size=batch_size,
   1613         topk=topk,
   1614         temp=temp,
   1615         allow_non_ascii=allow_non_ascii,
   1616         target_weight=target_weight,
   1617         control_weight=control_weight,
   1618         anneal=anneal,
   1619         anneal_from=0,
   1620         prev_loss=np.infty,
   1621         stop_on_success=stop_inner_on_success,
   1622         test_steps=test_steps,
   1623         log_first=True,
   1624         filter_cand=filter_cand,
   1625         verbose=verbose
   1626     )
   1628 return self.control, n_steps

File /kewei-ai/36 Universal and Transferable Adversarial Attacks on Aligned Language Models/kewei-attack/attack_manager.py:1018, in MultiPromptAttack.run(self, n_steps, batch_size, topk, temp, allow_non_ascii, target_weight, control_weight, anneal, anneal_from, prev_loss, stop_on_success, test_steps, log_first, filter_cand, verbose)
   1016 start = time.time()
   1017 torch.cuda.empty_cache()
-> 1018 control, loss = self.step(
   1019     batch_size=batch_size,
   1020     topk=topk,
   1021     temp=temp,
   1022     allow_non_ascii=allow_non_ascii,
   1023     target_weight=target_weight_fn(i),
   1024     control_weight=control_weight_fn(i),
   1025     filter_cand=filter_cand,
   1026     verbose=verbose
   1027 )
   1028 runtime = time.time() - start
   1029 keep_control = True if not anneal else P(prev_loss, loss, i+anneal_from)

File /kewei-ai/36 Universal and Transferable Adversarial Attacks on Aligned Language Models/kewei-attack/gcg_attack.py:190, in GCGMultiPromptAttack.step(self, batch_size, topk, temp, allow_non_ascii, target_weight, control_weight, verbose, opt_only, filter_cand)
    188     control_cand = self.prompts[j].sample_control(grad, batch_size, topk, temp, allow_non_ascii)
    189     print("kewei:",control_cand)
--> 190     control_cands.append(self.get_filtered_cands(j, control_cand, filter_cand=filter_cand, curr_control=self.control_str))
    191 del grad, control_cand ; gc.collect()
    193 # Search

File /kewei-ai/36 Universal and Transferable Adversarial Attacks on Aligned Language Models/kewei-attack/attack_manager.py:952, in MultiPromptAttack.get_filtered_cands(self, worker_index, control_cand, filter_cand, curr_control)
    950 print(control_cand.shape[0],"get_filter_cands", cands)
    951 if filter_cand:
--> 952     cands = cands + [cands[-1]] * (len(control_cand) - len(cands))

    955 return cands

IndexError: list index out of range

I ran into this issue earlier, and solved it by checking the consistency of tokenizer's decoding/encoding; e.g. x_str vs. encode(decode(x_str)). This was mentioned by the authors in one of the other issues.
decoded_str = worker.tokenizer.decode(control_cand[i], skip_special_tokens=True,
 clean_up_tokenization_spaces = False)
# print ("attack_manager.py line 631: control_cand[i] - ", control_cand[i])
# print ("attack_manager.py line 632: decoded_str - ", decoded_str)
# print ("attack_manager.py line 633: encoded - ", worker.tokenizer(decoded_str, add_special_tokens=False))

^ worked for me, by adding to get_filtered_cands function

@hoy021  Thanks for your help. However:

I changed as it shows above, and this still occurs
attack_manager.py:962, in MultiPromptAttack.get_filtered_cands(self, worker_index, control_cand, filter_cand, curr_control)
    960 print(control_cand.shape[0],"get_filter_cands", cands)
    961 if filter_cand:
--> 962     cands = cands + [cands[-1]] * (len(control_cand) - len(cands))
    965 return cands

IndexError: list index out of range

@hoy021  Besides, I found that I can't use a way to make encoding and decoding match:

Hey @Ethan-Chen-plus what you have found is the issue we were faced with for a while. It is basically that tokenizer is not invertible (which is not the purpose it is designed for) so that is why in the code we need to do re-tokenization to make sure the length does not change.
If you aren't able to get any string that has a fixed length and run into a runtime error, I would suggest to 1) increase the size of the batch and 2) try to directly truncate the strings so after tokenization it has the same length.
The setup in our code is shown to work when we run the experiment but it could be that under random conditions they will run into some edge cases like you are showing above.

I ran into this issue earlier, and solved it by checking the consistency of tokenizer's decoding/encoding; e.g. x_str vs. encode(decode(x_str)). This was mentioned by the authors in one of the other issues.
decoded_str = worker.tokenizer.decode(control_cand[i], skip_special_tokens=True,
 clean_up_tokenization_spaces = False)
# print ("attack_manager.py line 631: control_cand[i] - ", control_cand[i])
# print ("attack_manager.py line 632: decoded_str - ", decoded_str)
# print ("attack_manager.py line 633: encoded - ", worker.tokenizer(decoded_str, add_special_tokens=False))

^ worked for me, by adding to get_filtered_cands function

I faced this issue today and it was solved by adding this parameter(clean_up_tokenization_spaces = False), thanks!!!

clean_up_tokenization_spaces = False

Same here! Worked for llama3.1, for which i changed scripts inside the fastchat libraries to make it compatible


[bug] Events not reaching front-end

Describe the bug
Neither events emitted via &mut App, AppHandle, nor the main Window ever arrive at the front-end using any methods emit (on window), emit_to, emit_all.
Reproduction
Inside the set-up action of Tauri’s main function:
let window = app.get_window("main").unwrap();
let window_reference = window.clone();
let app_reference = app.handle();

window.on_menu_event(move |event| {
	println!("Hit menu event, dispatching event to front-end.");

	app_reference.emit_all("menu", "Test payload (emit all app) from Tauri.").unwrap();
	app_reference.emit_to("main", "menu", "Test payload (emit all app) from Tauri.").unwrap();
	window_reference.emit_all("menu", "Test payload (emit all) from Tauri.").unwrap();
	window_reference.emit("menu", "Test payload (emit) from Tauri.").unwrap();
});

println!("Set up window menu event listener and dispatch to front-end.");
Ok(())

Writing a custom asynchronous React hook to allow notification of window-related events:
export function useWindowEvent(callback: (event: Event<WindowEventPayload>) => void) {
	const [removeListenerBlock, setRemoveListenerBlock] = useState<Block | undefined>()

	useAsyncEffect(
		async () => {
			const removeListener = await listen("menu", event => {
				debugger
				callback(event as Event<WindowEventPayload>)
			})

			setRemoveListenerBlock(removeListener)
			console.log(`Registered async event listener for 'menu' event.`)
		},
		() => {
			removeListenerBlock?.()
		},
		[]
	)
}

All log messages are sent as expected, the debugger statement is never reached, no events are caught.
Expected behavior
I’d expect all log messages to be sent as expected, all four events or at least the two global app events are caught in the listener, and the debugger statement is reached.
Platform and versions
Environment
  › OS: Mac OS 12.4.0 X64
  › Node.js: 17.9.0
  › npm: 8.5.5
  › pnpm: Not installed!
  › yarn: 1.22.19
  › rustup: Not installed!
  › rustc: 1.59.0
  › cargo: 1.59.0
  › Rust toolchain:

Packages
  › @tauri-apps/cli [NPM]: 1.0.3
  › @tauri-apps/api [NPM]: 1.0.2
  › tauri [RUST]: 1.0.3,
  › tauri-build [RUST]: 1.0.3,
  › tao [RUST]: 0.12.1,
  › wry [RUST]: 0.19.0,

App
  › build-type: bundle
  › CSP: unset
  › distDir: ../build
  › devPath: http://localhost:3000/
  › framework: React

App directory structure
  ├─ node_modules
  ├─ docs
  ├─ public
  ├─ src-tauri
  ├─ build
  ├─ .git
  └─ src

Additional context
The other event-handling method appWindow.onMenuClicked does work but if the more general-purpose approach does not, it’s a dreadful sign for the rest of this application.

Tested this on a vanilla project and it works ok. Can you share a React project with your hook setup so I can check it?

Thanks, Lucas, I’ve investigated closer on the React side and just plainly setting up a listener in the App.tsx works. I thought it might the tear-down (clean-up) of the effect hook running prematurely but it just seems the useAsyncEffect hook in general (which is an external module) somehow prevents this from working. It’s easy to whip your own async effect hook, so my rewritten generic hook for all kind of events on the window is now:
type RemoveListenerBlock = () => void

export function useWindowEvent<Payload>(name: string, callback: (event: Event<Payload>) => void) {
	return useEffect(() => {
		let removeListener: RemoveListenerBlock | undefined

		const setUpListener = async () => {
			removeListener = await appWindow.listen(name, event => {
				callback(event as Event<Payload>)
			})
		}

		setUpListener().catch(error => {
			console.error(`Could not set up window event listener. ${error}`)
		})

		return () => {
			removeListener?.()
		}
	}, [name, callback])
}

Used as:
interface MyCustomEventPayload {}

useWindowEvent<MyCustomEventPayload>("menu", event => {
	// Handle event here.
})

This now works, all four events I’m sending are caught equally.

Damn React seems complicated :D good work on fixing it!

This is much easier actually:
useEffect(() => {
  const promise = listen("menu", event => {
    console.log(event);
  });

  return () => promise.then(remove => remove());
});

Even better way (note that we only need to run the hook once with [] )
useEffect(() => {
  const promise = tauriEvent.listen('eventName', ({ payload, ...evventObj }) => {
    console.log('got payload:', payload);
  });
  return () => promise.then(unlisten => unlisten());


Fix multiple history v4 upgrade regressions

Fixes regressions introduced in #125:

<Link> didn't append basenames to hrefs
basename disappeared from the router state after first user-initiated navigation
Empty query objects appended an extraneous question mark to URLs

/cc @baer @divmain @scottnonnenberg

Current coverage is 98.69% (diff: 100%)

Merging #126 into master will increase coverage by 0.01%

@@             master       #126   diff @@
==========================================
  Files            19         19
  Lines           227        230     +3
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits            224        227     +3
  Misses            3          3
  Partials          0          0

Powered by Codecov. Last update 1be0782...62ffb36

this looks good. hiw did yoi find that empty keys bug?

In express, query is always an empty object when no values are present AFAICT. We passed that to routerForExpress, which then thought that a query was around, so it appended the extra ? to the search field. Found it when I saw that all of our hrefs were asking questions 😆


[FEATURE] Indicate particular message being replied

Description
Currently, messages can only be quoted. Which makes it sometimes hard to know which message is being replied. It would be great to indicate which message is being replied to, and the user should be able to scroll to the original message by clicking on it in the reply.
It would also be very helpful if the original message is animated after the scroll.
Screenshots
No response
Additional information
No response
👀 Have you checked if this issue has been raised before?

[X] I checked and didn't find similar issue

🏢 Have you read the Contributing Guidelines?

[X] I have read and understood the rules in the Contributing Guidelines

Are you willing to work on this issue ?
None

Still thinking if I want this one, will Keep it inactive

Do you want it added now?

@Dun-sin sir can yo uplease assign this task to me

Do you want it added now?

Two things happen when they do, I ignore it or I correct them. Right now, I don't have the energy to correct them so the former is the default😂

Can i work on this issue @Dun-sin

@Dun-sin anyone working on this?

@Dun-sin anyone working on this?

no

Ok. I'd like to work on this.


Completion server locks in-place built pyd files

Environment data

VS Code version: 1.26.1
Extension version (available under the Extensions sidebar): 2018.7.1
OS and version: Windows 10.0.17134.228 x64
Python version (& distribution if applicable, e.g. Anaconda): CPython 3.7.0 x64
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): N/A or virtualenv

Actual behavior
The completion server seems to lock pyd files that are built in-place making it impossible to rebuild them without reloading VS Code, since there doesn't seem to be a way to restart it otherwise.
C:\hello_mod>python setup.py build_ext --inplace
running build_ext
copying build\lib.win-amd64-3.7\_hello.cp37-win_amd64.pyd ->
error: could not delete '_hello.cp37-win_amd64.pyd': Access is denied

Expected  behavior
It shouldn't lock them like that. It makes it a pain to develop extension modules in VS Code on Windows. At least a command to restart the completion server without reloading the entirety of VS Code can help a bit.
Steps to reproduce:

Download: hello_mod.zip
Build it in-place: python setup.py build_ext --inplace.
Play around with hello.py in VS Code.
Try to rebuild it in-place.

Eventually you are likely to hit an "Access is denied". Reloading VS Code will allow to build the module again.
Logs
Nothing relevant under default log level.

Is this while running the new Python Language Server, or while using Jedi?
What is the current setting for python.jediEnabled when you experience this?

I didn't change it. It's set to the default for this version: "python.jediEnabled": true.

@segevfiner please try setting it to "false" and let us know if the problem persists.

Yes, extension modules should work so please open an issue about your failure.
And Jedi will not being staying the default. Plus Jedi holding on to the file will probably turn out to be an upstream problem anyway which would mean a short-term fix may not even be possible.

Closing as this is an upstream issue which we don't have direct control or influence over.


Add error handling for datastore and other APIs

datastore.put could fail (either Illegal Argument, Concurrent Modification, or just a Datastore service error).
Try to look at other failures in these APIs. Program should recover gracefully wherever possible.

Should be resolved in the blobstore server. However, there could be other instances where handling this is important. Should be looked at.

After a quick glance, I don't think there is anywhere this needs to be fixed.
Yes, there are some areas that can fail (Google services, for example, could fail unexpectedly). However, these should not fail gracefully. If the Google Service is not working, any POST or GET requests should result in an error, and using the message passed from these services should suffice, at least for this MVP.
The Business Card drop is the only place where I see exception handling is essential, because the user has control of the type of file and # of files (either 0 or 1) that they can upload. In the forms, and the admin controls for viewing and deleting comments, the user's input is limited to specific types and would not / should not result in any error. Any error from those forms/functions would be caused by server, resulting in a 500 error anyways.
In the future, SQL injection and other things should be prevented (especially before launch). For now, this should be fine.


Sets timers format to hh:mm:ss

Actually I made a mistake here. There is a property called FormattedTimeLeft that already breaks the time down into smaller parts. What do you think would be the better solution?
We could add this here or stick with the hh:mm:ss fix and remove the other code. What I dislike about the code in FormattedTimeLeft is that it is rather verbose.

Actually no idea. Both solutions are legit.
A mixture would be nice .. something like "11h 20m 22s".

I went for the way we discussed yesterday. Thanks for the PR.


[WFCORE-4501] Provide XML schema for credential-reference

https://issues.jboss.org/browse/WFCORE-4501

It may be worth adding a readme describing how we manage versioning of a shared resource like this - e.g. would adding a new version of this schema place any requirement on updating all existing subsystems?

Good point. Added a readme.

@jmesnil I am happy to merge this one but did you want to check it over as you raised the original Jira issue?