Performance issues

Hi there,
first kudos, its amazing someone finally took to implement native vector maps for Flutter,
unfortunately, I just tried this package with our own tiles and I am having serious performance issues, basically the map freezes and I am unable to interact with it, after every zoom change/movement, for a few seconds.
This happens both in debug mode (Android emulator/device) and release mode.
Our config: https://gist.github.com/filipproch/a010f95fcab7a559ab3074c3ebd0e290
We use the map mostly over Czech Republic - Europe
I would like to contribute to help improve this, but not sure where exactly to look/start
or maybe you already have some ideas for performance improvements?

Hi, thanks for the kudos!
Performance is a real problem when it comes to vector tiles. So great to hear that you're interested in contributing.
There are a few things that you can do to tune the codebase as-is:

Check that you're using RenderMode.mixed, which uses raster tiles (rendered from vector data) while zooming, and then renders again with vector tiles when zooming is finished.
Check your tile data to verify that it doesn't include unnecessary information that would render outside of the tile bounds.
Tune your map theme to avoid rendering unnecessary polygons/lines
Check your memory usage - if the app is freezing, it's possible that you are using too much memory and hitting the upper limits.

I've done the above and come up with a decent frame rate while zooming/panning. There is some jitter (i.e. skipped frames) especially in dense regions (e.g. London, UK) however the user experience is still pretty good.
At one point I did experience freezing as you mentioned. I made some changes to the codebase to avoid rendering outside of the tile area for labels on the Canvas, and that made a huge difference.  There's potentially more that could be done here (e.g. with polygons/lines) I wonder if your tiles contain shapes outside of the tile rendering area? That could be a good place to start with a contribution.
The Flutter memory view is really helpful to see if you're hitting an upper memory limit.

Thanks for your fast reply, I will experiment and try to get into the code more. With Memory view didn't see immediate problems - so I started debugging it with the performance view (see below). I will write updates in this issue if I manage to get some improvements.

I've done some more work in this area since I was noticing jank on iPad.
It's still not perfect, but these two commits make a big difference:
https://github.com/greensopinion/flutter-vector-map-tiles/commit/b46c80a91855b6c8c0799bd6f561f55800235a32
https://github.com/greensopinion/flutter-vector-map-tiles/commit/55f38418e845ddf3b709465d98fa23333bda9d9c
Also, I've found that tuning the map style to reduce detail at different zoom levels can help. For example, the style that I'm using has minzoom=14 for road_minor, and minzoom=15 for road_minor_casing. By having casing for minor roads only render at zoom level 15 and above, there's half as much work to do at zoom level < 15.

Cool, I will try it out. I should get to experiment with it more during the upcoming week

I've done some more work in this area since I was noticing jank on iPad.
It's still not perfect, but these two commits make a big difference:
b46c80a 55f3841
Also, I've found that tuning the map style to reduce detail at different zoom levels can help. For example, the style that I'm using has minzoom=14 for road_minor, and minzoom=15 for road_minor_casing. By having casing for minor roads only render at zoom level 15 and above, there's half as much work to do at zoom level < 15.

Hi.
I have performance issues too , how do i tune my map style  to reduce details?
thanks

how do i tune my map style to reduce details?

to do that you'll need to edit the map style, which you could do with a text editor but it's easier with something like https://maputnik.github.io
The fewer things showing on a map, the less computationally expensive it is (i.e. the faster it will render.) There are a few ways to achieve fewer things while still having beautiful maps:

Consider rendering finer details only at higher zoom levels. For example, if your style shows buildings, consider having them show up only when the map is zoomed level 14, 15 or higher. That can be achieved with the minzoom property. The same applies for things like trails, alleys, and minor roads.

Consider reducing the level of detail of roads by eliminating borders at lower zoom levels. Most styles that I've seen render roads twice at different widths to give the effect of a border. By eliminating road borders, there's less work in rendering those tiles.

I hope that helps. If it's still not enough, you could use raster rendering which will provide the styling advantages of vector tiles, but without the crisp edges.

how do i tune my map style to reduce details?

to do that you'll need to edit the map style, which you could do with a text editor but it's easier with something like https://maputnik.github.io
The fewer things showing on a map, the less computationally expensive it is (i.e. the faster it will render.) There are a few ways to achieve fewer things while still having beautiful maps:
1. Consider rendering finer details only at higher zoom levels. For example, if your style shows buildings, consider having them show up only when the map is zoomed level 14, 15 or higher. That can be achieved with the `minzoom` property. The same applies for things like trails, alleys, and minor roads.

2. Consider reducing the level of detail of roads by eliminating borders at lower zoom levels. Most styles that I've seen render roads twice at different widths to give the effect of a border. By eliminating road borders, there's less work in rendering those tiles.

I hope that helps. If it's still not enough, you could use raster rendering which will provide the styling advantages of vector tiles, but without the crisp edges.

thanks.i have one more question! maputnik gave me a JSON file and where do i use it? the map controller didn't have any styling property (like google map controller ) .

@HusseinCopol You'd use that in your dart code like this:
const _themeData = /* { all of this is your json } */;

final mapTheme = ThemeReader().read(_themeData); // You can also supply logger: Logger.console() to the ThemeReader to spot parsing issues with your theme.

Then in your VectorTileLayerOptions, you can provide your theme by setting theme: mapTheme.
ThemeReader needs import 'package:vector_tile_renderer/vector_tile_renderer.dart'; which is part of this library so it should be available.

I've found that most of the time spent rendering is looping over the theme layers so to increase performance, you can start by throwing out all layers from your theme, that you won't need. For example I built an app with a map scoped in Germany but the default theme from Mapbox includes a theme layer for US highway symbols. After trimming down the layers from a few hundred to just over a dozen which suffice in my case, performance went from 5 FPS went around 50 when zooming, moving and rotating.
Of course if you have layers that should only show at specific zoom levels, using the minzoom/maxzoom properties works as well.

I read the rendering code today, it was really a pleasure. Great work @greensopinion !
I was also doing some canvas painting, but on top of the standard flutter_map tile layer. In my case I had a big performance improvement when actually adding lines via addPolygon() to the Path. It's a lot faster compared to lineTo()when having larger linestrings. However, I tried it out and couldn't feel a notably difference when applying to this project. Maybe it's a little bit faster, but I haven't done any benchmarking. I think because the tiled linestrings are already very small, most CPU time is took by the painting overhead.
When interested, take a look here: https://github.com/FaFre/dart-vector-tile-renderer/commit/f43fffe01f75fa3e462bddce68d5c75c2a88e088
But I think having 1-4 long running isolates and distribute the tile rendering jobs to them will bring a huge performance improvement. Downside, isolates are not supported for web.

I read the rendering code today, it was really a pleasure. Great work @greensopinion !

Thanks FraFre, great to hear.

I was also doing some canvas painting, but on top of the standard flutter_map tile layer. In my case I had a big performance improvement when actually adding lines via addPolygon() to the Path. It's a lot faster compared to lineTo()when having larger linestrings. However, I tried it out and couldn't feel a notably difference when applying to this project. Maybe it's a little bit faster, but I haven't done any benchmarking. I think because the tiled linestrings are already very small, so most CPU time is taken by the painting overhead.
When interested, take a look here: FaFre/dart-vector-tile-renderer@f43fffe

Feel free to issue a pull request - this is worth adding regardless of performance since it simplifies the code.

But I think having 1-4 long running isolates and distribute the tile rendering jobs to them will bring a huge performance improvement. Downside, isolates are not supported for web.

What do you have in mind for rendering jobs? From what I understand, calls to Canvas can only be done on the UI thread.

regarding canvas on UI thread, there is a Flutter issue for this FYI
https://github.com/flutter/flutter/issues/75755

Ohh too bad. I thought through the rasterization it could be also done in an isolate, but it seems like that it's not even possible to instantiate a Canvas inside an isolate. However, with the new release they put a lot of focus on isolates. Hopefully it will be possible soon.

@FaFre I couldn't wait - with some profiling I made a few changes inspired by your suggestion https://github.com/greensopinion/dart-vector-tile-renderer/commit/548c0ccc8f1583d24cab358b069ee1af134a03c7

I've had a run at some performance related changes. There's more work to do, but there are some noticeable improvements in the latest release.

eliminated unnecessary re-rendering of tiles while panning
reduced CPU overhead of text labels
reduced memory overhead
improved cache hit rate

The latest release has more performance improvements.
Some performance issues remain, clearly showing up as jank with the default theme.
Current recommendations:

Tune your theme:

reduce unnecessary map elements (such as buildings, minor roads, labels) at lower zoom levels using minzoom
avoid rendering unnecessary polygons/lines

Check that you're using RenderMode.mixed, which uses raster tiles (rendered from vector data) while zooming, and then renders again with vector tiles when zooming is finished.
Check your tile data to verify that it doesn't include unnecessary information that would render outside of the tile bounds.
Check your memory usage and avoid large cache sizes - if the app is freezing, it's possible that you are using too much memory and hitting the upper limits.

Closing based on recent optimizations including issues #21. Feel free to open a new issue if there continues to be a problem.

A lot of work has been done to improve performance. There is still room for more improvement, however in testing I've found that RenderMode.mixed is no longer needed for my use-case. See #21 for related discussion.

I'm just adding a few bits here, after recent thoughts on the removal of raster discussion...some has already been covered I think, so apologies, just getting it out of my head before it pops out again. Some may have already been done! But writing it down in case someone wants to explore (I don't think I've got quite time to dabble atm, but will if I do).
Generally in Flutter, canvas paths are typically very slow to render with strokewidths sub 1px (especially when zoomed) as it uses a different algorithm for linecaps etc. Sometimes there may be a path say at 1.2px...when pinchzooming, could anything between 1-2px be rendered at 0.99px for example (I experimented a bit with this in the renderer, and maybe it helped a little, in some cases, but not sure how the renderer would know if pinchzooming, or if thats better done outside the rendering code and figured before any calls to it)
Generally calls to drawPath should be minimised. Can paths with the same style be grouped into one superpath (with a Move inbetween them), cache that path and draw (may be fiddly if using expressions that change per zoom, but could be done in inbeween zoom level changes). This may be similar to the addPolygon idea, but we can also group multiple lines as well.
TextLayouts are typically expensive, so can that be cached (I know thats been discussed a bit before, not sure where that is currently), I was thinking about a simple storage of a previous frame type cache even. If a labels textlayout (based on latlng point and name for example) was stored in a hash of previous display frame/update, it should be able to be reused (I think!).
I played with the idea of "overZooming" (I think there's another term for it, but it eludes me). So let's say we are at zoom 14. We draw the tiles at zoom 13, which means you only need a quarter of the tiles (some loss of lines maybe, but for tiles that have too much data it can help a bit). (Does this also play into the mapbox tilesize of 512 vs 256, not sure ?)
Instead of drawPath, during a  pinchzoom, can Canvas drawPoints with a PointMode of lines/polygon be used ? It may not need to calculate linecaps etc do maybe faster (similar to earlier...if a strokeWidth is < 4 say, do we care about accuracy of joins ? Now, I did try this a year or two ago, and the odd time got an out of mem error, but I think this was referencing GPU memory for point storage, and it may well be all of that is fixed now.
If pinchzooming, could rendering be interrupted if a new zoom level comes in (bit fiddly this one...one may end up with blank tiles, not sure I like this...) ?
Out of interest, is there a way for the rendering code to know that a pinchzoom is taking place ? I couldn't quite tell from the code if that's possible or not...

Thanks for the ideas Ian!
On Tue, Feb 22, 2022 at 1:44 AM Ian @.***> wrote:

I'm just adding a few bits here, after recent thoughts on the removal of
raster discussion...some has already been covered I think, so apologies,
just getting it out of my head before it pops out again. Some may have
already been done! But writing it down in case someone wants to explore (I
don't think I've got quite time to dabble atm, but will if I do).
Generally in Flutter, canvas paths are typically very slow to render with
strokewidths sub 1px (especially when zoomed) as it uses a different
algorithm for linecaps etc. Sometimes there may be a path say at
1.2px...when pinchzooming, could anything between 1-2px be rendered at
0.99px for example (I experimented a bit with this in the renderer, and
maybe it helped a little, in some cases, but not sure how the renderer
would know if pinchzooming, or if thats better done outside the rendering
code and figured before any calls to it)
Generally calls to drawPath should be minimised. Can paths with the same
style be grouped into one superpath (with a Move inbetween them), cache
that path and draw (may be fiddly if using expressions that change per
zoom, but could be done in inbeween zoom level changes). This may be
similar to the addPolygon idea, but we can also group multiple lines as
well.
TextLayouts are typically expensive, so can that be cached (I know thats
been discussed a bit before, not sure where that is currently), I was
thinking about a simple storage of a previous frame type cache even. If a
labels textlayout (based on latlng point and name for example) was stored
in a hash of previous display frame/update, it should be able to be reused
(I think!).
I played with the idea of "overZooming" (I think there's another term for
it, but it eludes me). So let's say we are at zoom 14. We draw the tiles at
zoom 13, which means you only need a quarter of the tiles (some loss of
lines maybe, but for tiles that have too much data it can help a bit).
(Does this also play into the mapbox tilesize of 512 vs 256, not sure ?)
Instead of drawPath, during a pinchzoom, can Canvas drawPoints with a
PointMode of lines/polygon be used ? It may not need to calculate linecaps
etc do maybe faster (similar to earlier...if a strokeWidth is < 4 say, do
we care about accuracy of joins ? Now, I did try this a year or two ago,
and the odd time got an out of mem error, but I think this was referencing
GPU memory for point storage, and it may well be all of that is fixed now.
If pinchzooming, could rendering be interrupted if a new zoom level comes
in (bit fiddly this one...one may end up with blank tiles, not sure I like
this...) ?
Out of interest, is there a way for the rendering code to know that a
pinchzoom is taking place ? I couldn't quite tell from the code if that's
possible or not...
—
Reply to this email directly, view it on GitHub
https://github.com/greensopinion/flutter-vector-map-tiles/issues/10#issuecomment-1047607599,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEKDQ7XQHLAMA5WLWDMFBWTU4NLIZANCNFSM5B6ASJDA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you modified the open/close state.Message
ID: @.***
com>

Thinking out loud, I may be barking up the wrong street, so this may not apply! I was thinking about why I didn't really get any major text performance issues later on when experimenting with vector tiles.
When I was doing Vector stuff, in the end, I ended up not using Positioned Widgets (maybe a hybrid is interesting..can't remember if I tried this). I suspect what may be happening is that every zoom change, all the text layouts need to be recalculated, and then readjusted inversely according to the scale (and text layout is very expensive iirc).
However, in reality, we don't need to do any of this at all (unless we are either literally resizing the text to fit in with a road width for example, or curving text along a road). At all times, typically we want the text to be the same size (visually to the human eye). So we don't need to recalculate any of this, one can just reuse the old layout and draw on one none transformed Canvas.
For example...all on one single upper Canvas
///pos = tile position..
matrix = Matrix4.identity();
matrix..translate(pos.point.x.toDouble(), pos.point.y.toDouble())
          ..scale(pos.scale);

...

for (Label label in labels) {

    label.transformedPoint =
            MatrixUtils.transformPoint(matrix, label.point);

         // collision detection, maybe every x frames

         //canvas.save(); // only if map rotated

    canvas.translate(
            transformedPoint.dx, transformedPoint.dy);

         //canvas.rotate(-widgetRotation * 0.0174533); // only if map rotated
         //canvas.restore(); // only if map rotated

Apologies again, if this is a redherring.

Where is this RenderMode being set? could not find it anywhere with latest version of the package

Where is this RenderMode being set? could not find it anywhere with latest version of the package

RenderMode was removed, see discussion on #31 and implementation on #52

@HusseinCopol You'd use that in your dart code like this:
const _themeData = /* { all of this is your json } */;

final mapTheme = ThemeReader().read(_themeData); // You can also supply logger: Logger.console() to the ThemeReader to spot parsing issues with your theme.

Then in your VectorTileLayerOptions, you can provide your theme by setting theme: mapTheme.
ThemeReader needs import 'package:vector_tile_renderer/vector_tile_renderer.dart'; which is a dependency of this library so it should be available.

With this implementation you have to make sure that chosen theme and map providers are same. In my case I have been using Stadia map provider and picked a theme from GeoApify. I started facing problems because of incompatibilities. Then I switched to GeoApify map provider with the following link inside _urlTemplate()
https://maps.geoapify.com/v1/tile/klokantech-basic/{z}/{x}/{y}.pbf?apiKey=YOUR_API_KEY
I have picked theme style from here
https://maps.geoapify.com/v1/styles/klokantech-basic/style.json?apiKey=YOUR_API_KEY
This has completed my implementation ... rest anyone can pick a theme from the plugin's gallery and can add to application.


Docker "Failed to launch chrome!"

While running docker image, it's unable to get website screenshot.
changd-api  | https://google.com
changd-api  | Unable to get website screenshot: Error: Failed to launch chrome!
changd-api  | [57:57:0726/102936.931958:ERROR:browser_main_loop.cc(1485)] Unable to open X display.
changd-api  | TROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md
changd-api  | Unable to capture screenshot
changd-api  | Error: Image unable to be captured or differences detected.
changd-api  | Error: ENOENT: no such file or directory, unlink './screenshots/2/rmw9r_diff.png'
changd-api  |     at Object.unlinkSync (fs.js:1210:3)
changd-api  |     at Object.deleteFile (/app/db/file_handler.js:95:10)
changd-api  |     at executeVisualJob (/app/db/job_daemon.js:507:19)
changd-api  |     at processTicksAndRejections (internal/process/task_queues.js:93:5)
changd-api  |     at async /app/db/job_daemon.js:167:9 {
changd-api  |   errno: -2,
changd-api  |   syscall: 'unlink',
changd-api  |   code: 'ENOENT',
changd-api  |   path: './screenshots/2/rmw9r_diff.png'
changd-api  | }

It seems to me that after this commit (https://github.com/paschmann/changd/commit/51e386e530ca541d2970ad2657acc1b7deb3201d#diff-0b013f15ba214974a2b092d590d3608e033cac4e865b68209ae3d984e727d7caL521) the Chrome-Browser now needs a X11 display address.
Maybe a virtual framebuffer with xvfb that is launched just before the chrom(ium) binary would work?

Same problem here, any ideas on how to solve this issue?


Bluetooth in the crouton environment

name: precise
encrypted: no
Entering /mnt/stateful_partition/crouton/chroots/precise...
crouton: version 1-20161116131524~master:9db73043
release: precise
architecture: amd64
xmethod: xorg
targets: xfce
host: version 8872.76.0 (Official Build) stable-channel wolf
kernel: Linux localhost 3.8.11 #1 SMP Fri Jan 6 23:48:52 PST 2017 x86_64 x86_64 x86_64 GNU/Linux
freon: yes
Unmounting /mnt/stateful_partition/crouton/chroots/precise...

Please describe your issue:
(Just a note, the name of the chroot is precise but the installed version is trusty.)
I cannot understand bluetooth while within crouton and as a result of this, I cannot get bluetooth to work. I understand that, to an extent, crouton has to "interface" with chromeOS' existing buetooth system but no matter what I have tried, the bluetooth daemon fails to start from within Trusty.
I plugged an external USB Bluetooth adapter that I know that it works with Linux, hoping that I could control that one independently but the two distros are using the same kernel, so that is not going to work. lsusb shows the device but hcitool does not. In addition, modprobe btusb returns nothing but that could be due to the way crouton works with chromeOS. bluetoothctl is not available from within crouton but it is on chromeOS (and is working fine too with the builtin hardware).
I can see that there are hints towards solutions here and there and that bluetooth seems to work in some hardware + software combinations but not others (?)
Would it be possible for someone more knowledgable to put up a wiki page on how does bluetooth work from within crouton? Is it just a matter of renaming some interface so that the existing bluetooth subsystem is picked up by crouton or is it something more complex?
If known, describe the steps to reproduce the issue:
The problem manifests itself on a Dell Chromebook 11 with crouton which has been updated to Trusty

@aanastasiou,

name: precise
encrypted: no
Entering /mnt/stateful_partition/crouton/chroots/precise...
crouton: version 1-20161116131524~master:9db73043
release: precise

Your chroot still thinks the 'release' is precise too???
-DennisL

Yeah, there is something weird with the release, a chroot update might help.
For the original question, I'm not sure how Bluetooth is supposed to work in crouton, see #437 for a long-standing issue on the topic...


connections error

connections error:
{"level":50,"time":"2020-09-21T03:38:00.366Z","pid":1,"hostname":"sqlpad","name":"sqlpad-app","stack":"SyntaxError: Unexpected token o in JSON at position 1\n    at JSON.parse ()\n    at /usr/app/node_modules/sequelize/lib/dialects/mariadb/query.js:183:72\n    at Array.map ()\n    at Query.handleJsonSelectQuery (/usr/app/node_modules/sequelize/lib/dialects/mariadb/query.js:182:21)\n    at Query.formatResults (/usr/app/node_modules/sequelize/lib/dialects/mariadb/query.js:127:12)\n    at Query.run (/usr/app/node_modules/sequelize/lib/dialects/mariadb/query.js:72:17)\n    at processTicksAndRejections (internal/process/task_queues.js:97:5)\n    at async /usr/app/node_modules/sequelize/lib/sequelize.js:619:16\n    at async MySQLQueryInterface.select (/usr/app/node_modules/sequelize/lib/dialects/abstract/query-interface.js:938:12)\n    at async Function.findAll (/usr/app/node_modules/sequelize/lib/model.js:1741:21)\n    at async Function.findOne (/usr/app/node_modules/sequelize/lib/model.js:1905:12)\n    at async SchemaInfo.getSchemaInfo (/usr/app/models/schema-info.js:18:17)\n    at async getSchemaInfo (/usr/app/routes/schema-info.js:25:20)","type":"Error","msg":"Unexpected token o in JSON at position 1"}
env:
SQLPAD_BACKEND_DB_URI: 'mariadb://sqlpad:xxx@xxx:3306/sqlpad'
SQLPAD_SESSION_STORE: 'database'
version:
sqlpad images : sqlpad/sqlpad:5.6.0
mariadb: 10.5.5
thanks~

Thanks for the info @zhangzw001. Will have to see if this can be reproduced.
Is the connection you are getting the schema for also in mariadb?

I'm unable to reproduce this. I've added a mariadb example in the docker examples directory at https://github.com/rickbergfalk/sqlpad/tree/master/docker-examples/mariadb.
@zhangzw001 if you could get more info or reproduce the issue using docker-compose that'd be great.


Low Performance when using MaskableLayout in RecyclerView item layout

I am facing low performance and out of memory issues when i use MaskableLayout in RecyclerView item layout.
Let's assume that there are 2 types of items (A & B).
In my adapter i change the applied mask (setMask) according to the items type (A=A mask, B=B mask).
Is there a better way to handle that, because as far as i understand the masks keep invalidating causing me performance issues.

It should only invalidate once.
Try setting         setDrawingCacheEnabled(true); to false
Or attach a cpu profile trace ?
Thanks

I'm assuming the profiler showed that the maskable layout didn't invalidate more than it required.
Closing.


use file hashes to determine if a file is a duplicate, in which case the entire file does not need to be sent

if file hashes match, the server could just send the hash, or some other, smaller indicator. if the client doesn't have the file, they can always download it after the fact by request with the current protocol.

Issue created by BlueBubbles Bot in the BlueBubbles commnity on behalf of @isaac in #requests.

I dont think this is something we need to do anymore tbh


[IMPROVEMENT] System restore job name should be Longhorn prefixed

Is your improvement request related to a feature? Please describe (👍 if you like this request)
Longhorn system-restore job name is the SystemRestore name and the pod name is in format <system-restore-name>-<random-uuid>. This makes it hard to identify that they are from the Longhorn system restore job.
Describe the solution you'd like
Have a longhorn-system-rollout prefix for the system restore created job and pod.
Describe alternatives you've considered
None
Additional context
None

Validation - Pass
Verified with v1.4.x-head


misc: add line for test and build formatting

This is adds an extra line of padding so you can read the "Building from source" section (sorry).

Can one of the admins verify this patch?

Why are you sorry? Thank you for taking the time!


Never bypass if activate condition is met

Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
All Promptflow Contribution checklist:

[X] The pull request does not introduce [breaking changes].
[ ] CHANGELOG is updated for new features, bug fixes or other significant changes.
[X] I have read the contribution guidelines.
[ ] Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: suggested workflow.

General Guidelines and Best Practices

[X] Title of the pull request is clear and informative.
[X] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[ ] Pull request includes test coverage for the included changes.

Duplicated with https://github.com/microsoft/promptflow/pull/948


"Heisenberg" data gaps disappearing at different time ranges with the same step

This is done via Grafana over the same data series. The same query, with same step over different time ranges produces either gaps or no gaps. The smaller range makes the gap disappear.
Graphs and queries are attached.

gaps.txt
no_gaps.txt
Federated Prometheus with Thanos is used.
Prometheus Helm Chart 13.6.0

The issue is always reproducible for the same time ranges.

Does it help if you increase the step when there are gaps?

This is against query front end.
We haven't tried messing with the step, partially specifically because this is a very reproducible case - the only variable that changes is the start/end timestamp pair.
From the practical standpoint the step manipulation does not help as the dashboard in grafana are usually fixed at the same step. Furthermore, there may be a question on what exactly is being displayed as the graphs don't look similar for the same timestamps over the different time range, but those could be extrapolation artifacts.

This is still an issue.

This is still an issue.

This is still issue

This is still an issue.


Specify proposals from contributor summit

During the DockerCon contributor summit, we discussed to ideas related to the registry:

[ ] An authentication framework for building token issuers for the registry and other docker services.
[ ] Distributed images through bittorrent.

I have some detailed notes and would like etch some proposals around these two topics. I know that @pushrax had a Google Doc sketching out some of these ideas.
Let's do the following:

Participants in the summit: Please email me directly. I don't have all of your contact emails.
Let's drop two proposals in the distribution project covering this work. We can work together or I can just provide an outline and we can work from there.

cc @pushrax @vaceletm  @arneluehrs

Here's the doc referred to: https://docs.google.com/document/d/1rYDpSpJiQWmCQy8Cuiaa3NH-Co33oK_SC9HeXYo87QA
Feel free to start with some initial proposals – I don't think I have enough context to write a really solid migration roadmap. I will definitely comment and help flesh it out though.
cc @arneluehrs (I think that's the right account)

@pusrax, thanks for the initial Strawman, will circle back to it in the upcoming days.

I'll draft the use case we discussed next week

Hi all,
Just yo keep you updated, I was quite busy the last month and it's very unlikely that I will have more time in the upcoming weeks. So I prefer to be honest, I will not add energy to dedicate to this anytime soon. Sorry for the over-commitement.

@vaceletm This one is going to take some time.
As glimpse of effort, https://gist.github.com/stevvooe/74b46521688d171495c9 shows my attempt at creating a tracker framework that integrate with the registry API smoothly.
My research showed a few things:

The registry works excellently as a webseed. We'll have to integrate our token authorization into a torrent client to make it work well. It seems we can make this work quite well without sacrificing any security.
We will need to store the infohashes for the raw blob, minus tracker configuration on the disk. The repository specific torrent file will have to be generated at request time. Trackers are organized by info hash (sort of) and we may need to join several info hashes to support updating torrent urls.
At this time, selecting a client has been problematic. Most are focused on on-disk files and supporting variations in the implementations hiding the protocol aggressively. I'll play with this further but we still need to find a pure Go client that can accomplish this, well.

Sorry I haven't been great on the updates, but there is some progress.

@stevvooe do you have any kind of timeframe for the p2p push/pull registry? Trying to figure a way for intensive docker images pulling.
Thanks in advance!

@scalp42 My goal was to convince the docker project to properly store artifacts (layers + config). I've faced a lot of opposition to this but p2p cannot go forward without it. I cannot give you a timeframe for p2p.
The most approachable way to scale is probably readonly registry clones synced with a master using rsync. It will be a tricky setup but we've heard people have success with it.

@stevvooe understood, thanks for answering so fast! :smiley:

it out of date.

@xiaods Proposals here are still relevant. There are some novel ideas in p2p that we didn't consider at this summit, but the problems are still the same.

@stevvooe thanks for your clarify. got it.

This issue is way out of date. Closing, as this effort probably needs to be based on current functionality.


GitHub/issue 215/rotating credentials have lease

For issue #215 , expect rotating credentials to have a leaseId.  Also includes a fix for test bash script where keytool was already in path

@joeyslalom Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.

@joeyslalom Thank you for signing the Contributor License Agreement!

Thanks a lot for your pull request. Applying the change would keep issuing calls to sys/renew/{leaseId}. We can omit that call by checking for rotate-only leases (requested rotation for a non-renewable lease) and expire the lease. onLeaseExpired(…) will cause the actual rotation.
I applied the changes locally already.

Closing as per comment in the associated ticket #215.


Merge refs/heads/feature/chunked into alpha

Merge refs/heads/feature/chunked into alpha

This PR has been merged into alpha and tagged as 1.6.0-alpha-202304051401. Thanks for your contribution!


Be more explicit on seccomp availability

Seccomp is only compiled in binaries built for distros that ship with seccomp 2.2.1 or higher, and in the static binaries.
The static binaries are not really useful for RHEL and CentOS, because devicemapper does not work properly with the static binaries, so static binaries is only an option for Ubuntu and Debian.

LGTM

Author LGTM


feature Add BrokerLoad and DriverUrl white list

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall


Add support for svg and jpeg output to autobuild graph

This adds support for outputting jpeg or svg images from autobuild graph along with additional tests covering creation of those images

Codecov Report

Merging #13 (b87b6b8) into main (675bad0) will decrease coverage by 2.66%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main      #13      +/-   ##
==========================================
- Coverage   73.80%   71.13%   -2.67%
==========================================
  Files          22       22
  Lines        3157     3156       -1
==========================================
- Hits         2330     2245      -85
- Misses        827      911      +84

Impacted Files
Coverage Δ

autobuild/autobuild_tool_graph.py
49.35% <0.00%> (+0.31%)
:arrow_up:

autobuild/autobuild_tool_source_environment.py
41.15% <0.00%> (-30.46%)
:arrow_down:

autobuild/autobuild_tool_package.py
67.09% <0.00%> (-2.57%)
:arrow_down:

autobuild/common.py
74.90% <0.00%> (-2.00%)
:arrow_down:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Generate a git-like URL to enable links within dockstore UI for hosted workflows

#1746

Coverage decreased (-12.6%) to 32.366% when pulling 82a16dc6dd503b18de4142f16e5d71d610d7539c on feature/dockstore_hosted_fake_git_url into e4cf310b057200fdb8104875e7e3b138d4d3b1fd on develop.

Seems like codecov is delayed


Understanding different alerts

Hi guys , I'm developing a similar system and I'm wondering how exactly the platform is able to interpret and understand alerts in different languages and of different types . Thanks

We have an open extensible API that is very flexible. See http://docs.alerta.io/en/latest/api/alert.html


question about --triton-launch-mode=docker

The model requires for several other pip dependency not installed in
nvcr.io/nvidia/tritonserver:23.01-py3
So, I update the docker image by a docker file like
from nvcr.io/nvidia/tritonserver:23.01-py3
RUN pip install opencv-contrib-python-headless  onnxruntime-gpu
And create a tag like triton_onnx:0.1
But when I start the model_analyzer, I got error
07:31:58 [Model Analyzer] Initializing GPUDevice handles
07:31:59 [Model Analyzer] Using GPU 0 NVIDIA GeForce RTX 2060 with UUID GPU-f7572576-2bd0-ee4f-cd17-0a7a8ec6da5b
07:31:59 [Model Analyzer] Starting a Triton Server using docker
07:31:59 [Model Analyzer] Pulling docker image triton_onnx:0.1
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 268, in _raise_for_status
response.raise_for_status()
File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.40/images/create?tag=0.1&fromImage=triton_onnx
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/model-analyzer", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/model_analyzer/entrypoint.py", line 256, in main
client, server = get_triton_handles(config, gpus)
File "/usr/local/lib/python3.8/dist-packages/model_analyzer/entrypoint.py", line 130, in get_triton_handles
server = TritonServerFactory.get_server_handle(config, gpus)
File "/usr/local/lib/python3.8/dist-packages/model_analyzer/triton/server/server_factory.py", line 129, in get_server_handle
server = TritonServerFactory._get_docker_server_handle(
File "/usr/local/lib/python3.8/dist-packages/model_analyzer/triton/server/server_factory.py", line 216, in _get_docker_server_handle
server = TritonServerFactory.create_server_docker(
File "/usr/local/lib/python3.8/dist-packages/model_analyzer/triton/server/server_factory.py", line 69, in create_server_docker
return TritonServerDocker(image=image,
File "/usr/local/lib/python3.8/dist-packages/model_analyzer/triton/server/server_docker.py", line 72, in init
self._docker_client.images.pull(self._tritonserver_image)
File "/usr/local/lib/python3.8/dist-packages/docker/models/images.py", line 465, in pull
pull_log = self.client.api.pull(
File "/usr/local/lib/python3.8/dist-packages/docker/api/image.py", line 429, in pull
self._raise_for_status(response)
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 270, in _raise_for_status
raise create_api_error_from_http_exception(e) from e
File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation) from e
docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.40/images/create?tag=0.1&fromImage=triton_onnx: Not Found ("pull access denied for triton_onnx, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")
Seems, the model_anlyzer pull the image from server which not exists and stop to run.
How to fix this issue?
Thanks

The code incorrectly assumed that the docker image would be remote. I have put up a PR that should support locally created/tagged docker images as well.

The code incorrectly assumed that the docker image would be remote. I have put up a PR that should support locally created/tagged docker images as well.

Thanks for your quick aciton.

does the issue resolved? I got similar issue and just tried 23.03, the issue still there

@niuzheng168 yes this issue was resolved. If you are seeing a problem please open up a new issue.


visibleIf in matrixdropdown row not working in VueJs, but does in jQuery

Are you requesting a feature, reporting a bug or ask a question?
Bug
What is the current behavior?
Using the code with a matrixdropdown type
columns: [                 {                     name: "using",                     title: "Do you use it?",                     choices: [                         "Yes", "No"                     ],                     cellType: "radiogroup"                 }, {                     name: "rate",                     visibleIf: '{row.using} = "Yes"',                     title: "Please rate the framework itself"                 }             ]
The visibleIf shows the needed input field when the using field's answer is Yes with the jQuery version, however, the same code does not work with the VueJs version.
The VueJs version does work if there are multiple pages and you click previous and then next.
What is the expected behavior?
That the rate input field is also visible when the condition '{row.using} = "Yes"' equals true with the VueJs version.
How would you reproduce the current behavior (if this is a bug)?
Go to jQuery version in order to test the jQuery version.
Go to VueJs version in order to test the VueJs version.
Specify your

browser: Firefox + Chrome
browser version: FF 59.0, Chrome 65
surveyjs platform: vuejs
surveyjs version: 1.0.16

@exnihilo-creatio The issue is fixed.  The fix will be available in the next minor update on this week.
Thank you,
Andrew


Add a way to enter the Nether/End from within Machines

This would be necessary for a Skyworld WorldType.

Other mods add this functionality.


Fix deletion of project and org level connectors

Currently terraform fails to delete connectors that are at the org and project level because the identifiers aren't sent in the DELETE request.
This change includes them in the opts so that they're included in the query params.

Thanks for the contribution @FireGrey !


mysprinh to 0.0.1

Promote mysprinh to version 0.0.1

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approvers:
If they are not already assigned, you can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@devopsgym: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

promotion-build
c7930665fd50cf14e79db39d2dbfd36135813dc7
link
/test this

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


Component: Develop image - with caption component - Sprint 20-03

User Story

As a [user role below]:
IBM.com Library developer

I need to:
develop the image - with caption component

so that I can:
provide the component for the Learn page template

Additional information

Story within Storybook with corresponding knobs
Utilize Carbon
Design specs [sketch] [abstract]
Functional specs
Prototype for interactions [codepen]
Design QA issue (#1247)
Prod QA issue (#1248)

🚨 copy, icon and lightbox media viewer are not in scope for this release

Acceptance criteria

[ ] Image component is used
[ ] caption included
[ ] Follows functional specs and data model
[ ] Update cotentItem with this component.
[ ] The component development follows a modular component pattern
[ ] The component should be accessible via ES6 import
[ ] The component should be visible within the living documentation of the Dotcom components
[ ] A comment is posted in the Design QA issue, tagging Wonil and Roberta, when development is finished
[ ] A comment is posted in the Prod QA issue, tagging Praveen and Chetan, when development is finished

Latest design specs on Abstract: https://share.goabstract.com/c28dcfa6-a5a9-421b-b019-4a3390ff5184?


App crashes sometimes when launched scanner screen.

I have a recent stack trace in a below screenshot file.

This is a duplicate of https://github.com/juliansteenbakker/mobile_scanner/issues/1134 which was reported 2 days ago.
I'm going to close this issue in favor of that one.


API ML logon

Signed-off-by: Igor Kazmyr<EMAIL_ADDRESS>Your checklist for this pull request
🚨Please review the guidelines for contributing to this repository.

[ ] If the changes in this PR is part of the next future release, make this pull request against the docs-staging branch which will be published at the next release boundary. If the changes in this PR are part of the current release, use the default base branch, master. For more information about branches, see https://github.com/zowe/docs-site/tree/master#understanding-the-doc-branches.

[ ] If this PR relates to GitHub issues in docs-site or other repositories, please list in Description, prefixed with close, fix or resolve keywords.

Description (including links to related git issues)
Adding a sub-section on the 'Log in to API ML" feature.
Tech.implementation PR https://github.com/zowe/vscode-extension-for-zowe/pull/1062
:heart:Thank you!

I'm still not sure what the use case for Log out from API ML

The log out in the SSO case invalidates the JWT token.
If the JWT was issued by zOSMF then it asks zOSMF to invalidate
On top of that API ML keep cache of invalidated JWT token, which when queried against the API ML returns invalid.

Can this be merged now? @crawr

Also, release notes for the Zowe general RN page were added


Set allowbinary=false for console events when using the CLI

This will allow the CLI user to run scope events (after scoping an application with CLI defaults) without seeing binary data. If the CLI user does want to allow binary data anyway, they can use a custom config to set allowbinary=true for console events.


Couldn't find manifest for 'tail'

When I execute scoop install tail, I get the following error:

Couldn't find manifest for 'tail'.

Yet, tail is listed as on of the supported applications on the repo home under What does Scoop do?. Although, I've confirmed that the manifest for tail doesn't exist in the main bucket or extras.

please try this scoop search tail and choose you want to install with the list

tail.json was renamed to coreutils.json long time ago 0338f59c9854763b13ccc7263aebda0c4e145865 :smile:


az ad sp create-for-rbac --sdk-auth is DEPRECATED

az ad sp create-for-rbac --sdk-auth is DEPRECATED and should not be mentioned in the docs.
see also :

https://github.com/MicrosoftDocs/azure-dev-docs/issues/903
https://github.com/Azure/actions-workflow-samples/issues/91

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: cd35737d-566a-1a5c-6b0b-c9fbc2cd4ed8
Version Independent ID: fae96436-9730-460b-8c8e-9064a02cd970
Content: Publish revisions with GitHub Actions in Azure Container Apps
Content Source: articles/container-apps/github-actions-cli.md
Service: container-apps
GitHub Login: @craigshoemaker
Microsoft Alias: cshoe

@ezYakaEagle442 Thanks for your feedback. We will review and update as appropriate.

@ezYakaEagle442 Thanks for raising this issue. We have corrected the command, and the changes should be published soon.
#please-close


Merge pull request #1 from MicrosoftDocs/master

Draft Private Link TSG articles for review

Some of the images in Private Endpoint Troubleshooting article are missing, and will be updated by Monday. Please review the content and provide comments in the meantime.

Hi @ramandhillon84
This PR appears to contain 0 files. If you need to make content contributions, please do so in the private repo so the content can be staged and validated by the current validation rules.
After you review staging and fix all validation issues, you can add the sign-off comment to let PR reviewers know that the updates are ready to be merged.
For more information, see Should I work in a private or a public repo?
If you need help getting set up to work in the private repo, see Set up Git repository locally for documentation.
Would you create a pull request in the private repo to make these updates? We'll close this PR. Let us know if you need help moving the commits in this PR.
Thank you.

sorry, I appear to have used my local master as source instead of a branch of master. creating a new PR


Update deployment-prep-active-directory.md

the AD tools AsHciADArtifactsPreCreationTool requires both AD  and DNS tools to run on a jump box, otherwise the customer receive an error

@alkohli

Can you review this PR?
IMPORTANT: When this content is ready to merge, you must add #sign-off in a comment or the approval may get overlooked.

#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

Closing this PR as it now has merge conflicts. The method for AD prep has also changed significantly since this was opened and the contribution may not be valid. #please-close


macvtap err

{"component":"virt-launcher","level":"error","msg":"cannot open macvtap tap device /dev/tap46: Permission denied","pos":"virNetDevMacVLanTapOpen:179","subcomponent":"libvirt","thread":"36","timestamp":"2024-01-17T21:00:46.784000Z"}
deploy macvtap use https://kubevirt.io/user-guide/virtual_machines/net_binding_plugins/macvtap/
Environment:

KubeVirt version (use virtctl version): v1.1.1
Kubernetes version (use kubectl version): 1.25.9
VM or VMI specifications: N/A
Cloud provider or hardware configuration: N/A
OS (e.g. from /etc/os-release): N/A
Kernel (e.g. uname -a): N/A
Install tools: N/A
Others: N/A

Thanks for your interest in KubeVirt and macvtap.
That's an unsupported version of kubernetes (we only support the latest 3 versions).
Furthermore, you do not provide any meaningful info to address this issue; please provide information regarding:

which nodes (OS and version) you are using
which macvtap version you are using
are you using selinux ? if so, please upload the audit logs.

We would require at least that to have any idea of what is happening.


Using custom classes objects

I don't know if the type converter available before version 3 could help or not
I want to use custom data type in object like
class data{
Rxint speed;
}
Describe the solution you'd like
I want to save the speed value speed.value;
and read it as speed=(database value).obs;
Describe alternatives you've considered
I saw that there is a solution to make setters but to be saved but the getters are not available.
for the type converter code I saw there was a way to set getters and setters for each object which will clean too much of the code in my side
Version

Platform: iOS, Android, Mac, Windows, Linux, Web
Flutter version: any
Isar version: 3.*

there was a discussion with similar problem too https://github.com/isar/isar/discussions/730

I don't intend to add support for this. Convert your objects to json instead in Isar v4


Cannot access another VM

Describe the issue
According to network-mode:

Traffic is routed directly by the host operating system and the guest shares a VLAN with the host. Services running on the guest and the host can see each other without additional configuration. This is recommended for new virtual machines.

The docs do no not mention access between VMs, but the term Shared Network is a hint that this should work.
I'm trying to access one VM from another VM, using Apple virtualization in both, and it looks like there this works only with Bridged network. Is this expected?
When both vms are using Shared network, I can access the VM from the host, or the host from the VM, but I cannot access to another VM (tested with iperf3).
When both VMs are using Bridged network, I have have network access between the VMs.
Is shared network using VZNATNetworkDeviceAttachment and bridged network using VZBridgedNetworkDeviceAttachment? In this case it is clear why this does not work.
This looks like a documentation issue, the docs should help users that need access between VMs to choose the right network mode.
Also the Network page for a VM could have a simple description of each mode, or a link to the section in the docs for each type. The only thing we have is "Shared Network" and "Bridged (advanced)". These names do not help users that do not know the way networking is implemented in UTM.
Configuration

UTM Version: Version 4.5.3 (99)
macOS Version: 14.6.1
Mac Chip (Intel, M1, ...): M1 Pro

Is shared network using VZNATNetworkDeviceAttachment and bridged network using VZBridgedNetworkDeviceAttachment? In this case it is clear why this does not work.

This is true

Also the Network page for a VM could have a simple description of each mode, or a link to the section in the docs for each type. The only thing we have is "Shared Network" and "Bridged (advanced)". These names do not help users that do not know the way networking is implemented in UTM.

What would you recommend the names be?

Also the Network page for a VM could have a simple description of each mode, or a link to the section in the docs for each type. The only thing we have is "Shared Network" and "Bridged (advanced)". These names do not help users that do not know the way networking is implemented in UTM.

What would you recommend the names be?

Bridged is pretty good name, it described the network and it matches names used in other projects like libvirt. A short description can help people that need more help. For longer description a link to the docs would be useful.
Shared network is confusing since in other projects (e.g. lima) this is used to create user network providing access from host to vms and from vm to vm (like bridge, but 30 times slower). We can call it NAT (matching the underlying implementation) but I don't know if this will be more helpful to most users. Also in Libvirt NAT is used to describe an local network (<IP_ADDRESS>/24) allowing access from host to vms and between vms, so NAT can create more confusion. Since the name is more confusing, adding short description in the dialog is even more important.
When a user selects the network, they need to know the differences. Can I access the one vm from another? can I access a vm from another machine? The dialog should make this clear.
The dialog has lot of whitespace so adding a short description of the selected network mode is possible. Adding a (?) button linking to the docs or local help will be familiar to users.

It would be helpful if there was something between the current Shared Mode (Single VM to Host) and bridged with the Virtualization framework, which gives the VM access to the physical interface of the host. VMWare Fusion's Shared Mode allows VMs on a private host network to communicate with one another on a private network. I believe this is how UTM has historically worked with qemu emulation vs Virtualization framework. If it is a limitation with the Virtualization frameowrk, then I understand the current behavior.


Integrate with GitHub

:octocat:
Create new tasks from GitHub issues or PRs.
Components

[x] auto-link issue/pr in task text
[x] validate that issue/PR exists

It's good enough.


Add nvidia support

This adds support for containers to access NVIDIA GPUs for acceleration
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/overview.html
Adds related configuration and components from:

nvidia-container-toolkit
libnvidia-container

Also adds content interface plug for graphics-core22 provided by nvidia-core22 snap, containing required nvidia userspace libs.
Relates to: https://github.com/docker-snap/docker-snap/issues/91

Is nvidia-core22 content provider really needed? I think in some other snap we already pre-built support for this from source, without using the full graphics-core22 interface. Possibly in lxd or microk8s, will check.
Also security wise, lxd microk8s are much better choices. Are there any specific reasons to use very old docker snap, instead of better maintained lxd or microk8s? this docker snap really needs to get upgraded to never base and newer version.

Is nvidia-core22 content provider really needed? I think in some other snap we already pre-built support for this from source, without using the full graphics-core22 interface. Possibly in lxd or microk8s, will check.

It seems to be required.  Without it, I get the following when trying to run a container with gpu enabled:
nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown.

Also security wise, lxd microk8s are much better choices. Are there any specific reasons to use very old docker snap, instead of better maintained lxd or microk8s? this docker snap really needs to get upgraded to never base and newer version.

We are bound to docker in the cases we are working on, currently.  This is for a variety of "internal" reasons, and so is a hard requirement for us right now.

I would feel better to start docker track on core22 base, and maybe making it the default. Because we can end up with runtime incompatibilities between nvidia-ml library and other shared libraries from the base.

Using docker snap, during docker snap build is not acceptable. All other parts, set explicit options to by-pass and ignore lack of a docker container.
Similarly new nvidia parts may not use a nested docker container for the build; given the snap build is done in a constraint environment as is.

ok.  I was starting to dig into why build-snaps: [docker] was not working.
ok, so the nvidia parts will need to re-implement the build without then.  That'a a bit of a shame, as it means more code to maintain, but I will look into ti.

@xnox  Just a very quick look so far, but it looks like build solution in those nvidia repos only supported docker based builds.
I see two options, unless you have any other suggestions:

Re-implement the docker based build using using logic in the part and accept the ongoing maintenance of it

I don't know how complex or not it would be yet

use the already build packages from the nvidia apt repo

Probably much less work and ongoig maintenance, but providence supply chain are less well understood

Thoughts ?

FYI - I have reworked the libnvidia-container inspirded by the LXD snap, as originally suggested, it was pretty straight forward.
Just working on the nvidia-container-toolkit part now.

FYI - I haven't included the wrapper for nvidia-container-cli from the lxd snap at this point.  It seems like it could be useful perhaps for the classic use case, but wanted to discuss with you @xnox to see what you thought about it first.

This looks very good now from building point of view. Need to double check if we should call things with a command-chain wrappers, or not. Cause maybe we should be surfacing these new binaries as snap commands. Not sure.
The points raised above are minor, as this is probably in a mergeable state already.

snapcraft remote-build --build-on amd64,arm64,armhf,s390x,ppc64el

fail for multiple reasons. It seems we cannot specify default-provider: nvidia-core22, as it doesn't exist on arches beyond amd64/arm64, and yet we cannot make that slot arch specific.
similarly, since there are no nvidia drivers on amd64, arm64, we should skip building nvidia parts, and hooks on those architectures.
sigh.

fail for multiple reasons. It seems we cannot specify default-provider: nvidia-core22, as it doesn't exist on arches beyond amd64/arm64, and yet we cannot make that slot arch specific.

ok, I will remove the default provider. I believe auto-connection should still work if the content identifier matches, and the desired snap that provides the interfaces is installed first.  Possibly the publisher of the nvidia-core22 snap would have to be change to Canonical [ so publisher matches for both consumer and producer ], not completely sure.  Did you plan to migrate the nvidia-core* snaps to Canonical publisher eventually ?

similarly, since there are no nvidia drivers on amd64, arm64, we should skip building nvidia parts, and hooks on those architectures.

Makes sense, I will take a look at that.

I have made the nvidia parts amd64 and arm64 target only.
I will try and look at any remaining point tomorrow.

Imho this is enough for edge channel, will need to be retested with new nvidia-core22 (it restructured things) and I will need to retest myself on my instances. And For me it is easiest to do that form the snapstore.
And I will probably not land this into stable until after 2.59 snapd lands in stable anyway.

Great - thanks @xnox
I was intending to do further functional testing today, with latest version of nvidia-assemble and nvidia-core22.
I will discuss any further points in new issues or PR etc :+1:

Possibly the bin/ wrapper should be used when calling Nvidia binaries without all the Environment overrides.
That's exactly what I was thinking :+1:


chore: explicitly state platform support for cron time cycle

We want the users to know that cron is available only in Camunda Platform 8.1 or newer.

I think the message could be mistaken as "Everything listed is only available in 8.1, not just cron".
Is it possible to change the message depending on the engine version? This way we could only mention cron if the selected engine supports it. If this is not an easy fix, we can merge it as-is, as it is still an improvement over what is currently displayed

On the other hand, explaining that it's about cron expression only makes the error message super long. Note that the user can click the link to the documentation below the field.
@christian-konrad WDYT?

@barmac
Do we support showing different messages for different selected target versions?
If not, I guess there's no perfect solution for now; I agree that non-native speakers may think all options are only supported in 8.1 or higher
Let's add these two words "cron requires":
<Time cycle> should be an expression, an ISO 8601 repeating interval, or a cron expression (cron requires Camunda Platform 8.1 or newer)

Done.

Do we support showing different messages for different selected target versions?

This would require additional changes as it's not supported per default.


generator

Currently you cannot use generator because babel plugin is not inside @jsenv/jsenv-babel-plugin-map.
To fix that we should ensure Babel regenerator runtime ends up in a bundle when needed.
And also that it ends up in the node or chromium platform bundle used to dynamically import compiled file

Now fixed :)
Plugin added to @jsenv/jsenv-babel-plugin-map and regenerator-runtime import is forced when found in code and served from @jsenv/core dependencies.


Building the Docker image fails

I get the following error when trying to build the Docker image:
$ docker build -t csaf/validator-service .
[+] Building 11.5s (10/12)                                                                                docker:default
 => [internal] load .dockerignore                                                                                0.0s
 => => transferring context: 2B                                                                                0.0s
 => [internal] load build definition from Dockerfile                                                                                0.0s
 => => transferring dockerfile: 886B                                                                                0.0s
 => [internal] load metadata for docker.io/library/node:18-alpine                                                                                1.0s
 => [appbuild 1/4] FROM docker.io/library/node:18-alpine@sha256:435dcad253bb5b7f347ebc69c8cc52de7c912eb7241098b920f2fc2d7843183d                                                         0.0s
 => [internal] load build context                                                                                0.0s
 => => transferring context: 27.17kB                                                                                0.0s
 => CACHED [stage-1 2/5] WORKDIR /usr/src/app                                                                                0.0s
 => ERROR [stage-1 3/5] RUN apk add hunspell hunspell-en hunspell-de-de;  ln -s /usr/share/hunspell/en_US.aff /usr/share/hunspell/en.aff;  ln -s /usr/share/hunspell/en_US.dic /usr/sh  10.4s
 => CACHED [appbuild 2/4] WORKDIR /usr/src/csaf-validator-service                                                                                0.0s
 => CACHED [appbuild 3/4] COPY . .                                                                                0.0s
 => CANCELED [appbuild 4/4] RUN npm ci;     npm run dist                                                                                10.4s
------
 > [stage-1 3/5] RUN apk add hunspell hunspell-en hunspell-de-de;       ln -s /usr/share/hunspell/en_US.aff /usr/share/hunspell/en.aff;         ln -s /usr/share/hunspell/en_US.dic /usr/share/hunspell/en.dic;       ln -s /usr/share/hunspell/de_DE.aff /usr/share/hunspell/de.aff;         ln -s /usr/share/hunspell/de_DE.dic /usr/share/hunspell/de.dic:
0.388 fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
5.394 fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
5.394 WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.18/main: temporary error (try again later)
10.40 WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.18/community: temporary error (try again later)
10.40 ERROR: unable to select packages:
10.40   hunspell (no such package):
10.40     required by: world[hunspell]
10.40   hunspell-de-de (no such package):
10.40     required by: world[hunspell-de-de]
10.40   hunspell-en (no such package):
10.40     required by: world[hunspell-en]
10.40 ln: /usr/share/hunspell/en.aff: No such file or directory
10.40 ln: /usr/share/hunspell/en.dic: No such file or directory
10.40 ln: /usr/share/hunspell/de.aff: No such file or directory
10.40 ln: /usr/share/hunspell/de.dic: No such file or directory
------
Dockerfile:17
--------------------
  16 |     WORKDIR /usr/src/app
  17 | >>> RUN apk add hunspell hunspell-en hunspell-de-de; \
  18 | >>>      ln -s /usr/share/hunspell/en_US.aff /usr/share/hunspell/en.aff; \
  19 | >>>      ln -s /usr/share/hunspell/en_US.dic /usr/share/hunspell/en.dic; \
  20 | >>>      ln -s /usr/share/hunspell/de_DE.aff /usr/share/hunspell/de.aff; \
  21 | >>>      ln -s /usr/share/hunspell/de_DE.dic /usr/share/hunspell/de.dic
  22 |     ENV NODE_ENV=production
--------------------
ERROR: failed to solve: process "/bin/sh -c apk add hunspell hunspell-en hunspell-de-de; \tln -s /usr/share/hunspell/en_US.aff /usr/share/hunspell/en.aff; \tln -s /usr/share/hunspell/en_US.dic /usr/share/hunspell/en.dic; \tln -s /usr/share/hunspell/de_DE.aff /usr/share/hunspell/de.aff; \tln -s /usr/share/hunspell/de_DE.dic /usr/share/hunspell/de.dic" did not complete successfully: exit code: 1

It seems that the hunspell packages cannot be added.

I think this was a problem with my local Docker daemon. This can therefore be closed. Sorry for any inconveniences.


Vulnerability roundup 63: jquery-ui-1.11.4: 1 advisory

search, files

[ ] CVE-2016-7103 (nixos-19.03)

Scanned versions: nixos-19.03: 5847485e3ec. May contain false positives.

My PR was broken. The patch was for the unpacked source, but we use the release minified js as the derivation source.
I didn't find an obvious easy way to build jquery 1.11.4 from source. The simpler options are going to be either ignore this CVE (it's only in the nixos test framework and hard to trigger), or upgrade this derivation to jquery 1.12 (which has some incompatibilities, so I would have to go through the closure and check for problems).
Any opinions about the better options?

I think we can safely ignore this CVE.


Fix initialization when overriding drivers

Fixes logic so that if a user is providing a prompt_driver or embedding_driver, it will stick those into a generic StructureConfig, not an OpenAiStructureConfig. Additionally, if a user provides only one of prompt_driver or embedding_driver, it will default the null one to the defaults we provide in <0.23.
All of this can be removed after we remove the prompt_driver and embedding_driver fields for good.

Also including a lock file regeneration because tests were failing without it

is this intended to fix the missing prompt_driver exceptions when running integration tests?


Delete extracted downloaded files of a dataset

Will close https://github.com/huggingface/datasets-server/issues/753

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

Codecov Report
Base: 90.69% // Head: 92.08% // Increases project coverage by +1.38% :tada:

Coverage data is based on head (0b39356) compared to base (3ca72c6).
Patch coverage: 100.00% of modified lines in pull request are covered.

:exclamation: Current head 0b39356 differs from pull request most recent head c0388d0. Consider uploading reports for the commit c0388d0 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #794      +/-   ##
==========================================
+ Coverage   90.69%   92.08%   +1.38%
==========================================
  Files          70       33      -37
  Lines        3493     2249    -1244
==========================================
- Hits         3168     2071    -1097
+ Misses        325      178     -147

Flag
Coverage Δ

services_admin
?

services_api
?

workers_datasets_based
92.08% <100.00%> (+<0.01%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...datasets_based/workers/parquet_and_dataset_info.py
91.42% <100.00%> (+0.04%)
:arrow_up:

services/admin/tests/conftest.py

services/api/tests/routes/test_valid.py

services/api/src/api/routes/healthcheck.py

services/admin/src/admin/config.py

services/admin/src/admin/routes/jobs_duration.py

services/api/tests/conftest.py

services/admin/tests/test_prometheus.py

services/admin/tests/test_app_real.py

services/api/src/api/authentication.py

... and 28 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

@severo did I find all the place where this change should be applied? Also, do I need to write a test for it?

all the place where this change should be applied

maybe also in  splits, config_names and split_names, because the methods of https://github.com/huggingface/datasets/blob/b065547654efa0ec633cf373ac1512884c68b2e1/src/datasets/inspect.py also take DownloadConfig, but I'm not sure it's useful or not to get the list of splits and the list of configs.

do I need to write a test for it?

I don't think so, as it's already tested in datasets (I imagine)


fix unitest

所有的pr的ci都挂在test_sampler.py上，本地运行正常，libai的ci没有镜像，根据workflow file安装环境后也复现不了，暂时跳过test_sampler.py的测试
Run cd dev && bash run_unittest.sh
============================= test session starts ==============================
platform linux -- Python 3.[8](https://github.com/Oneflow-Inc/libai/actions/runs/4221417729/jobs/7345141366#step:6:9).16, pytest-7.2.1, pluggy-1.0.0
rootdir: /home/runner/work/libai/libai
plugins: hydra-core-1.1.2
collected 184 items

tests/test_file_io.py ......................                             [ 11%]
tests/test_optim.py ..                                                   [ 13%]
tests/test_scheduler.py .....                                            [ 15%]
tests/config/test_instantiate_config.py ........                         [ 20%]
tests/config/test_lazy_config.py ......                                  [ 23%]
Fatal Python error: Aborted

Thread 0x00007f15725b1740 (most recent call first):
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/oneflow/framework/tensor.py", line 450 in _numpy
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/oneflow/framework/tensor.py", line 370 in _tolist
  File "/home/runner/work/libai/libai/libai/data/samplers/samplers.py", line 77 in __iter__
  File "/home/runner/work/libai/libai/tests/data/test_sampler.py", line 35 in test_cyclic_sampler_iterable
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/unittest/case.py", line 633 in _callTestMethod
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/unittest/case.py", line 676 in run
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/unittest/case.py", line 736 in __call__
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/_pytest/unittest.py", line 330 in runtest
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/_pytest/runner.py", line 167 in pytest_runtest_call
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 3[9](https://github.com/Oneflow-Inc/libai/actions/runs/4221417729/jobs/7345141366#step:6:10) in _multicall
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/_pytest/runner.py", line 260 in <lambda>
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/_pytest/runner.py", line 339 in from_call
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/_pytest/runner.py", line 259 in call_runtest_hook
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/_pytest/runner.py", line 220 in call_and_report
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/_pytest/runner.py", line 131 in runtestprotocol
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/_pytest/runner.py", line [11](https://github.com/Oneflow-Inc/libai/actions/runs/4221417729/jobs/7345141366#step:6:12)2 in pytest_runtest_protocol
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/_pytest/main.py", line 324 in _main
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/_pytest/main.py", line 270 in wrap_session
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/_pytest/config/__init__.py", line 167 in main
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/_pytest/config/__init__.py", line 190 in console_main
  File "/opt/hostedtoolcache/Python/3.8.16/x64/bin/pytest", line 8 in <module>
run_unittest.sh: line 6:  1907 Aborted                 (core dumped) pytest --disable-warnings ./tests
tests/data/test_sampler.py
Error: Process completed with exit code [13](https://github.com/Oneflow-Inc/libai/actions/runs/4221417729/jobs/7345141366#step:6:14)4.


reuse cache function write

[x] Rebased/mergable
[x] Tests pass
[ ] CHANGELOG.md updated
[x] Sign CLA (if not already signed)

reuse cache function write in WriteMulti, make it more graceful.
func Cache.Write use write, why Cache.WriteMulti dont use it ?

@e-dard

This change is part #7228 which were still evaluating.


[Snyk] Security upgrade urllib3 from 1.26.17 to 1.26.18

This PR was automatically created by Snyk using the credentials of a real user.Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.
Changes included in this PR

Changes to the following files to upgrade the vulnerable dependencies to a fixed version:

requirements.txt

⚠ Warning
graphql-core 3.2.3 requires typing-extensions, which is not installed.
dj-database-url 2.0.0 requires typing-extensions, which is not installed.
asgiref 3.7.2 requires typing-extensions, which is not installed.
arrow 1.2.3 requires typing-extensions, which is not installed.
ariadne 0.20.1 requires typing-extensions, which is not installed.
ariadne 0.20.1 requires starlette, which is not installed.

Vulnerabilities that will be fixed
By pinning:

Severity
Priority Score (*)
Issue
Upgrade
Breaking Change
Exploit Maturity

496/1000  Why? Recently disclosed, Has a fix available, CVSS 4.2
Information Exposure Through Sent Data SNYK-PYTHON-URLLIB3-6002459
urllib3:  1.26.17 -> 1.26.18
No
No Known Exploit

(*) Note that the real score may have changed since the PR was raised.
Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.
Check the changes in this PR to ensure they won't cause issues with your project.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
🛠 Adjust project settings
📚 Read more about Snyk's upgrade and patch logic

Learn how to fix vulnerabilities with free interactive lessons:
🦉 Learn about vulnerability in an interactive lesson of Snyk Learn.

Codecov Report

Merging #397 (6258f3b) into develop (5e9cdf1) will increase coverage by 0.25%.
Report is 1 commits behind head on develop.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           develop     #397      +/-   ##
===========================================
+ Coverage    79.87%   80.13%   +0.25%
===========================================
  Files          148      148
  Lines         8623     8710      +87
===========================================
+ Hits          6888     6980      +92
+ Misses        1735     1730       -5

Files
Coverage Δ

parking_permits/forms.py
80.26% <100.00%> (+2.54%)
:arrow_up:

parking_permits/tests/factories/vehicle.py
100.00% <100.00%> (ø)

parking_permits/tests/test_forms.py
100.00% <100.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Add optional parameters: workgroup & request_token

Hello,
first of all thanks for this wrapper around the (rather horrible) generated AWS-SDK. 💐  It makes querying and result handling much more comfortable.
In our setup, direct usage of the output_location is heavily discouraged and instead I have to pass the work_group as parameter. This PR adds this as an additional, non-breaking option.
It also allows to pass in the optional request_token which we use for logging purposes.
I would be happy to see this merged and released, so we can rely on the official version of athens.
Thank you 😃

Just realized you've got workgroup for the parameter to the execute method, but the AWS client is using work_group.  I'm going to update the parameter so it's consistent - just a heads up on your end when you switch from your version to the official release

Ok, thanks a lot for merging and the heads-up! :-)


Allow Achievements doesn't appear to work

The 'Allow Achievements' functionality doesn't appear to be working, at least in DLC2.  I just played through the new DLC with it active and did not receive any achievements.
To Reproduce

Have mods loaded and running, including Toybox.
Play through long enough to reach an achievement you don't have.  Easiest to see in conversation-based achievements.
Perform the action or select the option to receive the achievement.

Expected behavior
A Steam achievement should be awarded.
An early save file is attached
SturmsohnBugSaveFile.zip
Screenshots
None
Settings
I couldn't find any 'settings.xml', but it looks like general_settings.json is what you want.
GeneralSettingsJson.zip
Version Info:

ToyBox Version: 1.4.16
OS: Windows 10
Version 10.0.19044

Additional context
I have not yet determined if this applies to the base game.  I have a save at the Lost Chapel, and can update this once I reach Drezen - there are Mythic paths I can take that I have not yet taken and thus don't have the achievement for.

I have noticed that in the DLC 3 that it is not awarding steam achievements either.
Win 10
Toybox 1.4.17 (beta)

Toybox's Allow Achievements only works for a specific subset of achievements within the game based on previous conversations between Narria and the developers. This is an intended limitation, and as a result, not a bug.


Add integration tests for ext_authz timeout

Commit Message: Add integration tests for ext_authz timeout
Additional Description: This PR adds integration tests for the http and grpc implementations of ext_authz. For each, it tests the fail open and fail closed modes. It also tests that ext_authz's failure_mode_allow_header_add config works as expected for the fail open cases.
Risk Level: none
Testing: test-only PR
Docs Changes: none
Release Notes: none
Platform Specific Features: none
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

/assign @yanjunxiang-google

/retest

/assign @tyxia

/lgtm
Thanks for adding timeout test to guard the behavior.

/retest


[bug]can not add datasource

dialoqbase           | An operation failed because it depends on one or more records that were required but not found. Record to update not found. dialoqbase           |     at vn.handleRequestError (/app/node_modules/@prisma/client/runtime/library.js:123:6730) dialoqbase           |     at vn.handleAndLogRequestError (/app/node_modules/@prisma/client/runtime/library.js:123:6119) dialoqbase           |     at vn.request (/app/node_modules/@prisma/client/runtime/library.js:123:5839) dialoqbase           |     at async l (/app/node_modules/@prisma/client/runtime/library.js:128:9763) dialoqbase           |     at async Queue.queueHandler (/app/queue/index.js:23:17) { dialoqbase           |   code: 'P2025', dialoqbase           |   clientVersion: '5.2.0', dialoqbase           |   meta: { cause: 'Record to update not found.' } dialoqbase           | } dialoqbase           | PrismaClientKnownRequestError: dialoqbase           | Invalid prisma.botSource.update()invocation: dialoqbase           |  dialoqbase           |  dialoqbase           |  dialoqbase           |  dialoqbase           | An operation failed because it depends on one or more records that were required but not found. Record to update not found. dialoqbase           |     at vn.handleRequestError (/app/node_modules/@prisma/client/runtime/library.js:123:6730) dialoqbase           |     at vn.handleAndLogRequestError (/app/node_modules/@prisma/client/runtime/library.js:123:6119) dialoqbase           |     at vn.request (/app/node_modules/@prisma/client/runtime/library.js:123:5839) dialoqbase           |     at async l (/app/node_modules/@prisma/client/runtime/library.js:128:9763) dialoqbase           |     at async Queue.queueHandler (/app/queue/index.js:77:17) { dialoqbase           |   code: 'P2025', dialoqbase           |   clientVersion: '5.2.0', dialoqbase           |   meta: { cause: 'Record to update not found.' } dialoqbase           | } dialoqbase           | Processing queue dialoqbase           | PrismaClientKnownRequestError: dialoqbase           | Invalidprisma.botSource.update()invocation: dialoqbase           |  dialoqbase           |  dialoqbase           |  dialoqbase           |  dialoqbase           | An operation failed because it depends on one or more records that were required but not found. Record to update not found. dialoqbase           |     at vn.handleRequestError (/app/node_modules/@prisma/client/runtime/library.js:123:6730) dialoqbase           |     at vn.handleAndLogRequestError (/app/node_modules/@prisma/client/runtime/library.js:123:6119) dialoqbase           |     at vn.request (/app/node_modules/@prisma/client/runtime/library.js:123:5839) dialoqbase           |     at async l (/app/node_modules/@prisma/client/runtime/library.js:128:9763) dialoqbase           |     at async Queue.queueHandler (/app/queue/index.js:23:17) { dialoqbase           |   code: 'P2025', dialoqbase           |   clientVersion: '5.2.0', dialoqbase           |   meta: { cause: 'Record to update not found.' } dialoqbase           | } dialoqbase           | PrismaClientKnownRequestError: dialoqbase           | Invalidprisma.botSource.update() invocation: dialoqbase           |  dialoqbase           |  dialoqbase           |  dialoqbase           |  dialoqbase           | An operation failed because it depends on one or more records that were required but not found. Record to update not found. dialoqbase           |     at vn.handleRequestError (/app/node_modules/@prisma/client/runtime/library.js:123:6730) dialoqbase           |     at vn.handleAndLogRequestError (/app/node_modules/@prisma/client/runtime/library.js:123:6119) dialoqbase           |     at vn.request (/app/node_modules/@prisma/client/runtime/library.js:123:5839) dialoqbase           |     at async l (/app/node_modules/@prisma/client/runtime/library.js:128:9763) dialoqbase           |     at async Queue.queueHandler (/app/queue/index.js:77:17) { dialoqbase           |   code: 'P2025', dialoqbase           |   clientVersion: '5.2.0', dialoqbase           |   meta: { cause: 'Record to update not found.' } dialoqbase           | } dialoqbase           | Processing queue
it is faild when i have add datasource.

Prisma is saying 'Record to update not found,' which means that the bot tried to update something, but it did not exist at that time, so it threw an error. Did you delete the bot while adding a datasource ?

Are you using version v0.0.27, right? I will look into it.

Are you using version v0.0.27, right? I will look into it.

i use the lastest of your github and the lastest master branch.
i run docker-compose with ./docker.

Which data source did you use? PDF?

source is :https://n4ze3m.com/projects
but i guess the reason, because i am in china, GWF(great wall firewall).
1.GWF, it can fetch openapi on GWF
2. WARP, openapi is banned WARP..
so bad... i hate GFW. fk

Sorry for that :/  dialoqbase support Llama v2 models via fireworks.ai ? Does it work there?

thanks. i will try.

Hey, just asking, how did you find the dialoqbase repo? The repo has received so many stars in the last few days from China 😊 Anyways, thanks for using it! ❤

i have found by DouYin.com.
i have try deploy once agent just now in USA VPS. but it is not work too. so bad.

i have try deploy once agent just now in USA VPS. but it is not work too. so bad.

Same issue ?

nothing... and pq BotDocument have nothing too.

Can you add a datasource once more and check it instead of the website?

I will try to reproduce what happened and release a patch :)

openai chatgpt key is valid.

Oh, okay. Are you using a premium OpenAI API key, not the trial one? Because a few months ago, a similar issue happened when a person used a free trial API key, which now won't work.
you can test your openai key on chatbotui : https://github.com/mckaywrigley/chatbot-ui

Can you add a datasource once more and check it instead of the website?

the same issue. it can not work.

my key is valid.

Oh, sorry, is dialoqbase docker's redis up too?

yes. redis pgvector dialoqbase is running and up ok.

Oh, sorry, is dialoqbase docker's redis up too?哦，对不起，dialoqbase docker 的 redis 也启动了吗？

Okay, I will check what happened and post an update here

Hey, v1 has been released with improved queue support, which may resolve your issue. If you test it, please let me know. Thanks!


HDDS-4667. XCeiverClientGrpc should give up if unexpected exception is thrown from read path

What changes were proposed in this pull request?
HDDS-4320 added the following logic in BlockInputStream:
https://github.com/apache/ozone/blob/77bf7ad7c1a70e9e1d7aaeb3ec006f19ae8d3089/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockInputStream.java#L287-L292
handleReadError tries to refresh the pipeline.  If it gets a new one, read is retried on that.  Otherwise it re-throws the exception to exit the loop.
https://github.com/apache/ozone/blob/77bf7ad7c1a70e9e1d7aaeb3ec006f19ae8d3089/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockInputStream.java#L171-L173
Pipeline equality includes the pipeline ID, but SCM constructs a new on-demand pipeline for reading from a single node.  The result is that the client thinks there is a new pipeline and retries.
This PR proposes to skip retry if the pipeline has the same set of nodes that failed in the previous iteration.
https://issues.apache.org/jira/browse/HDDS-4667
How was this patch tested?
Added unit test that reproduces the infinite loop without the fix.

Thanks @adoroszlai for working on this. I feel , we should have a retry policy configured to bail out after certain cycles of refreshing the pipeline.
Also, the refresh pipeline should only happen in case the all the dns fail to serve the read. In such cases, on the client we should see StorageContainerException on the client. Any exception on the grpc channel we should terminate without retrying..

Thanks @adoroszlai for working on this. I feel , we should have a retry policy configured to bail out after certain cycles of refreshing the pipeline.
Also, the refresh pipeline should only happen in case the all the dns fail to serve the read. In such cases, on the client we should see StorageContainerException on the client. Any exception on the grpc channel we should terminate without retrying..

Thanks @elek for reporting this problem and @bshashikant for reviewing and committing it.

Thanks @elek for reporting this problem and @bshashikant for reviewing and committing it.


Unable to run test due to "Error [ERR_MODULE_NOT_FOUND]: Cannot find module *.mjs"

Background
Current
><EMAIL_ADDRESS>test
> jest

 FAIL  tests/subtract.test.js
  ● Test suite failed to run

    Error [ERR_MODULE_NOT_FOUND]: Cannot find module 'D:\projects\repository\github.com\jessealama\decimal128\src\decimal128.mjs' imported from D:\projects\repository\github.com\jessealama\decimal128\tests\subtract.test.js

 FAIL  tests/rational.test.js
  ● Test suite failed to run

    Error [ERR_MODULE_NOT_FOUND]: Cannot find module 'D:\projects\repository\github.com\jessealama\decimal128\src\rational.mjs' imported from D:\projects\repository\github.com\jessealama\decimal128\tests\rational.test.js

 // ... repeat 12 more times

Test Suites: 14 failed, 14 total
Tests:       0 total
Snapshots:   0 total
Time:        0.434 s
Ran all test suites.

Expected
To be able to test
Reproduce
checkout to today's latest
git checkout 66a0909717894af09b3c613d77668f329dfceaf4
npm i

npm test

I manage to resolve the issue to a certain extend,
first - rename the tests to import mts (the real file extension) instead of the transpiled one
Then:

https://www.npmjs.com/package/jest-light-runner#using-custom-nodejs-esm-loaders - on using custom loader
https://github.com/nodejs/node/issues/51196 - on how to resolve the warning

to a certain extend

I said to a certain extends because coverage still doesn't quite work yet
Here is a PR that fix the issue with running npm test - https://github.com/jessealama/decimal128/pull/95

I am able to fix the coverage by using ts-jest following their guide on ESM Support in https://kulshekhar.github.io/ts-jest/docs/guides/esm-support
I have created another PR at #96
Let me know what you think

After inspecting .github/workflows/push.yaml, I come to realise that npm test and npm run coverage are meant to run after npx tsc.
Perhaps this is a no issue at all right now.
The original intention was actually to make test work for #93

Right, I always compile the TypeScript before running tests. Is there a better way we could document that? Should we compile TypeScript as part of running the tests? That would ensure that tests always run on the compiled JS (and wouldn't even run if there were TypeScript compile errors), but it seems a bit heavy to me.

To be honest, I'd love to just write my tests in TypeScript, too. The main thing, in my opinion, the transpiled JS for end-users.

oops, sorry for the late response.

Could case-(in)sensitivity be paying a role here?

Nope, i was using a case-insensitive file system (Windows).

Is this still an issue, given the changes in #121

I would say yes! but more of a documentation issue.
my last comment is still relevant.

The more I think about it, this issue doesn't warrant a code change https://github.com/jessealama/decimal128/pull/96 https://github.com/jessealama/decimal128/pull/95.
My original motivation is also best resolved in https://github.com/jessealama/decimal128/pull/93 (https://github.com/jessealama/decimal128/pull/93#issuecomment-2047790788) itself.
For this issue, I believe an update to the README should be sufficient.

This is from the perspective of a contributors when they first clone the project.
I'd assume most contributors will run npm run test, not knowing that they need to run npm run build first. Especially for project using jest or vitest.

let me know what you think.
I can help to:

update the documentation to add a contributor section
or we can use vitest to test directly on typescript code (vitest will internally transpile before testing)


Long literal lacking separators

A new Clippy lint is failing our build.
error: long literal lacking separators
  --> src/de/utf8.rs:17:26
   |
17 | const MAX_THREE_B: u32 = 0x10000;
   |                          ^^^^^^^
   |
   = note: `-D unreadable-literal` implied by `-D clippy`
   = help: consider: 0x1_0000
   = help: for further information visit https://github.com/Manishearth/rust-clippy/wiki#unreadable_literal

Fixed in d3e5dd9cd7e0c1c7f42d631cc217b72aaff5a399.


Adds SpringRabbitTracing.Builder.remoteServiceName

SpringRabbitTracing.Builder.remoteServiceName allows you to set the
name the broker will have in the service graph. Usually you'll want a
service-scoped name like a cluster or similar.
This also adjusts the span names to always be operation names like other
instrumentation.

@jonathan-lo I took liberties here, knowing we'll end up needing parsers like we have in grpc, cassandra etc at some point. This policy is most like other code (including OT).


Split EFS lifecycle into two blocks

Otherwise it succeeds on plan but fails on apply
https://github.com/hashicorp/terraform-provider-aws/issues/21862#issuecomment-1007115545
Follow-up to https://github.com/2i2c-org/infrastructure/pull/3183

terraform apply works with this now

Self-merging as a syntactic fixup


Better GetPassivePort (and GetEnhancedPassivePort)

See issues #1258, #409, #270
Sequential logic pattern makes this easer to maintain.
Supersedes #1260, if everyone is agreeable to this.

I will rebase and redo the PR


[hotfix][bugfix] The generic ingress requires an utf-8 key.

This PR fixes a bug in the event-generator at statefun-python-k8s example.
The statefun.kafka.io/routable-protobuf-ingress excepts a UTF-8 key to be present,
this PR adds that key to the records produced in the event-generator.py

Thanks! LGTM, merging ...