Querying DateTime64(1) columns results in exception

It seems that querying a DateTime64(1) column using the http_client.query_df (and probably any other query method) results in an exception as shown below. This does not happen with DateTime64(3) or DateTime64(6) columns. I verified that no exception is thrown if I remove the DateTime64(1) column from the select statement.
WARNING:clickhouse_connect.driver.httputil:Closed HTTP response due to unexpected exception
Traceback (most recent call last):
  File "/home/gpeev/.tox/default/lib/python3.9/site-packages/clickhouse_connect/driver/transform.py", line 42, in get_block
    column = col_type.read_column(source, num_rows, context)
  File "/home/gpeev/.tox/default/lib/python3.9/site-packages/clickhouse_connect/datatypes/base.py", line 128, in read_column
    column = self.read_column_data(source, num_rows, ctx)
  File "/home/gpeev/.tox/default/lib/python3.9/site-packages/clickhouse_connect/datatypes/base.py", line 143, in read_column_data
    return self._read_column_binary(source, num_rows, ctx)
  File "/home/gpeev/.tox/default/lib/python3.9/site-packages/clickhouse_connect/datatypes/temporal.py", line 151, in _read_binary_naive
    return numpy_conv.read_numpy_array(source, self.np_type, num_rows)
  File "clickhouse_connect/driverc/npconv.pyx", line 13, in clickhouse_connect.driverc.npconv.read_numpy_array
ValueError: cannot create an OBJECT array from memory buffer

Python version: 3.9
clickhouse-connect version: 0.5.7

As far as I can tell, there's no real numpy datatype for a time in 10ths of seconds (which is why this fails).  What is the use case for a DateTime64(1) column?

DateTime64(1) is supported by Clickhouse and, apparently, used in my organization.

Maybe you can cast it to a valid numpy type in your query?
-- Show that now64 is stored as a DateTime64(3) -- milliseconds
SELECT toTypeName(now64())

┌─toTypeName(now64())─┐
│ DateTime64(3)       │
└─────────────────────┘

-- cast now() to a DateTime(64, 1)

SELECT toTypeName(CAST(now64(), 'DateTime64(1)'))

Query id: cfc140f8-246a-453c-bbcc-a68822f93050

┌─toTypeName(CAST(now64(), 'DateTime64(1)'))─┐
│ DateTime64(1)                              │
└────────────────────────────────────────────┘

-- Cast a DateTime64(1) to microseconds

SELECT CAST(toDateTime(now64(), 1), 'DateTime64(6)')

┌─CAST(toDateTime(now64(), 1), 'DateTime64(6)')─┐
│                    2023-02-13 16:28:00.500000 │
└───────────────────────────────────────────────┘

So in your SELECT query I would suggest casting your DateTime64(1) column to either a DateTime column if seconds are good enough, or a DateTime64(3) column if you want to preserve 10ths of seconds in a numpy datetime64[ms] column (which then Pandas can convert to a Pandas Timestamp).
Note that DateTime64(1) actually works okay if you are using Python datatypes:

client.query('SELECT toDateTime64(100000000.2372, 2)').first_row
(datetime.datetime(1973, 3, 3, 9, 46, 40, 230000),)
client.query('SELECT toDateTime64(100000000.2372, 4)').first_row
(datetime.datetime(1973, 3, 3, 9, 46, 40, 237200),)

The workarounds with casting to precision 3 or 6, or querying via client.query can certainly be considered, but I don't think it's a great idea to force users to use workarounds to compensate for the lack of support for a particular and perfectly valid Clickhouse type.
If query_df will not support DateTime64(1) and will raise ValueErrors, that behavior should be properly documented so the user can at least know that workarounds will be necessary, as opposed to being surprised by an unexpected exception.

Pandas is based on numpy and numpy doesn't support a datatype of "10ths of seconds".  To deal with this edge case in the current optimized code for reading numpy datetime values doesn't make a lot of sense -- I think it would look something like multiplying the incoming ClickHouse value to a "recognized" milli/micro/nanosecond value, which would be much slower than the way it works now. There's also no advantage at to using a a DateTime64(1) over a DateTime(3) -- they use the same storage.
I can document it, but that would include a recommendation to use numpy recognized datatypes if you want to work in dataframes.

I don't think this is ideal, as it effectively means query_df won't support a perfectly valid Clickhouse type because, what ultimately amounts to, "handling the type is too much hassle" and "Clickhouse users should not use that type".

It's not a question of hassle, it's a question of tradeoffs.  I don't want to sacrifice performance for 99% of users who don't use a datatype that makes no sense, and that has an easy workaround.


Editor box for some Item types do not scroll

When trying to edit the description of an Armour or Ship Attachment item, the edit box does not produce a scroll bar. Nor can you scroll down using the mouse wheel. This also affects the Modifiers section - when you add more mods than the window size, it will not let you scroll down:
Here is a screen shot of a new gear item compared to an armour item. Both have text going down past the bottom of the editor window.

This only seems to affect the Armour and Ship Attachment item types, all the others item types function correctly.
This is using the Mandar theme and Item Sheet v2

Fixed as a part of #1386. Thanks for the report!


Telescope & NeoTree

Should distant.nvim work with Telescope & NeoTree already? Am I missing some configuration? I saw some telescope files in the repository from ~5 months ago & hoped that might be coming soon. If it's not in the plans, maybe it's something I could try to figure out? Distant.nvim looks like it solves the remote/container development problem, I just need to get it well integrated into my Nvim flow to justify not just copying Nvim into every container or machine I come across.
In any case, thank you so much for creating such a thoughtful technology & taking it to another level.
Regards

I see another issue shedding some light on the state of the telescope capability. I'll see if their config can get me going.

@josh-ramer feel free to start a discussion or comment on the issue you're describing if you have questions 😄 I worked on the telescope integration on-and-off, and it still seems a bit buggy. I think I haven't implemented the asynchronous logic for telescope quite properly, so any help you can give there is appreciated.
If you hit any bugs with the search logic from distant, it may mean we need to get a patch into the distant binary, just as a heads up. I've been building the binary along the same time as this plugin.
As for neotree, feel free to submit a PR for it if you get an implementation working!


ValueError on evaluate with generator.

output = classifier.predict(x=generator)
evaluate = classifier.evaluate(x=generator)

This code throws ValueError: Shapes (None, None, None) and (100, 4, 1, 200) are incompatible on the evaluate. The prediction works just fine.
I don't understand. Why does the predict work when the evaluate gives me a error? Anyone have any ideas?

@surGeonGG
Request you to fill issue template.
Request you to share colab link or simple standalone code to reproduce the issue in our environment.It helps us in localizing the issue faster.Thanks!

Figured it out. It was because I added another output.
Removed this and everything works fine,
    classifier.summary()
    classifier = Model(inputs=classifier.input,
                   outputs=[classifier.output, classifier.get_layer('attention').output])
    classifier.compile(loss='categorical_crossentropy', optimizer=SGD(), metrics=['accuracy'])

@surGeonGG
Glad to know the issue was resolved. Please, close this thread as your issue was resolved.Thanks!


[3.x] Add MergeGroup node to simplify merging Meshes at runtime

MergeGroup node will automatically merge suitable child / grandchild meshes when added to the scene.
Only active when running outside the editor.

Alternative approach to #61564 , using the functionality from #57661 .
Current Features

[x] Automatic or manual activation (via script) at runtime.
[x] Optional Baking in Editor for preview and more versatility. Click "bake" button and select a file to save the baked merged scene to. Features a progress bar with cancel button.
[x] Now utilizes SurfaceTool for merging vertex data. This reduces code duplication. SurfaceTool has been improved to use LocalVector rather than linked lists (this has already been done in master). This is necessary to get the speed required. It is slightly slower than the bespoke code that was previously in this PR by a factor of approx 4:3, but the benefits to reducing code duplication outweigh this, especially in terms of tangents.
[x] Meshes can now be stored in CPU, GPU or CPU & GPU memory. This allows faster access for geometry processing, and isn't reliant on GPU memory / drivers. This is used for intermediate steps during merging.
[x] Nodes can be ignored in merging by using the merging_allowed flag in the Inspector for Spatial, which is managed as an inherited property merging_mode.
(This replaces the allow_merging flag in CullInstance, as it is more versatile, and portals now respects this new flag. The legacy allow_merging flag is still respected, but deprecated. )
[x] Optional shadow proxy meshes, merging all opaque geometry, keeping only position vertex data and setting shadows_only flag.
[x] Optional Baking CSGs to MeshInstances, which can be further merged.
[x] Optional Baking GridMaps to MeshInstances, which can be further merged.
[x] Split by surface - split MeshInstances with multiple surfaces into one MeshInstance per surface prior to main merging step
[x] Optional Cleaning step to remove degenerate triangles. As well as being non-visible, degenerate triangles can cause problems especially for xatlas, should you choose to merge and bake UV2s for lightmap baking.
[x] Limit merging by locality - Instead of merging all matching meshes, you can choose to merge closest pairs, threes etc (now called "grouping").
[x] Split geometry by locality - Meshes taking up a large area can be split into multiple, for greater culling efficiency. This is the basic trade off between culling efficiency and reduced drawcalls.

Scene cleanup
After deleting meshes that were sources for merging, there is now a certain amount of cleanup that goes on, removing these source nodes, and moving their children to appropriate places (usually to be children of the parent of the removed node).
Shadow Proxies
This node can also be used for automatically creating merged "shadow proxies". I have added further backend to the merging to allow merging shadow meshes even that do not share the same materials, as long as they are opaque. If you tick the shadow_proxy checkbox, a single shadow proxy will be created, and set to cast shadows. The source meshes will be set to no longer cast shadows.
In this way rendering the shadowmaps can be done with far fewer drawcalls (at the cost of decreased culling accuracy). Additionally normals, uvs etc are stripped from the shadow proxy mesh.
Baking Merged Scenes

Example - Pirate Ship
If our pirate ship has 1000 objects using 10 different materials (woods, metals) and the ship root node is a MergeGroup, it can be converted at runtime to 10 MeshInstances (or 1 MeshInstance with 10 materials), and 1 shadow proxy MeshInstance. Shadow casting can be turned off for the main MeshInstances, and the shadow proxy can cast shadows only.
For a typical example with a directional light (4 splits) and 2 other spotlights, we might go from:

Before - 1000 draw calls + 1000 for prepass + (1000 * 6) drawcalls for shadow maps = 8000 drawcalls
After - 10 drawcalls + 10 for prepass + (1 * 6) drawcalls for shadow maps = 26 drawcalls

This is assuming the ship is opaque and all the objects in the ship are static relative to each other. If you wanted e.g. a moving sail / rigging this would be outside this MergeGroup.
Example - TruckTown demo

Download the truck town demo.
In town_mesh.tscn convert the root TruckTown node from a Spatial to a MeshGroup.
Run with and without auto_merge to turn merging on and off, examine debug monitor:

The shader changes actually went up.. not sure what that was about :grin: (ah it seems something to do with the 4 split directional light, maybe it's not culling as efficiently after merging. This should be helped by the shadow proxies though which I'm currently adding.). But the drawcalls etc are all a lot lower. The vertex count is slightly higher because the culling will be less accurate.
UPDATE: With the new shadow proxy mode, the drawcalls is reduced again for trucktown to between 30-40.
Merging techniques

[x] Split source meshes by surface before merging.
Although the original merging code was principally designed for merging identical (or very similar) objects, that shared identical materials, the new MergeGroup can now split source MeshInstances into several intermediate MeshInstances, one for each surface. This makes it more likely a match will be found for merging.
For example if object A contains wood and metal, and object B contains metal and plastic, with the old system they could not be merged. But if we split them into 3 objects based on material, then we can merge any number of these objects into just 3 MeshInstances.
[x] Combine MeshInstances with non-matching materials to form an UberMesh, that contains many surfaces, each with different materials. This can be culled as one, and forms a single node. The downside is that it culls as one, so culling accuracy may be reduced.

Notes

Naming can be changed, in particular I'm personally slightly preferring MergeNode to MergeGroup, but am happy to go with what we all think is most easy to understand.
It finds all similar meshes under the group and merges, there can be multiple lists of meshes that are suitable for merging (e.g. one list of meshes that uses a wood material, one using metal etc)
Can be used over an entire static part of a game level, or for a moving object that moves as one, such as a spaceship.
There are already a bunch of things which have to match for two MeshInstances to be considered mergeable - the GeometryInstance data, VisualInstance and CullInstance.
It may be worth adding to the verbose logging the reason why a particular pair of meshes can't be merged, and / or having a way to loosen the requirements for merging, as it could be annoying not knowing why meshes haven't merged if they have e.g. a slightly different cull_margin or something like that. But these are things that could be changed based on feedback in a later PR.
Baking in the editor is currently slower than merging at runtime.
Ideally it would be nice to hide most of the inspector settings behind an advanced tab, but we do not have this feature in 3.x.

If you add some sort of a display in-editor (AABB box around the merge group? view mode like portals?), this would effectively obviate the editor UI pr... achieving the same result while being more useful for scripting/procedural use cases

@lawnjelly what is the main benefit of this?

less draw calls

lawnjelly what is your strategy for getting this into godot 4? I wanted to try implementing texture merging.
This idea of cached merging resolved my concern with baking. So I want to try to do this for 4.

lawnjelly what is your strategy for getting this into godot 4? I wanted to try implementing texture merging.
This idea of cached merging resolved my concern with baking. So I want to try to do this for 4.

These two PRs are very much proof of concepts / probably to go with a proposal, and one of those things that it is as quick to write a draft PR as a proposal, so it made sense to do them together with some example working versions (these wouldn't be till 3.6 anyway so there is plenty of time).
Mesh merging is more immediately useful in 3.x than 4.x, because OpenGL is often bottlenecked by drawcalls, whereas with Vulkan the benefits may be less clearcut - although with the GLES3 renderer in 4.0, it still could be very useful. It would probably be best discussed with @reduz first for 4.x in terms of proposal, whether it is worth doing, and which approach should be taken, before spending a long time on it (or texture merging, which is more involved).
I also suspect the merging may be able to be done in a synergistic fashion with the HLOD. I'm not yet familiar with the HLOD in 4.0.

Is there a forward port of this to 4.0? I couldn't find it.

Is there a forward port of this to 4.0? I couldn't find it.

See discussion in https://github.com/godotengine/godot-proposals/issues/4630#issuecomment-1149724434 .
TLDR - This is more relevant for 3.x because of the cost of drawcalls / state changes in GLES. As I suspected reduz is more sceptical about adding this in 4.x because in vulkan (at least) these are much cheaper. We can experiment with it as an extension though (at a later date).

I now have the MergeGroup baking CSGs to MeshInstances, which themselves can be merged which should make rendering CSGs super efficient at runtime. (not pushed yet as I have to clean it up)
This is kind of related to https://github.com/godotengine/godot-proposals/issues/182 , although I think they are more concerned with the editor. Another additional option is to make this baking available in #61564 , and have a destructive bake operation available in the editor, which may be closer to that proposal.

Either approach is fine imho. I just wish this was merged in time for beta so that I can take it for a runtime spin and see how it works (after all, I have yet to see CSG being a editor runtime hog as opposed to e.g. GI) <3 Those approaches can be discussed later (hopefully with some more people than just me chiming in :P )

Moving this as ready for review, as I'm getting fairly happy with this as far as testing is concerned.
It's a pretty huge PR, but it contains a lot of super useful stuff, and enables a load of new possible workflows for level design / processing. I've moved the majority of the merging code out of MeshInstance and into a new MergingTool. This seems more sensible for a separation of concerns.
There are several optimizations which greatly speed up associated functions in the editor (which turned out to be bottlenecks in merging and baking). Duplicating scenes in particular can now be as much as 10-30x faster. Copying / pasting / saving scenes also benefits from this performance boost. This was necessary for the baking in editor functionality to make it non-destructive. Level loading is also approx 30% faster for large scenes.
One aspect which I suppose is an unsurprising consequence of allowing baking - baked scenes can be HUGE. This is for two reasons:

Baking means e.g. 10 boxes must store 10x the geometry, rather than a single set of geometry and 10 transforms.
The baked scenes are stored as text scenes. This seems very wasteful for high poly scenes.

On the other hand, although this occurs when testing extreme scenes like TPS demo (which has million poly models), most user created content is much lower poly. And baked scenes are mostly useful for smaller modular sections like houses, rather than pre-baking entire levels. Merging the entire level can be done at runtime, which will use far less storage.

The baked scenes are stored as text scenes. This seems very wasteful for high poly scenes.

Saving the scene as a .scn file instead of .tscn will alleviate this, although I recommend saving only the mesh resources as external .res files. This way, you can keep using .tscn for the scene.

Saving the scene as a .scn file instead of .tscn will alleviate this, although I recommend saving only the mesh resources as external .res files. This way, you can keep using .tscn for the scene file and keep the version control benefits this way.

Yes I would prefer saving binary if this is possible (or at least resources as you say for the geometry). I copied the code for saving a scene branch from the editor if I remember right, but there may be a better bit of code to copy for saving binary scene, do let me know if there are any areas that do this already. :+1:

Additionally you can choose the main scene to be a .tscn or .scn, and if .scn it will respect the compress checkbox.

Is there a reason not to use compression when saving a binary scene? It should be quite fast, especially if Zstandard compression is used.

You can set text for checkboxes using set_text(), so you don't have to create separate labels as headers like "Single Scene:". This also allows clicking the text to toggle the checkbox (like <label> in HTML).

Ah cool, I will try that out! :+1: :grinning:

Additionally you can choose the main scene to be a .tscn or .scn, and if .scn it will respect the compress checkbox.

Is there a reason not to use compression when saving a binary scene? Compression should be quite fast, especially if Zstandard compression is used. (I don't remember if compressed resource saving uses Zstandard by default.)

I just added this because there seems to be a flag used whether to compress binary files. But now that you mentioned it, it seemed pretty fast to compress / decompress and I can't think of an immediate reason not to use it, so maybe we can remove this checkbox. The choice for the main merged scene can be whether the user chooses a tscn or scn filename.

Drive by comment: I've given the simple, exposed merge_meshes() a test-drive in 3.5 with modular building pieces, and it massively reduced the draw call count on my test scene <3 so can't wait for this to land and make such modular workflows even easier

Marked as ready for review again. I managed to track down the largest (or most problematic) aspect of the runaway memory use, which is also a general problem in the editor. It was being caused by the gizmos, and most specifically the collision_meshes generated for mesh instance gizmos and picking with the mouse.
In discussion with @JFonS I have worked out some significantly much more efficient ways of handling this, but this will be in a later PR, as they aren't directly related, and I may need some functionality from this PR (specifically CPU meshes) to properly address this editor problem.
For this PR I have avoided the problem entirely by simply using a static bool to turn on and off the generation of gizmos. This is a super simple 5 line or so fix, but works great for this scenario, because when baking in the editor, the entire duplicated branch of the scene tree will be saved then deleted, so there is no need for gizmos at all.
It is possible we might decide later to have a more elaborate system for controlling gizmo generation, but for the purposes of this PR I'm of the opinion it would be over-engineering.

Based on first comment when I import Truck Town demo and covert root node in town_mesh.tscn to MeshGroup then run the demo, the merged mesh isn't visible when auto_merge is enabled.
This only work if I bake town_mesh, covert root node of the baked scene from MeshGroup to Spatial and replace town_mesh in town_scene by the baked one

Based on first comment when I import Truck Town demo and covert root node in town_mesh.tscn to MeshGroup then run the demo, the merged mesh isn't visible when auto_merge is enabled.

Oh yeah good point I've managed to cause a regression there, sorry was mostly working on the baking in the last weeks. I'll fix this. :+1:

Previous bug is fixed by your latest changes.
It seems I found another bug, when I duplicate town_mesh in town_scene multiple times for example as 2 * 2 grid, duplicated scenes aren't visible when game is running.

Previous bug is fixed by your latest changes. It seems I found another bug, when I duplicate town_mesh in town_scene multiple times for example as 2 * 2 grid, duplicated scenes aren't visible when game is running.

Ah this is interesting. I'm not exactly sure of your setup, but when duplicating 4 versions of townmesh (that are MergeGroups) they are showing up but something odd is going on with the transforms / houses, could be to do with these having shared data (resources or the subscene).

Also I had to increase the z far for the camera on the car_base, just in case they were being cut off by z depth.
This is not super surprising as although the merging itself is not too bad, the dealing with Godot scenes is fiendishly complicated to get right with owners and filenames (I'm fully expecting lots of bugs to sort)! :grin: I'll see if I can work out what is up.

Can you share your project because by latest change this is what I see now on duplicated scene:

Maybe the other way around might be better - my test scenes now work, but if you have a non-working scene you can share that would be great so I can debug it. I'm sure it will be a similar simple issue. :+1:

Thanks it works now :+1:

Another test:

Open town_mesh.tscn, change root node to MergeGroup, press bake button shows "Cannot bake MergeGroup when it is scene root". Why?
To skip this limitation, convert root node to spatial, add MergeGroup node, reparent subnodes of spatial to MergeGroup, press bake and save baked scene.
Add baked scene multiple times to town_scene.tscn(I added it 8 times)
Run demo, check fps and video memory in monitors tab in debugger.
This time select all baked scenes in town_scene and disable Auto Merge and Shadow Proxy in them and run the demo. check fps and video memory in monitors tab in debugger again, fps is bigger and video memory is half of the previous run.
Covert root node of baked scene from MergeGroup to Spatial and run the demo, monitor numbers are same as previous run.

Maybe its better to set the root node of the baked scene to Spatial by default.

Open town_mesh.tscn, change root node to MergeGroup, press bake button shows "Cannot bake MergeGroup when it is scene root". Why?

This is technically more difficult to do because in order to bake, it first has to duplicate the entire scene branch into a sibling node to the MergeGroup. This is so that the merge is non-destructive on the original data (it can't merge in place without altering the data).
So it is something that is possible, but not something I'm aiming for the initial implementation. Some other similar operations in Godot have similar limitations (I forgot which), presumably for the same reasons.

To skip this limitation, convert root node to spatial, add MergeGroup node, reparent subnodes of spatial to MergeGroup, press bake and save baked scene.
Add baked scene multiple times to town_scene.tscn(I added it 8 times)
Run demo, check fps and video memory in monitors tab in debugger.
This time select all baked scenes in town_scene and disable Auto Merge and Shadow Proxy in them and run the demo. check fps and video memory in monitors tab in debugger again, fps is bigger and video memory is half of the previous run.
Covert root node of baked scene from MergeGroup to Spatial and run the demo, monitor numbers are same as previous run.

Maybe its better to set the root node of the baked scene to Spatial by default.

Ah, this is a valid usability issue, as you say for baked scenes I should probably either disable auto-merge on the MergeGroup or convert the root of the baked scene from a MergeGroup to a spatial. Once baked there is no need to merge it again at runtime. I'll change this. :+1:
With the fps and video memory, I'm not able to completely follow your description, but just to re-iterate that in some cases merging won't lead to more fps - it is a trade off with culling. And video memory will also increase when merging - this is the trade off. This is expected:

Unmerged - Less GPU memory, less storage on disk (if baked), more drawcalls / state changes, more efficient culling
Merged - More GPU memory, more storage on disk (if baked), more startup time (if runtime), less drawcalls / state changes, less efficient culling

So you can see that merging will not always be a win, it depends very much on the scene etc, and where your bottlenecks are.
In order to deal with the specific case of merging large areas, where the lack of culling can be a problem, there are several options for merging by locality. This enables you to either merge everything, then split again on a grid pattern (with the grid scale decided by you), or you can choose to merge only the closest two objects, closest three etc. If you check the classreference for the parameters these should be described.
This can help with cases such as large voxel maps, or terrains. You can for instance merge an entire voxel map made of thousands of cubes, then split into a grid 5x5, which will give many of the benefits of reduced drawcalls while still allowing some decent culling (as you now have 25 sub-areas).

Ok now bakes to a Spatial, and I fixed another bug in the scene duplication that was causing a leak on exit. :+1:

All previous tested scenarios run OK now :+1:

Additionally normals, uvs are stripped from the shadow proxy mesh.

Can tangents and vertex colors also be stripped from the shadow proxy mesh, or is this already done? This could help bring down file sizes a bit more.

Yes, if I remember right, it's just the positions that are stored in the shadow proxy. I probably missed an etc from my text. :grin:

Bump! Still looking forward to this feature <3

Bump! Still looking forward to this feature <3

Is all done, just waiting for review and maybe an svg icon for the MergeGroup (that's more @Calinou 's forte :grin: ).

UPDATE: With fresh eyes, decided to change some naming of the properties, hopefully more sensible, but welcome improvements.
Banditted another icon (from icon_vcs_branches.png) until someone is able to make one specially.

Did a 4.0 version of this get created?

My bad cancelling the review broke, these aren't relevant

Did a 4.0 version of this get created?

See:
https://github.com/godotengine/godot/pull/61568#issuecomment-1150060118

I would definitely need this thing!!!

@SkanerSoft Please don't bump issues without contributing significant new information. Use the :+1: reaction button on the first post instead.

Tested locally (rebased against 3.x https://github.com/godotengine/godot/commit/23c142afe71be285e648312b4747bcb150eddd97), it works as expected. some feedback:

The Merging for "MergeGroup" took x ms message should mention the number of meshes considered for merging and the number of resulting meshes. It should also be printed in debug or verbose builds only, so that projects exported in release mode without verbose mode don't have this message printed (in case you don't bake the mesh merging, e.g. if you're procedurally generating levels). Other messages such as split by surface or per-mesh messages should be printed with verbose mode only.
The MergeGroup node description should clarify that it only merges child nodes together. It has no effect on sibling or parent nodes.
The MergeGroup node should have a node configuration warning when it has no child nodes that inherit VisualInstance, stating that it must have at least one child node that inherits VisualInstance to have an effect.
The Merging Mode documentation should state whether merging is allowed by default or not (if the root node also has the Inherit value).
Convert Csgs should be rendered as Convert CSGs by adding a "csgs" to "CSGs" remap below https://github.com/godotengine/godot/blob/23c142afe71be285e648312b4747bcb150eddd97/editor/editor_property_name_processor.cpp#L128.
When using Bake, saving over the current scene shouldn't be allowed as it can easily destroy your work. This happened to me while testing this project on the 3D platformer demo (I misunderstood what I was supposed to be doing).
The Single Scene checkbox in the bake dialog lacks a tooltip. It should have one so you can know what this checkbox does :slightly_smiling_face:
The Bake dialog should be replaced with an error dialog if you click it while the MergeGroup node has no child nodes that inherit from VisualInstance.
Is it expected that the baked MergeGroup isn't replaced with the instanced scene automatically? Right now, you have to remove the MergeGroup node and replace it with the instanced scene manually.

To make merging work in a less destructive manner, perhaps the MergeGroup node should be hidden and the scene should be instanced as a sibling of the MergeGroup.

[ ] The Merging for "MergeGroup" took x ms message should mention the number of meshes considered for merging and the number of resulting meshes.
[x] MergeGroup messages should also be printed in verbose mode in release builds for easier troubleshooting (at least some of them).
[x] The MergeGroup node description should clarify that it only merges child nodes together. It has no effect on sibling or parent nodes.

The MergeGroup node should have a node configuration warning when it has no child nodes that inherit VisualInstance, stating that it must have at least one child node that inherits VisualInstance to have an effect.

Not sure yet whether this can be done in efficient manner. Will see.

[x] The Merging Mode documentation should state whether merging is allowed by default or not (if the root node also has the Inherit value). It should also clarify whether Inherit is taken into account by nodes that are parents of the MergeGroup node.
[x] In the inspector, Convert Csgs should be rendered as Convert CSGs by adding a "csgs" to "CSGs" remap below this line: https://github.com/godotengine/godot/blob/23c142afe71be285e648312b4747bcb150eddd97/editor/editor_property_name_processor.cpp#L128
[x] When using Bake, saving over the current scene shouldn't be allowed as it can easily destroy your work. This happened to me while testing this project on the 3D platformer demo (I misunderstood what I was supposed to be doing).
[x] The Single Scene checkbox in the bake dialog lacks a tooltip. It should have one so you can know what this checkbox does 🙂
[x] The Bake dialog should be replaced with an error dialog if you click it while the MergeGroup node has no child nodes that inherit from VisualInstance.

Is it expected that the baked MergeGroup isn't replaced with the instanced scene automatically? Right now, you have to remove the MergeGroup node and replace it with the instanced scene manually.

Yes, to remain non-destructive.

To make merging work in a less destructive manner, perhaps the MergeGroup node should be hidden and the scene should be instanced as a sibling of the MergeGroup.

Possibly, I'll have a look whether this can be done.

Done. I also added one tiny change I forgot, a tooltip for the bake button, "Bake MergeGroup to Scene.".

Thanks!

I am here from the 3.6 release update, and have a question on this

Now you can blast through these barriers and potentially render any number of similar objects in a single draw call.

Does gridmap work the same way? As in, optimal rendering on detection of the same grid tiles/meshes?

Does Gridmap (in 4.x) work the same way in the renderer, as this MergeGroup feature?
As in, optimal rendering on detection of the same grid tiles/meshes?

Have to leave this to the 4.x guys, but afaik 4.x can do some degree of auto-instancing when duplicate objects are detected. This is not quite the same thing, as it relies on objects being duplicates, but could alleviate performance problems in some situations.
On the other hand with merging, you could even potentially procedurally shatter objects, then merge the results so they are rendered in a single draw call.
On the flip side, auto-instancing is afaik automatic, whereas merging requires a certain degree of thinking ahead for these situations. You may be able to e.g. place the shattered area under a MergeGroup, then manually call merge_meshes() after the explosion.

Have to leave this to the 4.x guys, but afaik 4.x can do some degree of auto-instancing when duplicate objects are detected. This is not quite the same thing, as it relies on objects being duplicates, but could alleviate performance problems in some situations.

Perfect, so Gridmaps are very efficient, thank you for the answer, and ofc for this new MergeGroup feature :tada:

potentially procedurally shatter objects, then merge the results so they are rendered in a single draw call.

:trophy:


Remove post from all timelines

Ability for admin/mod to remove certain post from all timelines
-admin/mod sees inappropriate post in timeline
-click post to review
-remove single post from all timelines (home/local/federated)
-all posts can still be seen in user profile
-user may have only one "bad" post while others are "safe"
Looked at https://docs.jointakahe.org/en/latest/features/#features but didn't see something similar listed

Yeah, good call - we need the ability to somehow "hide" it entirely without deleting (so that it won't reappear when another copy comes in via federation).


refactor: e2e chain initialization to support spinning up full nodes in tests (backport #1587)

Closes: #XXX
What is the purpose of the change

Add a description of the overall background and high level changes that this PR introduces

(E.g.: This pull request improves documation of area A by adding ....
Brief Changelog
(for example:)

The metadata is stored in the blob store on job creation time as a persistent artifact
Deployments RPC transmits only the blob storage reference
Daemons retrieve the RPC data from the blob cache

Testing and Verifying
(Please pick one of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added unit test that validates ...
Added integration tests for end-to-end deployment with ...
Extended integration test for ...
Manually verified the change by ...

Documentation and Release Note

Does this pull request introduce a new feature or user-facing behavior changes? (yes / no)
Is a relevant changelog entry added to the Unreleased section in CHANGELOG.md? (yes / no)
How is the feature or change documented? (not applicable   /   specification (x/<module>/spec/)  /  Osmosis docs repo   /   not documented)

Codecov Report

Merging #1613 (e84ba19) into v8.x (ef21a94) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             v8.x    #1613   +/-   ##
=======================================
  Coverage   20.66%   20.66%
=======================================
  Files         204      204
  Lines       25991    25991
=======================================
  Hits         5372     5372
  Misses      19637    19637
  Partials      982      982

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ef21a94...e84ba19. Read the comment docs.


ERROR Error executing verb "cluster_graph" in create_base_entity_graph: Columns must be same length as key

Below is the document I test：
"In the quaint village of Aeloria, a young girl named Elara discovered an ancient, enchanted locket in her grandmother's attic. When she opened it, she found a hidden map leading to a forgotten grove. Guided by curiosity and courage, Elara followed the map and uncovered a magical tree that granted one wish. With a heart full of hope, she wished for her village to prosper. The next morning, Aeloria was transformed, blooming with life and happiness. Elara's selfless wish had brought endless joy to her village, and she cherished the secret of the magical tree forever.magical places are those we discover when we dare to follow our curiosity and believe in the power of our dreams."
my cache file content is:
"{"result": "## Entity Recognition and Relationship Extraction Results\n\nEntity_types: person, location, organization, event, concept\n\nEntities:\n\n* Elara: young girl who discovers the enchanted locket\n* Aeloria: quaint village where the story takes place\n* Ancient tree: magical tree that grants wishes\n* Enchanted locket: object that reveals a hidden map\n* Hope: concept representing the power of dreams\n\nRelationships:\n\n* Elara discovers the enchanted locket in her grandmother's attic. (Person-Object)\n* Elara follows the map and uncovers the magical tree. (Person-Event)\n* The magical tree grants Elara one wish. (Object-Event)\n* Elara wishes for her village to prosper. (Event-Concept)\n* Aeloria is transformed with life and happiness. (Event-Concept)\n\nInterpretation:\n\nThe story revolves around Elara's discovery of an enchanted locket and her subsequent wish for the prosperity of her village. The magical tree symbolizes the power of hope and dreams, while the transformation of Aeloria highlights the positive impact of Elara's selfless wish.\n\nPossible Queries:\n\n* What is the significance of the enchanted locket in the story?\n* How did Elara's wish affect the village of Aeloria?\n* What role does hope play in the story's resolution?", "input": "\n-Goal-\nGiven a text document that is potentially relevant to this activity and a list of entity types, identify all entities of those types from the text and all relationships among the identified entities.\n\n-Steps-\n1. Identify all entities. For each identified entity, extract the following information:\n- entity_name: Name of the entity, capitalized\n- entity_type: One of the following types: [organization,person,geo,event]\n- entity_description: Comprehensive description of the entity's attributes and activities\nFormat each entity as ("entity"<|><entity_name><|><entity_type><|><entity_description>\n\n2. From the entities identified in step 1, identify all pairs of (source_entity, target_entity) that are clearly related to each other.\nFor each pair of related entities, extract the following information:\n- source_entity: name of the source entity, as identified in step 1\n- target_entity: name of the target entity, as identified in step 1\n- relationship_description: explanation as to why you think the source entity and the target entity are related to each other\n- relationship_strength: a numeric score indicating strength of the relationship between the source entity and target entity\n Format each relationship as ("relationship"<|><source_entity><|><target_entity><|><relationship_description><|><relationship_strength>)\n\n3. Return output in English as a single list of all the entities and relationships identified in steps 1 and 2. Use ## as the list delimiter.\n\n4. When finished, output <|COMPLETE|>\n\n######################\n-Examples-\n######################\nExample 1:\n\nEntity_types: [person, technology, mission, organization, location]\nText:\nwhile Alex clenched his jaw, the buzz of frustration dull against the backdrop of Taylor's authoritarian certainty. It was this competitive undercurrent that kept him alert, the sense that his and Jordan's shared commitment to discovery was an unspoken rebellion against Cruz's narrowing vision of control and order.\n\nThen Taylor did something unexpected. They paused beside Jordan and, for a moment, observed the device with something akin to reverence. \u201cIf this tech can be understood..." Taylor said, their voice quieter, "It could change the game for us. For all of us.\u201d\n\nThe underlying dismissal earlier seemed to falter, replaced by a glimpse of reluctant respect for the gravity of what lay in their hands. Jordan looked up, and for a fleeting heartbeat, their eyes locked with Taylor's, a wordless clash of wills softening into an uneasy truce.\n\nIt was a small transformation, barely perceptible, but one that Alex noted with an inward nod. They had all been brought here by different paths\n################\nOutput:\n("entity"<|>"Alex"<|>"person"<|>"Alex is a character who experiences frustration and is observant of the dynamics among other characters.")##\n("entity"<|>"Taylor"<|>"person"<|>"Taylor is portrayed with authoritarian certainty and shows a moment of reverence towards a device, indicating a change in perspective.")##\n("entity"<|>"Jordan"<|>"person"<|>"Jordan shares a commitment to discovery and has a significant interaction with Taylor regarding a device.")##\n("entity"<|>"Cruz"<|>"person"<|>"Cruz is associated with a vision of control and order, influencing the dynamics among other characters.")##\n("entity"<|>"The Device"<|>"technology"<|>"The Device is central to the story, with potential game-changing implications, and is revered by Taylor.")##\n("relationship"<|>"Alex"<|>"Taylor"<|>"Alex is affected by Taylor's authoritarian certainty and observes changes in Taylor's attitude towards the device."<|>7)##\n("relationship"<|>"Alex"<|>"Jordan"<|>"Alex and Jordan share a commitment to discovery, which contrasts with Cruz's vision."<|>6)##\n("relationship"<|>"Taylor"<|>"Jordan"<|>"Taylor and Jordan interact directly regarding the device, leading to a moment of mutual respect and an uneasy truce."<|>8)##\n("relationship"<|>"Jordan"<|>"Cruz"<|>"Jordan's commitment to discovery is in rebellion against Cruz's vision of control and order."<|>5)##\n("relationship"<|>"Taylor"<|>"The Device"<|>"Taylor shows reverence towards the device, indicating its importance and potential impact."<|>9)<|COMPLETE|>\n#############################\nExample 2:\n\nEntity_types: [person, technology, mission, organization, location]\nText:\nThey were no longer mere operatives; they had become guardians of a threshold, keepers of a message from a realm beyond stars and stripes. This elevation in their mission could not be shackled by regulations and established protocols\u2014it demanded a new perspective, a new resolve.\n\nTension threaded through the dialogue of beeps and static as communications with Washington buzzed in the background. The team stood, a portentous air enveloping them. It was clear that the decisions they made in the ensuing hours could redefine humanity's place in the cosmos or condemn them to ignorance and potential peril.\n\nTheir connection to the stars solidified, the group moved to address the crystallizing warning, shifting from passive recipients to active participants. Mercer's latter instincts gained precedence\u2014 the team's mandate had evolved, no longer solely to observe and report but to interact and prepare. A metamorphosis had begun, and Operation: Dulce hummed with the newfound frequency of their daring, a tone set not by the earthly\n#############\nOutput:\n("entity"<|>"Washington"<|>"location"<|>"Washington is a location where communications are being received, indicating its importance in the decision-making process.")##\n("entity"<|>"Operation: Dulce"<|>"mission"<|>"Operation: Dulce is described as a mission that has evolved to interact and prepare, indicating a significant shift in objectives and activities.")##\n("entity"<|>"The team"<|>"organization"<|>"The team is portrayed as a group of individuals who have transitioned from passive observers to active participants in a mission, showing a dynamic change in their role.")##\n("relationship"<|>"The team"<|>"Washington"<|>"The team receives communications from Washington, which influences their decision-making process."<|>7)##\n("relationship"<|>"The team"<|>"Operation: Dulce"<|>"The team is directly involved in Operation: Dulce, executing its evolved objectives and activities."<|>9)<|COMPLETE|>\n#############################\nExample 3:\n\nEntity_types: [person, role, technology, organization, event, location, concept]\nText:\ntheir voice slicing through the buzz of activity. "Control may be an illusion when facing an intelligence that literally writes its own rules," they stated stoically, casting a watchful eye over the flurry of data.\n\n"It's like it's learning to communicate," offered Sam Rivera from a nearby interface, their youthful energy boding a mix of awe and anxiety. "This gives talking to strangers' a whole new meaning."\n\nAlex surveyed his team\u2014each face a study in concentration, determination, and not a small measure of trepidation. "This might well be our first contact," he acknowledged, "And we need to be ready for whatever answers back."\n\nTogether, they stood on the edge of the unknown, forging humanity's response to a message from the heavens. The ensuing silence was palpable\u2014a collective introspection about their role in this grand cosmic play, one that could rewrite human history.\n\nThe encrypted dialogue continued to unfold, its intricate patterns showing an almost uncanny anticipation\n#############\nOutput:\n("entity"<|>"Sam Rivera"<|>"person"<|>"Sam Rivera is a member of a team working on communicating with an unknown intelligence, showing a mix of awe and anxiety.")##\n("entity"<|>"Alex"<|>"person"<|>"Alex is the leader of a team attempting first contact with an unknown intelligence, acknowledging the significance of their task.")##\n("entity"<|>"Control"<|>"concept"<|>"Control refers to the ability to manage or govern, which is challenged by an intelligence that writes its own rules.")##\n("entity"<|>"Intelligence"<|>"concept"<|>"Intelligence here refers to an unknown entity capable of writing its own rules and learning to communicate.")##\n("entity"<|>"First Contact"<|>"event"<|>"First Contact is the potential initial communication between humanity and an unknown intelligence.")##\n("entity"<|>"Humanity's Response"<|>"event"<|>"Humanity's Response is the collective action taken by Alex's team in response to a message from an unknown intelligence.")##\n("relationship"<|>"Sam Rivera"<|>"Intelligence"<|>"Sam Rivera is directly involved in the process of learning to communicate with the unknown intelligence."<|>9)##\n("relationship"<|>"Alex"<|>"First Contact"<|>"Alex leads the team that might be making the First Contact with the unknown intelligence."<|>10)##\n("relationship"<|>"Alex"<|>"Humanity's Response"<|>"Alex and his team are the key figures in Humanity's Response to the unknown intelligence."<|>8)##\n("relationship"<|>"Control"<|>"Intelligence"<|>"The concept of Control is challenged by the Intelligence that writes its own rules."<|>7)<|COMPLETE|>\n#############################\n-Real Data-\n######################\nEntity_types: organization,person,geo,event\nText: In the quaint village of Aeloria, a young girl named Elara discovered an ancient, enchanted locket in her grandmother's attic. When she opened it, she found a hidden map leading to a forgotten grove. Guided by curiosity and courage, Elara followed the map and uncovered a magical tree that granted one wish. With a heart full of hope, she wished for her village to prosper. The next morning, Aeloria was transformed, blooming with life and happiness. Elara's selfless wish had brought endless joy to her village, and she cherished the secret of the magical tree forever.magical places are those we discover when we dare to follow our curiosity and believe in the power of our dreams.\n######################\nOutput:", "parameters": {"model": "gemma:7b", "temperature": 0.0, "frequency_penalty": 0.0, "presence_penalty": 0.0, "top_p": 1.0, "max_tokens": 4000, "n": null}}"
❌ create_base_entity_graph
None
⠧ GraphRAG Indexer
├── Loading Input (text) - 1 files loaded (0 filtered) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:00
├── create_base_text_units
├── create_base_extracted_entities
├── create_summarized_entities
└── create_base_entity_graph
❌ Errors occurred during the pipeline run, see logs for more details.
BUG:16:31:37,751 datashaper.workflow.workflow ERROR Error executing verb "cluster_graph" in create_base_entity_graph: Columns must be same length as key
Not sure why this happen

i same

Same problem here.

Hi @mbt1909432
From the output provided I can see the LMM is answering in a different format.
Can you please provide information about the configuration you're using?
Working with models different than gpt-4 family may required a bit of prompt tuning, since some other models tend to be more verbose and we are actually aiming for structured responses.

Hi @mbt1909432 From the output provided I can see the LMM is answering in a different format. Can you please provide information about the configuration you're using?
Working with models different than gpt-4 family may required a bit of prompt tuning, since some other models tend to be more verbose and we are actually aiming for structured responses.

The LLM api backend I choose was ollama, the model that caused the result I mentioned above was gemma:7b. However when I used gemma2, the workflow can works normally

I tried using gemma2. It did help resolve the initial issue, but I encountered another one:
{"type": "error", "data": "Error executing verb \"text_embed\" in create_final_entities: 404 page not found...", "stack": "...from None\nopenai.NotFoundError: 404 page not found\n", "source": "404 page not found", "details": null}

I'm not using OpenAI, my embedding settings:
embeddings:
  ## parallelization: override the global parallelization settings for embeddings
  async_mode: threaded # or asyncio
  llm:
    api_key: ${GRAPHRAG_API_KEY}
    type: openai_embedding # or azure_openai_embedding
    model: nomic_embed_text
    api_base: http://localhost:11434/api/embeddings

The embedding api setting from ollama is not the OpenAI format. Use lm studio instead
发自我的iPhone
------------------ Original ------------------
From: hp0404 @.>
Date: Thu,Jul 11,2024 11:18 PM
To: microsoft/graphrag @.>
Cc: Xu Yilong @.>, Mention @.>
Subject: Re: [microsoft/graphrag] ERROR Error executing verb "cluster_graph"in create_base_entity_graph: Columns must be same length as key (Issue #426)
I tried using gemma2. It did help resolve the initial issue, but I encountered another one:
{"type": "error", "data": "Error executing verb "text_embed" in create_final_entities: 404 page not found...", "stack": "...from None\nopenai.NotFoundError: 404 page not found\n", "source": "404 page not found", "details": null}
I'm not using OpenAI, my embedding settings:
embeddings:   ## parallelization: override the global parallelization settings for embeddings   async_mode: threaded # or asyncio   llm:     api_key: ${GRAPHRAG_API_KEY}     type: openai_embedding # or azure_openai_embedding     model: nomic_embed_text     api_base: http://localhost:11434/api/embeddings
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>

I tried using gemma2. It did help resolve the initial issue, but I encountered another one:
{"type": "error", "data": "Error executing verb \"text_embed\" in create_final_entities: 404 page not found...", "stack": "...from None\nopenai.NotFoundError: 404 page not found\n", "source": "404 page not found", "details": null}

I'm not using OpenAI, my embedding settings:
embeddings:
  ## parallelization: override the global parallelization settings for embeddings
  async_mode: threaded # or asyncio
  llm:
    api_key: ${GRAPHRAG_API_KEY}
    type: openai_embedding # or azure_openai_embedding
    model: nomic_embed_text
    api_base: http://localhost:11434/api/embeddings

api_base: http://localhost:11434/api

I mean the api setting of embedding model from the ollama is not the OpenAI format. This is a standard. I know you use the embedding api from ollama，sadly，It does't support. Use another backend instead，like lm studio
发自我的iPhone
------------------ Original ------------------
From: T @.>
Date: Thu,Jul 11,2024 11:20 PM
To: reply+AQVVQALWW734EGEO3Q6UWOGETPMN5EVBNHHI5QG3RY @.>
Subject: Re: [microsoft/graphrag] ERROR Error executing verb "cluster_graph"in create_base_entity_graph: Columns must be same length as key (Issue #426)
The embedding api setting from ollama is not the OpenAI format. Use lm studio instead
发自我的iPhone
------------------ Original ------------------
From: hp0404 @.>
Date: Thu,Jul 11,2024 11:18 PM
To: microsoft/graphrag @.>
Cc: Xu Yilong @.>, Mention @.>
Subject: Re: [microsoft/graphrag] ERROR Error executing verb "cluster_graph"in create_base_entity_graph: Columns must be same length as key (Issue #426)
I tried using gemma2. It did help resolve the initial issue, but I encountered another one:
{"type": "error", "data": "Error executing verb "text_embed" in create_final_entities: 404 page not found...", "stack": "...from None\nopenai.NotFoundError: 404 page not found\n", "source": "404 page not found", "details": null}
I'm not using OpenAI, my embedding settings:
embeddings:   ## parallelization: override the global parallelization settings for embeddings   async_mode: threaded # or asyncio   llm:     api_key: ${GRAPHRAG_API_KEY}     type: openai_embedding # or azure_openai_embedding     model: nomic_embed_text     api_base: http://localhost:11434/api/embeddings
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>

api_base: http://localhost:11434/api

{"type": "error", "data": "...Error executing verb \"text_embed\" in create_final_entities: Error code: 404 - {'error': \"model 'nomic_embed_text' not found, try pulling it first\"}",

But you're correct; it doesn't work either way, but at least now I understand why. Thanks!

api_base: http://localhost:11434/api

{"type": "error", "data": "...Error executing verb \"text_embed\" in create_final_entities: Error code: 404 - {'error': \"model 'nomic_embed_text' not found, try pulling it first\"}",

But you're correct; it doesn't work either way, but at least now I understand why. Thanks!

try update the file /opt/miniconda3/envs/graphrag/lib/python3.12/site-packages/graphrag/llm/openai/openai_embeddings_llm.py as follows
Copyright (c) 2024 Microsoft Corporation.
Licensed under the MIT License
"""The EmbeddingsLLM class."""
from typing_extensions import Unpack
from graphrag.llm.base import BaseLLM
from graphrag.llm.types import (
EmbeddingInput,
EmbeddingOutput,
LLMInput,
)
from .openai_configuration import OpenAIConfiguration
from .types import OpenAIClientTypes
import ollama
class OpenAIEmbeddingsLLM(BaseLLM[EmbeddingInput, EmbeddingOutput]):
"""A text-embedding generator LLM."""
_client: OpenAIClientTypes
_configuration: OpenAIConfiguration

def __init__(self, client: OpenAIClientTypes, configuration: OpenAIConfiguration):
    self.client = client
    self.configuration = configuration

async def _execute_llm(
    self, input: EmbeddingInput, **kwargs: Unpack[LLMInput]
) -> EmbeddingOutput | None:
    args = {
        "model": self.configuration.model,
        **(kwargs.get("model_parameters") or {}),
    }
    #embedding = await self.client.embeddings.create(
    #    input=input,
    #    **args,
    #)
    embedding_list = []
    for inp in input:
        embedding = ollama.embeddings(model="nomic-embed-text", prompt=inp)
        embedding_list.append(embedding["embedding"])
    #return [d.embedding for d in embedding.data]
    return embedding_list

`# Copyright (c) 2024 Microsoft Corporation.
Licensed under the MIT License
"""The EmbeddingsLLM class."""
from typing_extensions import Unpack
from graphrag.llm.base import BaseLLM
from graphrag.llm.types import (
EmbeddingInput,
EmbeddingOutput,
LLMInput,
)
from .openai_configuration import OpenAIConfiguration
from .types import OpenAIClientTypes
import ollama
class OpenAIEmbeddingsLLM(BaseLLM[EmbeddingInput, EmbeddingOutput]):
"""A text-embedding generator LLM."""
_client: OpenAIClientTypes
_configuration: OpenAIConfiguration

def __init__(self, client: OpenAIClientTypes, configuration: OpenAIConfiguration):
    self.client = client
    self.configuration = configuration

async def _execute_llm(
    self, input: EmbeddingInput, **kwargs: Unpack[LLMInput]
) -> EmbeddingOutput | None:
    args = {
        "model": self.configuration.model,
        **(kwargs.get("model_parameters") or {}),
    }
    #embedding = await self.client.embeddings.create(
    #    input=input,
    #    **args,
    #)
    embedding_list = []
    for inp in input:
        embedding = ollama.embeddings(model="nomic-embed-text", prompt=inp)
        embedding_list.append(embedding["embedding"])
    #return [d.embedding for d in embedding.data]
    return embedding_list

`

Hi!
We are consolidating alternate model issues here: https://github.com/microsoft/graphrag/issues/657

api_base： http://localhost:11434/api

{"type": "error", "data": "...Error executing verb \"text_embed\" in create_final_entities: Error code: 404 - {'error': \"model 'nomic_embed_text' not found, try pulling it first\"}",

但你是对的;无论哪种方式都行不通，但至少现在我明白为什么了。谢谢！

可以解释一下你是怎么解决的吗

api_base： http://localhost:11434/api

{"type": "error", "data": "...Error executing verb \"text_embed\" in create_final_entities: Error code: 404 - {'error': \"model 'nomic_embed_text' not found, try pulling it first\"}",

但你是对的;无论哪种方式都行不通，但至少现在我明白为什么了。谢谢！

可以解释一下你是怎么解决的吗

bro，修改分块大小就行！！
chunks:
  size: 300
  overlap: 100
  group_by_columns: [id] # by default, we don't allow chunks to cross documents

size或大或小都可以，这样就不会报错了！！

api_base： http://localhost:11434/api

{"type": "error", "data": "...Error executing verb \"text_embed\" in create_final_entities: Error code: 404 - {'error': \"model 'nomic_embed_text' not found, try pulling it first\"}",

但你是对的;无论哪种方式都行不通，但至少现在我明白为什么了。谢谢！

可以解释一下你是怎么解决的吗

我他妈翻遍了整个issue，搞到凌晨终于成了

api_base： http://localhost:11434/api

{"type": "error", "data": "...Error executing verb \"text_embed\" in create_final_entities: Error code: 404 - {'error': \"model 'nomic_embed_text' not found, try pulling it first\"}",

但你是对的;无论哪种方式都行不通，但至少现在我明白为什么了。谢谢！

可以解释一下你是怎么解决的吗

我他妈翻遍了整个issue，搞到凌晨终于成了

所以你是一直尝试了切块部分的修改解决的吗？

api_base： http://localhost:11434/api

{"type": "error", "data": "...Error executing verb \"text_embed\" in create_final_entities: Error code: 404 - {'error': \"model 'nomic_embed_text' not found, try pulling it first\"}",

但你是对的;无论哪种方式都行不通，但至少现在我明白为什么了。谢谢！

可以解释一下你是怎么解决的吗

我他妈翻遍了整个issue，搞到凌晨终于成了

你用的什么llm模型和嵌入模型，模型供应商是？

你好，我的setting.yaml是这样的encoding_model: cl100k_base
skip_workflows: []
llm:
api_key: ollama
type: openai_chat # or azure_openai_chat
model: qwen2:1.5b
model_supports_json: true # recommended if this is available for your model.
embeddings:
parallelization: override the global parallelization settings for embeddings
async_mode: threaded # or asyncio
llm:
api_key: lm-studio
type: openai_embedding # or azure_openai_embedding
model: nomic-ai/nomic-embed-text-v1.5-GGUF/nomic-embed-text-v1.5.Q5_K_M.gguf
api_base: http://localhost:1234/v1
chunks:
size: 500
overlap: 100
group_by_columns: [id] # by default, we don't allow chunks to cross documents
可是为什么还是会报错呢？
报错信息是11:03:46,262 datashaper.workflow.workflow ERROR Error executing verb "cluster_graph" in create_base_entity_graph: EmptyNetworkError


⚠ ClamAV has degraded performance

In 1b86037, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 60 ms

Resolved: ClamAV performance has improved in 672ecd6.


quarto/integrals/mean_value_theorem.qmd

In the "quarto/integrals/mean_value_theorem.qmd", the third question, It seems no answer is correct.

Yes, there is something amiss, the integral should be int_0^t (v0 + as) ds. Thanks for pointing this oujt


Re-rendering of custom web-part is not happening while navigating between two similar pages

Category

[x] Bug

Custom web-part component is not re-rendering when navigating between two similar pages
I have created 3 SharePoint site pages from a custom template, which has a custom web-part that fetches title of the current page. Also, these 3 pages has tabbed button which links these 3 pages for navigation.
scenario:
When I open the page-1, my web-part showing correct page title-1, but if I navigated to page-2  web-part will show the old title value page title-1, same happens if I navigated to page-3 as well. This is happening because React component's componentDidMount lifecycle method is not being called while navigating between these pages.
My observations:

Webpart's render method is not called when navigating from Page-1 > Page-2

Webpart's onDispose method is not called when navigating from Page-1> Page-2 OR Page-2> Page-3

Pseudo code:
export default class SiteWebPart extends BaseClientSideWebPart<Props> {
  public async onInit(): Promise<void> {
    return super.onInit().then(() => {
      console.info("web-part", ` onInit ${this.context.pageContext.listItem.id}`);
      sp.setup({
        sp: {
          baseUrl: this.context.pageContext.web.absoluteUrl,
        },
      });
    });
  }

  public render(): void {
    const { pageContext } = this.context;
    console.info("web-part", `render `);

    /*
    SiteComponent is very simple react component,
    in which a list API call happening @ componentDidMount
   */

      const element: React.ReactElement<IProps> = React.createElement(SiteComponent, {
        pageContext: this.context.pageContext,
      });
      ReactDom.render(element, this.domElement);
  }

  protected onDispose(): void {
    console.info("web-part", `dispose`);
    ReactDom.unmountComponentAtNode(this.domElement);
  }

  protected get dataVersion(): Version {
    return Version.parse('1.0');
  }

  protected getPropertyPaneConfiguration(): IPropertyPaneConfiguration {
    return {
      pages: [
        {
          header: {
            description: strings.PropertyPaneDescription
          },
          groups: []
        }
      ]
    };
  }

Steps to reproduce

Create 3 pages from same template OR use copy of the page  feature to create the pages, add above custom webpart to the page and add navigation links between these pages using anchor tags/links. Note:  Do not ad any extra web-parts of information in these pages, content should be exact or similar.
Open the page-1, click and navigate to Page-2 using the link
Check console logs for render and dispose values of webpart
Similarly navigate between Page-2 and page-3, check the console logs of the webpart.

Expected behaviour

For every page navigation, webpart rendering method should be called
For every page navigation, webpart onDisopose method should be called

OR
Need a navigationChange event method in webpart to detect the navigation change
Developer environment

OS: Window 10
Browser(s): Chrome >= v79 ,  Chromium Edge >= v79
Tooling:  VS Code, SPFx v1.10.0

Additional context
Due to this issue we are not able provide any solution the business and customers are highly impacted, they are now refreshing this page. I have seen similar issues are https://github.com/SharePoint/sp-dev-docs/issues/5352 listed in the GitHub but nothing is solved or closed by answering. Thanks for the help and support, looking for your reply.
[EDIT]
This issue is not happening regularly, but frequently.
Thanks for your contribution! Sharing is caring.

Seems you're running into issues with the page router. That sits between you and SharePoint. It decides how much of the page needs to be re-rendered when you navigate from one page to another instead of the entire page being rendered.
There's a way to opt out of it when using custom links to navigate from one page to another (data-interception="off") see more in this post, and there's a way to hook into the navigation event with app customizers see this post, but I don't think either of these apply to your case.
When you say "same page template", are these regular pages or single part app pages you're creating? Would help in setting up a repro.
You also removed important parts of the Developer environment section in the issue template… specifically what SharePoint environment are you targetting & Node version.
There are a bunch of known/outstanding navigation related issues for these types of pages at this time: https://github.com/SharePoint/sp-dev-docs/issues?q=is%3Aissue+is%3Aopen+single+part+


Keep track of names & dtypes and introduce a "virtual" out DF

The idea of this change is to simplify the implementation of data frame backends
by applying the changes to a "virtual DF" and letting the backend decide how to apply those changes.
In this PR we have the following callbacks updated:

[x] select
[x] mutate
[x] rename
[x] pivot_longer
[x] group_by
[x] ungroup
[x] summarise
[x] join - a smaller change in the on specs
[x] join
[x] distinct

The PR is in WIP so we can evaluate if the changes makes sense.
PS: some of the changes here are duplicated from PR https://github.com/elixir-nx/explorer/pull/237, so after that merge I will rebase.

:+1: for this change. mmmmmm what you say @cigrainger?

Yep. Loving it. I think this is a really good approach.

I'm struggling a little bit with the pivot_wider operation. It is technically possible to create a new virtual representation, but it would require us to read all values of  the names_from column and match the dtype with the values from values_from. I'm not sure if we want to do that. Also, it's not clear how this would make easier for the backends.
I'm going to skip this function for now, but if you have ideas, please let me know.
PS: I made a small change in the callback signature to reflect the current implementation and the dplyr API.

@cigrainger hey Chris, would you mind to help me answering some questions about "grouping"?

for what I could understand, we keep the group names in the DF's struct, but we don't apply then when doing a DF.group_by/2. We use the groups list to perform operations lime summarise. But we really don't have the "grouped data" before performing that operation, right?
If so, why do we need to perform the grouping, taking the indexes for each group, applying in the intermediate DFs and doing the ungrouping for operations like mutate? Wouldn't be easier to just ignore the grouping and mutate the "original" DF?
My main question: if grouping is virtual (until we summarise), should we even consider the list of groups for operations like mutate or arrange? Could we try to "mutate" the original DF keeping the groups intact?

I added  a test to mutate/2: https://github.com/elixir-nx/explorer/pull/238/commits/363059337134038a809105f02a1e542109ba0927
The output DF is something like if it was not using groups.

So maybe let’s leave the group operations out of this particular refactor and perhaps work on their implementation next?

Actually, I think my suggestion is wrong. We can change those functions in this refactor, since the compatibility with mutate and friends is really about additional functionality. :)
however, I do have some questions about how dplyr manages the group/ungrouped columns. For example, what happens in your code after the first mutate? The other ungrouped columns now no longer make sense to be displayed, because they haven’t been “summarized”, or do they?
also what happens if I try to mutate a grouped dataset but the mutation is not a summarization operation such as cumsum? Say we mutate it so the new column is a negation of current values, does dplyr raise because operation then is not a grouped one? Or does it add a new regular “ungrouped” column?
I guess the question is: because group_by is “kind of a no-op”, as it just annotates the groups and still shows the full data, when does the group become concrete and effectively change the dataframe? Or does it always keep grouped and ungrouped stuff together?

@cigrainger awesome! Thank you for this great explanation! :purple_heart:
I'm going to add more tests for those operations against grouped DFs.
@josevalim yeah, I think we can go with the refactor. I think this won't impact :)

when does the group become concrete and effectively change the dataframe?

For what I could understand, this happens when each operation is applied. For example, the mutate does the grouping before adding the new column (or change an existing one). Is that correct, Chris?

@josevalim mutate and verbs other than summarise don't aggregate, so you keep all columns. You never need to change the inspect representation. The reason I tracked the groups in the struct is so we can pattern match and each verb does a split-apply-combine. Split on the groups, then apply the transform to each resultant datafframe. Once the combine step is done (with a row stack op), then you just have a regular dataframe again (keeping track of the groups in the struct). There are some things to think about re: implicit vs explicit ungrouping. But the groups are never really 'actualised' in any meaningful way outside of the operation.

For example, what happens in your code after the first mutate? The other ungrouped columns now no longer make sense to be displayed, because they haven’t been “summarized”, or do they?

So the key here is that mutate (and filter and others) differs from summarise in that you don't need to drop other columns because you're not reducing length. summarise does drop everything that isn't a group or summarised result. And see the .groups argument here for more info on ungrouping after a summarise operation.

I got it. So have the group information but that doesn't change the dataframe in any way. For example, the mutate operations above will add the same value to each element in the group.
A couple additional questions:

Do you have something like "pandas" get_group to get a representation of a particular group?

In the example above that you gave, what happens when you want to get the resulting the dataframe with the groups and the associated n_prev_participated? Do you just call summarize with the first/last/mean of n_prev_participated (given all values in the same group are the same, all of those operations will return the same result)?

Do you have something like "pandas" get_group to get a representation of a particular group?

Nah I've never seen the need when you can just use filter.

In the example above that you gave, what happens when you want to get the resulting the dataframe with the groups and the associated n_prev_participated? Do you just call summarize with the first/last/mean of n_prev_participated (given all values in the same group are the same, all of those operations will return the same result)?

n_prev_participated varies per row per group, so the values aren't all the same. The cumsum operation happens over the column within the group.

Oh, you are right. It is a cumsum. All clear now, thank you!

@philss @josevalim I forgot to mention something quite important, which is that I missed this (or didn't grok it) when I was first setting up grouped operations, but this is the direction I'm planning to go: https://docs.rs/polars/0.21.1/polars/frame/groupby/struct.GroupBy.html#method.apply. That is, my intent is to groupby then apply a closure of the intended operation.

I pushed just two final comments and we should be good!


Connection.get_dataverses only returns dataverses I have created?

I tried doing the following:
c = Connection('dataverse-demo.iq.harvard.edu', '...')
c.get_dataverses()

and found that this only returns one dataverse, which I created. Is there a way to access other dataverses on the server too? When I use the native API directly, I am able to list all dataverses on the server.

I believe this is a product of using the SWORD API to list Dataverses instead of the native API.
I would welcome switching over to the native API (there are other benefits as well), but I haven't done it because it would require a pretty hefty refactor of the Dataverse class and potentially the Dataset class as well.

@rliebz is right that the SWORD API only returns "your" data. In the DVN 3.x days this mean dataverses you were "admin" on. In Dataverse 4 we would like to move toward more granular permission checks as part of https://github.com/IQSS/dataverse/issues/1070 but this work is not complete so we generally use a permission that means the same as admin.
The Dataverse API provides other ways to list dataverses. Implementing the Search API as part of #21 is probably the best short term bet until the "contents" API can be sped up: https://github.com/IQSS/dataverse/issues/2122

Can confirm that Connection.get_dataverses() still does not return all dataverses. I like the idea of using the search API for now.


[Storage] Handling AzureNamedKeyCredential in Python SDK

Still TODO for Blob completion:

[x] Propagate all the changes to other places we accept credential

TODO for full support-complete:

[x] Propagate changes to other 4 packages
[ ] Ensure all base_client.py match
[x] Add a test for each package

API change check
API changes are not detected in this pull request.

/azp run python - storage - ci

/azp run python - storage - ci


NP33x Series Print Servers / PTSv3 English version

Is there an english version of this software.  I'm finding it difficult to use.

change i18n config in conf/app.conf:
default_lang="en-us"


basehub: faster hub restarts by letting template-sync react to SIGTERM

This change makes the hub pod restart ~29s faster consistently, and that means JupyterHub is unresponsive 29 seconds less during upgrades.

Closes #2129

Before

After

Starting template sync...

Info about local git repo to be synced:
+ git remote -v
origin	https://github.com/2i2c-org/default-hub-homepage (fetch)
origin	https://github.com/2i2c-org/default-hub-homepage (push)
+ ls -lhd /srv/repo
drwxrwsrwx    5 1000     1000         124 Aug 22 07:44 /srv/repo

Syncing local git repo /srv/repo against origin's branch master every 5 minutes...
HEAD is now at 9472884 Merge pull request #22 from GeorgianaElena/update-instructions
HEAD is now at 9472884 Merge pull request #22 from GeorgianaElena/update-instructions

Thanks for reviewing @yuvipanda!!!!!


Can I use images of WebP?

If I want to use WebP images, what can I do?

you can extend the webpack config to use a dedicated loader for those files: https://github.com/tcoopman/image-webpack-loader

OK, I got this. Build -> extend 'config.module.rules.push({/webp loaders/})'. ths

@ArvinQi @vuchl I also would like to use webp but don't have much experience with these configurations. Would you like to share more precisely how and where you solved that? Would appreciate it very much! Thanks a lot! :)

@silvio-e Here is my code in nuxt.config.js.
build: {
    extend(config, {
      isClient
    }) {
      config
        .module
        .rules
        .push({
          test: /\.(webp)$/i,
          loaders: [
            'file-loader?hash=sha512&digest=hex&name=[hash].[ext]'
          ]
        })
     }
  },


docs: add add-on docs

Closes #241
I have only started writing the docs for drizzle so that we can discuss what we want to show in the docs and what we don't. This should also be a great opportunity to test the docs preview generation.
Additional thoughts:

Is there a way / wish / need to create those docs pages automatically? We already have most of the information inside the add-on, but i don't see a way how we could add those texts in between to the adder. I think the manual way is the best way at the moment.
I was thinking about including the add-on logos here but i did not find a proper way to make this work. Other sites like the astro integrations don't use logos as well. If we don't want to use them , we should delete the logos in this pr.

preview: https://svelte-dev-git-preview-cli-272-svelte.vercel.app/
this is an automated message


Run linter only?

Is it possible to run eslint with configuration provided in craco.config.js in a single task?
"scripts": {
   "lint": "craco lint,
}

Hi @manuschillerdev
No, it is not possible. CRACO only support CRA scripts which are start, build and test.
Patrick


command: "hugo new theme" null -> minimum context

"hugo new theme" command makes null file -> minimum context.

 You should sign our Contributor License Agreement in order to get your pull request merged.小林 照幸 seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account ([for further information, click here] (https://help.github.com/articles/why-are-my-commits-linked-to-the-wrong-user/#commits-are-not-linked-to-any-user)).


Feature request: Bounce cell to reveal actions

Swipe actions are not always easy to discover by users. It would therefore be nice to have SwipeCellKit provide an easy mechanism to bounce a selected cell after x seconds to make the swipe discoverable.

Hi @pepejeria - This would probably best be addressed by a swipeOffset property and asetSwipeOffset(animated:) method.  That should give enough flexibility to the app to bounce however it needs.
Thoughts?

Hi @jerkoch, sounds good. I guess this would work for both orientations right?

Good point. Two options that I can see:

A negative offset would expose right-side actions, and a positive offset would expose left-side actions.

setSwipeOffset(offset:orientation:animated:) would be clearer but retrieving the current swipe offset wouldn't work as a single property.

I'll think some more about it.

Would wrapping the offset in an SwipeOffset enum be a viable solution? It seems to be an OK case for an enum – the two directions are mutually exclusive.
Something along the lines of:
enum SwipeOffset {
  case left(distance: UInt)
  case right(distance: UInt)

  func changed(to newDistance: UInt) -> SwipeOffset {
    switch self {
    case .left:
      return .left(distance: newDistance)
    case .right:
      return .right(distance: newDistance)
    }
  }

  mutating func change(to newDistance: UInt)  {
    self = changed(to: newDistance)
  }
}

Hey @Uupis - Good idea. I think it could work!  👍
Probably should add a .center case without an associated value, which would be the equivalent of .left(0) or .right(0).  And also a computed distance property that simply returns the UInt in cases where left or right isn't relavent.
I'll play around with this a bit when I get a chance.

Hi @Uupis - I scraped the .center idea since things got dicey trying to support + and - operators.   Here's where I landed:
public enum SwipeOffset: Equatable {
    case left(distance: Int)
    case right(distance: Int)

    public var distance: Int {
        switch self {
        case let .left(distance): return distance
        case let .right(distance): return distance
        }
    }

    public static func ==(lhs: SwipeOffset, rhs: SwipeOffset) -> Bool {
        switch (lhs, rhs) {
        case (let .left(distance1), let .left(distance2)): return distance1 == distance2
        case (let .right(distance1), let .right(distance2)): return distance1 == distance2
        default: return false
        }
    }

    public static func +(lhs: SwipeOffset, rhs: Int) -> SwipeOffset {
        switch lhs {
        case let .left(distance): return .left(distance: distance + rhs)
        case let .right(distance): return .right(distance: distance + rhs)
        }
    }

    public static func -(lhs: SwipeOffset, rhs: Int) -> SwipeOffset {
        switch lhs {
        case let .left(distance): return .left(distance: distance - rhs)
        case let .right(distance): return .right(distance: distance - rhs)
        }
    }
}

Hey @jerkoch – looks good!

👍 I was thinking this would be useful for my implementation too.
Happy to help out building the feature if needed.

Hey @Mutix - Thanks for the offer! 🍻
I had already started on this when I got sidetracked with a few other pull requests.  I'm hoping to get it finished this weekend.  If that doesn't happen, I'll gladly reach back out to you for help!

Thank you for implementing this, could you please provide an example how to slightly bounce the cell like iOS 6-7 did to reveal the camera on the home screen?

Sorry, I see now that setting a negative offset is required to expose the right side.

Is there any way of not triggering the willStart/didEnd delegate callbacks when using this? I need to tell if the user initiated the swipe or it's the programmatic setSwipeOffset(). Is there any other way of doing this?


Create a new script directory for data import, and move data conversion script to a sibling dir

This new node script will be used to set up a db -- primarily for setting up our local dbs prior to running servers (create db if it doesn't exist; collection if it doesn't exist; apply validator to collection; import sample data)
Choosing to use node because mongodb supports js for scripting for the mongo shell: https://docs.mongodb.com/manual/tutorial/write-scripts-for-the-mongo-shell/#scripting

 Add a script for fully setting up a local db with validation, indices, and more sample data

Maybe we'll want to move that folder one level higher (next to the dev/ folder for example) as I can see some DB setup coming for the curator service as well (setting up sources, acl roles, etc). I can do that when the time comes, no big deal.

good call!


Move upstream build logs from /tmp

Move upstream build logs from /tmp and create a sym link in
/var/www/html/cfme/upstream_build_logs

@miq-bot assign @simaishi

I think the logs should live in the same directory as the images that have been built.

@jvlcek moved this issue to manageiq-appliance-build.  Please close this one.


ci(sdk,app): upgrade github actions dependencies

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (508a097) 89.36% compared to head (db49c9a) 89.36%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #722   +/-   ##
=======================================
  Coverage   89.36%   89.36%
=======================================
  Files         108      108
  Lines        4438     4438
=======================================
  Hits         3966     3966
  Misses        316      316
  Partials      156      156

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Faster Azure CI Pipelines

[x] Move to MS public project
[x] Remove multi-root, debugger tests into nightly
[x] Update build badges

This should improve the completion times for our CI on Azure by around 2 minutes.

@d3r3kk

Already done


[CI/CD][SIMS #1442]Rename SIMS-Creds to be generic

Renamed sims-api-creds shared between API and Workers to the more generic name sims-creds.
Enabled the sims-creds to be deployed to the same environment multiple times using different names, for instance, sims-qa-creds to allow the use of the secrets by the QA sub-environment. This is needed to allow the different secret configurations, for instance, the Zeebe configurations.

Auto approving as talked with @abschwenker


export and import feature

first of all, mockit is great work.
it can be very good if you there is a feature that I can easily export and import my routes. for backing up stuff or sending to another one.
imagine that, I'm backend guy. and my front end guys wants to integrate with API. but API is under development, so I can mock my API and export my setting and routes and send it to my front developer. and then front developer than simply import my exported data, and use API and integrate with that.
I'm interested in to bring this feature.

Hey @amovah ,
Thanks for the feedback!
Yeah that sounds like quite an interesting feature, we have had requests for OpenAPI: #5 which might solve the problem too?
In your example would your back end dev be writing with express? Or...
In theory, you could have something that maps express routes / apps into the configuration.json that mockit requires and things should just work.
My initial thoughts are it will be alot of effort

@boyney123 we should consider this feature generally. for this project my server side is based on express. but maybe later, it can be python ruby java. so we should consider a better solution than mapping from express.
we dont need to map express routes to mockit app. I think only thing that we need, is exporting routes that we defined in mockit app and then import in another mockit in another machine.
where is mockit storing data? if I add a route into app, where is data going to be stored? in json file or a database or something like that ? if it's in file or database we can simply export that file or make a backup from database, then send it to user. so user can send that file (generated from backing up database or it's simply json file) to another user, and then that another user can simply import that file.
Or we can, bring a feature that will saves all routes into json file. then we can use that json to import routes into new app.
[{route: '/test', method: 'post', ..... }]

something like that, and another app will read data from this array and add routes that is inside of array.

Hey @amovah , thanks for confirming some stuff.
Yeah this sounds great! We currently have a routes.json file that holds all the information, so all we would need to do it maybe add some buttons to export or import (maybe in the settings modal?) some JSON configuration.
Export would save to a file on the machine
Import could just take the JSON import for now, and override the configuration file.
Sounds great!
Did you want to try and pick this up? I dont mind doing it / helping out

yes, I will be glad to help this out.
I try to bring this feature.

what is difference between these configurations file?
1: configuration/routes.json
2: client/config/routes.json
3: mockit-routes/configuration/routes.json
4: server/configuration/routes/json
they are same or not ? what is their job?

I should add some docs to this.
The way the whole thing works is they each can run in isolation. Meaning in the project you can run the client, mockit-routes and server as seperate apps.
The config itself is the same across all the apps.
When docker comes in it mounts the root configuration/routes.json into each app. Meaning all apps that touch it will read from it, from the same file on disk.
To get yourself up and running I would.
npm start in the client directory and this will read the configuration file in that project. If you want to export read that in from the directory and give it to the users. If you want to import you might need to add a route into the server (express) place.
If you create a new route to upload files, and have that uploaded file replace the configiration file in the server folder than things would be OK.
To test everything run docker-compose which will mount the 1 config file to all projects ready for consumption. I hope that makes sense?
I will write up some more contributing docs to help

why we can't use only one configuration file ? you confirm that, when docker comes they become one, so under development enviroment, I couldn't run this app untill I edited all of them and make them same. another thing is why we dont use one configuration file if all them are same as each other?


fsm2sv

Synthesizable SystemVerilog FSM generator from YAML description
https://github.com/mohamed/fsm2sv

Thanks! Added.


psram on wrover

I just got my hands on wrover module from analoglamb and i am trying to add spi ram to heap with this option:
 Integrate RAM into ESP32 memory map
Maybe my thinking is wrong, but my heap is not bigger. Here is log from hello_world example:
I (209) spiram: SPI RAM mode: flash 40m sram 40m
I (213) spiram: PSRAM initialized, cache is in low/high (2-core) mode.
I (220) cpu_start: Pro cpu up.
I (224) cpu_start: Starting app cpu, entry point is 0x40081108
0x40081108: call_start_cpu1 at D:/msys32/home/imper/esp/esp-idf/components/esp32/cpu_start.c:225

I (215) cpu_start: App cpu up.
I (1123) spiram: SPI SRAM memory test OK
I (1123) heap_init: Initializing. RAM available for dynamic allocation:
I (1124) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (1130) heap_init: At 3FFB3430 len 0002CBD0 (178 KiB): DRAM
I (1136) heap_init: At 3FFE0440 len 00003BC0 (14 KiB): D/IRAM
I (1142) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (1149) heap_init: At 4008D650 len 000129B0 (74 KiB): IRAM

And logs from example shipped with board:
I (700) cpu_start: PSRAM mode: flash 40m sram 40m
I (715) cpu_start: PSRAM initialized, cache is in even/odd (2-core) mode.
I (738) cpu_start: Pro cpu up.
I (749) cpu_start: Starting app cpu, entry point is 0x40080de4
I (0) cpu_start: App cpu up.
I (3506) heap_alloc_caps: SPI SRAM memory test OK
I (3506) heap_alloc_caps: Initializing. RAM available for dynamic allocation:
I (3512) heap_alloc_caps: At 3F800000 len 00400000 (4096 KiB): SPIRAM
I (3533) heap_alloc_caps: At 3FFAE2A0 len 00001D60 (7 KiB): DRAM
I (3554) heap_alloc_caps: At 3FFB2378 len 0000DC88 (55 KiB): DRAM
I (3575) heap_alloc_caps: At 3FFC0000 len 00008000 (32 KiB): DMAONLY
I (3597) heap_alloc_caps: At 3FFC8000 len 00018000 (96 KiB): DRAM
I (3617) heap_alloc_caps: At 3FFE0440 len 00003BC0 (14 KiB): D/IRAM
I (3639) heap_alloc_caps: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (3660) heap_alloc_caps: At 40089434 len 00016BCC (90 KiB): IRAM

Maybe its my lack of knowledge and i am expecting something different than menuconfig option suggests.
EDIT with Make RAM allocatable using malloc() as well spi ram is added to heap:
I (216) cpu_start: App cpu up.
I (1123) spiram: SPI SRAM memory test OK
I (1124) heap_init: Initializing. RAM available for dynamic allocation:
I (1124) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (1130) heap_init: At 3FFB3480 len 0002CB80 (178 KiB): DRAM
I (1136) heap_init: At 3FFE0440 len 00003BC0 (14 KiB): D/IRAM
I (1143) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (1149) heap_init: At 4008D768 len 00012898 (74 KiB): IRAM
I (1156) cpu_start: Pro cpu start user code
I (1161) spiram: Adding pool of 4096K of external SPI memory to heap allocator
I (1168) spiram: Reserving pool of 32K of internal memory for DMA/internal allocations
I (520) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
Hello world!
This is ESP32 chip with 2 CPU cores, WiFi/BT/BLE, silicon revision 1, 4MB external flash
stack size: 2084
free heap: 4477972

If you want the external memory to be available from standard malloc calls, you need to select SPIRAM_USE_MALLOC option.
We will update documentation page on PSRAM to point to the specific menuconfig option.
https://docs.espressif.com/projects/esp-idf/en/latest/api-guides/external-ram.html#software

Thanks.


Feature/tests

/cc @bambr1975

SonarQube analysis reported 9 issues

 definitionProvider.ts#L8: Missing trailing comma
 definitionProvider.test.ts#L18: Missing trailing comma
 definitionProvider.test.ts#L40: Missing trailing comma
 definitionProvider.test.ts#L59: Missing trailing comma
 definitionProvider.test.ts#L75: Missing trailing comma
 documentSymbolProvider.test.ts#L18: Missing trailing comma
 documentSymbolProvider.test.ts#L28: Missing trailing comma
 workspaceSymbolProvider.test.ts#L18: Missing trailing comma
 workspaceSymbolProvider.test.ts#L28: Missing trailing comma

Current coverage is 42.69% (diff: 100%)

Merging #36 into develop will increase coverage by 5.23%

@@            develop        #36   diff @@
==========================================
  Files            19         19
  Lines          1986       1972    -14
  Methods         110        110
  Messages          0          0
  Branches        419        414     -5
==========================================
+ Hits            744        842    +98
+ Misses         1242       1130   -112
  Partials          0          0

Powered by Codecov. Last update bdedb1a...20739fc


Unable to import file using Flat File Wizard when connected through AAD

Azure Data Studio Version: 1.3.6

Steps to Reproduce:

Open connection through "Azure Active Directory - Universal with MFA Support";
Open flat file import wizard (ctrl+i);
Select a file, schema and table name;
Click Next->Next->Import Data;
Error is shown: Cannot open server<EMAIL_ADDRESS>requested by the login. The login failed.
Import does work when repeating same steps when logged in with 'SQL Login'

This continues to be an issue for us as well.
SQL Version - Microsoft SQL Azure (RTM) - 12.0.2000.8
Using AAD with Universal MFA
ADS Version: 1.20.1 (user setup)
Commit: fe6312306647441b30102a64bb43abb06ebcf2be
Date: 2020-07-17T02:39:41.016Z
VS Code: 1.47.0
Electron: 7.3.2
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS: Windows_NT x64 10.0.18362


Point Convert Precision

Use the depth information collected in the DEPTH_LONG_THROW mode, and then use the save_pclouds.py script to convert it into a point cloud, but the accuracy of the point cloud is not enough. Are there any ways to improve it?

In Hololens2 research mode, is there any difference between the point cloud accuracy obtained by DEPTH_LONG_THROW and DEPTH_AHAT? After experimentation, it is found that the accuracy of the point cloud obtained in the two modes is not different from the effect.

In Hololens2 research mode, is there any difference between the point cloud accuracy obtained by DEPTH_LONG_THROW and DEPTH_AHAT? After experimentation, it is found that the accuracy of the point cloud obtained in the two modes is not different from the effect.


New Command - spo listitem roleinheritance break

New Command - spo listitem roleinheritance break
The aim of this PR is to add a new command which will allow to break role inheritance on list item.
Linked Issue
Closes #3199

Awesome! We'll review it asap

@waldekmastykarz thanks for your review and help in this PR👍. You rock 🤩

Merged manually. Thank you! 👏


[DOC] Typo on home page

There is a small typo on the home page in the fast & flexible section - https://refine.dev/

Thank you @ximenean !


useRxCollection returns null the first time it is called?

Hello and thanks for the awesome project!
While using useRxCollection I found it doesn't return the collection the first time it is called in a particular component for me. Is this by design?
ex:
const itemsCollection = useRxCollection('items')
 // ❓ useRxCollection returns null the first time????
  console.log('itemsCollection', itemsCollection)
const query = itemsCollection.find().where('owner').equals(userId)
const { result: items, isFetching: isFetchingItems } = useRxQuery(query)

I was expecting itemsCollection to get the right owner but it throws an error because the collection is null on the first render of the component. I made sure I did the setup with the provider correctly and used await on db.addCollections({ items: {...}}).
I might be messing up somewhere else but figured I could open here to see if anyone else has run into this.

Yes, by design useRxCollection returns null until the collection becomes available so you would have to account for this.
Alternatively, which is what I would recommend, you can use useRxData which is a thin wrapper around useRxQuery that handles this for you and makes for a more concise syntax.
So your example could be rewritten as follows:
const {
  result: items,
  isFetching: isFetchingItems
} = useRxData('items', col => col.find().where('owner').equals(userId));

Thanks for the response and clarification! I think this is good to close.