Improve DirectionalLight shadow rendering efficiency

Godot version
3.4.2
System information
Windows 11, GLES3, NVIDIA RTX 3090 (511.23)
Issue description
A screenshot was posted to the discord memes channel a few days ago, and I got curious. The screenshot is below, along with the original comment.

Godot demo project. 12 cubes. 100k vertices. 20! drawcalls.
And people asking why Godot so slow when they run demos.

I created a test scene with a few objects and found that DirectionalLight with shadows enabled was the biggest issue. There is also some odd behavior with this combination that looks like false positives in the renderer culling, but I haven't found any reason for it while looking through the spatial partitioning code. At certain camera angles, objects that are no where near the camera frustum will contribute to the "Vertices Drawn", "Surface Changes", and "Draw Calls" metrics in the monitor.
Here are some screenshots from the test procedure described below. First, the scene is running, and I have the monitor shown in the editor:

Then I hide the SportsCar node, and "Vertices Drawn" is reduced by over 20,000! Surface Changes and Draw Calls also see some improvements:

Then I turn SportsCar back on and disable the shadow on DirectionalLight. Vertices Drawn has reduced by almost 60,000 (a massive 76% improvement)! and Surface Changes and Draw Calls see a 75% reduction; another huge improvement!

And finally, changing SportsCar visibility with the shadow off has no impact whatsoever, exactly what is expected for a node that cannot be seen by the camera at all.

This might be the cause of #35746.
Steps to reproduce

Open the example repro project and run the scene.
Open the Debugger/Monitors tab and enable everything under Raster.
Change the visibility of the "SportsCar" node, which is behind the camera and its shadow is being cast away from it.

Notice that this object is being drawn when it should not.

Do the same SportsCar visibility test with the DirectionalLight shadow disabled.

Notice that the monitor numbers no longer change with the SportsCar visibility! The numbers are also much lower with the shadow disabled.

Do the very same test, but hide the DirectionalLight, and use either of the other lights instead.

I found that any combination of the omni light and spotlight with their shadows does not change how many things are drawn, regardless of the SportsCar visibility. Shadows for these lights are free???

Do the same test but this time use DirectionalLight+Shadow and change the camera rotation along its local X axis while SportsCar visibility is enabled.

Even a slight adjustment upward from the camera's default rotation will reduce the number of things drawn. And yes, these are all from the SportsCar. Do the same rotation test with SportsCar hidden.

Minimal reproduction project
Render Culling Test.zip

The first pic is in editor. I can't find it right now, but I recall discussing it, and it turns out the editor also counts the gizmos and the grid, hence more draw calls than you expect.
Directional light basically renders the whole scene again when casting shadows, so careful tuning of directional light shadow distance and what meshes cast shadows is needed. You also have the option of using a simplified mesh for shadow casting, i.e. placing a second invisible mesh set to shadows only.

Yes, but the gizmos and grid are not included in metrics while running the game.
Shadow map rendering is expected to be a little more expensive, but not 400% more expensive. Even if the camera encompasses the whole scene, I would expect the worst case for the shadow rendering pass to be 2x everything. Do that with the repro scene and you'll see it tops out around 127 draw calls with DirectionalLight shadows enabled. Disable those shadows and fake them with spot lights and you only have 49 draw calls.

There are also splits for a directional light, and depth prepass. The culling of the shadow volume against the view frustum is not that good either, I did a PR for this many moons ago.

Shadows in Godot generally need some optimizing, albeit things are much worse in 3.x compared to 4.0
However Primitive count and Draw calls don't mean much, Frame time contribution matters, but that's available in Visual Profiler in 4.0
On the topic of Culling https://github.com/godotengine/godot-proposals/issues/3890 proposes a method to Improve on Shadow rendering, with the occlusion culling part also being applicable to Directional Shadows if that is doable however I'm unsure if that would improve performance as they have to be rerendered every frame and the main benefit of it is to do a quick check to see if you need to render a new shadow or not.. But that is by using GPU's Shadow Maps, Using Occlusion culling available in 4.0 for Normal Rendering just used to help with Directional Shadows on the other hand might benefit performance.
Basically before rendering a given shadow in the shadowmap checks if there were any changes in the area that it's rendering (as in The radius around the Omni light, or Cone of the Spot Light, or less ideally their AABBs) and if there were changes there it pulls up the Last frame's Shadow to use as a Depth Buffer for Occlusion Culling, along with the Last frame's recorded object positions (to determine which objects were the casters in the last frame) and see if the Casters moved, or new Casters arrived in a possibly non shadowed area, in both cases requiring a new shadow rendering.

Even though draw calls and primitive counts don't mean much on their own, this issue gets much worse in a real scene where several nodes have multiple materials and shaders. Even if they are hundreds of units behind the camera, we still end up paying for shader switches for these nodes that should be culled early on.

And of course, some other tricks like rendering shadows with lower LOD meshes.

This is feasible to implement in Godot 4.0 with the automatic mesh LOD, see https://github.com/godotengine/godot-proposals/issues/2600.

I made a few minor changes to the repro project; Positioned the camera specifically so it can trigger a large change in the shadow map projection with just a minor change in camera angle. Use the space bar to switch between the two angles (the difference is only 1 degree). I also changed the shadow mode to 2 splits and the depth range to optimized which shows the issue pretty well.
There are other camera angles and shadow settings that are far worse. And throwing various things into the scene can completely resolve the "disappearing shadow" issue, but I don't know why.
Render Culling Test-optimized shadowmap.zip
And I also captured the frames between the two minor camera angles in RenderDoc. Here are the interesting textures.
Frame buffer at -50 degrees:

Shadow map at -50 degrees:

Frame buffer at -51 degrees:

Shadow map at -51 degrees:

Looks like the light-space projection is very inaccurate. I'll test the patch in #43207, I'm wondering how much it will help in this case.
This is definitely related to the issue. Hiding SportsCar makes both camera angles look like the second image, and fewer things get drawn.

Looks like the light-space projection is very inaccurate. I'll test the patch in https://github.com/godotengine/godot/pull/43207, I'm wondering how much it will help in this case.

@parasyte Bump :slightly_smiling_face: Did you have time to try it out?

Oof, sorry for the long delay! I finally got around to updating one of my projects to 3.5 (from 3.4.4!) so I'm still behind and getting caught up. I'll take a look at making sure my dev environment for Godot still works, so I can build the latest 3.x and try that patch. The repro scene with the cars definitely still has the issue shown above in 3.5, though. That is one of the things I tested while checking out 3.5...
I don't have a strict timeline on doing any of this, but I kind of have the interest again.

Fixed by https://github.com/godotengine/godot/pull/82584.


Add a script to build for RISC-V

Currently, all of the scripts that run elf2tab are for the ARM boards and
also call tockloader. None of the RISC-V boards use tockloader at the moment
but still need the final tbf binary. Add a simple script to build and then
run elf2tab.
This can certainly be expanded in the future but right now I'm looking for something simple that we can keep in tree to let RISC-V be built easily against master

Just as a note, I don't know about the other boards, but I use tockloader with the arty-e21 board.
tockloader install --board arty --openocd --app-address 0x430000

Instead of adding a script, would it be worth adding a Makefile that builds what you want and runs elf2tab? That way we line up with Tock and libtock-c.

How does a Makefile improve the situation here? Personally I find the combination of build.rs (for managing changes in dependencies like layout files or parameters) and shell scripts (for more or less sequential calls of external binaries) quite canonical in the Rust context.

@torfmaster The confusing part is that once this PR is merged we will now have multiple build_*.sh scripts. For a new user how do you know what build_*.sh script to run? Should I run build_riscv.sh or build_examples.sh?
A Makefile on the other hand it straight forward. We can just have PLATFORM=opentitan make examples

I can see about folding this into flash.sh. I wasn't sure if that was wanted or not so I figured I'd start with a separate file.

@alistair23 I think the content of this PR can be distributed over build.rs and flash.sh and there is no need for build_riscv.sh.

I can see about folding this into flash.sh. I wasn't sure if that was wanted or not so I figured I'd start with a separate file.

I think the opentitan board should be a first class citizen of libtock-rs

LGTM from a design perspective. I cannot tell whether the numbers are correct (as I do not possess the required hardware) but in any case this is a great improvement towards a multi-board build chain.

I like this a lot!
bors d+

bors +r

bors r+


Color column

Can you add color column support in a cleanish looking way?
I think currently color column does not work

Should be fixed now. If you want to override the color, you can do so in your config. E.g:
{
   "scottmckendry/cyberdream.nvim",
   config = function()
       local colors = require("cyberdream.colors").default -- Import the default cyberdream colors
       require("cyberdream").setup({
           transparent = true,
           italic_comments = true,
           hide_fillchars = true,

           theme = {
               -- Override highlights to your liking :)
               highlights = {
                   ColorColumn = { bg = colors.cyan }, -- hex codes are also supported e.g. bg = "#ff0000"
               },
           },
       })
       vim.cmd("colorscheme cyberdream")
   end,
},


Create a Data Specification Profile

A new profile is required that will hold the id of the root model that a Data Specification is linked to. There may be other data that we want to associate Data Specifications with as we develop the SDE Admin system, so this profile will be named with a view to additional fields being added to it.
The profile should be setup with the following information:
Profile Name: Data Specification Profile
Applicable For Domains: DataModel
Field name: Root model id
Data type: UUID (Or if that doesn't work, use String)
An example of creating a profile can be found at: https://github.com/MauroDataMapper-Plugins/mdm-plugin-explorer/pull/13/commits/7a61333cbbc00f5c92125d4a8f186a01eb985c02

This work is no longer required. Closing issue.


Transporter does not remove deleted document from index

Hi,
I have an issue here, where transporter does not remove deleted documents from index.
Since transporter replaces rivers, I expected such functionality. Is it out of scope of transporter or am I doing something wrong?
Thanks for help

I've had similar behavior with Mongo -> Mongo copies.  Deleted documents in MongoDB are basically _id only documents, i.e. {_id: ObjectId("<id>")}.
What is your source database type?

mongo -> elasticsearch . I worked around this issue using a typical  elasticsearch pattern, which is a rotating index. I build a new ES index every N minutes, and atomically update an alias after a transporter run.

+1 Same issue

There are a number of issues right now surrounding this functionality in transporter
on inserts and updates the document is run through the transformer which can potentially update the _id for insertion into other databases, however for deletes the document is passed through without transformation.  this behaviour needs to change

We've made a change in the recent transporter build,
transformers now have full access to the transporter message, not just the document.
have a  look at the changelog for the format of the msg, (we'll be writing a blog post soon about this)
https://github.com/compose/transporter/blob/master/CHANGELOG.md
deletes are now processed by transformers, so if your _id changes between databases and there is logic that needs to happen, that can be done in the transformer directly

+1 for the blogpost :D

@fredoche we believe this has been resolved with the 0.0.4 release. You can checkout a blogpost on the changes here

I am tailing mongo oplog and moving the data with transporter into elastic search.
So what exactly needs to be done so the documents will be automatically deleted from ES?

I don't know if there is actually some solution to remove data from transporter as soon as it's removed from mongo. Please could someone say if it's being worked on or if it's some hidden feature?

We are having the same issue. As there is no other solution for the moment, we are going to setup two different jobs for the moment.

A first one, running frequently updating data all day long.
A second one, running once a day, deleting all data from destination and re-inserting everything.

However, I consider this as a temporary workaround. If no solution is found, we will definitely find abandon transporter. Having a day long of potential out of data is a no-go.

@iammichiel if you abandon transporter please let us know what other solution did you go for.
I know that rivers are deprecated. Or I could tail mongo oplog by myself in my app...

So I abandoned transporter completely, and I created my own solution to sync MongoDB with ElasticSearch written in javascript using meteor.js framework, if anyone is interested I could make it open source. My solution also supports custom transformations, and AUTOMATICALLY DELETES removed mongodb documents from ElasticSearch...
I call it Mongo2ES.

@Alino , please share.

@Alino , please share.

@Alino :+1: was thinking similar since this project seems kind of abandoned?

@Alino @gdw2 @chiefly we apologize for not addressing this and other issues lately. We've been working to figure out how to best manage this project and will possibly (read: hopefully) be hiring someone to specifically run and manage the project full time soon. If you happen to know of someone, please send them my way.

@gdw2 https://github.com/Alino/Mongo2ES

Transorter v0.1.1 successfully syncs remove operations for me. Not yet in production, but for simple one collection replication from MongoDB to ElasticSearch it works.

closing this in favor of https://github.com/compose/transporter/issues/209


Differentiate write and write2 + simplify

UvWriteReq only uses pooled blocks
UvWrite2Req only has one buffer
List<GCHandle> isn't needed
From #519

Still needs a change as above

@benaadams Still working on this?

Not at the moment; would be a slight breaking change though would be weird if anyone but Kestrel took a dependency on them

Going to close this and raise as an issue


Step 7 location of Users JS

In step 7 it is shown as being in src/ when it is in libs/ currently in the es6-tutorial branch.
https://github.com/angular/material-start/tree/es6-tutorial#step-7

This should now be fixed on the es6-tutorial branch.


Propose "minimum" option for defanging

Great library!
It'd be great to see an option for defanging that would defang in a lesser intrusive manner. Potentially with the option to prioritise towards the left or right side. This may assist in the readability of some of the output strings (but maybe slightly more risky, hence making it optional!).
For example:
www.example.com = www[.]example[.]com (current)
www.example[.]com (minimum flag used, prioritise right by default)
www[.]example.com (minimum flag used, prioritise left)
http://www.google.com/this/is/bad.html = hXXp://www[.]google[.]com/this/is/bad[.]html (current)
http://www.google.com/this/is/bad.html = hXXp://www.google[.]com/this/is/bad[.]html (minimum flag used, prioritise right by default)
http://www.google.com/this/is/bad.html = hXXp://www[.]google.com/this/is/bad[.]html (minimum flag used, prioritise left)
<IP_ADDRESS> = 192[.]168[.]4.2 (current)
192.168.4[.]2  (minimum flag used, prioritise right by default)
192[.]168.4.2 (minimum flag used, prioritise left)

Thanks @geekscrapy ! Great idea, I hadn't thought of that. I'll work on getting this implemented over the next month or so.

Hey, hate to bug - any progress with this? I may have some downtime soon to look at it of you haven't already

Hey @geekscrapy ... thanks for the follow up and I apologize it has been so long. If you have time to take a look, that would be great! Otherwise, it is high time I work on it, so I'm happy to take a look at it in the near future.

I'm going to work on this today.

@geekscrapy : I did some investigation into how to implement this feature request and I do not think I'm going to do it for two reasons:
Reason 1
I don't think it is possible to implement this feature request without significant changes to how this package works.
The fundamental difficulty is that there is no way for this package to know which instances of a certain character to defang. For example, if the defang function is given www.example.com and we want to minimally defang (with priority on the right), we would expect to get www.example[.]com back from the function. The problem is that there is no way for the package to know that the first . should be left alone while the second . should be defanged. This issue becomes more poignant if one of the functions is given a large blob of text with multiple indicators in it. Does that make sense?
From what I can see, implementing this feature would require a significant over-haul to the way this package works (it seems to me we would have to update the package to first parse indicators and then defang them appropriately). This is certainly possible, but not something I'm interested in doing right now.
I'm open to suggestions on how I can implement this more easily, but I don't see any easy, non-invasive way to implement this.
Reason 2
In your original comment, you mentioned that using this feature would be risky (and I think you are correct); after thinking a bit more on this, I'm not comfortable introducing a feature with even a small amount of risk. When it comes to handling indicators of compromise, I'm comfortable compromising readability to maintain safety and security.
Conclusion
Does all of this make sense? I still think it is a great idea and I would love to implement it someday/somewhere, but I think this would require more extensive changes to this package than I'm willing to make right now. Again, I'm open to any ideas anyone has one this issue (I'll close it out in a few weeks - but I'm comfortable reopening it if a good implementation strategy arises).

You are spot on with your reasoning, and I was having the same thoughts about overall parsing.
My thought is that the library would probably need to understand the IOC type before it can make a decision on what to prioritise. This is made simpler, of course if you are dealing with a list of IOCs rather than a block of text.
Question: do people use this to convert blocks of text with IOCs in it? I would assume that IOCs are defanged "at source", during initial analysis/lookups and are therefore just a list (but we obviously can't ignore other use cases!) My particular use case is taking IOCs from analysis and pushing them to chat/emails etc. and so wouldn't be caught out by this.
Overall though, what type of IOCs would need to be recognised to implement this? As far as I see, it would just be two:

IP
Domain

Other considerations:

Email @ sign should come before a domain
URLs would only need the domain part converted so is covered above

So as far as I see, there would only be those two types that need to be recognised.
Thoughts?

To your question about people defanging blocks of text: Yes, there are use cases I'm aware of where people are defanging blocks of text and I wouldn't want to place any restrictions on the input format.
I agree that the system would have to be aware of IOC type and that is a great point that implementing this feature request really only requires us to handle IP addresses and domains.
Identifying IOC types is possible (I already do this here: https://github.com/fhightower/ioc-finder), but that is not a step I'm going to take right now. I'll keep this issue open, but will not be working on it in the near future.


Maintainer

Hi, I love this project and hope to contribute more to this. Please let me know how i can help?

Hi Harish! I am sorry for the long delay for me reply since I was away. I am thankful for your feedback and it really motivates me! :)
Well regarding contribution. The project was started as a container for the widgets I usually implement in my other repositories. As of now, I am really focused in the writing guidelines of the Slikker Design System and was planning to use this repository as a source for widgets that follow the guidelines.
Currently, the most of development goes internally, meaning I do not push changes since it's mostly used in my private repositories (which I haven't published yet 😉). As of yet the purpose and value of the package changes depending on the needs for these unpublished repos. I was aiming defining the stable development path for the package until the end of the Q2. Until then, it's probably clear that I would like to reimplement all of the material.dart library (still making both compatible) which definitely requires a lot of work, but the way of reimplementing it is not clear yet. I am very thankful for your interest, really! If you are interested, I may get in touch with you when the project development path will be defined :)


Show breadcrumb trail of past locations

Get past locations from firebase and display on map.

Already have location history showing from #3.  Next step is to make it look like a breadcrumb trail. Also get rid of accuracy radius.
Consider showing speed either with radius of color.

Switched map to satellite & replaced cirlces with a polyline.


Configure Petastrom to point to HDFS and SPARK

What are the configuration steps to point Hadoop cluster.
Just passing hdfs://nn:8020/file is not working. Do we need any specific setup required in pyarrow?
Similarly do we need to specify spark-defults.conf
I appreciate your input
PS: if you have specific instruction for Cloudera or any other kerberos enabled cluster that will be useful as well.

By default, Petastorm uses libhdfs3 driver. You need to make sure that your HADOOP_HOME environment points to a proper hadoop installation. If hadoop is configured properly, you should be able to run something like this from command line:
hdfs dfs -ls /

Once it succeeds, petastorm should work. You can also try specifying hdfs_driver='libhdfs' as make_reader parameter. That would force petastorm to use offical hadoop driver (this one is written in Java).
Might be a good idea to consult with the hadoop system administrator to get help configuring hadoop for your particular environment.


Removes docs from repo in favour of using cert-manager.io

The corresponding cert-manager.io PR is https://github.com/cert-manager/website/pull/997.
/hold
until the website PR is merged.

/retest

/hold cancel

/assign @jahrlin


stream-k v0.4

enable perf CI
adapt tune_streamk to be able to switch kernel tunning from command line
add persistent_gemm unit tests
make test_correctness work with multiple kernels
add perfci tuning shapes and fall back configs

LGTM for the CI part 👍


drag drop from zip files doesnt work

if i download a zip file, open it with 7zip and drag drop the files in the zip into vscode i get "unable to read file (error file not found ......local\temp.....\myfile)"
that is an issue since 3 years but the issue has always been closed AND NOT added to the roadmap
please i believe in you

/duplicate #51229


GraphQL: fix mongo order_by

Coverage increased (+0.08%) to 89.894% when pulling e676bb5cb59662bfe93d6794b9a7d8068cad1f6e on job-order-by into bcfdfd3ed9154b40157795c289e2676ad1eccc11 on develop.


fix(react) - Implement popper library

Fixes :- https://github.com/international-labour-organization/designsystem/issues/534
Notes

Implemented Popper.js to handle the tooltip.
Modified classes to fix icon

Screenshots
Long tooltip (More than 50 chars)

Normal tooltip (Less than 50 chars)

@justintemps In react it has the tooltip for both text and icon. Should I just leave the label as it is or just have the icon like the twig component. Also @justintemps might need your help in making the width bigger for the story in tooltip for react.

@justintemps In react it has the tooltip for both text and icon. Should I just leave the option to have a tooltip in a label as it is or just have the icon only like the twig component.

Also @justintemps might need your help in making the width bigger for the story in tooltip for react.

Sure let's talk during the standup today


Don't close Tags dropdown after selecting an option

I'm submitting a change request
Brief Description
Normally we want to select more than one Tag, after reviewing a changeset, but the dropdown closes automatically after selecting one option

What is the motivation / use case for this feature?
If it doesn't close automatically after selecting an option, we would improve the experience and motivate users to add more tags.

Can I work on this feature?

@ArslanYM Yes, you can. I noticed when we click on the input box, it doesn't close the dropdown, so maybe a solution is to include the label inside the input component.

@willemarcel Will definitely keep that in mind. Have started working on this.


Program hangs when instantiating a GP using multiprocessing

I'm trying to do what seems like a simple task: use multiprocessing to parallelize the optimize() call over many unique GPs. Here's a minimal example of what I'm trying to do.
from GPy.core.gp import GP
from GPy.kern import White
from GPy.likelihoods.gaussian import Gaussian
from GPy.inference.latent_function_inference.laplace import Laplace
from multiprocessing import Pool
import numpy as np

# Wrapper needed so the function is pickleable, which is required for multiprocessing.Pool
def opt_wrapper(gp):
   return gp.optimize() # Can replace with 'return 1' and program still hangs

size = 100 # Program works when this is low enough
inference_method = Laplace() # Program works when this is None
models = [GP(X=np.arange(size).reshape(size,1), Y=np.arange(size).reshape(size,1), kernel=White(1), likelihood=Gaussian(), inference_method=inference_method) for _ in range(1)]

print "Starting pool..."
pool = Pool(1)
print pool.map(opt_wrapper, models)
pool.close()
pool.join()

The program simply hangs after printing "Starting pool..." Annoyingly, it also results in a zombie process for each worker in the pool (just 1 in this example).
The program works just fine when size is less than about 60. However, when size is larger, it simply hangs after printing "Starting pool..." Note you can replace pool.map with the built-in map and it works just fine, so it seems to be an issue of creating a GP with Laplace over a certain size within a new process.
The program works just fine when any one of the following conditions are true:

When size is less than about 60. For larger values, it hangs.
When Laplace() is replaced with None. This is because the Gaussian likelihood then defaults to ExactGaussianInference(); however, my actual project's likelihood is custom and requires Laplace().
When pool.map is replaced with the built-in map.

Lastly, it still breaks when you replace return gp.optimize() with return 1. Similarly, the following program hangs (same imports):
def make_gp(dummy):
   inference_method = Laplace() # Again, program works when Laplace() becomes None
   gp = GP(X=np.arange(size).reshape(size,1), Y=np.arange(size).reshape(size,1), kernel=White(1), likelihood=Gaussian(), inference_method=inference_method)
   return 1

size = 100 # Again, program works when this is small
pool = Pool(1)
print pool.map(make_gp, ['dummy']) # Again, works with `map`
pool.close()
pool.join()

It seems to be an issue of instantiating/copying a GP--both with Laplace and above a certain size--within a new process. Seems highly odd and highly specific. Any help greatly appreciated.

Are GPs instantiated in the same node? Are you running out of memory? When my student @opkoisti tested distributed GP approach with GPy, there were some problems with the implementation of GPy. Unfortunately I don't remember whether they were properly fixed. Maybe @alansaul remembers?

Thank you for the response! Yes, they're all on the same node. I'm just trying this out on a single laptop with 4 cores. (I'm also considering distributed, though it would be with the multiprocessing package.) I'm not running out of memory either--good question. The memory requirements are very small.
It does seem to be a GPy implementation issue. I've tried toy examples with other objects and it works just fine. I know multiprocessing has issues with unpickleable data (i.e. the need to wrap the optimize() call). It could also be an issue with multiple instances of multiprocessing? I looked into what's happening under the hood when Laplace() is called, but it goes pretty deep with some other packages. I also tried replacing Laplace() with expectation_propagation.EP(), which resulted in some NotImplemented error about object 'super' not having __getstate__.
I look forward to help from @opkoisti and/or @alansaul. Thanks again.

I have just run your code and it seemed working fine for me. Which versions are you using? Which python are you running on?

On 7 Jul 2017, at 20:16, brendenpetersen<EMAIL_ADDRESS>wrote:
Thank you for the response! Yes, they're all on the same node. I'm just trying this out on a single laptop with 4 cores. (I'm also considering distributed, though it would be with the multiprocessing package.) I'm not running out of memory either--good question. The memory requirements are very small.
It does seem to be a GPy implementation issue. I've tried toy examples with other objects and it works just fine. I know multiprocessing has issues with unpickleable data (i.e. the need to wrap the optimize() call). It could also be an issue with multiple instances of multiprocessing? I looked into what's happening under the hood when Laplace() is called, but it goes pretty deep with some other packages. I also tried replacing Laplace() with expectation_propagation.EP(), which resulted in some NotImplemented error about object 'super' not having getstate.
I look forward to help from @opkoisti https://github.com/opkoisti and/or @alansaul https://github.com/alansaul. Thanks again.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/SheffieldML/GPy/issues/521#issuecomment-313769587, or mute the thread https://github.com/notifications/unsubscribe-auth/ABfcKcRU6Fo8mdyAxAu5FM4a0nv0Ct9Aks5sLoQHgaJpZM4OQIwA.

Python 2.7.9
GPy 1.7.7
I will try a more recent version of Python2 and report back.
On Sat, Jul 8, 2017 at 11:30 PM Max Zwiessele<EMAIL_ADDRESS>wrote:

I have just run your code and it seemed working fine for me. Which
versions are you using? Which python are you running on?

On 7 Jul 2017, at 20:16, brendenpetersen<EMAIL_ADDRESS>wrote:
Thank you for the response! Yes, they're all on the same node. I'm just
trying this out on a single laptop with 4 cores. (I'm also considering
distributed, though it would be with the multiprocessing package.) I'm not
running out of memory either--good question. The memory requirements are
very small.
It does seem to be a GPy implementation issue. I've tried toy examples
with other objects and it works just fine. I know multiprocessing has
issues with unpickleable data (i.e. the need to wrap the optimize() call).
It could also be an issue with multiple instances of multiprocessing? I
looked into what's happening under the hood when Laplace() is called, but
it goes pretty deep with some other packages. I also tried replacing
Laplace() with expectation_propagation.EP(), which resulted in some
NotImplemented error about object 'super' not having getstate.
I look forward to help from @opkoisti https://github.com/opkoisti
and/or @alansaul https://github.com/alansaul. Thanks again.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub <
https://github.com/SheffieldML/GPy/issues/521#issuecomment-313769587>, or
mute the thread <
https://github.com/notifications/unsubscribe-auth/ABfcKcRU6Fo8mdyAxAu5FM4a0nv0Ct9Aks5sLoQHgaJpZM4OQIwA
.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/SheffieldML/GPy/issues/521#issuecomment-313902113,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOkC3PF6LBYndMwJGVPZtU6U5WA9I2X4ks5sMHN3gaJpZM4OQIwA
.

I am running on Python 2.7.9 too...

@mzwiessele Can you try increasing size and see if it still works?
Could you check your versions of GPy and numpy (I'm on 1.13.1)?

Hi, this is probably problem with Jupyter Notebook.
You need to create an external python file (e.g. my_func.py) and put the opt_wrapper in there.
Then import the opt_wrapper and call your code normally.
from GPy.core.gp import GP
from GPy.kern import White
from GPy.likelihoods.gaussian import Gaussian
from GPy.inference.latent_function_inference.laplace import Laplace
from multiprocessing import Pool
import numpy as np
from myfunc import opt_wrapper

size = 100 # Program works when this is low enough
inference_method = Laplace() # Program works when this is None
models = [GP(X=np.arange(size).reshape(size,1), Y=np.arange(size).reshape(size,1), kernel=White(1), likelihood=Gaussian(), inference_method=inference_method) for _ in range(1)]

print "Starting pool..."
pool = Pool(1)
print pool.map(opt_wrapper, models)
pool.close()
pool.join()

If you are calling your script from commanline use if __name__ == '__main__' block to get multiprocessing working. This way you don't need external file for the function.
from GPy.core.gp import GP
from GPy.kern import White
from GPy.likelihoods.gaussian import Gaussian
from GPy.inference.latent_function_inference.laplace import Laplace
from multiprocessing import Pool
import numpy as np

def opt_wrapper(gp):
    return gp.optimize()

if __name__ == '__main__':
    size = 100 # Program works when this is low enough
    inference_method = Laplace() # Program works when this is None
    models = [GP(X=np.arange(size).reshape(size,1), Y=np.arange(size).reshape(size,1), kernel=White(1), likelihood=Gaussian(), inference_method=inference_method) for _ in range(1)]

    print "Starting pool..."
    pool = Pool(1)
    print pool.map(opt_wrapper, models)
    pool.close()
    pool.join()

Consider adding your pool.close and pool.join in Try-Finally block.
try:
    print "Starting pool..."
    pool = Pool(1)
    print pool.map(opt_wrapper, models)
finally:
    pool.close()
    pool.join()

Hi @ahartikainen, I was not using Jupyter Notebook. Python was executed from command-line. So I don't think those changes would fix the problem.
I've moved on from this project, but the issue was actually a limitation with Python's multiprocessing, which uses OS pipes under the hood and is therefore limited by buffer sizes. This explains why the program works when size is small enough, as it puts it under the buffer size, and why it worked for @mzwiessele, whose OS likely had a different buffer size.
One (of many) explanations here: https://sopython.com/canon/82/programs-using-multiprocessing-hang-deadlock-and-never-complete/

Thanks for the follow up. Interesting problem. What was your OS if I can ask.

@ahartikainen no problem! It's a bummer, since GPs can get quite data-intensive. I imagine it can be fixed by chunking up the size of the GP, but that was more work than I could afford at the time.
I'm on Mac OS X El Capitan, which I believe can handle up to 64 kB buffers, but I have no idea how they break that up.

I had a similar problem while doing benchmarking on AMD and Intel CPUs. AMD was too bad with GPy multiprocessing but Intel did well. Does anyone have a similar experience?

I have same issue with my Mac. When I try to run GPy parallel. on MacOS.


fix markdown on !pekopasta texts

since asterisks are italic signs on Discord
there's just a need to replace * to \* onto replies.pekopasta.sfw

HelIo
I would like to work on this. Before making any kind of change, I just want to confirm the place where the change is to be made which is https://github.com/bemxio/discord-pekofy-bot/blob/main/replies/copypasta.py#L20-#L37 right?


Switch MongoDB from local to MongoDB Atlas

Currently the database is running locally in each system, requiring additional setup.
Switching the database to cloud (MongoDB Atlas) ensures easier access and be able to host the platform later.

Yes, we will try to solve that.


Trying to use Dadaptation

Trying to use the new Dadaptation optimizer but it fails here.  The only values availaable for lr_scheduler.optimizers[0].param_groups[0]  when using --optimizer_type "DAdaptation" are:

lr
betas
eps
weight_decay
initial_lr
params

Where does 'd' come from?
Traceback (most recent call last):
  File "/root/data/fine_tune.py", line 353, in <module>
    train(args)
  File "/root/data/fine_tune.py", line 297, in train
    logs["lr/d*lr"] = lr_scheduler.optimizers[0].param_groups[0]['d']*lr_scheduler.optimizers[0].param_groups[0]['lr']
KeyError: 'd'

Ah, sill had --use_8bit_adam stuck in there.  Should probably assert that this flag is not used in conjunction with Dadaptation

'd' is a D in "Dadaptaion".
This 'd' means a distance-like value toward optimal minimum. more details in paper.
Unlike other optimizer, dadaptation use d*lr value as a learning rate.


feat: Add Deja Dup (backup utility) to utilities group

This closes #774.

Merged as https://github.com/ublue-os/bazzite/commit/61b9ec51aaf794ed01fc83a6051d158b94fe4b5a


Page "Warehouse Shipment": Lines editability

Please, call ActivateControls procedure also in OnAfterGetCurrRecord Trigger.
Currently it is not possible to dynamically change the editability of lines part based on the value of a header field.
We have used OnAfterShipmentLinesEditable event of the "Warehouse Shipment Header" table but this not works because the procedure ActivateControls  of the page is called only in the OnOpenPage trigger
Thanks

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: 13730.


[Maint] remove spurious print in layer_manipulator

This print leaked in via https://github.com/napari-threedee/napari-threedee/pull/168/files#diff-d6de47217f7a11c7748d0951b735668702bfd7d6ed7f4f6ad764b0074de8bc07
It's not informative nor really needed, so I suggest deleting.

Yep!


removes all text?

Hi! I was recently using this and used it as intended, and my input was named "input.md" and it returned nothing, and cleared the entire file?

thanks

Hi! Can you please share the file/content so that I can reproduce?


Unable to install any lib

I am trying to install any lib using vcpkg but it fails.
for example: installing zlib giving this error
Error: Building package zlib:x86-windows failed with: BUILD_FAILED
Please ensure you're using the latest portfiles with .\vcpkg update, then
submit an issue at https://github.com/Microsoft/vcpkg/issues including:
Package: zlib:x86-windows
Vcpkg version: 0.0.99-2017-12-01-34d8c77d35089484f66d80299dc6f8303a994a84
on  running \vcpkg update getting this
C:\Users\Desktop\vcpkg>.\vcpkg update
Using local portfile versions. To update the local portfiles, use git pull.
No packages need updating..

Thanks for posting this issue! Could you attach the log files listed slightly above that error message?

CMake Error at CMakeLists.txt:4 (project):
Failed to run MSBuild command:
C:/Program Files/Microsoft Visual Studio/2017/Community/MSBuild/15.0/Bin/MSBuild.exe

to get the value of VCTargetsPath:
Microsoft (R) Build Engine version 15.3.409.57025 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

Build started 12/7/2017 11:28:47 PM.
Project "C:\Users\gaur\Desktop\microsoft-vcpkg\vcpkg\buildtrees\zlib\x86-windows-rel\CMakeFiles\3.10.0\VCTargetsPath.vcxproj" on node 1 (default targets).
PrepareForBuild:
  Creating directory "Debug\".
  Creating directory "Debug\VCTargetsPath.tlog\".
InitializeBuildStatus:
  Creating "Debug\VCTargetsPath.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
FinalizeBuildStatus:
  Deleting file "Debug\VCTargetsPath.tlog\unsuccessfulbuild".
  Touching "Debug\VCTargetsPath.tlog\VCTargetsPath.lastbuildstate".
Done Building Project "C:\Users\gaur\Desktop\microsoft-vcpkg\vcpkg\buildtrees\zlib\x86-windows-rel\CMakeFiles\3.10.0\VCTargetsPath.vcxproj" (default targets).

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:03.98

this is the error log

Thanks for posting this issue. Please open a new issue if this is still a problem for you.


Is it possible to save data to cache after subscribe?

For example:

first open the app and subscribe the data
fetch the data using MeteorListView minimongo
in this case, when turning off ddp connection we can still get the data because data saved to minimongo
close the app completely (not only to background)
re open the apps, and the data still persist without ddp connection or resubscribe (is this possible)?

How do you think @Mokto

Yeah, why not !
I have to think about this but it is a good idea ;)
Thanks.

I'm doing something similar right now, well I was until my solution ended up not working, where I would write my cache (in that instance a redux store) to AsyncStorage whenever the app was put into the background (using AppState) or periodically.
The problem with AppState is that we can't listen to the close event (not just background) - it happens too fast and we can't fetch, stringify, and write data to AsyncStorage fast enough. Doing so periodically led to data being out of date and performance pains.
Would love to see other thoughts on this.

We could observe minimongo collections and save items whenever the collection changes.

@mokto dunno that your solution will be run well, because what @spencercarli said is true. I have tested myself when using ddp-client npm package and minimongo, the data just too fast to be saved to async or store. I hope that you have better solution than me

When you say the data is too fast to be saved to async, your mean that too much data is inserted into minimongo when the subscription is started, right ?
If it is the case, I have a ddp.on('ready') event available ;)

Dunno if I can help you to make it, for now I am only available for testing your plugin. Too many job on progress. I hope your minimongo able to do this ASAP. :D

any update to this @Mokto ?

No and it will be added soon, I have other things to do, sorry.
I'm currently trying react-native-meteor on a production app to stabilize it.

I've tested my self that almost all functionality work perfectly in my production apps, and what's missing is only this for now.
So I hope this is also part of priority task although will not be added soon. Thanks @Mokto

I'm less interested in getting this solved soon but am interested in furthering the discussion.
@spencercarli  what are the performance pains you mentioned above? Also referring to mokto's point above:

We could observe minimongo collections and save items whenever the collection changes.

Do you have any reason to think this wouldn't work sufficiently well performance-wise if you were using minimongo for caching data? Instead of saving periodic full snapshots of the collections we update asyncstorage whenever on the added, updated, removed collection events.
Alternatively I see realm is advertising faster speed than AsyncStorage for their react-native DB.
I saw https://github.com/rt2zz/redux-persist which might be more relevant to your use case... had you seen or tried that?
Ground DB is the biggest effort to make standard meteor and minimongo work offline. It uses localstorage so it probably could be adapted to AsyncStorage.
More generally properly invalidating, re-hydrating, and re-syncing anything other than immutable data  seems like a really difficult challenge. I'm not even sure of the common approaches or pitfalls for doing those thing in mobile or web.
Anyone know more on that or some things to look at / read up on?


Failed to connect to server (Windows)

I did the procedure correctly, but when I try to connect with the localhost ip it fails to connect.
(open router port to also connect my friend but without success)

Your target params should be -server=dcweb.pages.dev:80.

Ok, right, after I managed to create the character, this error occurs.


Pdfs in images from Wikimedia

Description

We have some pdf file data in the image table. It passes the dead link validation, but shows up as "Image Not Available" in the frontend.
If cleaning up the catalog is quick, we should do that. Then these items will be fixed after the image refresh.
I'm not sure if we want to also add a hot-fix in the API layer to prevent files with .pdf in the URL from showing up in the results. If we added a file type check to the dead link filter, we would stop sending unnecessary requests to the source. The status is currently "200" because the file is available, even though it doesn't show up in the UI.
Reproduction

See: https://api.openverse.engineering/v1/images/dffcffbe-3980-4cc7-bc61-804851cb9121/
and https://openverse.org/image/dffcffbe-3980-4cc7-bc61-804851cb9121
Additional context

@cc @krysal, I think you have done a similar cleanup in the catalog, haven't you?

@krysal, I think you have done a similar cleanup in the catalog, haven't you?

Yes, to remove audio files from the image table. But this is an interesting case, the link actually does return an image. It opens a debate: should we consider accepting PDF images or just remove them? Can Wikimedia provide a different format more traditionally accepted as an image?

Thanks for surfacing this Olga - I think if we can get an alternative from Wikimedia we should use that instead (the frontend shows that as "png", interestingly). But otherwise, we should probably remove PDFs from the image media set, I don't think most folks would consider PDFs as images 😅

Thanks for surfacing this Olga - I think if we can get an alternative from Wikimedia we should use that instead (the frontend shows that as "png", interestingly). But otherwise, we should probably remove PDFs from the image media set, I don't think most folks would consider PDFs as images 😅

I know why the frontend "considers" it PNG: we load the "Image not available" placeholder, and replace the image type with the placeholder type 😀 I'll open an issue for that.
@krysal , I didn't notice that it was considered an "image" in Wikimedia, that's interesting! We might need to add a check to the Wikimedia provider script if the pdf images are marked as images in the API responses, and we decide to not accept such pdfs as "image"s. I agree with @AetherUnbound that the users wouldn't expect an image to be a pdf.


robot.reset_new_pos() causes segmentation fault

I'm currently trying to use ROS to move the robot position via reset_new_pos(). To do this, I publish a PoseStamped message, which is received by examples/ros/gibson-ros/turtlebot_rgbd.py. turtlebot_rgbd.py then calls env.robot.reset_new_pos(data.pose.position, data.pose.orientation). An example message is the following:
header:
  seq: 1
  stamp:
    secs:<PHONE_NUMBER>
    nsecs: 781210899
  frame_id: "map"
pose:
  position:
    x: -10.4499994278
    y: 5.69999990463
    z: 0.5
  orientation:
    x: 0.0
    y: 0.0
    z: 0.0
    w: 1.0

However, when this is done, I get the following message:
[INFO] [1526417273.781602]: Teleporting robot
Fatal Python error: (pygame parachute) Segmentation Fault
[ INFO] [1526417273.983097364]: Got new plan
[turtlebot_gibson_sim-3] process has died [pid 4376, exit code -6, cmd ~/catkin_ws/src/gibson-ros/turtlebot_rgbd.py __name:=turtlebot_gibson_sim __log:=~/.ros/log/13cc672c-5881-11e8-80f1-305a3a540f92/turtlebot_gibson_sim-3.log].
log file: ~/.ros/log/13cc672c-5881-11e8-80f1-305a3a540f92/turtlebot_gibson_sim-3*.log
[ WARN] [1526417274.247098312]: Costmap2DROS transform timeout. Current time:<PHONE_NUMBER>.2471, global_pose stamp:<PHONE_NUMBER>.7352, tolerance: 0.5000
[ WARN] [1526417274.247133164]: Could not get robot pose, cancelling reconfiguration
[ERROR] [1526417274.489536942]: Could not get robot pose

I think you need to parse ros message and then assemble them into an array like [data.pose.position.x, data.pose.position.y, data.pose.position.z]


Stop allowing property access via getitem/setitem

Backwards incompatible change

Er, should this still be open?

right now with this library, one can do both of these:
In [12]: geojson.loads('{"type": "Feature","geometry": {"type": "Point","coordinates": [125.6, 10.1]},"properties":
    ...: {"name": "Dinagat Islands"}}').type
Out[12]: 'Feature'

In [13]: geojson.loads('{"type": "Feature","geometry": {"type": "Point","coordinates": [125.6, 10.1]},"properties":
    ...: {"name": "Dinagat Islands"}}')['type']
Out[13]: 'Feature'

when i first opened the issue, i had interest in wanting to disallow indexing the with ['type'] for the geojson.feature.Feature object (removing the __getitem__ implementation). i don't feel strongly it should be removed, i'm okay keeping it if people want it. what do you think?

I think it's fine to leave it; I'm in favor of more rather than less ways to access things.

sounds good to me 👍


Enrichment

Use $TID from a new wrapped install script.
See https://github.com/firezone/steve
Fixes firezone/marketing#128

Pull Request Test Coverage Report for Build 0a9c21b36b2fe9566c7e5e01269f9340b4923902-PR-1220

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 66.13%

Totals

Change from base Build ed6784d5986b681d9262f0b5241513c12dfd3b9b:
0.0%

Covered Lines:
1191

Relevant Lines:
1801

💛  - Coveralls

No real way to test this without going live so, merging.

Note: this will require 0.6.13 to be deployed for the new server script changes.


Seeking Assistance with Connecting GYML8511 Sensor to Arduino

Hello everyone,
I hope you're doing well. I'm currently working on connecting a sensor to my Arduino project and could use some guidance.
I've been following the connection instructions from this page   link ], but I'm encountering some uncertainty. Given the unique nature of my sensor (GYML8511), I'm unsure if the connection might differ from the instructions provided.
Could someone kindly assist me with understanding how to properly connect the GYML8511 sensor to Arduino?
Any help or insights would be greatly appreciated. Thank you very much in advance for your assistance.
Best regards,

Thanks for your question
I will look into it later this week. Have some other tasks with prio.
Did you read the datasheet?
Do you have a link to the datasheet.?

I'm currently working on connecting a sensor to my Arduino project and could use some guidance.

OK, please note that the Arduino Forum is a better place to discuss project guidance.
This is a place to discuss the working (or not) of this library.

Could someone kindly assist me with understanding how to properly connect the GYML8511 sensor to Arduino?

Looked at the link of the sensor - https://learn.sparkfun.com/tutorials/ml8511-uv-sensor-hookup-guide/all - you provided and I have little to add to their code. OK, they do not use my library however it should work,
See also - http://wiki.sunfounder.cc/index.php?title=GYML8511_UV_Sensor - roughly identical

... but I'm encountering some uncertainty.

So what is the problem?
PLease note that UV sensors do not work in house (unless you use a UV emitting light)

@Olvera-Orozco-Miguel-Angel
As there is no answer for a week I assume the problem is solved, so I close the issue.


Export issues (in Haiti); lots of points/lines exported, can't limit to one type of feature?

Some observations from doing some exports for Haiti (for inclusion in the HDX). These probably apply more broadly. As we'd like to automate these (see #126), we'll need to find a way around both of these. See the test export at http://export.hotosm.org/en/exports/150ae762-eb56-4071-9af6-97281e25c70c/. The preset file only request the following:
<item name="Building" type="closedway,relation">
    <combo key="building" values="yes,commercial,residential"/>
    <text key="name"/>
</item>

This yields shapefiles with the following:

A great deal of points and lines? They're nodes like this church, which do have a name attribute. These points/lines never seem to get discarded. However, the final export doesn't actually contain the name attribute, just geometry and the osm_id.
In the polygon shapefile, also all features that aren't buildings but do have a name. Meaning you need to export, then move to QGis for an additional filter step (building IN ('yes','commercial','residential')).

The first one seems like a bug to me.
Not sure what to do about the second one, any ideas? As far as I can tell, it's related to using JOSM preset files as a querying mechanism, where you can't really be very specific, for example by querying "I want these features (buildings), and for those features give me the following list of attributes: x, y and z"? Or is there another way to do that currently?

Since presets aren't intended as filters, we'd either need to assume "AND" or "OR" for each key included in an item. At the moment, the choice appears to be "OR". Changing that to "AND" could produce a different set of problems.
osmfilter is used under the hood, so we could store a separate filter string (initially derived from the presets) that could be modified to do more advanced filtering that can't be described just by using presets.

Should a different filtering method be applied in general? Say a combination of the tree tag and a key=value combo that could be entered into a text box that pops up when a feature is selected from the tree.
And the preset capability removed from the Export Tool entirely? I know we were discussing this in Uganda Paul, but you've done a lot more investigating since then and might now see benefits to the preset.

Honestly, I think the total number of people able to craft preset files is (and that have taken the trouble to do so) is very low? I get asked to make even the simplest ones for people all the time.. That's just a hunch though; can you see how many people have actually used/uploaded custom preset files?
IMHO, it would perhaps make more sense to standardize around feature selection in the same way as http://overpass-turbo.eu/ does, which is in JOSM nowadays (File->Download from Overpass).
Possibly in the same way as JOSM, which brings what Overpass Turbo calls the "wizard" more to the forefront. Meaning you can use 'simplified' queries (key=value1 AND key=value2), which get translated to a full overpass query, while maintaining the flexibility to adjust either one. We could then call the full Overpass query "advanced mode" and hide it from view entirely, initially?

That would ease querying considerably, but still not take care of filtering. Not in favor of changing to AND, excluding all buildings without a name in the example used here :). Perhaps have a separate input for filtering as an advanced option (expose in UI as well?).
Separate question: why does the export linked here seem to have disappeared? Is this #125 in action?

For a revision of the presets functionality I would suggest a YAML or JSON based format. They should be 1:1 translatable with each other. Perhaps a subset of the mapping format Imposm3 uses?
https://github.com/omniscale/imposm3/blob/master/example-mapping.yml
Extra win if it could be done in a way that we can remove the tags table from the database and instead store the preset/mapping in a blob column

https://github.com/hotosm/osm-export-tool2/blob/master/hdx/example_preset.yml
proposal for YAML preset format. Notes:

it handles both AND and OR via a sql fragment
Thematic Shapefiles: one top level item per theme (one theme has 1-3 shapefiles for points,lines,polygons)
OSM Shapefiles: 3 shapfiles: planet_osm_point, planet_osm_line, planet_osm_polygon files, each with the union of all attributes specified in preset

for OSM PBFs, PBFs will either be all data or filtered to union of above attributes
Not sure about how this preset maps to Garmin IMG, KML, OsmAnd OBF formats.

So for the filtering, I think YAML is definitely much better suited to advanced filtering than the JOSM presets are. Proposal above looks good.
However, it is still mostly suited to advanced users. In addition to that, would it be an option to add a similar syntax to that used in JOSM (Windows->Filter) or the 'wizard' in Overpass Turbo (http://overpass-turbo.eu/)? Or perhaps something even simpler..
Examples of that filtering syntax:

All highways: highway=*
Waterways: waterway=* | natural=wetland | natural=water | natural=coastline | landuse=basin | landuse=reservoir
Buildings that don't have levels or materials: building=* & ( -building\:levels | -building\:material )
Banks and mobile money agents: amenity=bank or amenity=mobile_money_agent

@PaulUithol Would the existing "Tree tag" GUI in the current Export Tool be sufficient for non-advanced users?
That could become a GUI that creates YAML behind the scenes.
I think adding another JOSM/Overpass wizard syntax is doable, but then we would have

"tree tag" gui (simple use cases)
JOSM/Overpass wizard syntax (medium-advanced users?)
YAML (advanced users)

The YAML could also be made more friendly by selecting a pre-existing YAML (just like stored presets) and modifying it in a text area

What would be nice is if the tree tag GUI could switch between the GUI, and the ability to modify the generated YAML? I think we're planning on improving the tree tag chooser anyway, isn't it? To get rid of most of the duplication that is there at the moment between "OSM" and "HDM".

#158

#158

We've implemented the YAML format and tag tree checkboxes. @PaulUithol this is unidirectional from checkboxes -> YAML, as in you can use the checkboxes to create YAML and then tweak it, but changing the checkbox selections will erase your tweaks. It seemed too complex to make it bidirectional.


fix water temp display units

This PR will fix the units in stations environment table.

Deployed on dev


Adds more feature to sentry

Added Replay and Browser Tracing
idk what we have set up in Posthog, in this is the case please close the pr)

@jog1t poke


Feat/dev server improve error handling

This improves some rough edges around error handling across the board.

when es-module-lexer fails to parse a file it doesn't give a great error message. with this change, we hand off the file to babel parser to generator a developer-friendly error message.

when there is an error compiling (node resolve, compatibility mode, etc.) one of the modules that are early in the request chain it might happen before the message channel endpoint is connected. right now we don't display this error message, with this change we cache error message and send them if they occur within 1 sec

right now we only log compile errors to the browser and not the terminal because if you're reloading frequently it's hard to know whether an error has been solved or not since your terminal is just full of errors. with this change, we log reloads of your index.html so that errors are visually grouped per reload so that we can turn logging to the terminal back on

if there is an error in a module the first time it is served, it is not watched for reloads because we only check for 2xx response codes. now we watch any status code and use generic util for finding the served file path

when you have multiple browser tabs open (for example due to restarting the server a few times) and have watch mode turned on, all the browser tabs will reload on file changes spamming the terminal and slowing down the server. we now only allow one message channel to be active at any given time, which improves the developer experience

because we only allow one message channel, we have to change the way karma-esm listens for file changes. we now pass a custom fileWatcher from karma-esm to the dev server and listen to that directly. this is a change I was intending to do anyway because it's more robust than relying on a network call

I like that summary 🤗
should we put the content of this MRs message in a DesignDecisions.md file or so?

Well maybe not the literal text, since it speaks about before and after. But yeah we could add some docs about some design decisions.

sure... error message should be more helpful imho
edit it with this
    if (error.code === 'MODULE_NOT_FOUND') {
      const relativeImportFilePath = path.relative(rootDir, sourceFilePath);
      const resolvedImportPath = toBrowserPath(relativeImportFilePath);
      const realtivePathToErrorFile = resolvedImportPath.startsWith('.')
        ? resolvedImportPath
        : `./${resolvedImportPath}`;
      throw new Error(
        `Could not resolve "import ... from '${importPath}';" in "${realtivePathToErrorFile}"`,
      );
    }

which imho improves the message
Error compiling: Could not resolve "import { ... } from 'asdf';" in "./demo/node-resolve/module-a.js"
// vs old
Error compiling: Could not find module "asdf".

maybe for another time... having two "failed" imports after each other only results in one error
import 'foo/bar.js'; // will only show error for this
import 'asdf'; // but not for this

We used to add the filename of where the import could not be resolved in the middleware, but it looks like that was removed at some point. Definitely need to add that back in.
I'm not sure import { ... } from 'asdf' is the right way, because you might be doing a default import, an import for side effects or it could even be a dynamic import.
Stopping at the first error is pretty common, one error might be the cause of other errors and you end up printing a lot of messages otherwise. I'm not sure if it really improves the DX.

Stopping at the first error is pretty common

sounds good 👍
When I got the error I really didn't understand what that means Error compiling: Could not find module "asdf".

Why does it search for this module?
Where does it search for this module?
Is it an es module or module in a sense of a package?

the best would be
Error in "await import('asdf');" in "./path/to/file.js:300:20"
                                    ^ Could not resolve es module "asdf"

but yeah that is probably tougher to do so, for now, I think we could go with
Error compiling: Could not resolve "import { ... } from 'asdf';" in "./demo/node-resolve/module-a.js"

Why does it search for this module? => because it is an import be it static named, static default or a dynamic on (we don't know it here) => so let's show the "most common one" with ... and the user will need to find out for himself
Where does it search for this module? => path to file is there (actual line + character is sadly not that easy to get)
Is it an es module or module in a sense of a package? => clear due to import that it is actually a bare module of a package

Yeah, you're right. It'd be great if we can use babel's code frame error messages here. I'll see what we can do.


Remove extra call to deallocate input to move op and throw if input cannot be reallocated

lgtm.
I am wondering for the no move case, if it's better to assert out or not. Since the user has no control/access to see if they can move a tensor without calling the move op itself, or if the move op is part of a repeated layer, then it might become clunky for the user to try and only have some of the layers have move and the others not to.

Yeah good point, I will add a warning print that the move is a noop for this case and just return.
I can’t check this PR in yet because bloom for question answering looks like it’s failing due to oom now that the bug of double deallocating is resolved

@arakhmati could you help take a look at test_performance_of_bloom_for_question_answering which is failing. It seems like whatever tensor is being moved can actually be deallocated since the test passes with the double deallocate

Sounds good. I will take a look


Accept keyword list as a tags

This should provide better user experience and ergonomic to the library.
I will provide PR later.

Closing in favour of the PR.


refactor: Replace usages of Popover from react-bootstrap with Antd

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

ADDITIONAL INFORMATION

[ ] Has associated issue:
[x] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@a071393). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #11163   +/-   ##
=========================================
  Coverage          ?   61.05%
=========================================
  Files             ?      391
  Lines             ?    24491
  Branches          ?        0
=========================================
  Hits              ?    14954
  Misses            ?     9537
  Partials          ?        0

Flag
Coverage Δ

#python
61.05% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a071393...eb421f1. Read the comment docs.

@suddjian Could you please take a look?

E2E tests still failing here :(

@mistercrunch I fixed the tests, they should pass now. However, this PR might have some changes conflicting with the Tabs PRs, so depending which feature we merge first, the tests in the other might start failing again. I'll keep an eye on that!

@mistercrunch Python / pre-commit has failed several times in a row, though no changes were done to Python codebase. Is it possible that a faulty code was merged to master? Can we get a fix for that?
superset/config.py:959: error: Incompatible import of "CeleryConfig" (imported name has type "Type[superset_config.CeleryConfig]", local name has type "Type[superset.config.CeleryConfig]")
superset/config.py:959: error: Incompatible import of "CELERY_CONFIG" (imported name has type "Type[superset_config.CeleryConfig]", local name has type "Type[superset.config.CeleryConfig]")
Found 2 errors in 1 file (checked 529 source files)


Safari fixes

Script dashboard error SyntaxError: Unexpected use of reserved word 'let' in strict mode
Fixed in https://github.com/Clever/grafana-service/commit/062b1b06889dd7cd61f9c9fb45d34a3768f92376.
Script dashboard error SyntaxError: Strict mode does not allow function declarations in a lexically nested statement.
Fixed in https://github.com/Clever/grafana-service/commit/a3226e7a7067fa5b9b0d6095a17c51b28b827ac9. Do we want to disable strict mode though? We could also check what the browser agent is, and only skip use-strict for Safari.

Tested in recent versions of Chrome, Safari, Firefox locally.

lgtm

Hm. I would prefer to keep use strict enabled, but I've tried the other following workarounds that were unsuccessful:

replace function doSomething(){} with var doSomething = function(){} even though is this not what our style guide recommends. The Strict mode does not allow function declarations in a lexically nested statement error message persisted.

check browser user agent and only use strict mode for non-Safari browsers. It seems navigator is undefined when the dashboard is being created, and navigator.userAgent is the best method I've found of browser detection so far.

I informally surveyed our eng team on Slack, and roughy ~25/40 people responded: 22 Chrome, 2 Safari, 1 Firefox (people could have chosen > 1).

I see a lgtm. Can we push this ❄

I don't like the idea of removing use strict (but perhaps this isn't as bad as it sounds), and can't figure out how to only disable this for Safari. 😢
Merging for now, will see if this is fixable later (the dashboard will error/crash anyway, so we'll know if something is wrong).


FailureReason: Unrecoverable/Unexpected BadResponse, Newtonsoft.Json.JsonReaderException

NEST/Elasticsearch.Net version: 5.6.5
Elasticsearch version: 5.6.9
Description of the problem including expected versus actual behavior:
Getting sporadic error while using delete_by_query to delete documents
The query always try to delete few documents, not more than 50
FailureReason: Unrecoverable/Unexpected BadResponse while attempting POST on https:******/_delete_by_query?routing=&refresh=true

[1] BadResponse: Node: https://*********/elasticsearch/ Exception: JsonReaderException Took: 00:00:00.0312570

Audit exception in step 1 BadResponse:
Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
at Newtonsoft.Json.JsonTextReader.ParseValue()
at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
at Nest.JsonNetSerializer.Deserialize[T](Stream stream)
at Elasticsearch.Net.ResponseBuilder1.SetBody(ElasticsearchResponse1 response, Stream stream)
at Elasticsearch.Net.ResponseBuilder1.ToResponse()    at Elasticsearch.Net.HttpWebRequestConnection.Request[TReturn](RequestData requestData)    at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TReturn](RequestData requestData)    at Elasticsearch.Net.Transport1.Request[TReturn](HttpMethod method, String path, PostData`1 data, IRequestParameters requestParameters)
Inner Exception: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
at Newtonsoft.Json.JsonTextReader.ParseValue()
at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
at Nest.JsonNetSerializer.Deserialize[T](Stream stream)
at Elasticsearch.Net.ResponseBuilder1.SetBody(ElasticsearchResponse1 response, Stream stream)
at Elasticsearch.Net.ResponseBuilder1.ToResponse()    at Elasticsearch.Net.HttpWebRequestConnection.Request[TReturn](RequestData requestData)    at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TReturn](RequestData requestData)    at Elasticsearch.Net.Transport1.Request[TReturn](HttpMethod method, String path, PostData`1 data, IRequestParameters requestParameters)
Exception:
Elasticsearch.Net.UnexpectedElasticsearchClientException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0. ---> Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
at Newtonsoft.Json.JsonTextReader.ParseValue()
at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
at Nest.JsonNetSerializer.Deserialize[T](Stream stream)
at Elasticsearch.Net.ResponseBuilder1.SetBody(ElasticsearchResponse1 response, Stream stream)
at Elasticsearch.Net.ResponseBuilder1.ToResponse()    at Elasticsearch.Net.HttpWebRequestConnection.Request[TReturn](RequestData requestData)    at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TReturn](RequestData requestData)    at Elasticsearch.Net.Transport1.Request[TReturn](HttpMethod method, String path, PostData1 data, IRequestParameters requestParameters)    --- End of inner exception stack trace ---    at Elasticsearch.Net.Transport1.Request[TReturn](HttpMethod method, String path, PostData1 data, IRequestParameters requestParameters)    at Nest.LowLevelDispatch.DeleteByQueryDispatch[T](IRequest1 p, PostData1 body)    at Nest.ElasticClient.Nest.IHighLevelToLowLevelDispatcher.Dispatch[TRequest,TQueryString,TResponse](TRequest request, Func3 responseGenerator, Func3 dispatch)    at syngo.Cloud.RD.Data.Access.Storage.CategoryRepository1.Delete(Int64 tenantId, Int64 id) in C:.cs:line 153
at syngo.Cloud.RD.Service.Controllers.FleetController.<>d__32.MoveNext() in C:\n\124\s******.cs:line 271
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Threading.Tasks.TaskHelpersExtensions.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Tracing.ITraceWriterExtensions.d__171.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__1.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at System.Web.Http.Tracing.ITraceWriterExtensions.<TraceBeginEndAsyncCore>d__171.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Filters.ActionFilterAttribute.d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Web.Http.Filters.ActionFilterAttribute.d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Filters.ActionFilterAttribute.d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.ActionFilterResult.d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Filters.AuthorizationFilterAttribute.d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Tracing.Tracers.HttpControllerTracer.d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Tracing.ITraceWriterExtensions.d__17`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Dispatcher.HttpControllerDispatcher.d__15.MoveNext()
Steps to reproduce:

Not able to reproduce, its happening sporadically

Provide ConnectionSettings (if relevant):
Provide DebugInformation (if relevant):

Describe the feature:

Hi @amitsingh26
The error message:

Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.

This message strongly indicates that response coming back from the server is HTML, which then fails JSON parsing. It is unlikely that this is an error in either the client or the server, and more likely a proxy / network configuration issue.
Are you able to enable the debug information on the client (see: https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/troubleshooting.html) and provide further information? It would be interesting to see the response from the server that is unable to be parsed.
PS. 5.6.5 is currently end of life (see: https://www.elastic.co/support/eol) but we will do our best to help you here.

Unfortunately no, it has happened very few times and I have got these logs from the productions.
I cannot get the debug information.
One more problem we have seen, in some of the responses Nest.IResoponse.isValid is false but serverError object is null
We try to log the serverError but that is coming as null

Hi @amitsingh26,
In light of not being able to review the debug information it is going to be very difficult to determine what is going on in this situation.
I am going to close this ticket; you are welcome to open another issue in the future if you are able to provide the additional information requested.
As mentioned in my previous reply, I suspect the issue here is environmental, and suspect a proxy or some networking issue - as indicated by the parsing problem.

I've also encountered a similar situation, the problem lies in the data conversion process because the data has an integer field type but the retrieved value is null, causing an error.

@Bathe0603it Hi! Did you encounter that issue in a modern (8.x) version of the ES client?


Full Space Immersion auto fade in outside of 1.5 meter from the origin

After updating the ALVR AVP app from the AppStore it is not possible to have the Full Space immersion mode outside of 1.5 meter from the origin. Earlier the passthrough was automatically ca. 40% faded in during the movement and afterwards faded out.
Now if you go outside of the 1.5 meter origin the passthrough is 100% visible. And its only possible to enable the Full Space Immersion with a long click of the Digital Crown. I tried all possible settings. Nothing worked for me. And actually I don't even remember how I enabled it with the version before. May be it was a bug of the visionOS?
Is there any information on the topic?
Thank you one more time for this amazing software!!!

Only the testflight has the visionOS 2.0 features, which includes .mixed mode for the non-experimental renderer. So you'll need to swap over to the experimental renderer to get .mixed again

Thank you for a quick answer. I tried the experimental renderer out. The .mixed mode worked but the streaming quality is kinda bad.
<< Only the testflight has the visionOS 2.0 features, which includes .mixed mode for the non-experimental renderer.
does that mean, that you have it on the roadmap and when the visionOS 2.0 is officially out, the non-experimental renderer will have the .mixed mode?


Updated Migration from 2.2 to 3.0.

Fixes #14362 and #14364.

There's an include for the 2.2 SDK that could be used in the corresponding 2.1 to 2.2 migration guide too. I can make that change as an additional commit for consistency if desired.

@serpent5  nice work thanks!

It's no problem at all. Did you see the last comment I added after your commit? Did I misunderstand something or is it a non-issue?

@Rick-Anderson The live version of this is rendering with unwanted whitespace before the included content. Using docfx the whitespace was after, which is why I moved it. Should we drop the include approach for the SDK or is there a better approach? It's also underlined without hover.


added grid for tech stack to readme

added grid for the tech stack
did basic linting for markdown
removed extra trailing spaces
removed multiple headings of same level (1), and changed to leavel 2 heading.

Looks good to me :100:


ResourceRule named resource option

Feature
Add a name field to the ResourceRule object such that individual resources can be referenced. This allows for a more finely scoped validation to occur when we potentially only need to evaluate one specific resource.
Alternatives
To-date we can accomplish this already in Rego with the some function and identifying information.

Possible options:
return all with matching name
Implement functionality that returns all items in the search area (cluster, namespace, etc) that matches the name of the targeted resources.
Targeting a single resource of the payload in the Rego would look like:
item := input.items[0]
Return single instance
More aligned with kubectl - assumes that the name identifier means that you are targeting a single resource in a single namespace.
Targeting a single resource in rego is now more streamlined - as the payload will be a single object vs a list of many objects. (See the json output of kubectl get po -A vs kubectl get po <name> -n <namespace>


[Retrospective] Release 2.8.0

This is Retrospective for the release 2.8.0:

#3434

How to use this issue?
Please add comments to this issue, they can be small or large in scope. Honest feedback is important to improve our processes, suggestions are also welcomed but not required.
What will happen to this issue post release?
There will be a discussion(s) about how the release went and how the next release can be improved. Then this ticket will be updated with the notes of that discussion along side action items.

Request to merge [AUTO] issues that increment snapshot versions quickly so as not to block bwc CI on downstream dependencies.  These are raised by bots and are super-simple to approve/merge.
e.g. https://github.com/opensearch-project/common-utils/pull/444

Request to merge [AUTO] issues that increment snapshot versions quickly so as not to block bwc CI on downstream dependencies. These are raised by bots and are super-simple to approve/merge. e.g. opensearch-project/common-utils#444

Yes! It's more involved than that -- the manifest file needs to be updated incrementally as the version bump PRs are merged. Currently only OpenSearch snapshot are built.

Hi!
On behalf of OpenSearch SQL plugin team I share our observations of current weaknesses of the release process.

Dependency chains
Dependency chains should be defined all across the project. This is crucial to make breaking changes as painless as possible.
Some people could be surprised that SQL plugin depends on ML plugin, which depends on ml-commons, which depends on common-utils, which depends on OpenSearch core.
Unblocking dependencies
For all components which have dependents, responsible people should be defined. Those people should be first responders for unblocking dependents and applying fixes which required for that. This task should be on the highest priority always.
Notification for breaking changes
When a component team is going to do a breaking change, a notification (email, or GH tagging) should be sent to all first responders of dependent modules/components. I also propose to postpone the merge of breaking changes for one day after making a notification, this will give people time to get prepared.
Fallback strategy
As I understand from discussing troubles of the current release, if a component is not ready for the release, a previous version of it could be taken as part of the release build. Unfortunately, a version N of a component could be incompatible with version N+1 of core or other components. Keeping a fallback version ready for the release is a good strategy to avoid release delays.
Fallback maintenance
Unfortunately, an extra effort is required from all teams to keep version N of a component ready for release N+1. Some changes need to be backported to make version N compatible with N+1. That version won’t be N anymore, it will be N*, which requires additional testing.
Branch integration testing
CI workflow described in sql#1242 should be done for all project repositories. Given a PR from branch A to B has CI workflows running on branch A (let me name this run as CI_1). Once merged, branch B is updated and new CI started (CI_2), but nobody sees the results. If CI_1 fails, it blocks the merge, but failures of CI_2 don’t, and it also produces troubles for upcoming PRs much later. The goal is to run CI_2 in parallel with CI_1 as part of PR checks. It is pretty simple to do in GHA.
Flakey tests
OpenSearch release build runs tests for N components K times (there are builds for rpm, deb and so on), where every component has M thousands of tests. When 1% of tests fall with 1% probability, total chance to fail the entire build becomes pretty high. I propose to prioritize fixing flakey tests, maybe second after unblocking dependents and dependencies.

Thanks everyone's feedbacks and retro here.
I will close this issue for now.
Thanks.


Hide prayers within the prayer tab

There's a large number of prayers within the tab that one will never use after a certain level. Why can't we just remove those?
I'd love to be able to remove prayers while reordering, and the possibly resize them a tad.


Links with a  dot are not processed by turbo

We have links in our platform that are user profiles like this:
domain.com/users/some.username
Whenever you click on those links, a normal request is made instead of a turbo request.
Is there any way to fix this? (I don't want to write a stimulus "link_controller#click" that calls Turbo.visit if it can be fixed from turbo side)
Thanks!

If anyone else was reading this issue and asked themselves "are periods even allowed in the path?", apparently they are.

I have the same issue, it would be ideal if it could be solved directly from turbo.

If anyone else was reading this issue and asked themselves "are periods even allowed in the path?", apparently they are.

Yes, instagram does it all day https://www.instagram.com/ruby.tuesday/

TLDR: Turbo can handle periods, as long as the URL doesn't end in a way that suggests the URL is NOT an html page.
DETAILS:
Links with dots are processed with turbo:

domain.com/users/some.username.html => fetched with turbo
domain.com/users/some.username/ => fetched with turbo
domain.com/users/some.png => NOT fetched with turbo
domain.com/users/some.username => NOT fetched with turbo

Note in the instagram example, the URL ends in "/", indicating it's a directory
Turbo checks to see if a url is a directory or an html page
function isHTML(url) {
  return !!getExtension(url).match(/^(?:|\.(?:htm|html|xhtml))$/)
}

Turbo skips non-html links (e.g. .json, .svg, .jpg, .png, etc).
  locationIsVisitable(location: URL) {
    return ... && isHTML(location)
  }

In the example of domain.com/users/some.username, how would you propose the javascript change to determine if the url is an html page vs a file with a .username extension?

Thanks, we solved it adding trailing_slash to the route.rb


PoC: Use orjson instead of stdlib's json

Description
TODO: probably it's better to create a new extra, e.g. wandb[perf] that would improve the performance by installing and using things like orjson at the cost of adding dependencies.
Replace stdlib's json.dumps and json.loads with orjson's counterparts. This provides a significant speed up.
For example, consider the following script:

Current performance:

With orjson:

I also considered python-rapidjson -- results were also much better than the stdlib, but ~10-20% worse than orjson.
Testing
Let's see what the full test suite has to say...
Checklist

Name PR "[WB-NNNN][WB-MMMM] Add support for..." similar to entries in CHANGELOG.md
Include reference to internal ticket "Fixes WB-NNNN" (and github issue "Fixes #NNNN" if applicable)

implemented elsewhere, closing


Issues about the vina score

Hi,
I have used your vina score script "docking.py" recently. However, I have met the following error,

Traceback (most recent call last):
File "/sample_geo.py", line 519, in
g_vina_score = vina_task.run_sync()
File "/evaluation/docking.py", line 168, in run_sync
while self.get_results() is None:
File "/evaluation/docking.py", line 182, in get_results
self.results = parse_qvina_outputs(self.docked_sdf_path)
File "/evaluation/docking.py", line 24, in parse_qvina_outputs
suppl = Chem.SDMolSupplier(docked_sdf_path)
OSError: File error: Invalid input file /tmp/olzbpempbkdjotcpxfbqvwjfnxltvc_ligand_out.sdf

I found that error was caused by the missing script “prepare_receptor4.py ”.
Could you please help me solve it or provide this script?
Thank you for your attention.

Hi, to use docking.py you should install conda environment defined in adt.yml. When you installed the environment, the prepare_receptor4.py file were automatically installed. You can check whether you have successfully installed the environment by directly tapping prepare_receptor4.py and qvina2 in the terminal
.


Latest version of mod (1.17.1-1.0.0-beta) with Optifine glitch

Whenever I break a block, no block breaking particles appear, no matter what kind of settings I choose or whatever, it does not make any block breaking particles and its really weird and annoys me a lot ;-; please fix! Thank you <3

Alright, I'm gonna look into that issue. But there's probably nothing that I can do, I don't even use the particle system of MC in my mod IIRC.
I should also add that OptiFine is known to cause bugs on mods that extend the rendering system of MC.
Also, please check your OptiFine settings to see if you accidentally disabled particle effects.
- Budschie

Alright, I can definitely confirm that this happens.

Alright, this is not an issue of mine but an OptiFine issue, because if I remove my mod (without changing the loader, so that I still use Forge to start MC), there are still no block breaking particles.

In the preview release of OF, the block break particles seem to work again. So, you could try that. Or, you can just use alternatives of OptiFine, there are some pretty great options out there (although Sodium+Iris is Fabric only).


Change console colors

The way to change console colors today works as follows: The user writes the name of the color.
The suggestion would be to change to hexadecimal, so that the color palette gets bigger.

Need

RGB to CMYK
HEX to CMYK
....


Multiple conditions in Query::where() method

According to db-query-builder doc for now multiple conditions in where() are supported only for hash format.
Could it be more useful to add support of multiple conditions for mixed format, including operand?
Eg:
$query->where([
    'type' => 2, // hash format
    ['like', 'name', 'alex'], // and operand format
]);

For now the above query currently possible to write only this way:
$query->where([
    'type' => 2,
])->andWhere(['like', 'name', 'alex']);

No problem with that. But adding additional andWhere() is not possible to do for methods with conditions like delete/update where we should hardcode such query:
Yii::$app->db->createCommand()
    ->delete(
        'table',
        'type = :type AND name LIKE :name',
        [
            'type' => 2,
            'name' => 'alex',
        ]
    )->execute();

So by adding this support we could write:
Yii::$app->db->createCommand()
    ->delete('table', [
        'type' => 2, // hash format
        ['like', 'name', 'alex'], // operand format
    ])->execute();

I was sure it's supported, looks obvious for me. It could be consistent.
Maybe it was discussed before and there are some breaking downsides behind such approach, but I couldn't find any discussion regarding this proposal.
Thanks.

How to create sql like this?
select * from film_actor fa
where
actor_id = 1
AND (actor_id, film_id) not in ((1,1),(1,23),(1,25),(1,980),(1,970))

wait yiibot =)

related https://github.com/yiisoft/yii2/issues/10805


vacuum_clean_zone not working | Failed to call service dreame_vacuum/vacuum_clean_zone. 'int' object is not subscriptable

Describe the bug
Whenever I try to call service "vacuum_clean_zone" I get an error saying: "Failed to call service dreame_vacuum/vacuum_clean_zone. 'int' object is not subscriptable"
To Reproduce
Follow documentation to create clean zone service call. e.g. below.
show_name: true
show_icon: true
type: button
tap_action:
  action: call-service
  service: dreame_vacuum.vacuum_clean_zone
  target:
    entity_id: vacuum.your_vacuum_name
  data:
    zone:
      - -3551
      - -546
      - -1275
      - 2200
    repeats: 2
entity: vacuum.your_vacuum_name
name: Vacuum that zone
icon: mdi:robot-vacuum
show_state: true

Expected behavior
Self explanatory.
Screenshots
Can be provided if the error message in text is not sufficient.
Additional Information (please complete the following information)

Model Name: Dreame Z10 Pro
Firmware Version: 4.1.8_1156
Home Assistant Version:
Core 2023.11.2
Supervisor 2023.11.3
Operating System 11.1
Frontend 20231030.2
Integration: v2.0.0b8
Configuration Type: With map I quess?
Errors or warnings shown in the HA logs (if applicable): "Failed to call service dreame_vacuum/vacuum_clean_zone. 'int' object is not subscriptable"

Same service call worked perfectly with v1.0.1 but I was forced to upgrade to beta as v1.0.1 stopped working completely due some breaking changes in HA.

Ooh. v1.0.2 is released. I'll try that quickly.

@Hount thanks, I agree, we should make sure that @Tasshack is aware.

@Tasshack thanks for the quick fix


ST_Transform error with doughnuts

I am facing an error when tryng to transform geometries that include doughnouts  (holes)
One example is:
SELECT  ST_Transform( ST_GeomFromText('POLYGON ((3301892.7081 2305424.6332, 3297641.4146 2293854.3149, 3311252.1571 2288743.8019, 3322410.9401 2280361.8837, 3322085.273 2275041.252, 3318903.5687 2269530.8181, 3315052.0551 2262870.2843, 3313598.2675 2255611.9221, 3304356.0358 2255544.4336, 3299995.8432 2251719.8828, 3300298.3817 2248820.3076, 3304266.6753 2247578.2129, 3301776.1263 2238082.5652, 3283221.0582 2241578.9782, 3280799.4261 2245695.1826, 3281142.5189 2253859.4624, 3276773.4563 2256390.7988, 3270589.2771 2251898.2402, 3268897.3401 2260428.3755, 3263191.4445 2269557.4174, 3249571.2601 2278693.3564, 3249773.3834 2281355.1971, 3250227.7287 2287323.6797, 3247573.221 2288352.4905, 3242476.1795 2290332.0758, 3243878.6509 2296591.2115, 3246594.6023 2298063.0509, 3249026.1976 2297136.9709, 3259898.26 2293011.2743, 3262495.8129 2295521.9721, 3260589.2171 2300629.207, 3263318.1856 2300530.7404, 3264794.3518 2302409.892, 3262145.1912 2304764.0635, 3262200.7379 2307365.4, 3260625.93 2307497.7578, 3256755.1595 2303490.781, 3252545.7222 2307836.9957, 3253121.686 2318845.3297, 3255442.84 2321747.7637, 3260779.9173 2319719.2841, 3268846.0956 2323205.8617, 3271079.6228 2317750.6139, 3270372.2206 2315806.1767, 3268797.8507 2311480.5666, 3279698.4348 2303347.7128, 3292652.3093 2299850.1394, 3294313.4361 2305713.7012, 3301892.7081 2305424.6332), (3270969.4748 2271758.2972, 3272874.4151 2267708.8757, 3275976.4969 2266056.1655, 3276737.0802 2262242.684, 3280734.3288 2261171.7626, 3285660.5305 2261369.8463, 3286896.4781 2258685.4039, 3291253.9138 2258101.4314, 3293829.2967 2256241.5158, 3296529.1289 2256778.6072, 3296722.7488 2260852.6801, 3292037.8003 2260114.8086, 3290215.1655 2263282.8501, 3294438.177 2267457.5003, 3291842.5399 2271360.9896, 3288472.2535 2271349.9963, 3281847.6405 2275670.851, 3278803.0097 2276290.0461, 3273515.1866 2277201.8981, 3270969.4748 2271758.2972))',3035),4326)
The error is:
IllegalArgumentException: Points of LinearRing do not form a closed linestring

Hi @bilbomaticaeugis.
Thank you for the expressed interest in my project.
I will have a look and revert back to you shortly.
Please state:

Which Library version are you using?
Which Dremio version are you using?
Which JVM Runtime are you using?

Hi again
I am using a Docker image based on Dremio 23.1 running on Kubernetes ( https://github.com/dremio/dremio-cloud-tools/blob/master/charts/dremio/README.md )
Library version 0.8.8.jar
JVM :
openjdk version "1.8.0_352"
OpenJDK Runtime Environment (Temurin)(build 1.8.0_352-b08)
OpenJDK 64-Bit Server VM (Temurin)(build 25.352-b08, mixed mode)
Thanks

Given the original geom is valid, seems like a PROJ4 issue.

I am using proj4j-1.1.0
Do you suggest any version in version in particular?

Not sure what you mean by that. jts-proj4j is baked into the library via jts-proj4

In the Dockerfile I copy the following jar files into /opt/dremio/jars/3rdparty/ folder
FROM dremio/dremio-oss:23.1
COPY "esri-geometry-api-2.2.2.jar"   "proj4j-1.2.3.jar"  "proj4j-epsg-1.2.0.jar" "dremio-udf-gis-0.8.4.jar" /opt/dremio/jars/3rdparty/
I am testing with  different proj4j-1.x.j.jar files with the same result.

So, at best, you'll be getting classpath collosions. You should not be copying any additional jar aside from dremio-udf-gis, as it's what you call a fat jar (includes all dependencies within). Again, it's seems to be an issue with the PROJ4 library implementation itself. as coordinate transformation creates a non closed polygon (could also be an issue with my interface).
Please allow for a few days for me to test it out

Sure, no problem

So, good news, it's a  problem with my interface, I do not break down the rings for multi-ring polygons.
I will release a fix in the upcoming version. Great catch!

Good news indeed! Many thanks for looking into this.

Many thanks !!