Adds the ability to set a FunctionListenerFactory to be notified of function calls

For #451 (maybe fixes if this is enough functionality for now)
This allows registering a FunctionListenerFactory to a runtime. Anytime a module is instantiated, the factory is invoked while building the functions within the module. If the factory returns a listener, it is invoked anytime the defined function is called with context. The context can be used to propagate stack across function invocations in a stack.
We'll keep the hooks internal for now to let us bake within our own tests, so registration is defined using an internal interface implemented by runtime.

I would recommend squashing and trying to contain changes as much as possible to new files. since a lot is in the same package, that should be possible. That way this low priority change doesn't have so much drift when we try to focus on things important to complete in the next 2 weeks

This is great, @anuraaga!
As discussed offline, I squashed this draft into one commit then will add another to make it completely internal (ex no leak to wazero package) so we can iterate without effecting anyone.

@anuraaga if you are keen, mark ready for review

nice work @anuraaga!


Not an issue, but trying to understand the purpse of this icon, for the other icon, it does jump to others place, but this icone just refresh

Not an issue, but trying to understand the purpose of this icon, for the other icon, it does jump to others place, but this icon just refresh the page, understand its name as page. so we can't find this icon on other page other than home page, so it only shows the total pages of a site? Thank you.

Yep, it just jumps to the homepage, since there isn't a page describe about this info so far, perhaps it would be better that removing the link of that icon.

Maybe we can move it to the end of the icon list, and rename it as about, which link to a page where describes the site and give a overall view?


Remove extra slash from UrlHasPrefix

URL.Path already has a leading slash, so adding our own slash causes strings.HasPrefix to fail.

Thanks!
We need to add a test for UrlHasPrefix...


[WinUI3 preview] DEP1000: DirectoryNotFoundException for nuget\microsoft.winui\3.0.0-preview NoiseAsset.png

Error DEP1000: Cannot copy file
"C:\Users<username>.nuget\packages\microsoft.winui\3.0.0-preview3.201113.0\runtimes\win10-arm64\native\Microsoft.UI.Xaml\Assets\NoiseAsset_256x256_PNG.png"
to layout "C:\Users<username>\Source\Repos\Xaml-Controls-Gallery\XamlControlsGallery\bin\XamlControlsGallery.Desktop.Package\XamlControlsGallery.Desktop.Package\AppX\XamlControlsGallery.Desktop\runtimes\win10-arm64\native\Microsoft.UI.Xaml\Assets\NoiseAsset_256x256_PNG.png".
DirectoryNotFoundException - Could not find a part of the path
'C:\Users<username>\Source\Repos\Xaml-Controls-Gallery\XamlControlsGallery\bin\XamlControlsGallery.Desktop.Package\XamlControlsGallery.Desktop.Package\AppX\XamlControlsGallery.Desktop\runtimes\win10-arm64\native\Microsoft.UI.Xaml\Assets\NoiseAsset_256x256_PNG.png'. [0x80070003]	XamlControlsGallery.Desktop.Package
Steps to reproduce the bug
xcg: branch winui3preview
run LocalMachine: Debug/Win32 XamlControlsGallery.Desktop.Package
Visual Studio:

[ ] 2019 16.9 preview

Windows 10 version:

[ ] November 2019 Update (18363)

Device form factor:

[ ] Desktop

System Type

[ ] x64-based PC

(Is it possible to get xcg winui3preview pre-built version?)

CC: @MikeHillberg @marb2000 @anawishnoff

I've just tried it and it worked pretty well:

Did you clean the solution?
Are you using VS 2019 16.9? Sometimes, if you have more versions of VS you start the wrong version when double click on the sln.

-M

Same problem here. Your path is too long. I copied the project to C:\w, set the start project to the XamlControlsGallery.Desktop.Package project and everything was fine :)


[CRYPTO] Adds NetflixVaultGenerator

Adds Netflix Vault Generator.

https://jenkins.code.rdkcentral.com/job/github-triggered/739/ started

https://jenkins.code.rdkcentral.com/job/github-triggered/890/ started

@sebaszm , thanks for you review feedback, I agree with all the points raised and amended a commit. Please have another look at this PR. Thanks

https://jenkins.code.rdkcentral.com/job/github-triggered/894/ started

https://jenkins.code.rdkcentral.com/job/github-triggered/895/ started

https://jenkins.code.rdkcentral.com/job/github-triggered/902/ started


Adding retries to the java install and other installs

Description
Some ppl have noticed some hosts failing to install java, giving it 10 retries
Also giving other jar/cli installs retries
Lowering the number of retries on the component health checks
Fixes # (issue)
Type of change

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
Hard to replicate the bug, but retries are only helpful so this should be good.
Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] Any dependent changes have been merged and published in downstream modules

@JumaX I'm glad you are putting thought into it! Most of them are 40s with 5s delay (not 10). C3 has the 10s delay.
We definitely want to go over by a few minutes but probably not 5! I don't have data on how long a 100 node kafka cluster might take, so giving buffer room is important. The service that stands out in my mind is c3 which can take a really long time to start up. Off of my recollection start up times in order from shortest to longest go (ill also include the retries/delays in this PR):

Zk with 30*5 = 2.5mins
SR with 30*5 = 2.5mins
KR with 30*5 = 2.5mins
Kafka with 40*5 = 3mins
KSQL with 40*5 = 3mins
Connect with 40*5 = 3mins
C3 with 40*10 = 6.6mins

My goal was not to get them dead on but lower some. LMK how you think each should change!


Name clash while using the same name for resource and resource type

When trying the example https://github.com/telia-oss/github-pr-resource#example we discovered a problem by using the same name for resource and resource type. We are using concourse v5.6.0.
We got a Error 500: base resource type not found: pull-request
Seems that there is a name clash. After renaming the resource_type to e.g. pull-request-resource it worked.
Before
resource_types:
- name: pull-request
  type: docker-image
  source:
    repository: teliaoss/github-pr-resource

resources:
- name: pull-request
  type: pull-request
  check_every: 24h
  webhook_token: ((webhook-token))
  source:
    repository: itsdalmo/test-repository
    access_token: ((github-access-token))

After
resource_types:
- name: pull-request-resource
  type: docker-image
  source:
    repository: teliaoss/github-pr-resource

resources:
- name: pull-request
  type: pull-request-resource
  check_every: 24h
  webhook_token: ((webhook-token))
  source:
    repository: itsdalmo/test-repository
    access_token: ((github-access-token))

Could you update the example please?

This is working fine from my testing - is it something that changed in v5.6.0, and is it documented and intended?

We upgraded to v5.7.2 already. With which concourse version did you test it? I mean if the name clash is not happening anymore you can close this issue.


Discussion isn't enabled

Requesting @steelthedev to enable "Discussions" for this repository for connect and collaborating more easily.

@devsujay19 I have enabled that.

Yep.


[generic/png.c] Add missing error handling

The PNG manual shows how to clean up after setjmp(png_jmpbuf(...)). We were missing the cleanup.

The other setjmp calls (like this one) should be handled as well, right? (even though the current test will not be enough to cover them)

@deltheil: Hm, yes, probably. Done and force-pushed, please have a look.

Now that I touched multiple parts of the code, I also uniformized the formatting.

Any reason not to handle the setjmp calls from libpng_(Main_save) and libpng_(Main_size)? It's true that there are a lot of code paths where errors occur, and most of them do not release resources properly right now... (the file handle, the PNG struct, etc)
IMHO a goto approach for error handling would be of great help, like replacing every call to luaL_error(L, "foo %s", "bar") by lua_pushfstring(L, "foo %s", "bar"); goto err; and use a final err section like:
err:
  /* release all resources here, e.g. */
  if (fp) fclose(fp);
  /* ... */

  return lua_error(L);

@deltheil: The only reason for why I do and don't do any one thing is this: I didn't write the code (otherwise there'd be nothing inconsistent or missing :-) ), and I only spot problems when they surface in tests.
If there's anything else you'd like done to this code, you can either point me to it (within scope of a single commit), or perform a separate, systematic code review of this package later (that's probably a better idea in the long run). In particular, wasn't comfortable installing a goto-based error handling system here, because to me that would be part of a fundamental design idiom that I would want broader project lead support and guidance on, and I wouldn't want to just randomly use that in one place. A lot of the Torch code suffers from similar problems of not handling multiple exits well, and it would benefit from a general code design guideline, but that's not something I'd want to start in an ad-hoc way like this.
All I want to achieve here is fix a memory leak in a common code path (bad file format), and I'm more willing to ignore the more obscure code paths (like no more file descriptors, out of memory). In other words, code paths that most likely lead to whole-process termination aren't quite as interesting in leaf libraries like this as those arising from regular operation.

Right, makes sense. I had similar concerns while improving error reporting (like in #100).

All I want to achieve here is fix a memory leak in a common code path

Allright. So no more remarks :)

Alright. So no more remarks :)

Hehe - to be clear, I just want to avoid scope creep for this change, so that it doesn't end up stranded forever because it doesn't achieve perfection. I'm totally in favour of a systematic review and overhaul of the existing codebase.

@soumith: could you please review this PR?

Looks good to me! Thanks @tkoeppe


"Resource temporarily unavailable" on NVIDIA Jetson TX2

Required Info

Camera Model
D400

Operating System & Version
Ubuntu 18.04.4

Kernel Version (Linux Only)
4.9.140-tegra

Platform
NVIDIA Jetson TX2

SDK Version
3.35.2 }

Language
python

Segment
other

Issue Description
I'm currently working on a system involving 3 D435 cameras connected to a NVIDIA Jetson TX2 platform via a USB3 hub. Quite often when trying to access the cameras, I get a "Resource temporarily unavailable" from one of the camera and I can't access the device that throws this error, even in the RealSense Viewer. It happens quite randomly, on any of the cameras. To get things back on track, I have to shutdown the TX2 platform and reconnect the USB hub manually. Is there any way to do this (programmatically or other) without having to manually reconnect things ? Since this system will soon go to production, it won't be easy to manually fix things up. Any help or suggestion on how to get rid of this issue ? Many thanks :)

Hi @rcroset   You can add a hardware_reset() routine to perform an automated hardware reset of the camera that has a similar effect to physically unplugging and re-plugging the camera in the USB port.  The link below has example scripting for doing so with Python and multiple cameras using the camera serial numbers:
https://github.com/IntelRealSense/librealsense/issues/5428

I unfortunately can't perform the hardware_reset() routine because I cannot access the device. When I want to query a device from the context, I get the error (see below):
>>> ctx = rs2.context()
>>> list = ctx.query_devices()
>>> list[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: set_xu(ctrl=1) failed! Last Error: Resource temporarily unavailable

The CPU usage of the TX2 is quite high (but still not near 100%) when running my project, but even when running the snippet above the error occurs.

@tispratik had a similar situation with this error when using Python, and doing a hardware_reset() did not correct the problem for them either.
https://github.com/IntelRealSense/librealsense/issues/6132

Thank you for pointing this link. Unfortunately, it doesn't seem to provide a solution. Rebooting the system is not enough in our case.

In past cases where doing a hardware_reset() on the camera has not been practical due to detection problems and you cannot reset the computer, there has been the possibility of achieving the same effect by using a USB port reset script with Ubuntu    Please google for ubuntu usb port reset script for more details.

I've already tried that kind of script, unsuccessfully. It appears that only a manual reconnection works.

Thanks for your patience.  I have been considering further possibilities.  To aid that analysis, could you tell me please if the project is being used outdoors, with the cameras exposed to sunlight?

The project is used indoor.

Okay, thank you very much.  My reading of your situation is that 3 cameras are attached to one TX2 board.  And when one camera stops responding, the other two remain accessible.  Is that correct, please?
What FPS speed are the cameras running at, please?

The 3 cameras are actually connected to a USB3 hub connected to the TX2 board. When a camera stops responding, it also shuts down the cameras plugged below on the hub (e.g. when the camera plugged on the USB slot 2 of the hub stops responding, it also affect the one plugged in slot 3). So when the first cameras stops responding, it blocks all cameras.
The cameras are running at 6 FPS with maximum resolution for depth and color stream.

6 FPS is a speed that is prone to errors that disappear at 15 FPS.  Would it be possible to try 15 FPS and see if you continue to experience the problems, please?

Ok I'll try this and I'll get back to you in a few days to inform you if the problem comes back.
How is it possible that low FPS can introduce such errors that disappear at higher framerate ?

I believe it is because when the FPS is updating slowly, it can lead to timeouts while waiting for frames to arrive.

We've been running at 15 FPS for almost one week and this problem didn't occur (yet). But a new problem appeared. The depth frames seemed not to update anymore. They update every minute or so. I've tried using wait_for_frames and poll_for_frames and nothing changes. When I switch back to 6 FPS, everything is back to normal. How is this possible ?

As you are using Python, how you are storing the frames could be a factor.  Like in the case below:
https://github.com/IntelRealSense/librealsense/issues/946

Thanks for this link. However, I am not storing the frames in an array. All post-processing is done on a copy of the frame, and the variable containing the original frame is overridden at each iteration of the loop.

My understanding is that when a frame is modified (for example, by a post-processing filter), the original is not destroyed but instead a copy of the frame is automatically created.  Every frame has a counter stating how many copies of the same frame are being held in the frameset (which is a collection of frame objects).
Old frames get pushed out when a new frame enters the frame queue, and waitforframes() pulls a frame from the queue.

Indeed, but the original is destroyed and overridden by the return value of wait_for_frames. How to access this copy counter?
Is it possible to have an example on how to properly use wait_for_frames or poll_for_frames for multicam in Python? I can't find any.

Remember from earlier in this case that for multicam projects, it is recommended that poll_for_frames() is used instead of wait_for_frames().  This includes programs that do not use multicam hardware sync, such as rs-multicam.
https://github.com/IntelRealSense/librealsense/issues/6780#issuecomment-655517695
I could not find a short and neat example of multicam use of poll_for_frames() in Python, though a device manager script in the multi-camera box_dimensioner_multicam example program makes use of it:
https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/box_dimensioner_multicam/realsense_device_manager.py#L208
You can read the frame counter from metadata with Python, though I'm not sure that is what you were asking for.
https://github.com/IntelRealSense/librealsense/issues/3179#issuecomment-462093873

I remember, thanks :)
I tried to do the same thing as in the box_dimensioner_multicam example, but still, the depth frames do not update. The color frames are fine. And when switching back to 6 FPS, the problem disappears.

I apologise for the limitations of my Python programming knowledge, which may be slowing this diagnostic process.
Some Python users who have multiple streams active have found that it can help to separate the stream types into different pipelines.  The Python script in the link below separates RG / depth in one pipeline and IMU in the other.  Whilst you are not using IMU, conceivably you could put RGB in one pipeline and depth in the other pipeline.
https://github.com/IntelRealSense/librealsense/issues/6773#issuecomment-655364659

Thanks for the link and the idea but I'm not sure this will suit our project. We need to gather color frames and the corresponding depth frames to be able to look at them at the same time. Will those two pipelines be perfectly synchronous with each other?

I would think so.  Librealsense can freely pass data between threads for purposes such as having a different processing pipeline for each stream type.
https://dev.intelrealsense.com/docs/frame-management#section-frames-and-threads

Thanks, I will try this. Do you recommend continuing to use pipelines or switching to something else like a syncer ?

My understanding is that syncer is useful for synchronizing between different streams (e.g all sensors across all cameras).  So it may be appropriate for your three-camera project.

I'm trying to use a syncer, but I have a strange behavior. Depth frames are always empty (i.e. I get the following RuntimeError: null pointer passed for argument "frame_ref") on multicam. If I'm working with only one camera, it works fine. Is there any good practices or things to do when doing multicam with a syncer ?

This case sounds similar to what you are experiencing with this error in multicam:
https://github.com/IntelRealSense/librealsense/issues/2656

Thanks for the link. However, it doesn't solve my problem. If I want to ensure that the size of the frameset coming from the syncer is 2 (i.e. color frame + depth frame), it takes approximately 50 seconds per syncer, which is way too long. Even if I use one syncer per  stream instead of one syncer per camera, it still takes way too long for gathering the two frames. From what I saw, the poll_for_frames returns a frameset containing only one frame (either the color or the depth one) but rarely the two frames in one frameset.
This problem occurs only at a framerate of 15 FPS or more. At 6 FPS everything works fine, but the initial error from this thread occurs again (I got one Resource temporarily unavailable error this morning while testing at 6 FPS and I had to reboot everything and reconnect the cameras to make it disappear)

Is the any depth to color alignment being done in your project?

Yes, depth to color alignment. But in the tests I've done for using the syncers, I didn't do the alignment

Ok, the reason I asked is that if the CMake build flag BUILD_WITH_OPENMP is set to True when building Librealsense then multiple processor cores can be utilized during depth to color spatial alignment.  This reduces latency, at the expense of higher processor usage.

So you recommend setting this flag to True ?

If you are going to be using depth to color alignment and have a powerful computer board like the TX2 then it may certainly be worth trying in a future version of the program.  If the benefits of the flag are limited to depth to color (and YUY to RGB conversion, which the flag can also assist) though then it may not be worth trying right now with your current test program.

Alright, I'll try it in a few weeks, I cannot do it right now.
In the mean time, any suggestions on how to make those syncers (or pipelines) work at 15 FPS (recall that everything works fine at 6 FPS) ? Or on how to remove this Resource temporarily unavailable error without reconnecting the cameras manually ?

Some multicam problems may occur simply because of the particular model of USB hub being used.  Though Intel recently introduced improvements in SDK 2.35.2 for how multicam is handled, these improvements do not help fix problems if the issue is related to a specific USB hub model.
Intel has successfully tested RealSense 400 Series cameras with the AmazonBasics mains-powered USB 3 hub with up to 4 cameras attached to the hub.  I have one myself that I bought based on this recommendation and have had no problems with it.  The link below references the subject of the hub.
https://github.com/IntelRealSense/realsense-ros/issues/1184#issuecomment-620466170
Checking through the case from the start, I notice that the subject of USB cable lengths has not been addressed yet.   Are you using the official short cables supplied with the camera or a longer cable of your own choice, please?

Thanks, I'll see if we can update the SDK soon.
I don't use the official cables because they are too short for the purpose of our project. I'm using longer cables. When I tested them (not multicam, one camera at the time) everything worked fine.
A weird thing is that in realsense-viewer, with the multicam, color and depth frames are coming correctly, at any framerate with all cameras streaming. The problem occur only in code

Because RealSense can transmit a lot of data through the USB cable, using a high quality cable is important with cable lengths of 2 meters or more, otherwise performance deterioration or disconnections can occur.  Intel's data sheet document for the 400 Series cameras officially recommends the high quality USB equipment supplier Newnex as a purchasing source for USB cables.
https://www.newnex.com/realsense-3d-camera-connectivity.php
Having said that, your success with your own choice of cable when using the camera on its own would suggest that your cable is okay.
When creating your own program, differences in the running of that program compared to the results of the RealSense Viewer may differ because the RealSense Viewer may have post-processing filters enabled.  In self-created programs, filters have to be deliberately programmed in.
You can check whether post-processing filters are being applied by default in your Viewer if you expand open the Stereo Module side-panel options, scroll down to the Post Processing option and see whether it has a blue colored icon (Enabled) beside it.

Post Processing is enabled in RealSense Viewer (as well as all filters) and everything works fine. In my program it doesn't work even when not doing any post processing at all. Just gathering the frames with poll_for_frames takes ages to get both depth and color frames together.

There is a detailed discussion here about how the RealSense Viewer processes streams:
https://github.com/IntelRealSense/librealsense/issues/3836
As the Viewer program is open-source, RealSense users are welcome to use it as a basis to adapt into a custom viewer to build their own application around.
https://github.com/IntelRealSense/librealsense/tree/master/tools/realsense-viewer
Having got as far as you have with your own program, that works fine with single cameras at least, I can understand that you would want to continue with your own solution.

Thanks for those informations.
The frustrating part is that my solution works fins at 6 FPS (besides the resource unavailable error) and not at all at 15 FPS

An approach that Intel have tested with multicam at 6 FPS is to connect the multiple cameras to a central computer with ethernet networking, using a tool called rs-server.
https://dev.intelrealsense.com/docs/open-source-ethernet-networking-for-intel-realsense-depth-cameras

Thanks, I'll have a look at it. But ideally, for this project we'd like to stick to USB only and not using another device as a central computer and avoid networking

I am starting to wonder if it is a Linux kernel related issue.  If it is, building librealsense from source using the RSUSB backend installation method may resolve it.  It has a tendency to clear up weird little issues.  For example, one Linux user this week had a green line on the corner of their images that wasn't present on the Windows version of the SDK, and building with backend fixed it.
I also saw a case where the user was testing streaming at 6 and 15 FPS.  they could get frames at 15, but could barely get frames at 6 FPS at all, and only if they used 1280x720 resolution.
https://github.com/IntelRealSense/librealsense/issues/2771

Fine, will try it probably next week. Thanks!
This issue with the Xavier seems to be the opposite of mine. He has troubles with 6 FPS and I have issues with 15 ;) Unfortunately, this thread doesn't seem to give any solutions

Okay, do you want to take a break from it for the weekend and come back to it next week after you have had the opportunity to test some of the things discussed?  Thanks!

Sure, I won't be able to try anything during the weekend and I have to check with my supervisor before trying to rebuild the library with those flags. I will come back as soon as possible when some things have been tested. Thanks for your help!

Hi @rcroset   Do you require futher assistance please, or can this case be closed now?  Thanks!

Hi! Yes I'll need further assistance soon. I still didn't had the opportunity to try those things we talked about. I'll come back to you as soon as possible in the coming weeks to discuss the results of those tests.

Okay, thanks very much for the update.  I will keep the case open for the next month.   Please do keep us updated about your progress in the coming weeks.   Thanks!

Hi! So we've tried building librealsense from source using the RSUSB backend as you suggested but the problems are still here, i.e. at 6 FPS the camera crash (resource temporarily unavailable) and at 15 FPS the depth frames don't update at the correct rate (once every minute or so). Any suggestions?

We have already established that using poll_for_frames() is not a fix for the problems, and that getting the color and depth frames together takes "ages".  I don't think you said whether you were also putting the CPU to sleep though.
With poll_for_frames, you have to manually control when the CPU is put to sleep, otherwise you end up "maxing out" the CPU.  You said that !he CPU usage of the TX2 is quite high (but still not near 100%) when running my project".
I wonder if poll_for_frames() is still the solution and it is the CPU sleep period value that needs adjusting for it to work properly.

Hi @rcroset   Do you still require assistance with this case, please?   Thanks!

Hi @MartyG-RealSense ! We are still trying some things in order to make everything working. We still have some issues and things to tests. I'll come back to you when we're done to tell you if we solved this case or not.

Okay, no problem.  Thanks for the update!

Hi @MartyG-RealSense ! Here are some news about those issues.
We've rebuilt the whole library using OpenMP and RSUSB backend and our problems evolved.
At 15 FPS, we had issues to get synchronized color and depth frames (the color frames were up to date but the depth frames updated once every minute or so). Now we have synchronized frames and we're very glad about this (using both wait_for_frames or poll_for_frames).
With wait_for_frames the "Resource temporarily unavailable" still occur and requires a manual intervention to reboot/restart everything as before.
With poll_for_frames and putting the CPU to sleep as you suggested, this error disappears BUT sometimes it fails to get both color and depth frames, so the resulting frameset contains only one frame, which is unusable for our project. When this situation occurs, it lasts forever until we kill the program and restart it, which is not very convenient. We've tried to detect this situation and restart the faulty pipelines programmatically but it raises a "Failed to set power state" error.
Fun fact, this problem seems to affect all cameras at the same time, so it might be related to USB.
Any suggestions ?

If wait_for_frames() is the ideal but is prone to error, I wonder if you could use try_wait_for_frames() instead.  This blocks until a frame is available like wait_for_frames() does, but returns a False state if timeout occurs instead of throwing an error.
https://github.com/IntelRealSense/librealsense/issues/2422#issuecomment-423254709

Isn't there a risk of still having the Resource temporarily unavailable error with try_wait_for_frames(), like as with wait_for_frames() ?

I am not personally aware of whether it blocks 'resource temporarily available' being thrown and returns a 'False' logic state instead.  It's probably a quick tweak to find out one way or the other though.

Alright, I'll try with this function. I'll come back to you soon to tell you if it worked well !

So I've just tried using try_poll_for_frames the results are not as good as expected. Fortunately, no "Resource temporarily unavailable" error, which is a good point, but after 20 minutes this method was unable to gather frames anymore (i.e. it continuously returns 'False'). This lead to a USB disconnection as I mentioned before.
What should I do to avoid this and keep my pipelines open and healthy ?

If you are using both color and depth streams, you could introduce some latency into the pipeline by changing the frame queue size from its default of '1' to a value of '2'.  Changing the frame queue size is not generally recommended as you can accidentally break the streams with a chosen value, but for only increasing it to '2' (as suggested in the documentation link below) you should be okay.
https://github.com/IntelRealSense/librealsense/wiki/Frame-Buffering-Management-in-RealSense-SDK-2.0#latency-vs-performance

Yes we are using both color and depth streams and they are required to be synchronized. We'll try to change this frame queue size, I'll keep you posted!

Hi @rcroset   Do you have an update that you can provide about this case, please?   Thanks!

Hi @MartyG-RealSense we still didn't tried your idea, and we have issues with the cameras disconnecting by themselves forcing us to reboot the platform every hour, so things are moving slowly. I'll come back to you as soon as possible.

Thanks @rcroset for the update - good luck with your work.

Hi @MartyG-RealSense ! We still didn't try your idea (increasing the frame queue size) but we noticed something strange during out investigations... Just before the cameras die and crash the whole USB system (which forces us to reboot to have again access to the cameras), it seems that librealsense calls USBDEVFS_CLEAR_HALT (similar dmesg messages as mentioned in issue #6123). Recall that the cameras might run perfectly fine for about one hour before crashing.
Any suggestions ?
FIY we recompiled the library with the CMAKE flag FORCE_RSUSB_BACKEND set to True, as suggested in issue #6123

Hi @MartyG-RealSense! Sorry for the long delay answer. We don't perform checking of events. We'll try to do as suggested in the issue you pointed soon.
In the meantime, I have another question. It seems that those issues mainly come from the USB traffic and the USB bus that has trouble to handle frames coming from 3 cameras at the time. In our project, all cameras need to be streaming together but only for a given amount of time, and then can be stopped, but should be able to start again quite quickly. For now we let them stream and just drop the frames when we don't need them. To reduce the amount of USB traffic while we don't need the frames, we are looking for a way to pause the pipelines (i.e. keep the cameras alive but not streaming on USB) without having to perform auto-exposure. We can't just stop the pipeline and start it again later because we need them to be reactive and the auto-exposure takes too much time. Is there a way to do so ? I've tried looking at the documentation and examples but didn't found anything.
(Sorry if this is a little off-topic for this issue)

Hii @rcroset   What some projects do to keep the cameras running in 'low power' mode when capture is not currently required is to set the Laser Power value to zero.  Doing so turns off the projector but keeps the pipeline active.  When a capture needs to be performed, the Laser Power value is increased above zero, activating the projector.  When Laser Power is minimized, the depth image will be sparse in detail.  When the capture is completed, Laser Power is set to zero again.
A way to have fine-control over timing and camera triggering that is compatible with D435 is external synchronization (genlock).
https://www.intelrealsense.com/depth-camera-external-sync-trigger/
https://dev.intelrealsense.com/docs/external-synchronization-of-intel-realsense-depth-cameras

Thanks for your answer.
So if I understood correctly, when the projector is off, no frames (color and depth) are sent through the USB cables ?

The projector is a component that is separate from the imagers.  It can enhance the image by providing light and a dot-pattern projection that the camera can use as a texture source to perform depth analysis of surfaces that have low texture or no texture (doors, walls, desks, etc).  The streams will still be active if the projector is turned off.  By having Laser Power minimized, you can reduce the camera's power draw and lower operating temperature during periods of low / no activity between capture periods.

Hi @rcroset   Do you still require assistance with this case, please?  Thanks!

Hi @MartyG-RealSense ! Yes, we still have issues with the cameras disconnecting for no reasons and killing the whole usb bus. We need to fix this before going any further with this issue. I'll come back to you as soon as we have some results.
Btw if you have any hints for those cameras disconnecting, we'll be happy to read about them ;)

SDK 2.35.2 was the version where improvements to the handling of multicam were introduced.  This was during a period where there were a number of cases where Jetson boards were having non-detection problems with more than one camera (with the failed to set power state error particularly), so this SDK version is a good choice for Jetson multicam.
The improvements mainly address problems related to rs2::pipeline though, and Jetson issues related to specific models of USB hub may still occur.  A brand of hub that Intel have successfully tested with when developing their multiple camera white-paper document is AmazonBasics.

Great, thanks for your answer ! We'll try to upgrade the SDK version as soon as possible. I'll keep you posted

Hi @rcroset   Do you have an update for us please?   Thanks!

Hi @MartyG-RealSense Not yet, we still have some other issues to solve before going any further

Okay, thanks very much @rcroset for the update.  I will keep this case open for a further time period.

Adding a note to keep this case open for a further time period.

Adding a note to keep this case open for a further period.

Adding a note to keep this case open for a further period.

Adding a note to keep this case open for a further period.

Hi @rcroset   Do you have an update about whether you are ready to proceed with the subject on this case, please?   Thanks!

Hi @MartyG-RealSense. Not yet, sorry. But we have observed that the cameras crash less often when also enabling the infrared stream. Unfortunately, we cannot investigate further yet as we have some other things to take care of before. Sorry the the long delay.

No problem @rcroset I totally understand - thanks for the continued updates.

Adding a note to keep this case open for a further time period.

Adding a note to keep this case open for a further time period.

Adding a note to keep this case open for a further time period.

Adding a note to keep this case open for a further time period.

Hi @rcroset  Do you have an update that you can provide, please?   Thanks!

Case closed due to no further comments received.

Case closed due to no further comments received.


How much does it cost to solve this problem

for GPT3 I wonder about the cost of money and the cost of time in 4 dataset
for flan-T5 I wonder about the cost of time in 4 datasets with different size
can you provide the actual data?

Hi, I sent an email to<EMAIL_ADDRESS>Did you receive it?

I think this has been long resolved. I am closing it now. Feel free to open it again if there is a follow-up.


fix(typeahead): dangling event listeners

Fixes #4632

Appreciated, but I would like to see tests.

@Foxandxss, before adding that comment, I looked and we don't even have tests to check to see that any event listeners are added which is why I didn't ask @oliversalzburg to add them.  The onus shouldn't be on him to create tests for checking the adding and removing of all the event listeners.  If you're ok with him just adding checks to see that they are removed, I'm ok with that as well.  But note that we don't even have a framework in place yet for checking them at all.  How do you want to proceed?

The responsibility to add tests is for the person who creates a PR.
In this case, creating a typeahead with the append to body, check that there are some events registered ( I guess that is possible) and then kill it and see if those events are gone.

@oliversalzburg, please add a test to check that those two event handlers are removed once the typeahead is destroyed.  Thanks.  It should be self-explanatory, but let us know if you need help figuring out how to do that.

Sure thing. I'll work on those tests. It'll might take me a while, since the weekend is approaching.
I just wanted to put the PR out there to be able to discuss it :)
Thanks for the feedback.

I'd appreciate some help with these tests. There are confusing things going on here :P
Right now, both tests fail for me:
Chrome 45.0.2454 (Windows 8.1 0.0.0) typeahead tests event listeners should register event listeners when attached to body FAILED
        Expected spy addEventListener to have been called with [ 'resize', <jasmine.any(Function)>, false ] but it was never called.
            at Object.<anonymous> (d:/bootstrap/src/typeahead/test/typeahead.spec.js:1023:39)
        Expected spy addEventListener to have been called with [ 'scroll', <jasmine.any(Function)>, false ] but it was never called.
            at Object.<anonymous> (d:/bootstrap/src/typeahead/test/typeahead.spec.js:1024:46)
Chrome 45.0.2454 (Windows 8.1 0.0.0) typeahead tests event listeners should remove event listeners when attached to body FAILED
        Expected spy removeEventListener to have been called with [ 'resize', <jasmine.any(Function)> ] but it was never called.
            at Object.<anonymous> (d:/bootstrap/src/typeahead/test/typeahead.spec.js:1033:42)
        Expected spy removeEventListener to have been called with [ 'scroll', <jasmine.any(Function)> ] but it was never called.
            at Object.<anonymous> (d:/bootstrap/src/typeahead/test/typeahead.spec.js:1034:49)
Chrome 45.0.2454 (Windows 8.1 0.0.0): Executed 1014 of 1014 (2 FAILED) (25.43 secs / 25.22 secs)

When I remove the append to body tests (typeahead.spec.js:862`, the first test will succeed. I don't understand how that test is affecting mine.
Secondly, the test for removeEventListener always fails. It appears that $destroy is never triggered in the test.

@oliversalzburg, why not test to see if a flag in your event handler has been toggled (or something) when the event has been fired and not toggled when it's been removed and the event has been fired?  Will that work?

@icfantv As I said in my previous post, the spies work right when the other "append to body" tests aren't run. So I'm pretty confident in the approach I just don't understand how the other test is affecting it.
And the second test fails because the handler for $destroy is never triggered, so the listeners actually are not removed and the test fails as it should.

Not sure why the tests would fail, probably would need to look into it

@oliversalzburg, did you mean line 872?

At first glance, I don't see how/why that test would cause yours to fail - each describe() is a separate unit and there shouldn't, in theory, be any spillover.

The code style and whitespace issues should be fixed now.
@icfantv Yes, the tests starting on line 872 are what I was referring to (there was a typo in my other post).

I guess the answer is actually pretty simple.

The previous tests cause the event listeners to be registered. jQuery remembers those listeners.
Because $destroy is never invoked, the listeners are not removed.
When my test runs, jQuery will not invoke addEventListener, because the listeners are already attached.

So it all boils down to, why is there no $destroy?

angularjs: why isn't $destroy triggered when I call element.remove? sounds pretty relevant…

Perhaps instead of element.remove, you should take the $scope and execute $scope.$destroy()

@wesleycho But the other tests aren't causing the scope to be destroyed (or the $destroy event being emitted from it). So, calling $scope.$destroy() in my tests won't have the desired effect.
When I change originalScope.$on('$destroy', fn) to element.on('$destroy', fn), the handler is invoked and all tests are still passing. Well, except my test that checks for removeEventListener being called. I'm not quite sure why that is, but at least the surrounding code then works as expected.

I fixed and merged it.

@Foxandxss Thanks a lot for taking care of this!
However, clearing the listeners manually in the tests doesn't seem like the right approach IMHO.
In general, it seems like $destroy is not broadcast from the scope when the DOM element is removed. You can observe this in http://plnkr.co/edit/XBWPaITD4v9RY29FwR0U?p=preview In fact, due to this behavior, when using jQuery's .remove(), my originally reported issue is still present, as the event listeners are not removed.
As mentioned in angularjs: why isn't $destroy triggered when I call element.remove?, the $destroy event is being emitted from the DOM nodes themselves in this case. Because the typeahead directive does not account for this case, it can still leak event listeners.
The docs even note:

Note that, in AngularJS, there is also a $destroy jQuery event, which can be used to clean up DOM bindings before an element is removed from the DOM.

I would propose that the event should be respected and the listeners should be cleaned up.

I am sorry, but I am bit confused.
When you test something, those objects will be created and disappear after each test, but if you bound some of the to the body, they will remain in there for all the eternity.
In the describe for append to body, it is always better to make a cleanup after each test (Perhaps I would even need to delete the element entirely).
For you concrete tests, first one works after you clean the previous ones (as you said, jquery was not adding new listeners because they were there yet). Second one works as soon as your start clean and you trigger $destroy.
I don't see any issue with this approach, but, if you think there is a better way, I am open for it.

Well, the tests aren't accurately testing the directive. The directive is supposed to clean up after itself. But it doesn't. You're cleaning up the event listeners yourself in the test.
That being said, there is still a real issue in the code. If the DOM element (or any of its parents) is removed via jQuery (as it is done in the tests), the event listeners will not be cleaned up, because the $destroy event is not handled. In fact, none of the cleanup code is run in that case.

About the first point, yes, the directive is cleaning itself correctly.
  describe('event listeners', function() {
    it('should register event listeners when attached to body', function() {
      spyOn(window, 'addEventListener');
      spyOn(document.body, 'addEventListener');

      var element = prepareInputEl('
<input ng-model="result" uib-typeahead="item for item in source | filter:$viewValue" typeahead-append-to-body="true">
');

      expect(window.addEventListener).toHaveBeenCalledWith('resize', jasmine.any(Function), false);
      expect(document.body.addEventListener).toHaveBeenCalledWith('scroll', jasmine.any(Function), false);
      $scope.$destroy();
    });

    it('should remove event listeners when attached to body', function() {
      spyOn(window, 'removeEventListener');
      spyOn(document.body, 'removeEventListener');

      var element = prepareInputEl('
<input ng-model="result" uib-typeahead="item for item in source | filter:$viewValue" typeahead-append-to-body="true">
');
      $scope.$destroy();

      expect(window.removeEventListener).toHaveBeenCalledWith('resize', jasmine.any(Function), false);
      expect(document.body.removeEventListener).toHaveBeenCalledWith('scroll', jasmine.any(Function), false);
    });
  });

If you change your tests like that. No manual cleanup it works. If you comment out the $destroy on the first test, the second one won't work (because we have "stale" code in the dom).
About the second point, not sure if I follow. If a $destroy is triggered in the parent, all the children will die too.

I'm really not sure how to make this clearer than I already have with the provided explanation, links to the Stack Overflow question, the AngularJS documentation and the Plunker.
So, in the interest of saving everyone's time, I'll just move on.

I now see what you mean. There is nothing wrong with asking.
I will fix it.


Feat/groups

Familiarizing myself with PRs, branches, etc. The "groups" feature is far from complete, and I probably should have given it a more detailed name. I'll


Feature: Role Based Access Control(RBAC)

So far Squest has 2 persona: Admin and User.
We need to implement RBAC system so admins can creates groups and give them roles like capability of creating a service, an approval config,...

TODO:

[ ] Rename Billing group --> organization
[ ] complete the RBAC yaml descriptor with all perms for all objects
[ ] Create OrgRoleBinding class
[ ] enforce permissions on views
[ ] delete "admin" role linked to user on instance (we can only have reader or operator)
[ ] migration script to move existing admin to operator role


SSL Error when Accessing DataLake via Java Driver

Description
Issue Description:
I am experiencing an SSL error when trying to access the Data Lake through a connection via driver and Java library. The system cannot establish a secure connection due to a problem with verifying the SSL certificate chain. Below is the detailed error message in attach file.
Impact:
It is not possible to access the Data Lake via the Java connection.
The system fails to establish a secure connection with the server, directly affecting operations dependent on the Data Lake.
Requested Action:
Please verify the validity and configuration of the certificates on the Data Lake server accessed via the Java driver.
Guidance is needed to fix the certificate chain if required, or to update the keystore if the issue is on the client side.
DBeaver Version
Community Edition 24.2.1
Operating System
Windows 11
Database and driver
Infor Data Lake
Driver: infor-compass-jdbc-2020-09.jar
Steps to reproduce
No response
Additional context
No response

Please send us log files. You can collect them to archive automatically with Help->Collect Diagnostic Info or manually following this guide: https://dbeaver.com/docs/wiki/Log-files/. Thank you.

Please try changing the setting Use Windows trust store in Preferences->Connections and restarting DBeaver. If you haven't overridden the trust store, it might help.

Uploading dbeaver-diagnostic-info-1727794312160.zip…

Something went wrong while uploading an archive. Could you try again and wait until Uploading dbeaver-diagnostic-info-1727794312160.zip… will disappear?

dbeaver-diagnostic-info-1727794312160.zip

it's ok now

thank you

Use Windows trust store  in Preferences->Connections is not available in the Community version

The error in logs is exactly the same as in this issue https://github.com/dbeaver/dbeaver/issues/35686
The reporter said that the issue was resolved for him by providing updated TLS certs for their datalake access

I can find this setting in Community Edition, is it disabled or missing for you?

This option is enable

I'm going to test again and restart DBeaver, and test the connection with datalake.

I see, could you try disabling it? Maybe it's vise versa and Java certs will help

When i test disable the Windows trust store option and restart DBeaver, the connection with DataLake has been reestablished.
Thanks for helping me!


docs & readme: what/why/how

[x] update README
[x] update registry docs
[x] maybe restructure registry docs a bit (credentials, getting started, and full reference are in 3 different counter-intuitive places?)
[x] depends on #414 (or revert be6aa67)

fixes #415

@casperdcl Looking good!

Update from the discussion in Slack here. Agreed to use the universal TPI banner with the dark background.

I'd suggest merging for now @dmpetrov @0x2b3bfa0 since it's probably best to iterate on docs in follow-up issues/PRs.

naturally :)
I think it's just the "end-to-end" point in #363, but let me know if anything else is unresolved.

@jorgeorpinel, even if your review is a bit late (post–merge), you make lots of good points that should be considered and addressed. 😅

@casperdcl, should this be discussed here and then applied with a separate pull request?


Run dependabot on this repo

Automatically keep all the deps up-to-date.

I set it up, no idea if it is going to work or choke on the curlies in Cargo.toml


Not accepting * in a database account password

Bug description
A postgres database connection with valid credentials does not work because there is a special character in the password(*).
We tried escaping the password string directly in the URL and in the DB with no change in behavior.
How to reproduce the bug

Add database connection with a password containing *
Try to load schemas in SQL Lab
get an error

Screenshots/recordings
No response
Superset version
3.1.1
Python version
3.10
Node version
I don't know
Browser
Chrome
Additional context
Only error message in logs we get is when requesting schemas when exploring in sqllab.
DatabaseRestApi.schemas.error
All of the sqlalchemy libs we use.
flask-sqlalchemy 2.5.1

marshmallow-sqlalchemy 0.28.2

sqlalchemy-dremio 3.0.3

sqlalchemy-utils 0.38.3

sqlalchemy 1.4.51

Checklist

[X] I have searched Superset docs and Slack and didn't find a solution to my problem.
[X] I have searched the GitHub issue tracker and didn't find a similar bug report.
[X] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

Pinging @dpgaspar @betodealmeida in case there's good reason for not accepting that particular character.

When using tools like Amazon RDS with a Secrets Manager that is rotating the RDS credentials for you, you cannot enforce how strong the password will be.  This is going to be a problem in large enterprise environments that have very strict password policies.

I also encounter the same issue, particularly when the * is at the end of my password in the SQLAlchemy URI.

I'll fix this, thanks for the report!

I have the same problem with a password with a % character in it.  I've tried using a backslash to escape it and URL-encoding it, to no avail.

Facing the same issue. Hit problems with both * and %. Any potential updates on this?


Fix Javadoc mention a not existing method

This looks like a fine change but I'd like to keep bug fixes daggy (https://wiki.monotone.ca/DaggyFixes/) so that they can be easily merged and tracked through the various branches.


SVG ids are not valid

When exporting SVG images from Wavedrom, I discovered not all IDs are valid. According to the XML spec, IDs cannot begin with a numeric value, they must start with an alpha character. However, Wavedrom is using a combination of them.
    <defs>         <g id="socket">             <rect y="15" x="6" height="20" width="20"/>         </g>         <g id="pclk">             <path d="M0,20 0,0 20,0" class="s1"/>         </g>         <g id="nclk">             <path d="m0,0 0,20 20,0" class="s1"/>         </g>         <g id="000">             <path d="m0,20 20,0" class="s1"/>         </g>         <g id="0m0">             <path d="m0,20 3,0 3,-10 3,10 11,0" class="s1"/>         </g>         <g id="0m1">             <path d="M0,20 3,20 9,0 20,0" class="s1"/>         </g>
   Here socket, pclk, and nclk are all valid, but all remaining values are not. Most tools seem to be allowing these, however, we have to validate all our content and images created with wavedrom are all failing due to this problem.

does it cause XML validation issues somewhere?

It does we use epubcheck and when we have documents that contain wavedrom derived SVGs it will not pass those documents and thus we can't publish them. Looking at the raw SVG it really is a simple matter of following the standard XML ID rules and not using a numeric character.


Add markdown support

Any chance N1 will get Markdown support somewhere down the line?

It seems like you should be able to create a plugin for it

Hey! This sounds great. I'd love to be able to type Markdown into the composer. We're creating a public Trello board to keep track of package ideas. I'm going to file this there—feel free to upvote it on Trello.
https://trello.com/b/hxsqB6vx/n1-open-source-roadmap

@nunofgs: I have started on a package here

+1 Most email clients are missing markdown and I would willingly switch to one that supports it.

It's ready for public testing, but it is blocked by #283

Any updates on this?
I would love to switch from Inbox by Gmail and the fantastic Markdown Here but not having support for Markdown in Nylas N1 is preventing me so far.
Thankfully,
~Robert

@bengotow I can’t find a card with a Markdown feature request on your Trello board.
Would you mind linking it from here? With Markdown all over the place recently, I’m curious to know how much this is in demand.

Can this issue be reopened?  The plugin referenced hasn't been updated in 2 years.  I'm using 2.0.32 and I don't see any markdown features included by default, or perhaps I'm missing something?

@RTbecard you're missing #3564


Find a mistake in server/election.go

Hi, I think there is a mistake in server/election.go#L235
should be
	if !ok {
		s.logger.Infof("leader %s does not exist", s.leaderID)
		return false                   <--- here
	}

am I right?

that's correct, that should be false. You can make a PR to fix it.
@malwaredllc , are you happy to accept some  hacktoberfest PR on this project. If yes, you can add it using this Guide

@moooofly Sorry for the delayed response, feel free to submit a PR with the fix! (Ideally with a test case that will catch this next time)


Remove all build warnings

Cleanup the code to remove build warnings.

I have commented out the offenders in origin/master and added TODOs so that we can manually review and make sure they are safe to delete. It looks mostly like copy/paste things and left over code from when Matt added ROI. It should be safe to remove the offending code bits, but wanted another set of eyes.

Check clang / gcc ... clang has much better warning detection.


Add JMESPath support for operation write property

Looks great... Ah - one thing - there might be one last reference to operation.write in the long, awful looking line that has ... message ?? operation.write ?? operation.request ....
If you'd like to push a commit to change that last reference then I'll merge this PR. Thanks again!

Oops, I might have missed that one. Will fix now.


Kafka Indexing Service failed - could not allocate segment for row with timestamp[*]

Affected Version
0.16.0
Description
Small cluster ingesting data using Kafka Indexing Service. Was ingesting fine. A task failed and every subsequent restarted task failed reporting the same error. I had to reset the supervisor and skip some messages to get it to continue ingesting - resulting in a little data loss.
Note that the row timestamp is a few months old (2019-12-11T00:00:01.000Z). Note that there are segments on the historical and in deepstorage that cover that time period.
Digging through the error message and druid repository - the exception has been thrown because the segmentIdentifer for that row is null??? Link to offending line of code.
I think the preferred way of handling this would be to log the data to a file and continue, rather than stop ingesting altogether. Do you agree?
Below is the task report (formatted slightly for readability):
{
    "ingestionState": "BUILD_SEGMENTS",
    "unparseableEvents": {},
    "rowStats": {
      "buildSegments": {
        "processed": 2945168,
        "processedWithError": 0,
        "thrownAway": 0,
        "unparseable": 0
      }
    },
    "errorMsg": "
org.apache.druid.java.util.common.ISE: Could not allocate segment for row with timestamp[2019-12-11T00:00:01.000Z]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:651)
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:259)
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:177)
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:419)
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:391)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)"
}

Below is the task payload:
{
    "type": "index_kafka",
    "id": "index_kafka_DataProd_ac63ef887e4e88d_behocdcc",
    "resource": {
      "availabilityGroup": "index_kafka_DataProd_ac63ef887e4e88d",
      "requiredCapacity": 1
    },
    "dataSchema": {
      "dataSource": "DataProd",
      "parser": {
        "type": "string",
        "parseSpec": {
          "format": "json",
          "timestampSpec": {
            "column": "ts",
            "format": "iso"
          },
          "dimensionsSpec": {
            "dimensions": [
              {
                "name": "control_group",
                "type": "string"
              },
              {
                "name": "country",
                "type": "string"
              },
              {
                "name": "device_name",
                "type": "string"
              },
              {
                "name": "metric_name",
                "type": "string"
              }
            ]
          }
        }
      },
      "metricsSpec": [
        {
          "type": "count",
          "name": "count"
        },
        {
          "type": "doubleSum",
          "name": "sum",
          "fieldName": "metric_value",
          "expression": null
        },
        {
          "type": "doubleMin",
          "name": "min",
          "fieldName": "metric_value",
          "expression": null
        },
        {
          "type": "doubleMax",
          "name": "max",
          "fieldName": "metric_value",
          "expression": null
        }
      ],
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "DAY",
        "queryGranularity": "MINUTE",
        "rollup": true,
        "intervals": null
      },
      "transformSpec": {
        "filter": null,
        "transforms": []
      }
    },
    "tuningConfig": {
      "type": "KafkaTuningConfig",
      "maxRowsInMemory": 1000000,
      "maxBytesInMemory": 0,
      "maxRowsPerSegment": 5000000,
      "maxTotalRows": 20000000,
      "intermediatePersistPeriod": "PT10M",
      "basePersistDirectory": "/var/tmp/druid/1580855764252-0",
      "maxPendingPersists": 0,
      "indexSpec": {
        "bitmap": {
          "type": "concise"
        },
        "dimensionCompression": "lz4",
        "metricCompression": "lz4",
        "longEncoding": "longs"
      },
      "indexSpecForIntermediatePersists": {
        "bitmap": {
          "type": "concise"
        },
        "dimensionCompression": "lz4",
        "metricCompression": "lz4",
        "longEncoding": "longs"
      },
      "buildV9Directly": true,
      "reportParseExceptions": false,
      "handoffConditionTimeout": 0,
      "resetOffsetAutomatically": false,
      "segmentWriteOutMediumFactory": null,
      "intermediateHandoffPeriod": "P2147483647D",
      "logParseExceptions": false,
      "maxParseExceptions":<PHONE_NUMBER>,
      "maxSavedParseExceptions": 0,
      "skipSequenceNumberAvailabilityCheck": false
    },
    "ioConfig": {
      "type": "kafka",
      "taskGroupId": 0,
      "baseSequenceName": "index_kafka_DataProd_ac63ef887e4e88d",
      "startPartitions": {
        "type": "end",
        "stream": "switchdin_rollups",
        "topic": "switchdin_rollups",
        "partitionSequenceNumberMap": {
          "0": 419705763,
          "1": 247769423,
          "2": 395630862,
          "3": 555085414
        },
        "partitionOffsetMap": {
          "0": 419705763,
          "1": 247769423,
          "2": 395630862,
          "3": 555085414
        }
      },
      "endPartitions": {
        "type": "end",
        "stream": "switchdin_rollups",
        "topic": "switchdin_rollups",
        "partitionSequenceNumberMap": {
          "0":<PHONE_NUMBER>854776000,
          "1":<PHONE_NUMBER>854776000,
          "2":<PHONE_NUMBER>854776000,
          "3":<PHONE_NUMBER>854776000
        },
        "partitionOffsetMap": {
          "0":<PHONE_NUMBER>854776000,
          "1":<PHONE_NUMBER>854776000,
          "2":<PHONE_NUMBER>854776000,
          "3":<PHONE_NUMBER>854776000
        }
      },
      "startSequenceNumbers": {
        "type": "start",
        "stream": "switchdin_rollups",
        "topic": "switchdin_rollups",
        "partitionSequenceNumberMap": {
          "0": 419705763,
          "1": 247769423,
          "2": 395630862,
          "3": 555085414
        },
        "partitionOffsetMap": {
          "0": 419705763,
          "1": 247769423,
          "2": 395630862,
          "3": 555085414
        },
        "exclusivePartitions": []
      },
      "endSequenceNumbers": {
        "type": "end",
        "stream": "switchdin_rollups",
        "topic": "switchdin_rollups",
        "partitionSequenceNumberMap": {
          "0":<PHONE_NUMBER>854776000,
          "1":<PHONE_NUMBER>854776000,
          "2":<PHONE_NUMBER>854776000,
          "3":<PHONE_NUMBER>854776000
        },
        "partitionOffsetMap": {
          "0":<PHONE_NUMBER>854776000,
          "1":<PHONE_NUMBER>854776000,
          "2":<PHONE_NUMBER>854776000,
          "3":<PHONE_NUMBER>854776000
        }
      },
      "consumerProperties": {
        "bootstrap.servers": "kafka.vpca.switchd.in:9092"
      },
      "pollTimeout": 100,
      "useTransaction": true,
      "minimumMessageTime": null,
      "maximumMessageTime": null
    },
    "context": {
      "forceTimeChunkLock": true,
      "checkpoints": "{\"0\":{\"0\":419705763,\"1\":247769423,\"2\":395630862,\"3\":555085414}}",
      "IS_INCREMENTAL_HANDOFF_SUPPORTED": true
    },
    "groupId": "index_kafka_DataProd",
    "dataSource": "DataProd"
  }

Note I am also seeing this warning message from the coordinator:
WARN [KafkaSupervisor-DataProd-Reporting-0] org.apache.druid.indexing.kafka.supervisor.KafkaSupervisor - Lag metric: Kafka partitions [0, 1, 2, 3] do not match task partitions []

There were issues with the segments and/or segment metadata for a particular period. Nothing to do with the ingestion. I compacted the segments for the offending periods - creating new segments. The old segments were removed (kill tasks). Everything seems to be working now.


fix serialization

Fixes https://github.com/OpenDevin/OpenDevin/issues/1782

Confirmed working. Going to get this in to unbreak main. CC @neubig for any post-hoc code review

Thank you, LGTM!


It returns Invalid params.

I am trying to make a custom invoke function. Using NetworkFacade.transferToken, it works fine however my code returns invalid params
try {
  const facadePromise = await api.NetworkFacade.fromConfig({
    node: props.nodeUrl,
  });

  const intent = {
    from: new wallet.Account(props.wallet.privateKey),
    to: "NaES6pHFRBibbUkRBm59KNujaxFGaDX69z",
    decimalAmt: 0.00000001,
    contractHash: Neon.CONST.NATIVE_CONTRACT_HASH.GasToken,
  };

  const signingConfig = {
    signingCallback: api.signWithAccount(
      new wallet.Account(props.wallet.privateKey)
    ),
  };

  const sb = new sc.ScriptBuilder();
  sb.emitAppCall(props.script.scriptHash, "symbol", []);
  const txBuilder = new TransactionBuilder();
  txBuilder
    .addScript(sb.build())
    .addSigners({
      account: intent.from.scriptHash,
      scopes: tx.WitnessScope.CalledByEntry,
    })
    .addEmptyWitness(intent.from);

  const txn = txBuilder.build();
  const validateResult = await facadePromise.validate(txn);
  if (!validateResult.valid) {
    throw new Error("Unable to validate transaction");
  }
  const signedTx = await facadePromise.sign(txn, signingConfig);
  const sendResult = await facadePromise
    .getRpcNode()
    .sendRawTransaction(signedTx);
} catch (e) {
  throw new Error(e);
}

I don't see anything obviously wrong at first sight. Note that you're using the deprecated emitAppCall instead of emitContractCall.
Which neon-js version are you using and what version is the RPC node that you're talking to? You can request the node version by RPC using getversion

this worked for me
sb.emitContractCall({
      scriptHash: Neon.CONST.NATIVE_CONTRACT_HASH.GasToken,
      operation: "symbol",
    });

The  specifying empty args or not doesn't matter, so I'm going to guess that the scriptHash provided via your props is invalid.
Full script
const Neon = require("@cityofzion/neon-js");
const wif = "L1QqQJnpBwbsPGAuutuzPTac8piqvbR1HRjrY5qHup48TBCBFe4g";
const acc = new Neon.wallet.Account(wif);

async function issue713() {
  try {
    const facadePromise = await Neon.api.NetworkFacade.fromConfig({
      node: "http://<IP_ADDRESS>:20332",
    });

    const intent = {
      from: acc,
      to: "NaES6pHFRBibbUkRBm59KNujaxFGaDX69z",
      decimalAmt: 0.00000001,
      contractHash: Neon.CONST.NATIVE_CONTRACT_HASH.GasToken,
    };

    const signingConfig = {
      signingCallback: Neon.api.signWithAccount(
        acc
      ),
    };

    const sb = new Neon.sc.ScriptBuilder();
    sb.emitContractCall({
      scriptHash: Neon.CONST.NATIVE_CONTRACT_HASH.GasToken,
      operation: "symbol",
      args: [],
    });
    const txBuilder = new Neon.api.TransactionBuilder();
    txBuilder
      .addScript(sb.build())
      .addSigners({
        account: intent.from.scriptHash,
        scopes: Neon.tx.WitnessScope.CalledByEntry,
      })
      .addEmptyWitness(intent.from);

    const txn = txBuilder.build();
    const validateResult = await facadePromise.validate(txn);
    if (!validateResult.valid) {
      throw new Error("Unable to validate transaction");
    }
    const signedTx = await facadePromise.sign(txn, signingConfig);
    const sendResult = await facadePromise
      .getRpcNode()
      .sendRawTransaction(signedTx);
    console.log(sendResult);
  } catch (e) {
    throw new Error(e);
  }
}

issue713();

I assume that the symbol call was just a minimal example and that you plan to call a function that should be persisted to the chain. If not, note that you can query read only information (like "symbol", "decimals" etc) for free by using the invokefunction RPC call.
If you want to do a single call grabbing multiple method invoke results you can chain the calls using scriptbuilder as done before and call the invokescript RPC call. Basically all RPC calls except the ones starting with send are read-only. Of the send* ones sendrawtransaction is the only one that does not require an open wallet on the RPC side. Be aware that there is a limit to how many instructions you can execute for free when using invokescript, so if it fails with (iirc)  insufficient gas, then split the script up in multiple RPC calls.

Weird. This works now. Thanks for trying!
But it has the same issue as https://github.com/CityOfZion/neon-js/issues/714
I am following https://dojo.coz.io/neo3/neon-js/docs/en/guides/basic/transfer.html this instruction to break down to get the right system fee and will compare with NetworkFacade methods.


Update version to 24.08.1-SNAPSHOT [skip ci]

Wait for the pre-merge CI job to SUCCEED

build

this was due to 24.08.0 artifact got released accidentally


Please invite me to the Recode-Hive GitHub Community Organization

Name
Venkata Ramana Billana
Discord Username (if applicable)
venkataramana2074

Invitation sent to join the GitHub Organisation. Welcome to the community 🎉
Don't forget after accepting to make it public so it appears on your GitHub profile for everyone else to see. You can do this by finding your name in the GitHub organisation list and change the dropdown to public https://github.com/orgs/recodehive/people


src/video/* static assets are not deployed.

I get 404 for https://ampbyexample.com/video/tokyo.mp4 (Noticed from https://ampbyexample.com/components/amp-video/ example). The file is in the source, I think just not deployed.
Problem does NOT exist for other assets like images in src/img/* so https://ampbyexample.com/img/Hovawart.jpg works fine.

Thanks for reporting!


Question: exporting raw column data into separate files, sorted by all columns

Hi
First of all, compliments for your software (I tested it the first time in 2017 but at that time the lack of custom partitioning was a problem and insert-performance wasn't good; now with custom partitioning and the availability of the "Buffer"-engine it's incredibly fast - and thanks as well a lot for making it available directly in the main portage tree of Gentoo :) ).
Situation
(version being used: 18.10.3)
I have a base/source table like this one (simplified example)...
create table testtbl (col01 UInt8, col02 UInt8, col03 String, col04 UInt64,...) engine=MergeTree partition by(col01) order by (col02,col03,col04,...) settings index_granularity=65536

...and I keep on inserting data into it => works perfectly.
What I need
I need an extract of all raw/original data that belongs to a specific partition, one file per column, where each column has its contents perfectly sorted by all columns.
What I'm currently doing
I'm running scripts like these ones:
create table export_mypart123_col02 engine=File(RowBinary) as select col02 from testtbl where col01 = 123 order by col02,col03,col04;
create table export_mypart123_col03 engine=File(RowBinary) as select col03 from testtbl where col01 = 123 order by col02,col03,col04;
create table export_mypart123_col04 engine=File(RowBinary) as select col04 from testtbl where col01 = 123 order by col02,col03,col04;
...

Problem
Exporting a single column takes in my case ~35 minutes for a table that contains ~5 billion records.
FYI I had to activate the option "max_bytes_before_external_sort" (currently set to 5GBs, which is enough for ~2.5% of the data) as the whole table does not fit in my server's RAM (32GBs).
I see that retrieving the data from the DB is superfast but when Clickhouse starts writing the temporary files needed for the "sort" and then later the final output file into the "RowBinary"-table it's veeery slow (nothing much going on there; I see 1 CPU-thread being fully used and ~20MB/s being written to the temporary sort files and later to the final file).
Question
Is there a more intelligent way to do this kind of export?
Currently, I'm just splitting all "create table"-statements (1 per column) into groups (e.g. 3 groups of 5 "create table"-statements each), start up 3 clickhouse-clients in parallel and make each client run sequentially all 5 queries of their own group => in this example, running 5 such statements would need ~3 hours.
For example MariaDB has the vector-table type (https://mariadb.com/kb/en/library/connect-vec-table-type/) which writes the contents of each of its columns into single files (some fields, e.g. "varbinary" which more or less matches "String" in Clickhouse is a problem but I was handling it in a different way) => is it possible in Clickhouse for example to make a "File(RowBinary)"-table write the contents of its columns into separate files?
Or is there any other altenative so that I don't have to keep re-running the "order by" (e.g. some sort of table that would have its contents ordered perfectly by rows inserted into it, from which I could therefore just "select" when creating the single columns)?
Reason
I'm doing some experiments with indexing of data; I want to check how the amount of data and/or the order or the columns impact the singlecolumn&total compression ratio of the data.
From time to time I therefore export the raw contents of the columns, compress it and then compare the file sizes to the previous result.
Thanks a lot :)
Stefano

I'm doing some experiments with indexing of data; I want to check how the amount of data and/or the order or the columns impact the singlecolumn&total compression ratio of the data.

are you aware about existence of system.parts , system.columns, system.parts_columns? They have different metadata, including real sizes of columns/parts and their compressed sizes. So if you need that data for analyzing how Clickhouse stores the data you don't need to export it outside :)
Clickhouse already stores columns in filesystem in 'one file per column' format (spit to partitions / parts) sorted by PK.
if you will create another copy of your table with another PK and copy data there you will see how compression ratio changes.

Thank you, no I wasn't aware of "system.parts_columns", great! I will use the sum of the bytes directly from that table when I test changes to the order of columns or other stuff :) .
On the other hand, if I compare "sum(column_data_compressed_bytes)" of a specific column and partition against the compressed size of the file I generated as explained above with "File(RowBinary)" and then compressed manually, the size is more or less similar for small partitions (e.g. for a part that contains 186'150'292 rows I get from the DB ~390MBs and ~380MBs from the external file), but for bigger partitions there is a big difference (e.g. for a part containing 5'170'344'578 rows I get from the system-table a sum of ~5033MBs but the manually exported and compressed file is only 3820MBs big) => can it be that the compressed data in Clickhouse needs more storage because it is compressed in smaller "chunks" (and is therefore less efficient) than when fully exported into a single file and compressed all together?
Fyi I have this in "config.xml"...
<compression incl="clickhouse_compression">
      <case>
         <method>zstd</method>
      </case>
</compression>

...and when compressing manually the exported file I used zstd compression level 3 ("zstd -3  -o ").
Thank you

ClickHouse use zstd compression level 1 by default. You can change it by adding
<level>3</level> in <case> near <method>.
And yes, ClickHouse use smaller compressed blocks by default (from 64 KB to 1 MB, depending on your data and index granularity).
This is controlled by
min_compress_block_size
and
max_compress_block_size
per user settings, that can be tuned in users.xml in default profile.

Great, very useful to know, thanks a lot!


POC tool castlist doesn't handle duplicate titles properly.

In this code:
https://github.com/cjcodeproj/medialibrary/blob/226747af68f5f5a625b6b3d9b643c53b2e61b4ec/src/media/tools/movies/castlist.py#L15-L22
The key for the map should actually be the movie object, not the movie title object.  The title object will not be 100% accurate if there are movies in the repository with the same title.  However, the movie object has sorting routines and uniqueness guarantees to reduce the chance of that happening.

Commit.
https://github.com/cjcodeproj/medialibrary/commit/889a180df8885f43ddcb67030010af0a63f74ced

Closing notes:
This issue was discovered while working on https://github.com/cjcodeproj/medialibrary/issues/46.
The large internal movie test dataset has 8 movies that share titles (3 remakes of 3 other movies, 2 movies with the same title, but no relation), so using the title as a key was producing non-sensical data, and merging actor jobs erroneously.  Changing the key value in the grab_cast_names() function essentially fixed this.
https://github.com/cjcodeproj/medialibrary/blob/889a180df8885f43ddcb67030010af0a63f74ced/src/media/tools/movies/castlist.py#L90-L103


Feature/examples folder

Moved source files from repo root directory into dedicated examples folder

Or maybe not. Do they run correctly?

Compilation itself works, OE_Test seems to run correctly

Yeah, my brain though that if we change the .cpp position that the root directory changes, but it is only important where the execution happens and not where the .cpp is. :P


Memory Leak

Their appears to be a minor memory leak.

After disabling & reloading:

Are you sure about this? I've never had this problem before. 🤔🤔

Are you sure about this? I've never had this problem before. 🤔🤔

Positive. It resolved itself after uninstalling the extension & reloading vscode. Hours later still no issue.

Hmm, ok, very weird, I'll take a look to see what I did wrong. 👍

cheers.

The problem should be fixed in the newly released version, please upgrade to the extension and let me know if the problem is fixed.

https://github.com/LeonardSSH/vscord/issues/7#issuecomment-896199539


macOS: support case sensitive file system

VSCode Version: 1.46.1
OS Version: macOS 10.15.5 in case sensitive volume

Steps to Reproduce:

Open any file.
Change file name case
Close file.
Open again and get error.

Does this issue occur when all extensions are disabled?: Yes/No
Yes.

We currently do not support case-sensitive file systems on macOS or Windows. It may have worked by chance in the past but we never officially did anything to enable this scenario.
//cc @jrieken

Dup of https://github.com/microsoft/vscode/issues/94307


Generalize flag handling in memap.py

affects:

-t/--toolchain
-e/--export

Both now accept upper or lower case and hyphens or underscores and will convert as appropriate

It tried on the third commit (Refactor export ...), but the diff is still much bigger looking than it actually is. I just moved code around and changed indenting.

cc @mlnx

@theotherjimmy thanks for looking into this. It looks good to me.

@0xc0170, @mlnx would you like to see similar changes across the tools, making them all have a similar interface?

Well, I did it anyway. Let's see if this breaks anything.

+1 from me, this is nice stuff

+1

Working on more porting of argument parsing to the more liberal style.

I think I got everything I wanted to for now. LGTM? @0xc0170 @bogdanm

At first glance, these changes look good, thanks @theotherjimmy.
One thing that stands out to me is in make.py and project.py, if an incorrect test name or number was provided (via -n or -p respectively), a list of available tests were printed to help the user. As far as I can, tell this has capability was removed? Here in make.py: https://github.com/mbedmicro/mbed/pull/1976/files. And here in project.py: https://github.com/mbedmicro/mbed/pull/1976/files. Unless this is handled by argparse?
I'd like to test this locally too, but so far it's looking good.

outputs!
$  python tools/make.py -p -1
[WARNING] Using default settings. Define your settings in the file "./mbed_settings.py"
usage: make.py [-h] [-m MCU] [-t TOOLCHAIN] [-c] [-o OPTIONS]
               (-p PROGRAM | -n PROGRAM) [-j JOBS] [-v] [--silent] [-D MACROS]
               [-S] [-f GENERAL_FILTER_REGEX] [--automated] [--host HOST_TEST]
               [--extra EXTRA] [--peripherals PERIPHERALS]
               [--dep DEPENDENCIES] [--source SOURCE_DIR]
               [--duration DURATION] [--build BUILD_DIR] [-N ARTIFACT_NAME]
               [-d DISK] [-s SERIAL] [-b BAUD] [-L] [--rtos] [--rpc] [--eth]
               [--usb_host] [--usb] [--dsp] [--fat] [--ublox] [--testlib]
               [-l LINKER_SCRIPT]
make.py: error: argument -p: -1 does not index a test. The accepted range is 0 to 135

and
$ python tools/project.py -n foo
[WARNING] Using default settings. Define your settings in the file "./mbed_settings.py"
usage: project.py [-h] -m MCU -i IDE [-c] (-p PROGRAM | -n PROGRAM) [-b] [-L]
                  [-S] [-E] [--source [SOURCE_DIR [SOURCE_DIR ...]]]
                  [-D MACROS]
project.py: error: argument -n: Program with name 'foo' not found.  Define your settings in the file "./mbed_settings.py"
usage: project.py [-h] -m MCU -i IDE [-c] (-p PROGRAM | -n PROGRAM) [-b] [-L]
                  [-S] [-E] [--source [SOURCE_DIR [SOURCE_DIR ...]]]
                  [-D MACROS]
project.py: error: argument -m/--mcu: foo is not a supported MCU.  Supported toolchains are:
atmelstudio,  coide,        ds5_5,        e2studio,     emblocks,
gcc_arm,      iar,          kds,          lpcxpresso,   simplicityv3,
sw4stm32,     uvision,      uvision4,     uvision5

and
$  python tools/project.py -n foo
usage: project.py [-h] -m MCU -i IDE [-c] (-p PROGRAM | -n PROGRAM) [-b] [-L]
                  [-S] [-E] [--source [SOURCE_DIR [SOURCE_DIR ...]]]
                  [-D MACROS]
project.py: error: argument -n: Program with name 'foo' not found. Supported tests are:
MBED_A1,              MBED_A2,              MBED_A3,
MBED_A4,              MBED_AT30TSE75X,      MBED_A5,
MBED_A6,              MBED_A7,              MBED_A8,
MBED_A9,              MBED_A10,             MBED_A11,
MBED_A12,             MBED_A13,             MBED_A14,
MBED_A15,             MBED_A16,             MBED_A17,
MBED_A18,             MBED_A19,             MBED_A20,
MBED_A21,             MBED_A22,             MBED_A23,
MBED_A24,             MBED_A25,             MBED_A26,
MBED_A27,             MBED_BLINKY,          MBED_BUS,
MBED_BUSOUT,          BENCHMARK_1,          BENCHMARK_2,
BENCHMARK_3,          BENCHMARK_4,          BENCHMARK_5,
PERF_1,               PERF_2,               PERF_3,
MBED_1,               MBED_2,               MBED_3,
MBED_4,               MBED_5,               MBED_6,
MBED_7,               MBED_8,               MBED_9,
MBED_10,              MBED_11,              MBED_12,
MBED_13,              MBED_14,              MBED_15,
MBED_16,              MBED_17,              MBED_18,
MBED_19,              MBED_20,              MBED_21,
MBED_22,              MBED_23,              MBED_24,
MBED_25,              MBED_26,              MBED_27,
MBED_28,              MBED_29,              MBED_30,
MBED_31,              MBED_32,              MBED_33,
MBED_34,              MBED_35,              MBED_36,
MBED_37,              MBED_38,              MBED_39,
CMSIS_RTOS_1,         CMSIS_RTOS_2,         CMSIS_RTOS_3,
CMSIS_RTOS_4,         CMSIS_RTOS_5,         CMSIS_RTOS_6,
CMSIS_RTOS_7,         CMSIS_RTOS_8,         RTOS_1,
RTOS_2,               RTOS_3,               RTOS_4,
RTOS_5,               RTOS_6,               RTOS_7,
RTOS_8,               RTOS_9,               NET_1,
NET_2,                NET_3,                NET_4,
NET_5,                NET_6,                NET_7,
NET_8,                NET_9,                NET_10,
NET_11,               NET_12,               NET_13,
NET_14,               UB_1,                 UB_2,
USB_1,                USB_2,                USB_3,
USB_4,                USB_5,                USB_6,
USB_7,                CMSIS_DSP_1,          DSP_1,
KL25Z_1,              KL25Z_2,              KL25Z_3,
KL25Z_4,              KL25Z_5,              EXAMPLE_1,
EXAMPLE_2,            UT_1,                 UT_2,
UT_3,                 UT_BUSIO,             UT_I2C_EEPROM_ASYNCH,
UT_SERIAL_ASYNCH,     UT_SPI_ASYNCH,        UT_LP_TICKER,
DTCT_1

and
$  python tools/project.py -p -1
usage: project.py [-h] -m MCU -i IDE [-c] (-p PROGRAM | -n PROGRAM) [-b] [-L]
                  [-S] [-E] [--source [SOURCE_DIR [SOURCE_DIR ...]]]
                  [-D MACROS]
project.py: error: argument -p: -1 does not index a test. The accepted range is 0 to 135
The test mapping is:
0:MBED_A1,                1:MBED_A2,                2:MBED_A3,
3:MBED_A4,                4:MBED_AT30TSE75X,        5:MBED_A5,
6:MBED_A6,                7:MBED_A7,                8:MBED_A8,
9:MBED_A9,                10:MBED_A10,              11:MBED_A11,
12:MBED_A12,              13:MBED_A13,              14:MBED_A14,
15:MBED_A15,              16:MBED_A16,              17:MBED_A17,
18:MBED_A18,              19:MBED_A19,              20:MBED_A20,
21:MBED_A21,              22:MBED_A22,              23:MBED_A23,
24:MBED_A24,              25:MBED_A25,              26:MBED_A26,
27:MBED_A27,              28:MBED_BLINKY,           29:MBED_BUS,
30:MBED_BUSOUT,           31:BENCHMARK_1,           32:BENCHMARK_2,
33:BENCHMARK_3,           34:BENCHMARK_4,           35:BENCHMARK_5,
36:PERF_1,                37:PERF_2,                38:PERF_3,
39:MBED_1,                40:MBED_2,                41:MBED_3,
42:MBED_4,                43:MBED_5,                44:MBED_6,
45:MBED_7,                46:MBED_8,                47:MBED_9,
48:MBED_10,               49:MBED_11,               50:MBED_12,
51:MBED_13,               52:MBED_14,               53:MBED_15,
54:MBED_16,               55:MBED_17,               56:MBED_18,
57:MBED_19,               58:MBED_20,               59:MBED_21,
60:MBED_22,               61:MBED_23,               62:MBED_24,
63:MBED_25,               64:MBED_26,               65:MBED_27,
66:MBED_28,               67:MBED_29,               68:MBED_30,
69:MBED_31,               70:MBED_32,               71:MBED_33,
72:MBED_34,               73:MBED_35,               74:MBED_36,
75:MBED_37,               76:MBED_38,               77:MBED_39,
78:CMSIS_RTOS_1,          79:CMSIS_RTOS_2,          80:CMSIS_RTOS_3,
81:CMSIS_RTOS_4,          82:CMSIS_RTOS_5,          83:CMSIS_RTOS_6,
84:CMSIS_RTOS_7,          85:CMSIS_RTOS_8,          86:RTOS_1,
87:RTOS_2,                88:RTOS_3,                89:RTOS_4,
90:RTOS_5,                91:RTOS_6,                92:RTOS_7,
93:RTOS_8,                94:RTOS_9,                95:NET_1,
96:NET_2,                 97:NET_3,                 98:NET_4,
99:NET_5,                 100:NET_6,                101:NET_7,
102:NET_8,                103:NET_9,                104:NET_10,
105:NET_11,               106:NET_12,               107:NET_13,
108:NET_14,               109:UB_1,                 110:UB_2,
111:USB_1,                112:USB_2,                113:USB_3,
114:USB_4,                115:USB_5,                116:USB_6,
117:USB_7,                118:CMSIS_DSP_1,          119:DSP_1,
120:KL25Z_1,              121:KL25Z_2,              122:KL25Z_3,
123:KL25Z_4,              124:KL25Z_5,              125:EXAMPLE_1,
126:EXAMPLE_2,            127:UT_1,                 128:UT_2,
129:UT_3,                 130:UT_BUSIO,             131:UT_I2C_EEPROM_ASYNCH,
132:UT_SERIAL_ASYNCH,     133:UT_SPI_ASYNCH,        134:UT_LP_TICKER,
135:DTCT_1

I'm pretty much done here. @0xc0170 @mlnx @bridadan @bogdanm LGTM?

Looks good. I'm not completely sold on the replacing "-" with "_" or vice versa (I find that it can be more confusing than helpful), but all in all looks good. +1.

(I find that it can be more confusing than helpful)

I'm curious, confusing for the user of the tools or the developer of the tools?

cc @screamerbg

One thing I noticed is if you don't supply -t or -m to build.py, it by default builds the mbed lib for all targets and toolchains. This is actually really awesome! It's basically a release script, but I wanted to make sure that this was intended. If this is unintended then there might be other unintended behavior.
$ python tools/build.py -m K64F

...[SNIP]...

Completed in: (68.20)s

Build successes:
  * ARM::K64F
  * GCC_ARM::K64F
  * IAR::K64F
Build skipped:
  * uARM::K64F
  * GCC_CR::K64F

@bridadan:

One thing I noticed is if you don't supply -t or -m to build.py, it by default builds the mbed lib for all targets and toolchains.

Actually that was the behaviour before all the recent development. I'm pretty sure it was used as a release script at one point.

Instead of predicting what you meant and doing it for you, the parsers now predict and tell you what you should have typed. For example:
$  python tools/build.py -t gcc-arm
usage: build.py [-h] [-m MCU] [-t TOOLCHAIN] [-c] [-o OPTIONS]
                [--source SOURCE_DIR] [--build BUILD_DIR] [--no-archive] [-r]
                [--rpc] [-e] [-U] [-u] [-d] [-F] [-b] [--cpputest] [-D MACROS]
                [-S] [-f GENERAL_FILTER_REGEX] [--cppcheck] [-j JOBS]
                [-N ARTIFACT_NAME] [-v] [--silent] [-x]
build.py: error: argument -t/--tool: gcc-arm is not a supported toolchain. Did you mean GCC_ARM?

Why moving away from autocorrecting name casing? That change seemed like a great way to resolve naming issues between tools.

make.py: error: argument -t/--tool: GCC-ARM is not a supported toolchain. Did you mean GCC_ARM?

Great!
+1

Why moving away from autocorrecting name casing? That change seemed like a great way to resolve naming issues between tools.

Too much magic. Perhaps solve the issue between tools by fixing the tools?

And FYI, here is how git handles it
$ git config -global
error: did you mean `--global` (with two dashes ?)

It doesn't auto-correct it for you. It teaches you by showing the right name, but you have to type it.

No I agree it would be magic to change arguments that are not character-equivalent. But being case insensitive isn't magic.
There's no penalty to avoiding confusion between -t GCC_ARM and -i gcc_arm.

Maybe magic in mbed-cli in particular when the --build option is not specified to test.

build is green; @sg- LGTM.

I think I took it to far by saying that - should be equivalent to '_' where I think case sensitivity should be stripped is generally accepted.

Agree that case sensitivity makes more sense than '-' and '_'

Agree that case sensitivity makes much more sense than '-' and '', but then we should propagate across TARGET, TOOLCHAIN_ and FEATURE_ as well?

I still think - <-> _ is reasonable, but it's debatable.

Agree that case sensitivity makes much more sense than '-' and '', but then we should propagate across TARGET, TOOLCHAIN_ and FEATURE_ as well?

At this point no for backwards compatibility. But the convention for macro names is all uppercase.


[patterns] Function pattern inconsistency between implementation and spec.

Hello, again 😃
Consider:
void main() {
  if (() {} case Function()) {
    print("foo");
  }
}

On DartPad, it prints foo and terminates successfully. However, it looks like the spec does not intend for that to be possible. Function is not a typeIdentifier so objectPatterns do not describe Function() and there's no other pattern that explicitly introduces Function patterns.
 === Analyzer ===
Scan errors: 0
Parse errors: 0
...
        ┃  ┣━ <CaseClauseImpl> [26-41]
        ┃  ┃  ┣━ 'case' [26-30]
        ┃  ┃  ┗━ <GuardedPatternImpl> [31-41]
        ┃  ┃    ┗━ <ObjectPatternImpl> [31-41]
        ┃  ┃      ┣━ <NamedTypeImpl> [31-39]
        ┃  ┃      ┃  ┗━ 'Function' [31-39]
        ┃  ┃      ┣━ '(' [39-40]
        ┃  ┃      ┗━ ')' [40-41]
...
--------------------------------------------------------------------------------
 === ANTLR ===
line 2:17 missing ')' at 'Function'
Errors of type 1: [[@15,41:41=')',<7>,2:27], [@15,41:41=')',<7>,2:27], [@15,41:41=')',<7>,2:27], [@15,41:41=')',<7>,2:27]]
Errors of type 2: [<missing ')'>, Function, (, ), )]
line 2:17 missing ')' at 'Function'
<startSymbol>
┗━ <libraryDefinition>
  ┣━ <metadata>
...

Good catch!
The implementations (the analyzer as well as the common front end) do already support a pattern of a form like Function() or Function(: var toString), and they treat it as an <objectPattern>. This treatment is also consistent with the treatment of Function as the name of a class, and the treatment of object patterns otherwise.
I created a PR that introduces support for this construct into the grammar: https://github.com/dart-lang/language/pull/3496, and a spec_parser update: https://dart-review.googlesource.com/340181.

Not sure we should allow it, actually.
We generally say that when Function is followed by (, it's a function type, not a reference to the name Function exported by dart:core.
Allowing Function() here to refer to that Function name would be inconsistent.
And it's not like there are any useful getters on Function where it would be necessary to use an object pattern.
To match any function, you can do Function _ instead.
I'd rather that we made it an error to write case Function():, for great consistency.

We generally say that when Function is followed by ( ...

I don't think we've specified a rule of that nature. We did discuss the need to disambiguate, but I think we concluded that it wasn't needed, and I can't immediately find anything like a specified disambiguation mechanism. Also, Dart.g doesn't have any special treatment of FUNCTION followed by (. Are you aware of a specified rule about this?
Next, in an object pattern like C(...), C is a reference to a declaration that introduces a type (could be a class, mixin, typedef, extension type, enum, etc.). We do treat Function as the name of a class when it is used an a context like this: Function foo() => print;, and this does make MyClass(:var toString) similar to Function(:var toString): In both cases, we're denoting a type and then specifying a sequence of field patterns. Finally, in the situation where a pattern is expected, the grammar does not allow a function type to occur.
So we may be able to say that Function followed by ( implies function type, but we can also list several reasons why there is no such rule, or it isn't applicable right here.

I'd prefer Dart not to support constructs that essentially add no value. As Lasse said, such a Function objectPattern isn't useful, and I agree.
The ability to extend Function types was deprecated for a long time and removed recently, and I think it would make sense not to support Function objectPatterns, too.
In system libraries, the implementation already ignores the fact that classes can't be called Function. The fact that objectPatterns support Functions appears to be an accident.

I'd prefer Dart not to support constructs that essentially add no value.

I can follow that, but I'm not convinced that it is a good idea to create exceptions in order to avoid having support for certain things, just because they appear to add little value.
We allow Function to be used as the name of a class in a number of situations, including Function foo() => print; and List<Function>, but we will now insist that we cannot do the same thing in an object pattern, in spite of the fact that it isn't even difficult to parse? And we will even pay for it in terms of a breaking change process? ;-)
Note that we can always emulate Function(...) as an object pattern by declaring a different name for it:
typedef MyFunction = Function;

void main() {
  Object? x = print;
  switch (x) {
    case MyFunction(): x('Hello again!');
    case Enum(): print(x.index);
    case var other: ...;
  }
}

That exception already exists for void

It is true that void gets a very special treatment (it's a reserved word, no other type is a reserved word), and it hasn't been fully "normalized" in the grammar. However, that's a consequence of the history, it isn't an exception that the language team wanted and subsequently introduced, it's an oddity that had already existed for several years when, for instance, I joined Google.

I think we can't remove or not introduce any exceptions unless we remove support for dynamic patterns & Function patterns, or keep Function patterns and add void patterns.

Well, I think a pattern like void(...) is going to be difficult to use in a meaningful way: void is a top type, which means that testing whether something is void (except that we can't even do that unless we use a type alias) is true for every object whatsoever; hence, void(...) isn't capable of selecting any set of objects, it just accepts all objects. Next, any void pattern with field patterns (like void(hashCode: int h)) should arguably be a compile-time error, because it is doing the same thing as int h = s.hashCode; where s is the scrutinee object, and s has the static type void, and that is a compile-time error. In other words, a void pattern can basically never be anything other than a no-op (unless we introduce a bunch of other exceptions to the treatment of void typed objects).
In contrast, Function() will match all function objects, and that could actually be useful now and then. Also, it seems natural to me that you should be able to do things like this:
SomeType foo(Object arg) => switch (arg) {
  case Function(): ...; // Deal with functions.
  case String(): ...; // Deal with strings.
  ... // Deal with other things.
};

This works today. I'm just saying that I don't think it's going to be an improvement for anyone if we turn that usage into a compile-time error.
Granted, you would only encounter this kind of code in a "rather dynamically typed" library, because we won't know anything about which kind of function we're working with unless we proceed to test arg is void Function(int) or some other specific function type. However, I don't think we should specifically aim for making that style of code less convenient to write (and read).
For me, this points in the direction of preserving the support for Function(...) patterns, and not introducing void patterns. One might say that we'd get rid of an exception if we did add support for void patterns, but I'm not convinced that void patterns are useful anyway, whereas Function(...) patterns can actually make sense.

I do.
Whenever you see Function() I would like it to mean the same thing.
It's the design rule of "similar syntax means similar things".
Function() should always be a function type, and should not be allowed where other function types are not allowed.
If we managed to actually disallow it in patterns, and not allow it by accident, we should hold onto that.

OK, so A() should always be an instance creation and never an object pattern? ;-)


pgupgrade: Ensure that old pg deployments are terminated before upgrade job is run (PROJQUAY-8092)

/cherry-pick redhat-3.13