Digits with Caffe on CPU

Hello,
Not sure where this information belongs, just wanted to share.
I wanted to use digits on my MacBook Pro 13 without GPU.
(also have another system, digits+coreos on top of gtx1070)
Pulled the latest docker image 6.0
meanwhile searching around the net how to run digits with caffe on CPU.
The suggestions that i found in multiple places was to compile caffe on OSX,
then use the CAFFE_ROOT environment variable to point it to my local build of caffe.
I did not want to install caffe on my MacBook, because i want to use docker for these kind of situations.
Looked at caffe cpu Dockerfile
and compared it with digits 6.0 Dockerfile
to see what is the difference in parameters for cmake.
Fired up VSC, created a Dockerfile then pasted the contents of digits 6.0 Dockerfile.
Changed the line 70:
cmake -DCMAKE_INSTALL_PREFIX=/usr/local/caffe -DUSE_NCCL=ON -DUSE_CUDNN=ON -DCUDA_ARCH_NAME=Manual -DCUDA_ARCH_BIN="35 52 60 61" -DCUDA_ARCH_PTX="61" .. && \
to
cmake -DCMAKE_INSTALL_PREFIX=/usr/local/caffe -DCPU_ONLY=1 .. && \
and run docker build.
While docker was building, prepared a small dataset for object detection.
After docker build, run the image, configured the dataset and model parameters in digits,
then started training.
Caffe was running in CPU mode, training my model.
Question, can this information be included in the docs?
For people who want to use digits with caffe on cpu for whatever reasons.
Or there are complications when doing so and i'm not aware because not knowing enough.
Thanks.

@laszl0 Hello, I followed the above instructions and build the docker image. I am new to docker environment. Upon completion, tried the following commands to run Digits:
docker run --runtime=nvidia --name digits -d -p 5000:5000 nvidia/digits
It worked. Then, tried the following to get see formerly-trained models within this Digits;
sudo nvidia-docker rm digits
sudo nvidia-docker run --runtime=nvidia --name digits -d -p 5000:5000 -p 6006:6006 -v /media/user01/bd00e3c9-7644-472e-b940-d88866d46538/home/user01/digits/digits/jobs:/jobs nvidia/digits:6.0

It also succeeded, old jobs were available in Digits web interface. Tried detectnet object detection inference while "nvidia-smi -l 1" running, unfortunately all process was still taking place in GPU. And, digits main page was saying 1/1 GPU available. Here is the output to $ sudo nvidia-docker images  command:
REPOSITORY          TAG                              IMAGE ID            CREATED             SIZE
<none>              <none>                           a2e8b6178bad        2 hours ago         2.37GB
<none>              <none>                           a891b2b8bd61        3 hours ago         2.99GB
<none>              <none>                           1eab0a961686        3 hours ago         1.38GB
nvidia/cuda         latest                           9337ecb4311e        12 days ago         2.24GB
nvidia/cuda         8.0-cudnn5-devel-ubuntu14.04     9ef12db61cfd        12 days ago         1.89GB
nvidia/cuda         8.0-cudnn5-runtime-ubuntu14.04   b3c5f37b54b1        12 days ago         983MB
ubuntu              14.04                            8cef1fa16c77        2 weeks ago         223MB
hello-world         latest                           e38bc07ac18e        4 weeks ago         1.85kB
nvidia/digits       6.0                              fb4bfabb5acd        7 weeks ago         2.8GB
nvidia/digits       latest                           fb4bfabb5acd        7 weeks ago         2.8GB

First three repositories with no name was produced after (your) docker build command, as far as I know. I am not sure what they are. I am using ubuntu 16.04, by the way.
All I need is, CPU-only inference to be able to make use of larger memory, for detectnet. Can you or anyone interested tell me what I am doing wrong?

Hi @ontheway16 ,
upon reading what you did, i would like to suggest that,
the first part of command
sudo nvidia-docker run --runtime=nvidia ....
( this command is for running docker containers with Nvidia GPU support )
should be just
sudo docker run .....
because you want to run the image on CPU.
If that does not work,
what i wrote in the post, the docker image is available here on docker hub,
if someone wants to use the image without doing the modifications that i have explained.
The command that i have used:
sudo docker run --name digits -d -p 5000:5000 -p 6006:6006 -v /Users/laszl0/Documents/test-digitsv6/jobs:/jobs laszl0/nvidia-digits-caffe-cpu
(And yes in the Info tab, when running digits, it will still write 'Caffe flavor: Nvidia' , but that should not be a problem)
This solution worked for me.

@laszl0 Thank you very much for the reply. Before start trying, I just want to verify my idea on using CPU-inference due to larger memory. My idea was to train the model with digits-GPU (which I did already), then apply inference to larger images (normally not fit into GPU memory) using digits in CPU mode. Is it applicable or am I on the wrong route?

@laszl0 I appreciate for such a detailed answer.
Despite years passed since the introduction of deep learning frameworks, object detection systems are only focusing to 'speed' which is something I dont need, at all.. My needs are centered around two things; Highest accuracy, and inference of small images (like 40-70 pixels WH) in very high resolution images (20-80 MegaPixels range, DSLR digital cameras are capable to produce this).
I was thinking of alternatives to 'how we can apply inference to very large images' for literally several months and the only solution possible come up as 'crop->infer->stitch', I guess you know what I mean. Unfortunately, I have failed to detect a single example for such a process, in object detection solutions, something hard to understand, since there are no cameras left around with 640x480 or 1280x800, etc.  resolutions, inference parts of these systems should already be started to be designed to handle resolutions around 10MP (thinking all mobile phones)...
I have pulled your docker image, and succesfully started the digits in CPU mode.
I want to share results of my experiments;
I choose two models trained last year, 1st one was trained with 1248x384x3 images, with stride 8;
Inference results;
1248x384: success.
2560x456: success.
2560x912: success.
Then I switched to second model which was trained with 5120x896x3 images, stride 16, results;
2560x456: success.
2560x912: success.
5120x912: success.
5120x1824: success.
5120x2048: success.
5120x2296: success.
5120x3000: success.
5120x3200: success.
With the last one, I hit to my swapspace limit (poorly defined as 15GB, about 13.7GB used, along with 15.4GB free RAM. All filled in the last example)
Detection accuracy was as good as regular sized inferences, but do not know what happens If I further extend the image length, just ordered a couple of 16GB sticks to see this ;)
I am not sure how this happens since its opposite to your descriptions above, but I feel the "Do not resize input image(s)" option at inference page of Digits is responsible for this. Or maybe I didnt hit to actual walls of network design, yet...
I am really thankful for the cpu mode you provided, I hope one day someone starts considering on an allinone way for applying inference to todays image dimensions.

@ontheway16 I might have misunderstood you...sorry...was to generic...
Thanks for sharing the idea that you are working on. :)
What i know is, progress is made on this path, it's just that it's internal most of the time,
so we have to wait until results are made public.
For your mobile example, i would say, it makes sense to not use the camera output of 10+MP,

Sending that image to a server for inference, takes time and bandwidth costs.
Storing it locally and then processing offline sometime in future, problematic, not user friendly,
(maybe for specific use case might work).
Mobile resources(CPU and RAM) for inferencing locally are limited, instead it's better to resize the image to (640x480 or 1280x800) and then run inference on a model that's fits into the limits.
So using the raw images of a mobile camera, use cases are very specific, my opinion.

So coming back, i understand that there are raw images (20-80MP) for some problems,
but usually some of the questions before starting to solve the problem are:

what do you want to classify, detect, segment in these images?
can you use today's solutions to solve your problem, or have to invent a new solution?

About your experiment results, i'm very interested in the further progress (16GB sticks solution).

I am not sure how this happens since its opposite to your descriptions above, but I feel the "Do not resize input image(s)" option at inference page of Digits is responsible for this.

Indeed.

what do you want to classify, detect, segment in these images?
can you use today's solutions to solve your problem, or have to invent a new solution?

@laszl0 I forgot to answer these two. I need to detect and count small objects in very large images, and I believe todays sw/hw tech is more then enough for this, only thing is, as you pointed out above, not released to public I believe. The only similar implementation for processing large images in parts and stitch results is in https://github.com/thstkdgus35/EDSR-PyTorch
The author is using a "chop" option to apply super resolution to very large images, parts are stitching back together for the final image, not the same thing with object detection of course, but may give some ideas.

@ontheway16 congrats on managing with docker 👍 .... don't worry about digits still shows 1/1 GPU and about the docker image being based on ubuntu 14.04 and your OS is 16.04...no worry for now.
regarding the parameters for "param_str", did you adjusted also the rest of the values?
sample:

param_str: "5120, 3200, 16, 0.06, 3, 0.02, 10"

Just asking because as explained the param_str are passing the values to opencv groupRectangles, which in turn needs proper values because as it's pointed out here.
I have seen implementation documented, where the network architecture is created in a way that it crops the big image then does inference and then puts it back together at the end in one forward pass. So there are solutions to this, just not public, and it was created with tensorflow if remember correctly.
Great job doing the Alp's tool and even the custom solution for large images 👍 .

I hope, one day, the steps above can be done within DIGITS framework. Until that day, enjoy the toolset and good luck !

You can fork DIGITS and make the modifications in DIGITS yourself if you want it, just saying.
About using CPU features, you have to build the docker image yourself on your machine, just create a folder, drop there my Dockerfile and build the docker image, then it will take advantage of your cpu features. Because my only reason to go into that direction is of your small objects, if you agree...
Definitely is a pity to have so many 1080ti's lying and not doing anything, so what do you think about the second approach they describe in this article?

@laszl0

regarding the parameters for "param_str", did you adjusted also the rest of the values?

Hi, I have tested many different numbers and the level of success is changing depending on objects' pixel dimensions (I am taking images at fixed distance so my objects are almost always at same dimension, a perfect target for a detector :) ) See an example below;

param_str: "5120, 864, 16, 0.01, 1, 0.020, 10, 2"

I guess this one was for two classes.

I have seen implementation documented, where the network architecture is created in a way that it crops the big image then does inference and then puts it back together at the end in one forward pass. So there are solutions to this, just not public, and it was created with tensorflow if remember correctly.

I know such projects exist, like the one below,
https://blogs.flytbase.com/arabian-oryx-detection-counting/

You can fork DIGITS and make the modifications in DIGITS yourself if you want it, just saying.

I wish I could but my knowledge, on modifying a network architecture is very limited, currently I just modify some parameters etc, therefore I have no high hopes for that route. I can plan the function very well, but implementation is weak here :) Data science || programming is not my background.

Great job doing the Alp's tool and even the custom solution for large images +1 .

Thank you, I am putting all the code there, in hope of helping out someone else, and according to downloads, I think it did..

Definitely is a pity to have so many 1080ti's lying and not doing anything, so what do you think about the second approach they describe in this article? Because my reason to go into that direction is of your small objects and using the Ti's, if you agree...

Yeah, I did ask this when I first discovered object detection thing, and at that time, Greg offered segmentation, but the problem is, I am not interested with locational/area information of objects detected, I only need numbers per object class, so, how many of them per image.
Therefore segmentation is no or little help for me since some of the same-class objects are touching each other and segmentation will see them as one, while detectnet is (weakly) capable to seperate them, and results are countable.
While 80Ti's are working very well for training purposes, blaming Nvidia since they are not coming with a solution to high resolution images, like unifying RAMs of connected GPUs, etc. Should I wait for 32GB Tesla V100 price drop, to make use of GPU inference?
Nowadays I am thinking on an intermediate solution, cropping images and saving them to RAMdisk, then stitch together with some faster NMS, like the one below,
https://www.pyimagesearch.com/2014/11/17/non-maximum-suppression-object-detection-python
or this one;
https://www.pyimagesearch.com/2015/02/16/faster-non-maximum-suppression-python/
But of course non of these replaces natural code. And physically saving each cropped part means a lot of PNG encoding, which is taking ton of time, killing the GPU advantage. I even offered some small payment in hire sites for a singleshot solution, but seems no one interested. Therefore, CPU inference currently is invaluable to me. Its too bad Nvidia integrated Tensorflow into Digits, but they did not provided an official example for object detection, I am stuck with detectnet (Inception v2 I guess?). I even not sure if Nvidia keeps developing Digits..
Regards,
Alper

@laszl0
5120x6000, success.
3x16gb sticks plus 15gb swap, was barely enough for 40MP image. swap was @ 13.7GB so 4x16gb will make it I guess.

@ontheway16 Lot of info, thanks! And sorry was busy and still busy these days, but congrats with the 40MP image, keep it up, i want to see where all this goes :), I'l be back with a couple of answers in the following days!

@laszl0 Now I am looking to find a way to modify two lines in two files, in the docker image.
https://github.com/NVIDIA/caffe/blob/v0.15.13/python/caffe/layers/detectnet/clustering.py#L8
https://github.com/NVIDIA/caffe/blob/v0.15.13/python/caffe/layers/detectnet/mean_ap.py#L5
I need to make both of them 2000, but could not find how to do it while things are in docker...?

@ontheway16 Hi, about the other topics, i have an idea for big images, in the next weeks i will develop it, once i finish my current work.
About your caffe problem, if you have not figured it out already, check out this file and the line #Edit your caffe files. There you can edit other caffe files.
You can use that dockerfile, it's the same as the last one, except i added these lines.
All the best!

Great to hear this. I have several ideas for it too,but again, implementation. If we communicate, I believe we can develop a good solution to it.
Meanwhile;
$ sudo docker ps -a
CONTAINER ID        IMAGE                            COMMAND              CREATED             STATUS                  PORTS                                            NAMES
d75dde1216f9        laszl0/nvidia-digits-caffe-cpu   "python -m digits"   30 minutes ago      Up 30 minutes           <IP_ADDRESS>:6006->6006/tcp, <IP_ADDRESS>:5005->5000/tcp   digits
ef364c773449        hello-world                      "/hello"             3 days ago          Exited (0) 3 days ago                                                    kind_darwin

sudo docker exec -it d75dde1216f9 sh

cd /usr/local/python/caffe/layers/detectnet

sed -i "/MAX_BOXES = 50/c\MAX_BOXES = 2000" clustering.py

sed -i "/MAX_BOXES = 50/c\MAX_BOXES = 2000" mean_ap.py

exit

solved the problem without building it again.

@ontheway16
Indeed, sudo -ing into the docker container is easier...my bad!
Sounds good, let's try to come up with something together!

@laszl0 Hi, it appears above MAX_BOXES commands are not permanent, so its better to build a new one with your mods.
What else, I made some modifications in network and now training/inferencing with stride 8 instead of 16.
The cost is a lot. Applying inference to a 40MP image is filling 48GB RAM plus 79GB swap space, at stride8, took 25mins :) . Therefore, I have to admit that its not practical to try to infer whole large image at once, even by CPU.
This is taking us back to applying gpu based inference to cropped parts.
I can imagine a process like below;

crop a part thats capable to fill the mem. of GPU during inference (predefined crop size here), and store crop offsets.
feed the cropped part to first available GPU for inference
collect detected bounding box coordinates, apply crop offset and store in an array etc.
goto 1 until large image finishes.
apply non-maximal supression to b_boxes to get rid of duplicate boxes.
save bboxes of detected object class to disk, in Kitti format.

I think, very roughly, it should be something like this..?

Oh... indeed that's a lot of ram and time 👍 .
I was looking into the scenario that you described, how far did you got with implementation.
Could you drop me an email instead, thanks!

@laszl0 Please send me an email since I dont know if its possible to find mail info from here.  mail is   al,p9,7 at ho,tma,il.com  please remove all commas.


Sample style sheets are uncompressed

Another drive-by report:
The /dist/css/*.min.css files suggest minified output. The style sheets all seem processed but uncompressed. Am I missing something or should these indeed be minified?

There's probably a Sass compiler option that can be used to compress whitespace:
https://github.com/blinkkcode/amagaki-starter/blob/experimental/webpack.config.js#L19
That said I'm not sure how much it will help in practice; any server worth its salt should be gzipping the content so any repeating whitespace should likely be compressed over the wire. That said I think it's still a good thing for our starter to have so why not. :)

Maybe I miss something else, but why not export the style sheets as foo.css then? It’s probably the “.min” that this is exported to by default that’s confusing then.

This was fixed with the addition of our production webpack rule. Now, when npm run build is invoked, the prod build is run, and the generated CSS files are fully minified.
https://github.com/blinkkcode/amagaki-starter/commit/ca7dab941ed2514d998e5ff5767448f6b0602cea


fix(utils): Check for the presence of the rule's operator

Extension of the util/crs-rules-check/rules-check.py script to check for the presence of the rule's operator.

Looks good to me. Did not test the implementation though.
@RedXanadu : You OK with the wording?

Made some suggested changes for clarity and consistency.

Thanks guys both of you. I added all suggestions, I think the PR is ready to merge.

Thank you @airween. Also to @theseion and @RedXanadu for the constructive feedback. Merging now.


How to fail execution when Kafka server is not available?

In our project, we need to periodically fetch data from the 3rd party by many threads and to push this data to the Kafka then. And if Kafka server is not available at the moment, the flow should be terminated, the fetched data should be lost and re-fetched during the next scheduled execution. Also, there's a need to use transaction management, because we need to send the batch of messages to Kafka to different topics. And if one message is not sent, all others should be rolled back.
We've been experiencing an issue with terminating execution if Kafka Server is not available. When transaction management is disabled, everything works fine and we're getting org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
But with transaction management enabled, Kafka producer tries to reach the server infinitely and all threads that initiate pushing messages to Kafka sticking. We've tried different settings to make it fail after some time of unavailability, but it didn't help.
Is there some combination of settings that make Kafka Producer fail after some timeout when Kafka server is not available and the transaction management is enabled?

First of all this is question and we don't manage questions here on GitHub. Consider to move it to some community forum, e.g. StackOverflow.
Another concern that it doesn't sound like a Spring Kafka problem, but Apache Kafka per se and we don't provide here support for it.
We may answer some Kafka related questions, but if it doesn't fall to deep and doesn't expose something what is fully out of Spring control.
Thank you for understanding.
Maybe transaction.timeout.ms property can help you on the matter though?

It's a bug in the kafka client see KAFKA-6446. It will be fixed in the 2.0.0 client.

All 1.x users are recommended to upgrade to at least 1.3.5 because it has a much simpler threading model thanks to KIP-62.
2.1.7 is the current release and has significant new features above 1.x.


[Bug]: OBSNotifier does not open and crashes silently in background

Commit or version
1.3.2
On which operating systems the error occurs
OS: Windows 10 22H2 (19045.5131) with all updates installed along with .NET Framewework 4.8
OBS: 30.2.3 (64 bit) with websockets enabled
PS C:\Users\PC> Get-ChildItem 'HKLM:\SOFTWARE\Microsoft\NET Framework Setup\NDP' -Recurse | Get-ItemProperty -Name version -EA 0 | Where { $_.PSChildName -Match '^(?!S)\p{L}' } | Select PSChildName, version

PSChildName                      Version
-----------                      -------
v2.0.50727                       2.0.50727.4927
v3.0                             3.0.30729.4926
Windows Communication Foundation 3.0.4506.4926
Windows Presentation Foundation  3.0.6920.4902
v3.5                             3.5.30729.4926
Client                           4.8.09037
Full                             4.8.09037
Client                           <IP_ADDRESS>

Issue description
Hi, I wanted to use this app once again as I reinstalled my Windows but whenever I open OBSNotifier.exe v1.3.2, the app silently crashes in the background without any prompt or any helpful logs.
[11/16/2024 19:26:10]	// Time format - MM/dd/yyyy HH:mm:ss

As I've used this in the past on my previous Windows installation without any issue, I tried to open older version 1.3.0 but the result was the same although, this time I saw something in the error logs. I'm not sure if this is related.
Logs:
[11/16/2024 19:22:25]	// Time format - MM/dd/yyyy HH:mm:ss
[11/16/2024 19:22:25]	--- The program crashed due to an exception ---
[11/16/2024 19:22:25]	Exception:
[11/16/2024 19:22:25]-	Sequence contains no elements
[11/16/2024 19:22:25]-	StackTrace:
[11/16/2024 19:22:25]-	   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
[11/16/2024 19:22:25]-	   at OBSNotifier.App.SelectLanguageOnStart(CultureInfo start_ui_info)
[11/16/2024 19:22:25]-	   at OBSNotifier.App.Application_Startup(Object sender, StartupEventArgs ee)
[11/16/2024 19:22:25]-	   at System.Windows.Application.OnStartup(StartupEventArgs e)
[11/16/2024 19:22:25]-	   at System.Windows.Application.<.ctor>b__1_0(Object unused)
[11/16/2024 19:22:25]-	   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
[11/16/2024 19:22:25]-	   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
[11/16/2024 19:22:25]-	   at System.Windows.Threading.DispatcherOperation.InvokeImpl()
[11/16/2024 19:22:25]-	   at System.Windows.Threading.DispatcherOperation.InvokeInSecurityContext(Object state)
[11/16/2024 19:22:25]-	   at MS.Internal.CulturePreservingExecutionContext.CallbackWrapper(Object obj)
[11/16/2024 19:22:25]-	   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[11/16/2024 19:22:25]-	   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[11/16/2024 19:22:25]-	   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
[11/16/2024 19:22:25]-	   at MS.Internal.CulturePreservingExecutionContext.Run(CulturePreservingExecutionContext executionContext, ContextCallback callback, Object state)
[11/16/2024 19:22:25]-	   at System.Windows.Threading.DispatcherOperation.Invoke()
[11/16/2024 19:22:25]-	   at System.Windows.Threading.Dispatcher.ProcessQueue()
[11/16/2024 19:22:25]-	   at System.Windows.Threading.Dispatcher.WndProcHook(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
[11/16/2024 19:22:25]-	   at MS.Win32.HwndWrapper.WndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
[11/16/2024 19:22:25]-	   at MS.Win32.HwndSubclass.DispatcherCallbackOperation(Object o)
[11/16/2024 19:22:25]-	   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
[11/16/2024 19:22:25]-	   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
[11/16/2024 19:22:25]-	   at System.Windows.Threading.Dispatcher.LegacyInvokeImpl(DispatcherPriority priority, TimeSpan timeout, Delegate method, Object args, Int32 numArgs)
[11/16/2024 19:22:25]-	   at MS.Win32.HwndSubclass.SubclassWndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam)
[11/16/2024 19:22:25]-	   at MS.Win32.UnsafeNativeMethods.DispatchMessage(MSG& msg)
[11/16/2024 19:22:25]-	   at System.Windows.Threading.Dispatcher.PushFrameImpl(DispatcherFrame frame)
[11/16/2024 19:22:25]-	   at System.Windows.Threading.Dispatcher.PushFrame(DispatcherFrame frame)
[11/16/2024 19:22:25]-	   at System.Windows.Application.RunDispatcher(Object ignore)
[11/16/2024 19:22:25]-	   at System.Windows.Application.RunInternal(Window window)
[11/16/2024 19:22:25]-	   at System.Windows.Application.Run(Window window)
[11/16/2024 19:22:25]-	   at OBSNotifier.App.Main()

Steps to reproduce

Open OBSNotifier.exe > App silently crashes in background

I've now also checked EventViewer and upon opening 1.3.2, the event viewer shows the same log I've shared for 1.3.0 above although the log in %APPDATA%/OBSNotifier/logs is empty.

I've now also checked Event Viewer and upon opening 1.3.0 under the Windows Logs>Application I was able to see the same error log as I shared above for 1.3.0 so the issue is related.
For some reason this error is not being shown in the %APPDATA%/OBSNotifier/logs for 1.3.2

all updates installed along with .NET Framewework 4.8

My program is actually targeting 4.7.2.
https://github.com/DmitriySalnikov/OBSNotifier/blob/04937bfe7338b943f0d495acc8fcdafed59ec5e1/OBSNotifier/OBSNotifier.csproj#L11

[11/16/2024 19:22:25]	Exception:
[11/16/2024 19:22:25]-	Sequence contains no elements
[11/16/2024 19:22:25]-	StackTrace:
[11/16/2024 19:22:25]-	   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
[11/16/2024 19:22:25]-	   at OBSNotifier.App.SelectLanguageOnStart(CultureInfo start_ui_info)
[11/16/2024 19:22:25]-	   at OBSNotifier.App.Application_Startup(Object sender, StartupEventArgs ee)

Based on these logs I will assume that your language is not added to the list of supported localizations.
https://github.com/DmitriySalnikov/OBSNotifier/blob/04937bfe7338b943f0d495acc8fcdafed59ec5e1/OBSNotifier/App.xaml.cs#L375-L380
I'll fix that soon.

Can you check this build?

My program is actually targeting 4.7.2.

Yeah, I noticed. Although having a newer minor version shouldn't be a problem.

Based on these logs I will assume that your language is not added to the list of supported localizations.

Also, yes. I didn't submit any translation yet :)
Although, I thought that might be an issue, when I switched my Windows to English as a primary language, the app still didn't open without this fix.

Can you check this build?

You are awesome! :) Now it starts without any issue. Thanks for the quick fix!

Now it starts without any issue.

I hope the program restarts and the language changes without problems too?
If so, I will prepare and release this version.

Yes it does, already tried to auto start it along OBS. No issues here :)


Env configuration for resource quota/limit are ignored

Describe the bug
None of the container platform resources are configured with any limit/quota.
To Reproduce
Steps to reproduce the behavior:
$ helm install eapfoo --debug -f app.yaml -f values.yaml -n default --repo https://jbossas.github.io/eap-charts eap-xp4
Please include if possible the debug output to get the values provided to the chart with
Here we can see the deployer configured with cpu 2000m.
$ helm install eapfoo --debug -f app.yaml -f values.yaml -n default --repo https://jbossas.github.io/eap-charts eap-xp4 |grep -B 5 cpu
install.go:194: [debug] Original chart version: ""
install.go:211: [debug] CHART PATH: /home/whitingjr/.cache/helm/repository/eap-xp4-1.0.0.tgz

client.go:133: [debug] creating 8 resource(s)
  uri: https://github.com/hpehl/first-responder-demo.git
deploy:
  containers:
  - resources:
      requests:
        cpu: 2000m
--
  - name: MP_MESSAGING_CONNECTOR_SMALLRYE_KAFKA_BOOTSTRAP_SERVERS
    value: frdemo-kafka:9092
  maxReplicas: 10
  metrics:
  - resource:
      name: cpu
--
  uri: https://github.com/hpehl/first-responder-demo.git
deploy:
  containers:
  - resources:
      requests:
        cpu: 2000m
--
      port: admin
    initialDelaySeconds: 60
  maxReplicas: 10
  metrics:
  - resource:
      name: cpu

Expected behavior
At least one resource with a cpu 2000m limit.
Component Versions(please complete the following information):
$ helm version
version.BuildInfo{Version:"v3.11", GitCommit:"", GitTreeState:"", GoVersion:"go1.19.6"}
$ oc version
Client Version: 4.12.0-202304070941.p0.g31aa3e8.assembly.stream-31aa3e8
Kustomize Version: v4.5.7
Server Version: 4.10.43
Kubernetes Version: v1.23.12+8a6bfe4
$

Additional context

@whitingjr just to be sure, the app.yaml you showed is not properly indented. Is it the actual YAML you are using or was it a copy/paste issue when you created the gist?

Great advice and thank you for the pointers.
Yes these are good candidates for RFEs.

@jmesnil I created #50 . Is this the RFE candidate you had in mind ?


MAINT: Clean up whats_new and doc versions

Closes #11885
Closes #10932
Closes #10789

Refactor whats_new to show latest changes and link to all others
Refactor rst_prolog so we don't need to .. include:: links.inc and .. currentmodule:: mne all over the place
Add links to old doc version archives
Streamline links.inc to be much shorter and remove a lot of stuff we don't use (didn't check everything but checked some stuff at least)

@drammock let me know what you think. @sappelhoff also feel free to look since you had a preference for the long, continuous whats_new.html in #10789 . Will post CircleCI links once they render.
WIP until approved, then I'll:

[ ] revert the link to the versions.json from this branch
[ ] merge
[ ] remove 0.11 through 0.20 folders and versions_temp.json from mne-tools.github.io
[ ] update release Wiki with new instructions

whats_new is now a landing page
zip links to old docs

This almost works:

But it links to https://mne.tools/dev/old_versions.htmlindex.html (note the added index.html). @drammock in theory it would be nice if pydata-sphinx-theme had some logic to say whether or not to add this. But in the meantime it's easy enough for me to move this to a doc/old_versions/index.rst and link to dev/old_versions/ as the URL for the older ... one.

But it links to https://mne.tools/dev/old_versions.htmlindex.html (note the added index.html). @drammock in theory it would be nice if pydata-sphinx-theme had some logic to say whether or not to add this. But in the meantime it's easy enough for me to move this to a doc/old_versions/index.rst and link to dev/old_versions/ as the URL for the older ... one.

ouch that is not pretty.  There are some changes to version switcher JSON in the dev version of the theme, so there's a chance it's already fixed but I'd have to check. For now please do switch so that it lives mne.tools/dev/old_versions/index.html
Relatedly, do we need to specify a numeric version for "older..."? I think it might work to just say the version is "old" (but again, maybe only with dev version of theme).
FYI I'm trying to get a new version of the theme released soon so feel free to just get it working and don't worry about testing against dev version of theme; I'll be doing that soon enough anyway.

For now please do switch so that it lives mne.tools/dev/old_versions/index.html

Done!

Relatedly, do we need to specify a numeric version for "older..."? I think it might work to just say the version is "old" (but again, maybe only with dev version of theme).

I actually kind of like the idea of changing it to ≤ 0.20 ..., it's more explicit. I'll make that change

Comments addressed and I wanted to get the mne-tools.github.io archiving done to hopefully speed up deployments (one timed out recently :scream: ) so I self-merged, happy to continue tweaking in follow-up PRs (or review from others).

And I'm a bit confused by the fact that version 1.3.1 is shown in the new changelog (notice the patch version increment):

Whereas all other options are without patch increments (even if such versions were released, see: https://github.com/mne-tools/mne-python/releases).

Other than that, and the comment about the error 404 above, I am happy with the changes here. thanks a lot @larsoner!

I get this for all versions except dev:

The version dropdown it works for me (on dev and stable):
https://mne.tools/0.21/index.html
As does the whats_new:
https://mne.tools/dev/changes/v0.20.html
But it does look like there are corner cases:

If you are on a very old version like https://mne.tools/0.21/index.html and try to get to 0.11 it does 404
If you are on a page like https://mne.tools/dev/changes/v0.20.html and then try to nav to an older version, you get a URL like https://mne.tools/0.24/changes/v0.20.html that didn't exist back then.

The first one we can fix with trivial redirect index.htmls to the old_versions page, I'll do that manually in mne-tools.github.io now.
The second one is hopefully low impact since I think if people want docs like 0.22 they'll go there first from the landing index.html rather than trying to go from a nested page like changes/v0.20.html. But it's actually a problem in main even before this PR, so I opened https://github.com/mne-tools/mne-python/issues/11893

Okay, works for me as well now -- thanks for the explanations.
However what about this odd-one-out item?

And I'm a bit confused by the fact that version 1.3.1 is shown in the new changelog (notice the patch version increment)

In my opinion either:

this should be 1.3.0
all other "patch releases" should be documented in the changelog (both heading and content) as well (e.g., 1.4.0 should actually be 1.4.2)

... snuck 1.3.1 cleanup into https://github.com/mne-tools/mne-python/pull/11890 since that's marked for merge-when-green


unable to parse package name "libxgboost-0.6+218_omp_0.tar.bz2"

Hello. I get an error when parsing repodata.json of a channel (internal channel at work). I already posted an issue to rattler-build here https://github.com/prefix-dev/rattler-build/issues/944 and @wolfv suggested I should post here instead. The error message was:
ERROR resolve_conda{group=default, platform=linux-64}: rattler_repodata_gateway::gateway: error=failed to parse repodata.json
  failed to parse repodata.json
    invalid filename at line 38856 column 38

and the snippet below shows the entry starting at the 38856'th line (or 'line 38855' if you call the first line 'line 0'). I don't think the error refers to the previous line though as it's only a few characters long (a few spaces plus a close bracket). We also have noarch and win-64 sections in the channel, but the repodata.json files for those are shorter than 38856 lines so can't be the cause of the error.
    },
    "libxgboost-0.6+218_omp_0.tar.bz2": {     <---- Line 38856
      "build": "0",
      "build_number": 0,
      "depends": [],
      "extra": {
        "git": {
          "branch": "HEAD",
          "commit": "2715bae",
          "sha": "2715baef64c07546314977caf858d1cce1043b66"
        },
        "package": {
          "md5": "43b62a5ba9470354c6244353bfcad65d"
        },
        "system": {
          "cwd": "/some/path/to/a/dir",
          "node": "name.company.com",
          "os": "Linux",
          "prefix": "/some/path",
          "time": "2017-04-06 14:13:26.296584",
          "user": "some_user"
        },
        "versions": {
         "conda": "4.1.12",
          "conda-build": "1.21.7",
          "conda-tox": "0.6.5",
          "python": "3.5.2"
        }
      },
      "features": "omp",
      "md5": "477ec8a7faa700d27138e1cdf8fec967",
      "name": "libxgboost",
      "sha256": "4005a7034a294aa0d063aa5c4273f09e21cd04e1997e436f5bf0aada7e5c91da",
      "size": 912390,
      "subdir": "linux-64",
      "version": "0.6+218"
    },

Yes, that filename has only a single -. We assume that a valid filename has at least two dashes (e.g. libxgboost-0.6+218-omp_0 would be valid).
You can also see that somethign went wrong because the build is just "0" while it probably is mean to be something like _omp_0.
Not sure if we can solve this on our end easily. We could be more lenient but its' really quite broken metadata wise.

If there's some pieces like the build that are just broken then fair enough. I don't know how this particular entry came to exist.
I think more specific logging would have helped me with diagnosing the problem or getting help. Like 'could not parse the entry for libxgboost-0.6+218-omp_0 on line X column Y' would have been slightly nicer than just line X column Y. And anything else you can give 😄
So I think your point about better logging was a good one.

Yep - we already merged a PR for that (#222)

Closing this since #757 was merged.


feature-wip support deltalake catalog step1-metadata

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.51 seconds
stream load tsv:          534 seconds loaded<PHONE_NUMBER>9 Bytes, about 133 MB/s
stream load json:         20 seconds loaded<PHONE_NUMBER> Bytes, about 112 MB/s
stream load orc:          64 seconds loaded<PHONE_NUMBER> Bytes, about 16 MB/s
stream load parquet:          31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select:          28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size:<PHONE_NUMBER>7 Bytes


iProc pax dma driver support

This patch set series adds support for Broadcom PAX(PCIE<->AXI) DMA driver.

@olofj @xxie24 I addressed your comments where I could do and provided my feedback on requested changes, could you please review again ?

Hi Xin Xie,
I have now added support to handle dma_block_config's list or scatter/gather list handling in dma_iproc_pax_v2.c driver,
And also handling 15 pair of PCIE/AXI descriptors in one header BD. Please review the same.

@olofj @xxie24 :  Addressed your review comments, please have a look.

@xxie24 Addressed comment related to **non_hdr_bd_count, made single pointer.

@olofj Could you please review and provide feedback.

Hi, my outstanding comments seem to have been addressed, and I'm fine with this driver going in. I believe it still needs review from a DMA subsystem maintainer before merging though.

Hi, my outstanding comments seem to have been addressed, and I'm fine with this driver going in. I believe it still needs review from a DMA subsystem maintainer before merging though.


Query planner fails to plan nested query when an inner query is to topN and outer one has a filter

Description
Below query throws a planner exception -
SELECT * FROM
(
  SELECT "channel", count(*) as c
  from wikipedia
  group by 1
  ORDER BY 2 desc
  limit 10
)
where "channel" = '#en.wikipedia'

From the druid broker logs -
     Suppressed: org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There are not enough rules to produce a node with desired properties: convention=BINDABLE, sort=[].  │
│ Root: rel#37119:Subset#10.BINDABLE.[]                                                                                │
│ Original rel:                                                                                │
│ LogicalSort(subset=[rel#37068:Subset#5.DRUID.[]], fetch=[100]): rowcount = 1.5, cumulative cost = {1.5 rows, 12.0 cpu, 0.0 io}, id = 37066                                      │
│   LogicalFilter(subset=[rel#37065:Subset#4.NONE.[1 DESC]], condition=[=($0, '#en.wikipedia')]): rowcount = 1.5, cumulative cost = {1.5 rows, 10.0 cpu, 0.0 io}, id = 37064      │
│     LogicalSort(subset=[rel#37063:Subset#3.NONE.[1 DESC]], sort0=[$1], dir0=[DESC], fetch=[10]): rowcount = 10.0, cumulative cost = {10.0 rows,<PHONE_NUMBER>395237 cpu, 0.0 io}, │
│       LogicalAggregate(subset=[rel#37061:Subset#2.NONE.[]], group=[{0}], c=[COUNT()]): rowcount = 10.0, cumulative cost = {11.25 rows, 0.0 cpu, 0.0 io}, id = 37060             │
│         LogicalProject(subset=[rel#37059:Subset#1.NONE.[]], channel=[$1]): rowcount = 100.0, cumulative cost = {100.0 rows, 100.0 cpu, 0.0 io}, id = 37058                      │
│           LogicalTableScan(subset=[rel#37057:Subset#0.NONE.[]], table=[[druid, wikipedia]]): rowcount = 100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 37019    │
│ Sets:

Note - this is a simplified version of a complex query generated by Tableau.

If this issue if open, I would like to take this up

I checked a similar query shape and that is working in the master branch. Going to close this.


Search example doesn't work

I am running the example search_google.py  and i get the following error. But I put a set & uncomment from the config file. GOOGLE_API_KEY: "xxx"   and take the # away from it so its all uncommented. technically it should work.
❯ python search_google.py
2023-10-02 22:26:46.137 | INFO     | metagpt.config:__init__:44 - Config loading done.
2023-10-02 22:26:47.243 | INFO     | metagpt.roles.seacher:_act_sp:54 - Alice(Smart Assistant): ready to SearchAndSummarize
2023-10-02 22:26:47.243 | WARNING  | metagpt.actions.search_and_summarize:run:118 - Configure one of SERPAPI_API_KEY, SERPER_API_KEY, GOOGLE_API_KEY to unlock full feature

Have you removed the "" also?

SERPAPI_API_KEY, SERPER_API_KEY, and GOOGLE_API_KEY are all reported as unavailable in the log. Please ensure that the key you used is uncommented and that the value is usable.


[Bug]: 400 Bad Request

Your current environment
使用docker run --gpus all -v D:\docker\huggingface:/root/.cache/huggingface --env "HUGGING_FACE_HUB_TOKEN=****************" -d -p 8000:8000 --ipc=host --name vllm vllm/vllm-openai:latest --model nvidia/Llama3-ChatQA-1.5-8B --api-key vllm-abc123456 --gpu-memory-utilization 0.9部署成功了server端
客户端运行下面的代码报错：
from openai import OpenAI
Set OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "vllm-abc123456"
openai_api_base = "http://<IP_ADDRESS>:8000/v1"
client = OpenAI(
api_key=openai_api_key,
base_url=openai_api_base,
)
chat_response = client.chat.completions.create(
model="nvidia/Llama3-ChatQA-1.5-8B",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Tell me a joke."},
],
extra_body = {
"guided_choice": ["positive", "negative"]
}
).choices[0].message.content
print("Chat response:", chat_response)
错误：
D:\docker\vllm\venv\Scripts\python.exe D:\docker\vllm\llm_client.py
Traceback (most recent call last):
File "D:\docker\vllm\llm_client.py", line 14, in
chat_response = client.chat.completions.create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\docker\vllm\venv\Lib\site-packages\openai_utils_utils.py", line 277, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "D:\docker\vllm\venv\Lib\site-packages\openai\resources\chat\completions.py", line 579, in create
return self._post(
^^^^^^^^^^^
File "D:\docker\vllm\venv\Lib\site-packages\openai_base_client.py", line 1240, in post
return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\docker\vllm\venv\Lib\site-packages\openai_base_client.py", line 921, in request
return self._request(
^^^^^^^^^^^^^^
File "D:\docker\vllm\venv\Lib\site-packages\openai_base_client.py", line 1020, in _request
raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError:

400 Bad Request

Bad Request
Your browser sent a request that this server could not understand.
Reason: You're speaking plain HTTP to an SSL-enabled server port.
 Instead use the HTTPS scheme to access this URL, please.

🐛 Describe the bug
D:\docker\vllm\venv\Scripts\python.exe D:\docker\vllm\llm_client.py
Traceback (most recent call last):
File "D:\docker\vllm\llm_client.py", line 14, in
chat_response = client.chat.completions.create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\docker\vllm\venv\Lib\site-packages\openai_utils_utils.py", line 277, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "D:\docker\vllm\venv\Lib\site-packages\openai\resources\chat\completions.py", line 579, in create
return self._post(
^^^^^^^^^^^
File "D:\docker\vllm\venv\Lib\site-packages\openai_base_client.py", line 1240, in post
return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\docker\vllm\venv\Lib\site-packages\openai_base_client.py", line 921, in request
return self._request(
^^^^^^^^^^^^^^
File "D:\docker\vllm\venv\Lib\site-packages\openai_base_client.py", line 1020, in _request
raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError:

400 Bad Request

Bad Request
Your browser sent a request that this server could not understand.
Reason: You're speaking plain HTTP to an SSL-enabled server port.
 Instead use the HTTPS scheme to access this URL, please.

Have you tried using HTTPS instead of HTTP according to the error message?
- openai_api_base = "http://<IP_ADDRESS>:8000/v1"
+ openai_api_base = "https://<IP_ADDRESS>:8000/v1"

Have you tried using HTTPS instead of HTTP according to the error message?

- openai_api_base = "http://<IP_ADDRESS>:8000/v1"

+ openai_api_base = "https://<IP_ADDRESS>:8000/v1"

我是本地部署的，没有https只有http

I'm seeing a 400 bad request with 0.4.2 that does not occur with 0.4.1 with nothing changing.... It's strange because if I submit a CURL request, it works fine, but my application using axios returns the 400...  Reverting to 0.4.1 and everything works.  This has to be a regression.

The issue I mentioned above still occurs with the latest, 0.5.0-post1.  I'm stuck on 0.4.1 to have a functioning openai api server.

I'm seeing a 400 bad request with 0.4.2 (vllm.entrypoints.openai.api_server) that does not occur with 0.4.1 with nothing else changing.... It's strange because if I submit my normal test CURL request, it works fine, but my application using axios returns the 400... Reverting to 0.4.1 and everything works. This has to be a regression.

I encountered this issue as well. I think it's caused by the current default output being set to streaming. As long as you set 'stream': false, you should be able to get the correct response.

In my case, it was items in the messages array containing extra fields... date, and _id...
This is the core of the breaking change that occurred after 0.4.1.  The API is extremely rigid now in validating the input.
I've worked around it in my app by stripping out the fields before submission, but not excited about that.

same error , please fix

In my case, it was items in the messages array containing extra fields... date, and _id... This is the core of the breaking change that occurred after 0.4.1. The API is extremely rigid now in validating the input. I've worked around it in my app by stripping out the fields before submission, but not excited about that.

This was an intentional change in response to #4399. Sorry for the late reply!

same error , please fix

Please provide more information about this.


pythonPackages.matplotlib: enable tk backend by default

Motivation for this change
We currently do not build mathplotlib with any backend. This can be very
confusing for users. They will try to use matplotlib and it will simply
display nothing (see #51337). We should ship at least one backend. tk
was chosen somewhat arbitrarily. The gtk backend is problematic (see
https://github.com/NixOS/nixpkgs/issues/50959#issuecomment-441327767)
so tkinter seems like a good choice.
Closure size before: 256190712
After: 275376184 (7.5% increase)
CC @lovek323
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Assured whether relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

We currently do not build mathplotlib with any backend.

You mean interactive backend. There are, of course, working non-interactive backends (e.g. agg).
We are already unconditionally providing "MacOSX" backend on macOS, so it should be fine to have a working interactive backend on Linux too. And since macOS already has one, it doesn't need tk enabled by default.

I don't think we should add this back-end. Matplotlib as is can be used for plotting, but just not for interactive plots. If one wants interactive plots, one can choose to override matplotlib, or simply add the dependencies (e.g. tkinter) to their Python environment.
Longer-term we should get rid of the options, and simply have ways of saying how an environment should be extended when a certain option (a back-end) is desired.

@FRidh I had the same idea, but if we really believed that matplotlib should have no windowed interactive backends, we would also want to disable cocoa on darwin by default...

@FRidh is this still a hard "no" from you?

I definitely get a Tk window when I add tkinter to an environment:
$ nix-shell -p "(python3.withPackages(ps: with ps; [ ipython matplotlib tkinter]))" -I nixpkgs=.
these derivations will be built:
  /nix/store/0qinnl18hln46d5f2m4g256f5ayx7p4y-python3-3.7.2-env.drv
building '/nix/store/0qinnl18hln46d5f2m4g256f5ayx7p4y-python3-3.7.2-env.drv'...
created 390 symlinks in user environment

[nix-shell:~/Code/libraries/nixpkgs]$ ipython
Python 3.7.2 (default, Dec 24 2018, 03:41:55)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.1.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: %pylab
Using matplotlib backend: TkAgg
Populating the interactive namespace from numpy and matplotlib

In [2]: plot(range(10), range(10))
Out[2]: [<matplotlib.lines.Line2D at 0x7fb3c579c780>]

In [3]:

I didn't get it directly to work with notebook, but if there's an issue there, it's unrelated to the matplotlib build.

I just ran into this as well and can confirm that, even when having tkinter in the python environment, a pyplot.show() just does nothing.

Thats apparently because %pylab does some magic. With plain python or ipython no graphical plot is shown:

Turns out it actually enabled the WxAgg backend.

But when tkinter was present at compile time of matplotlib, that does work.

Indeed, so it seems this backend (just as Cocoa) actually need to be build.
Go for it, but include this observation that these two backends, contrary to the others, require building.

I just encountered such problems when testing osmnx for https://github.com/NixOS/nixpkgs/pull/57426. That package really wants to have a working graphical backend for matplotlib. And simply overriding with enableTk = true; doesn't work, as two other dependencies also depend on matplotlib, so you end up with something like
osmnx.override rec {
  matplotlib = super.matplotlib.override { enableTk = true; };
  descartes = super.descartes.override { inherit matplotlib; };
  geopandas = super.geopandas.override { inherit descartes; };
}

I think this PR is alright like this, the slight closure size increase is worth the reduced trouble for users. If there are no good objections, I'll merge this today.


Add M8 Additional Resources Notebook for P1

Hi @apvoges
It seems the Additional Resources notebook was removed. Could we add this back for M8 of P1?
Thanks,
Dean

Not to worry @apvoges!
Just spotted that M8 should already be populated. I'll take this through Asana to update instructions.
Thanks,
Dean


Fix missing istio-ingressgateway Service in istio yaml

A missing --- was causing the istio-ingressgateway Service to not get created when applying this yaml via oc apply.

thanks!


'.src(buffer)' image filename is a random number

When I use
.src(buffer)

I got optimized images success, but these images filename looks like some random number, e.g.
 "5d720201-1e25-4e35-b84c-da5b6ad82c18.png"
How to change the filename? I can't use that 'gulp-rename' plugin because my project is not based on gulp.
Thansk!!!

You can create your own renamer plugin with through2.ctor.
'use strict';

const Imagemin = require('imagemin');
const through2 = require('through2');

new Imagemin()
	.src('test.jpg')
	.dest('./')
	.use(Imagemin.jpegtran({progressive: true}))
	.use(through2.ctor({objectMode: true}, (file, enc, cb) => {
		file.path = 'new_file_path.jpg';
		cb(null, file);
	}))
	.run((err, files) => {
		// ...
	});

Hi, I try to do something like, in your previous post, but it  doesn't work. Console log inside through2 is not probably called.
const through2 = require("through2"); //2.0.3
const imagemin = require('imagemin'); //5.3.1
const imageminJpegtran = require('imagemin-jpegtran'); //5.0.2
const imageminPngquant = require('imagemin-pngquant'); 5.1.0

var myUniqueImages = [...];

imagemin(myUniqueImages, 'img/minified', {
        use: [
            imageminJpegtran({progressive: true}),
            imageminPngquant({quality: '65-80'}),
            through2.ctor(function(file, enc, cb) {
                console.log("Im here...");
                cb(null, file);
            }),
        ]
    }).then(data => { ... })

Console will not write anything, but images are minified without any problem.
Where is problem please? Thank you.


⚠ Bofrost.at has degraded performance

In 8d9e3a0, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 928 ms

Resolved: Bofrost.at performance has improved in 8830eb3.


feat: btc send transfer

Try out this version of the Hiro Wallet - download extension builds.

This PR is a draft which starts to implement working with the new rpc request api pattern. There is a draft PR also up in btckit to add support for this request type, proposed as sendTransfer with the params passed from the app of amount and address.

@kyranjamie appreciate the review, but most of this was just temp/placeholder code until I received actual designs. I'll go back thru it today knowing we want to release it and address your comments.

Ok @kyranjamie, let me have it! Opening for review ...addressed your first round of comments.

@kyranjamie let's merge this in?

@kyranjamie let's merge this in?

This is waiting for btckit to be published so I can remove the hardcoded request types.

Let me do this first then

I'm actually getting an error while QA'ing:

I'm actually getting an error while QA'ing:

I never ran into this, user error? 😆

I installed this PR's build then went to the test app  then "Bitcoin" tab and "Send transfer" button having authenticated with the first account in my wallet (with positive mainnet and testnet BTC balances). Then got this error when selecting the mid-range fees level.

https://user-images.githubusercontent.com/28991/233981175-99c90d67-c657-4b2b-b04c-a9f059d7ab32.mov


Fixes #35; Declare build dependency on Cython using pyproject.toml

Description
Add pyproject.toml file to declare Cython as build dependency
Related Issue
#35
Motivation and Context
Previously, running pip install ppsim without cython leads to import errors. Adding this file tells pip to install cython if the user does not have it, allowing the build to succeed.
How Has This Been Tested?
Deployed the changes to Test PyPI: https://test.pypi.org/project/ppsimtest/<IP_ADDRESS>/
To test installation removed cpython and ppsim installations:
pip3.8 uninstall cython ppsim ppsimtest

Then installed from Test PyPI
pip3.8 install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple --no-cache-dir ppsimtest==<IP_ADDRESS>

Command Explanation: The -i https://test.pypi.org/simple/ tells pip to look in Test PyPI. The  --extra-index-url https://pypi.org/simple tells pip to look in regular PyPI for certain packages like setuptools. --no-cache-dir insures that pip is installing from the Test PyPI repository and not some local cached file. This long command is only needed for testing purposes. Once deployed to PyPI, the regular pip install ppsim command will run the installation.

Then, verified that
import ppsim

runs without issue.

Note that this does require pip to be version 21.3 or newer, as mentioned here https://snarky.ca/what-the-heck-is-pyproject-toml/

Here'a a potential problem for Linux users, they will need to have python3.7-dev or later installed in order to build the project. Ohterwise, they will run into this error message when they try to pip install the package:
  Building wheel for ppsimtest (pyproject.toml) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3.8 /home/ben/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmpths5pyuh
       cwd: /tmp/pip-install-x07mrme9/ppsimtest_c1e920d9b09b49b1b6506e3ccf6e54e7
  Complete output (21 lines):
  ppsim version = <IP_ADDRESS>
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.8
  creating build/lib.linux-x86_64-3.8/ppsim
  copying ppsim/__version__.py -> build/lib.linux-x86_64-3.8/ppsim
  copying ppsim/crn.py -> build/lib.linux-x86_64-3.8/ppsim
  copying ppsim/__init__.py -> build/lib.linux-x86_64-3.8/ppsim
  copying ppsim/snapshot.py -> build/lib.linux-x86_64-3.8/ppsim
  copying ppsim/simulation.py -> build/lib.linux-x86_64-3.8/ppsim
  running build_ext
  creating build/temp.linux-x86_64-3.8
  creating build/temp.linux-x86_64-3.8/ppsim
  x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/tmp/pip-build-env-f2mo96tc/overlay/lib/python3.8/site-packages/numpy/core/include -I/usr/include/python3.8 -c ppsim/simulator.c -o build/temp.linux-x86_64-3.8/ppsim/simulator.o
  ppsim/simulator.c:36:10: fatal error: Python.h: No such file or directory
   #include "Python.h"
            ^~~~~~~~~~
  compilation terminated.
  error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for ppsimtest
Failed to build ppsimtest
ERROR: Could not build wheels for ppsimtest, which is required to install pyproject.toml-based projects

I'm not sure if this is something that can be resolved by specifying it in pyproject.toml because the problem is a missing software package rather than a missing Python package

If pip is not at least version 21.3, then import error occurs:

As far as this, and the dependency on python3.7-dev, can these themselves be declared as build dependencies?

Actually, I am unable to reproduce the import error. I tried downgrading pip as low as version 10 but still the pip installation runs find, so I'm not sure how I ran into this import issue. Seems like there's no requirement for pip version.
There still a requirement for python3.7-dev specifically for Ubuntu OS, (I imagine different OS will requirement a different package). I don't think it would be possible to specify these system dependencies for every single OS and I can't find any information for how to specify system packages as build dependencies in the first place for pyproject.toml.
The pyproject.toml we have now is the best we can do as far as I can tell.

Saving these commands in case I want to test this further:
pip3.7 uninstall cython ppsim ppsimtest
python3.7 -m pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple --no-cache-dir --user ppsimtest==<IP_ADDRESS>
echo "import ppsim" | python3.7

Thanks for the help on this.
I'm sorry I'm basically incapable of providing any help, as I have very
close to 0 understanding of what this issue even is. But I'm happy to see
that some action is being taken to improve things.
Toward my confusion, where have the changes made been incorporated? Is it
just that the master branch will do a better job uploading to PyPI in
future uploads?
I don't see anything different out on PyPI history for the package:
https://pypi.org/project/ppsim/#history
So it there still some uploading that needs to happen for changes to take
place?
Best,
Eric
On Mon, Jan 10, 2022 at 3:55 PM Benjamin Lee @.***>
wrote:

Saving these commands in case I want to test this further:
pip3.7 uninstall cython ppsim ppsimtest
python3.7 -m pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple --no-cache-dir --user ppsimtest==<IP_ADDRESS>
echo "import ppsim" | python3.7
—
Reply to this email directly, view it on GitHub
https://github.com/UC-Davis-molecular-computing/ppsim/pull/40#issuecomment-1009466481,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQLFGD42O34MFBIXIRXQYB3UVNWVPANCNFSM5LLDVSFA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because your review was requested.Message ID:
@.***>

@EricESeverson

Toward my confusion, where have the changes made been incorporated?
The changes have been incorporated in this branch and [Test PyPI] (https://test.pypi.org/project/ppsimtest/<IP_ADDRESS>/). These changes have not been pushed to master or the official PyPI repository yet.

The next steps are:

Merge this PR into dev.
Merge dev into main
Push to PyPI following the instructions you have listed here: https://github.com/UC-Davis-molecular-computing/ppsim/issues/35#issuecomment-1003440497

I merged it into dev.
Eric, whenever you're ready, merge dev into main and then update the PyPI package.


Error: "FATA Provided address [] is not an EVM smart contract"

Based on the bugs from #653:
When removing Ropsten from the Bridge config, if the client is left in node.yml, there's an error: FATA Provided address [] is not an EVM smart contract
Note after some research: https://github.com/LimeChain/hashport-validator/blob/main/bootstrap/services.go#L90 - to test if there's RouterContractAddress.

I will skip creating tests for bootstrap files because this will increase afterward the work on the project because will need to do changes in those tests for each change in each client, service, and repository.


Add note about failing upgrade tests

Add a short note about fixing failing upgrade tests.

Should we close @VenelinMartinov or still editing this?

I think I forgot to answer here, does this version make sense?
I think this is what I meant by "the test changes"

You have a good point that any time source changes are made to the program under test, the snapshots need to be recorded again. We could enforce that by a checksum. So you would never forget to record anew.
The verbiage in the PR is still a little confusing to me though.

Something like this?

** Fixing failing tests

- If the tests fail by flagging unwanted resource updates or replacements that are actually
  acceptable, configure or custom
  [DiffValidation](https://github.com/pulumi/providertest/blob/5f23c3ec7cee882392ea356a54c0f74f56b0f7d5/upgrade.go#L241)
  setting with more relaxed asserts.

- If the tests flag legitimate upgrade issues, fixes are necessarily specific to the provider and resource being tested.

- Remember to re-record the test snapshots when making changes to the example program or the
  baseline provider dependency.

Thanks for clearing that up, that looks very sensible!


break out preamble header

Breaking out the preamble header into its own template so N&C repo can override to style as necessary.
Adding inline SVG to N&C repo will make it look like this:

eregs/notice-and-comment#36

Just to check, are there any icons in the icon font that do this already? We don't really need SVGs here; unlike the left-side, there is an accessible fallback (the text).

So unless I'm mistaken and just couldn't find it, there isn't an icon font. But I decided to break this out so N&C template/style can override and add inline SVG to show the read/write icons.

Ok so I realize it does use a CF icon font, but it doesn't contain the icons used for read/write tabs.


Win 10 Pro start menu customization issues feedback

Hi jdeckerMS,
Re: article: https://technet.microsoft.com/en-au/itpro/windows/manage/customize-windows-10-start-screens-by-using-group-policy
The problem is that us IT Admins trying to administer 500+ staff on Win 10 pro and not having all the options to preset layout for new user and then allow  them to freely rearrange is not currently possible. I would like Microsoft to add group policy settings like in Win 8.1  in gpo: user config > Administrative Templates > Start Menu and Taskbar > "Pin Apps to Start when installed" - it needs to work on Win 10 as well. Can this be flagged as a priority for next version of Win 10 ADMX Templates?
thanks,
Michael

@jdeckerMS
Thought I check with you.

Hi Michael, this would be a good suggestion to submit using the Feedback Hub in Windows 10. That's the best avenue for feedback on the product.


Is the SCM commit view action button redundant?

Testing #153366
There's the giant commit button just below, is the view action just causing clutter?

It is, and I would love to hide it in case the "Commit" action button is enabled.
@joaomoreno, do you think that would be too disruptive given that the "Commit" action button is enabled by default?

I think that in order for us to do this, we should also be doing https://github.com/microsoft/vscode/issues/153504

I'm too scared to hide this action. You will definitely get angry mobs.
Plus, what of the people who have entirely disabled the scm button altogether? Wouldn't you have to understand this setting from the git world, in order to pull this off? Ie, the action still needs to be there, in case the button is disabled altogether.
I wouldn't lump this change together with the work you've done here.

Feel free to close off if it's not likely to happen

I will keep this issue opened, as long term I would like to hide that action.

Users can now choose to hide this button by right-clicking on it.

Given that users can now hide the button, I think that it is fine to close this issue.


backing vocal separation

can you add that? this for example:
import os
from audio_separator.separator import Separator
input = "/content/input.mp3"
output = "/content/output"
separator = Separator(output_dir=output)
Vocals and Instrumental
vocals = os.path.join(output, 'Vocals.wav')
instrumental = os.path.join(output, 'Instrumental.wav')
Vocals with Reverb and Vocals without Reverb
vocals_reverb = os.path.join(output, 'Vocals (Reverb).wav')
vocals_no_reverb = os.path.join(output, 'Vocals (No Reverb).wav')
Lead Vocals and Backing Vocals
lead_vocals = os.path.join(output, 'Lead Vocals.wav')
backing_vocals = os.path.join(output, 'Backing Vocals.wav')
Splitting a track into Vocal and Instrumental
separator.load_model(model_filename='model_bs_roformer_ep_317_sdr_12.9755.ckpt')
voc_inst = separator.separate(input)
os.rename(os.path.join(output, voc_inst[0]), instrumental) # Rename file to “Instrumental.wav”
os.rename(os.path.join(output, voc_inst[1]), vocals) # Rename file to “Vocals.wav”
Applying DeEcho-DeReverb to Vocals
separator.load_model(model_filename='UVR-DeEcho-DeReverb.pth')
voc_no_reverb = separator.separate(vocals)
os.rename(os.path.join(output, voc_no_reverb[0]), vocals_no_reverb) # Rename file to “Vocals (No Reverb).wav”
os.rename(os.path.join(output, voc_no_reverb[1]), vocals_reverb) # Rename file to “Vocals (Reverb).wav”
Separating Back Vocals from Main Vocals
separator.load_model(model_filename='mel_band_roformer_karaoke_aufr33_viperx_sdr_10.1956.ckpt')
backing_voc = separator.separate(vocals_no_reverb)
os.rename(os.path.join(output, backing_voc[0]), backing_vocals) # Rename file to “Backing Vocals.wav”
os.rename(os.path.join(output, backing_voc[1]), lead_vocals) # Rename file to “Lead Vocals.wav”

This can be added, but it will require me to change a lot of my current code. So for now I'll put it on my TO-DO list.

thanks


Optimization Issues with Pencil Tool Strokes on Page Zoom

Appreciate the fantastic software crafted by the development team. I've come across an issue while using it and would like to report it here. I hope you can consider further enhancements to the handwriting feature.
When the page is enlarged, there can be issues with writing using the pencil tool, often resulting in overly smooth strokes. This might be due to the visual effects seen in videos. Is there any plan to address this? If not, could you advise on how to adjust the length of the "guide line" at the pen tip in the video? Thanks!
Also, would it be possible to include a feature to disable pressure sensitivity? I'm concerned that this might be causing severe lag during writing after a while. Thanks!

https://github.com/user-attachments/assets/6e69e9d6-d863-4b63-b689-efcdc6c486f9

I tried, and it's working fine. Seems like this issue is from user's end.

Is this excalidraw.com?

Also, would it be possible to include a feature to disable pressure sensitivity?

Yeah, we were just discussing that. Assuming you dont have a pen with hardware pressure sensitivity?

I hope you can add a function to disable the pressure sensitivity of the stylus. When there is too much handwriting, there is a noticeable severe lag, which I suspect is caused by the pressure sensitivity feature. Although my stylus has pressure sensitivity, I do not want to use it during normal handwriting.
I hope you can add a function where the thickness of the pen does not change with document zoom. This way, no matter how much the canvas is zoomed in or out, the handwriting with the stylus will not have the issue shown in the video I uploaded maybe.
Thank you ！！ @Mrazator


Run Western scenarios in containerized framework on laptop

Scenarios involving a Western interconnect fail when run in the containerized framework on a laptop. It runs successfully on the bedev01.

This is now documented in the troubleshooting guide


Client Management/MDM: rename page names & titles

Description:
As requested by @yannisle (Yannis Lempidakis) in issue ticket #6610 (Can we change the title of this page to Poilcy "CSPs" supported by HoloLens 2? It makes it clear because the word Policy alone confuses customers), this PR suggests to rename the page title and the file name according to the request, to say "Policy CSPs supported by" instead of only "Policies supported by", because that seems to confuse customers.
Thanks to @yannisle for posting the request in ticket #6610.
Changes proposed:

Rename the page titles, from "Policies supported by"[...] to "Policy CSPs supported by"[...], in 3 pages (so far)
Rename the page names, from "policies-supported-by"[...] to "policy-csps-supported-by"[...], affecting 3 files (so far)
Whitespace changes:

Reduce the bullet point link list spacing, from 3 to 1 (all 3 pages)
Remove 1 redundant end-of-line space in the "div class" section (all 3 pages)
Add NewLine (line break) at the end-of-file (EOF)  (all 3 pages)

Additional notes:

[ ] Confirm whether 6 more files need the same change:

Policies supported by Group Policy
ADMX-backed policies
Policies supported by Windows 10 IoT Enterprise
Policies supported by Windows 10 IoT Core
Policies supported by Microsoft Surface Hub
Policies that can be set using Exchange Active Sync (EAS)

(See https://github.com/MicrosoftDocs/windows-itpro-docs/issues/6610#issuecomment-623102343 for details)
Ticket closure or reference:
Closes #6610

#sign-off

Update: As confirmed in ticket comment https://github.com/MicrosoftDocs/windows-itpro-docs/issues/6610#issuecomment-623283503 , The remaining 6 pages are also requested to be changed in the same manner as the current 3 files in this PR. Working on adding the files now, focused on being careful to only make the same changes as already documented in the main PR comment.

Finally done. Funnily enough, the last file in the set (Policies that can be set using Exchange ActiveSync (EAS) ) shows up in a combined file rename & diff view, which I really wish we could have had for the other 8 files in this PR content. Oh well. "You can't win them all", can you. Anyway, that last file contains the gist of the main changes in all the files, only omitting the bullet point indent spacing reduction.

#sign-off

There you go, @yannisle . The changes should be complete now. Feel free to add your thoughts, suggestions or other comments.

Ahaaa... Right. Sure, I can get that included too. Just need another day to make sure that the files are updated correctly (had a plumber visiting my flat and need to sort that out too).

Fair enough. I reminded myself that I only needed to rename the file name links and the link texts. As noted in the commit message:
2 more files with link references to the already added files in this PR:

TOC.md
policy-configuration-service-provider.md

Changes done here are limited to the file names in the links and the link texts. No further work has been done to improve these 2 files.

#sign-off


add imagePullPolicy to grafana job tasks

fixes: issue #40
added "imagePullPolicy": "IfNotPresent" to ./manifests/grafana/import-dashboards/jobs.yaml.

Thank you!


Level save with populated forest results in "FileStream::open::empty filename"

When saving a level following addition of trees with the forest brush the editor asserts with "FileStream::open::empty filename".
Even when ensuring an existing forest data file is selected in the inspector and also when previously saving a level with an empty forest successfully.
v.3.6.3 from git
File:
game\tools\forestEditor\main.cs ->
function ForestEditorPlugin::onSaveMission( %this, %missionFile )
{
ForestDataManager.saveDirty();
if ( isObject( theForest ) )
theForest.saveDataFile();
ForestBrushGroup.save( "art/forest/brushes.cs" );
}
Temp Workaround:
Hardcoding a path/filename:
if ( isObject( theForest ) )
theForest.saveDataFile("tempForestFile.forest");

Thanks for the report! Is this on the development branch?

I think it was 3.6.3 development.

Okay, thanks!

yw!
Just now built and tested 3.7 release with the same behavior.
I need to get up to speed and get on the forums.
Perhaps grab the forest "dataFile" somehow from the .mis or %mission in main.cs...
Thanks for the hard work guys!

HI, don't know if the same bug, I was playing around t3d 3.7rc with walkabout. Added some trees save the level and the next day I open my level and all trees was gone.

Fixed with the assocaited PR. Apparently formatted the comment incorrectly to have github automatically associate them. Closing manually.


When the code will be available

Hi, do you have the plan to release the code?
Thx!

Yes, but this work is supported by an enterprise, they want to apply for a patent. After they finish the patent application, we will release the code. It may still take one month for them.

@yqtianust Yes, we will release the code. However, this work is supported by an enterprise, they want to apply for a patent. After they finish the patent application, we will release the code. And I'm sorry to say that it may still take one month for them.

I see. I noticed that you also re-implemented SG. Will you release it as well?
Thx.


nativefier about install ActiveX

Description
<Detail your problem here as clearly and concisely as possible 🙂. Help us help you by saying why and giving some context!>
Steps to reproduce issue
Details

Are you nativefying a public website?
Feature request? Have you looked at nativefier --help to see if an existing option could fit your needs?
Full nativefier command used to build your app: <command here>
Version of Nativefier (run nativefier --version): v0.0.0
Version of node.js (run node --version): v0.0.0
OS: <OS here>
Error message / stack trace (if any):

<Error message / Stack trace here>

er,maybe there is no error  or i can't find it , what i want to ask actually is 'is the nativefier support ie ActiveX plugin?' thanks 🙂🙂🙂🙂🙂🙂🙂

nativefier uses Chromium inside Electron framework
ActiveX is a software framework created by Microsoft that adapts its earlier Component Object Model (COM) and Object Linking and Embedding (OLE) technologies for content downloaded from a network
it's apples and oranges! <3

nativefier uses Chromium inside Electron framework
ActiveX is a software framework created by Microsoft that adapts its earlier Component Object Model (COM) and Object Linking and Embedding (OLE) technologies for content downloaded from a network
it's apples and oranges! <3

fine,i just a novice to use this tool and i have solved the problem by other way thanks :)


Discussion for Aggregated Trace Metrics (ATM)

Requirement - what kind of business use case are you trying to solve?
We are trying to build a way to aggregate traces into metrics for operational use cases you often see in monitoring requirements.
Problem - what in Jaeger blocks you from solving the requirement?
There is no current feature to do this.
Proposal - what do you suggest to solve the problem or improve the existing situation?
As per this document: https://docs.google.com/document/d/1EqIhkm7FLTEJQiDlPDlwR8f2KbJvPx6HMUHKCxQpqLg/edit which was discussed on the Jaeger community call on 10/16 I agreed to open this issue to address issues and have a discussion.
Any open questions to address
We have a few ways to design this feature. The goal would be to export metrics from trace aggregation to a Prometheus compatible backend (remote_write or scrapable endpoint). In the discussion on the 10/16 call  @yurishkuro suggested this to be in Otel versus in a streaming system post ingest. This is a good design IMO, but the challenge is that after discussing with @albertteoh Otel doesn't allow for mixed telemetry types you can only export traces from traces or metrics from metrics. This would not come until Otel v2 which is too far away for us.
We would like to propose modifying the Jaeger exporter to include metric export along with trace export (we would do the same with logz.io exporter).
As a second phase we'd figure out how to use these metrics from within the Jaeger UI for topology views and possibly a more useful homepage, but that will require building some type of MetricQuery service.
Discussion open.
Thanks!

@yurishkuro suggested this to be in Otel versus in a streaming system post ingest

Correction: I suggested it to be in the Jaeger collector, not necessarily in the OTEL project.
The reason for collector is because it is the closest backend component to the source of data. In an architecture that supports tail-based sampling, for example, the collectors have access to highest fidelity / least sampled data, compared to all other stages of the backend processing. Building this on top of Kafka means we'd always have to pay disk serialization cost, whereas extracting metrics from spans is a lightweight operation that could be easily done prior to that.

Otel doesn't allow for mixed telemetry types you can only export traces from traces or metrics from metrics.

I still don't follow why this is a problem. A TraceConsumer just takes trace data and does something with it. If that something happens to be extracting metrics and sending them somewhere, what does that have to do with processing pipelines of the OTEL collector?

A TraceConsumer just takes trace data and does something with it.

My thoughts exactly: the processor can just capture the metrics based on the traces from the pipeline and export them as metrics data points.

A TraceConsumer just takes trace data and does something with it.

My thoughts exactly: the processor can just capture the metrics based on the traces from the pipeline and export them as metrics data points.

I tried to draw this up:
+-----------------------------------------------------------------------------------------------------------+
|                                         OTEL Collector Pipeline                                           |
|                            +---------------------+     +----------------------+                           |
|     +---------------+      |                     |     |     (optional)       |      +---------------+    |
|     |               |      | agg trace metrics   |     |other trace processors|      |               |    |
|     |trace receiver |----->|(passthrough traces) |---->|like tail-based       |----->|trace exporter |    |
|     |               |      |                     |     |sampler               |      |               |    |
|     +---------------+      +---------------------+     +----------------------+      +---------------+    |
|                                       |                                                                   |
+-----------------------------------------------------------------------------------------------------------+
                                        v
                           +--------------------------+
                           |                          |
                           |    a metrics consumer    |
                           |  (m3, prometheus, etc.)  |
                           |                          |
                           +--------------------------+

I like this because it's very simple and we should be able to get something working pretty quickly with the existing OTEL Collector design. It's certainly feasible, especially (as @yurishkuro mentioned) that extracting metrics from spans is lightweight and the network serde and I/O could be done out of band in a seperate goroutine, taking the cost away from the trace processing hot path.
However, I feel it diverges from the design philosophy of OTEL collectors. An exporter sends data to other systems/backends and processors simply pre-process data or help with reliable completion of pipelines.
Why this could be an issue is:

(Usability) It would be unexpected to users to need host:port configuration on a processor when this should be on receiver or exporter config.
How will those metrics be sent to the backend? Would we need to implement support for multiple metrics shippers (i.e. prometheus scrape endpoint, prom remote write, metricbeat, etc.) and if so, would this be duplicated work that is already provided (or might be in future) by the open-source community through OTEL Collector exporters?

Although, a question that comes to mind, and is on me to research/learn more about OTEL Collectors, is how do we monitor the processors? Do processors need to instrument themselves and expose/ship their metrics? If so, we could perhaps piggyback off that precedent.

processors simply pre-process data or help with reliable completion of pipelines.

I think that the definition in the readme has only the case of processors acting on data of the same type. Using traces to generate metrics is acceptable, IMO, but you might want to explicitly ask in the Collector SIG.

It would be unexpected to users to need host:port configuration on a processor when this should be on receiver or exporter config.

I had a similar concern with the routingprocessor and with the loadbalancingexporter. In this case, I would just allow OTLP endpoints to be on the receiving end of these metrics. Users can then configure their metrics pipeline to export this to their preferred backend.
Processors are indeed able to instrument themselves. Take a look at the loadbalancingexporter or groupbyprocessor for examples.

Do processors need to instrument themselves and expose/ship their metrics? If so, we could perhaps piggyback off that precedent.

Given the volume of metrics that could be created I think that Prometheus remote_write would be the right choice here over exposing a page to scrape.  remote_write is well supported amongst tsdb backends and so it would give users options for where to store their data.

Given the volume of metrics that could be created I think that Prometheus remote_write would be the right choice here over exposing a page to scrape. remote_write is well supported amongst tsdb backends and so it would give users options for where to store their data.

The downside is that we can't remote_write into prometheus itself, or maybe we can and I am just not aware of it. In order to process real-time metrics (ie alertmanager) you'd want to scrape as part of your pipeline to ingest. I could see both methods being valid for users depending on their specific requirements.
If we want to use these metrics inside the Jaeger UI we also need to build a service and determining how we support multiple backends, but at least promql would be the query language.

Given the volume of metrics that could be created I think that Prometheus remote_write would be the right choice here over exposing a page to scrape.

Why not export to OTLP metrics, and let users configure their exporters the way they need it?

Correction: I suggested it to be in the Jaeger collector, not necessarily in the OTEL project.

@yurishkuro I was under the impression that the Jaeger collector would be replaced by the Otel Collector hence we should be focusing efforts on Otel vs Jaeger.
Based on this discussion we should look at Otel and utilize OTLP to create metrics which can then be sent to any backend. (thanks for the great advice @jpkrohling). We just need to determine how to use this data within Jaeger UI eventually.
Thanks for the discussion.

the Jaeger collector would be replaced by the Otel Collector

Yes and no. Jaeger collector will be based on OTEL Collector, but I think it will always remain a custom build, especially because it includes implementations of direct-to-db exporters.

Why not export to OTLP metrics, and let users configure their exporters the way they need it?

@jpkrohling, for my understanding, why specifically OTLP metrics?
Could these metrics, theoretically, be shipped in any other format (opencensus, prometheus, etc.) so long as there is an OTEL receiver available to accept this data and normalise it to the OTEL metrics data model, then users can, as you say, configure their exporters the way they need?

for my understanding, why specifically OTLP metrics?

The otlp receiver is more likely to exist in an OpenTelemetry Collector than other receivers. And if it's not there, I think it's more reasonable to require it to exist than, say, the OpenCensus receiver.

Wanted to flag up my interest in this topic in general.  My research group at MPI-SWS is currently doing some work on trace visualization and aggregate analysis, and our interests extend into the systems side of driving visualizations.  Our starting point at this moment is mentioned in @jkowall's google doc writeup:

Aside from getting ATMs into Prometheus or other metric systems, we would ideally like to include or overlay metrics on the dependency view, or maybe a more usable Jaeger homepage which shows operational status of the services being monitored

We've been looking at this in a few ways.  We've been thinking about visualizations that (a) extend single-trace visualizations to provide aggregate context; and (b) visualizations for sets of traces in general.  Metrics are a big part of this, plus we are also interested in how to represent structural aggregations too.  We have also been thinking about the backends needed for driving the visualizatoins.  In case anybody is interested, a PhD student in my group (Vaastav Anand) prototyped and wrote up some ideas for a class project earlier this year (link)
We're planning to reach out to the community within the next few months to find participants for a user study.  We'll also be contributing the visualizations back into Jaeger.
A question that has come up repeatedly over the years (I must sound like a broken record) is that of trace datasets.  Currently we're working with pretty simple traces generated from the DeathStarBench which is a small microservices benchmark.  This dataset is quite limited in terms of trace complexity and diversity, and means we end up making (often incorrect) assumptions as we design and develop visualizations.
Would anybody reading this be interested in sharing some "real" trace datasets and examples of metrics you use in practice?  We can sign NDAs where needed.  If so please reach out to me<EMAIL_ADDRESS>

I wanted to report that @albertteoh has a working POC for this using Otel Collector. We will be submitting a PR for the new pipeline in the coming weeks, and I'll update this issue. The reason for the delay is competing deadlines, but we'll get back on it, since it's an area which is critical for making tracing operational in nature.
@JonathanMace that sounds interesting. I'll be waiting to hear from you!
We have some UX resources here that want to also work on improvements in the visualizations, and I can envision a more usable main page when you login to Jaeger which leverages the work that @albertteoh is doing on ATM. I've been getting resources for some design work aligned internally and I'll keep the community updated as that progresses.


Uses etcdv3 for calico 3 rr_v4 resources

Given that Calico 3 is using etcdv3, this ensures that rr_v4 resources are added via the etcdv3 api
For example:
$ ETCDCTL_API=3 etcdctl --endpoints=https://localhost:2379 --cacert=/etc/ssl/etcd/ssl/ca.pem --cert=/etc/ssl/etcd/ssl/admin-etcd01.pem --key=/etc/ssl/etcd/ssl/admin-etcd01-key.pem get --prefix /calico/ --keys-only

/calico/bgp/v1/rr_v4/<IP_ADDRESS>

/calico/bgp/v1/rr_v4/<IP_ADDRESS>

/calico/ipam/v2/assignment/ipv4/block/<IP_ADDRESS>-26
...

ci check this

/lgtm
/approve


Fix test script

The test script was erroneously exporting MIX_ENV.

Coverage remained the same at 81.625% when pulling 007f15eeb7fa37828d4b97603955b329cc54eb8c on fix-test-script into 01be6be35d2f0cbc1496df87ce6bc076ab16f850 on master.


Fix counters path in lwaftr/query

Update location of counters path and counters_name array.

LGTM, good catch


Update spell_palette.sass

@nwinter, could you see if this change is fine? I made it because when I maximized the level editor, it was broken.

Great idea!


Disable Shipping V2 SymbolTask on OnPrem TFS

@arunkm,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

take or close

Closing (not taking) as we handled it in a different way of customer explicitly selecting "FileShare" vs "Service"


Refactor ledger storage

Describe your changes
Refactor of the ledger/account stores to persist all ledgers as column families of accounts, instead of serializing/deserializing the entire ledger.
Link issue(s) fixed
Resolves #1354
Checklist before requesting a review

[ ] I have performed a self-review of my code.
[ ] I have added tests that prove my fix is effective or that my feature works.
[ ] I have verified new and existing tests pass locally with my changes.
[ ] I verified whether it was necessary to increment the database version.

@Isaac-DeFrain, test failures here.

Opening a different PR instead...


Rename --job to --step for artifact upload/download

Fixed in https://github.com/buildkite/agent/commit/bda99cedbfdd7131c9ee5d1b8e7be4b6cd8761cd


Error: invalid module name: 'demon.x64RemoteShinjectNimPackt'

I am needing help to understand where is my error.

Hi @calex80! It looks like this bug occurs since your filename excluding .bin (demon.x64) has a period (.) within it. This name is used as a Nim identifier, but Nim does not support periods I believe.
If you did not get this to work yet, could you please try again with a name such as demon.bin for the input file?