[Q] Any suggestions as to how I can speed up channel changes in my Windows tuner app

So, currently my pseudo code is like this when tuning to a channel, or changing channels:

initialise COM
new ts::Tuner
tuner_config.searchService(channel name)
tuner_config.serviceToTuning(tuning params)
tuner->tune()
tuner->start()
new ts::Service, SectionDemux etc, and my code then produces a minimal TS stream by only keeping the packets for the channel I am interested in)
forward this data to FFmpeg AVFormat
AVformat demuxes, decodes....

This is all done on a server app, which then filters, scales, etc, re-encodes and streams to the player.
The slowest bit in the above, is tuner->start(). This takes 4-5 seconds. After that the TSDuck demux is more or less instant, and it's maybe 0.5s-1.0s for AVFormat to fully demux the retained packets in a format that AVCodec etc can use.
Of course, Ideally I would enhance my usage of the TSDuck demux to directly fill out the AVFormatContext fields with the codec info etc and skip the avformat demux altogether, but that's a larger job I think and by the time I've done it I might just end up having transferred the 0.5s-1.0s avformat demux time to TSDuck demux time without really gaining anything.
Is there anything I can do to speed up the tuner->start() though? 5s is a long time before I've even started demuxing, let alone going through my transcode -> {INTERNET} -> player-decode step and means channel changes are a bit slower than I'd like. I thought that maybe this 4-5s included the time taken to receive the first packets, and maybe it was buffering them, but if that was the case I'd expect to receive a large chunk of them straight after start() returns, but I don't, they just arrive gradually.
For comparison WinTV 10, is maybe 2-3 seconds all in for tuning and buffering.
Maybe I don't need to tear down the tuner each time I change channels? Would that help if I just tuner->stop(), tuner->tune(), tuner->start()? Or is this just an artefact of the Windows tuner code?

Hi @oviano
There are several tracks of improvements.
The first one is that TSDuck is probably not the right tool to do that on Windows. WinTV, and other proprietary TV-watching applications, are pure DirectShow applications. DirectShow is designed for audio/video streaming. TSDuck, on the other hand, is a transport stream processing toolkit. Roughly, DirectShow is designed to implement device-side applications while TSDuck is designed to implement head-end processing (not only but mostly).
DirectShow is a streaming engine, including the rendering endpoints. TSDuck does not need DirectShow in itself. DirectShow is a pain in the xxx of TSDuck. The problem on Windows is that DirectShow is only standard way of interfacing DVB tuners. TSDuck uses DirectShow for this sole reason. But, then, the problem with DirectShow is that you cannot extract data from it. Audio/video data flow from source (here a tuner) to the target (often a rendering engine) using the underground plumbing of DirectShow without any possibility to escape from it. Moreover DirectShow has a push model while TSDuck has a pull model. So, to do something as simple as reading raw packets from a tuner, TSDuck needs to insert a custom filter inside a DirectShow graph, use that filter to extract data out of the DirectShow plumbing, using asynchronous queues to convert the push model into a pull one. This has a cost in terms of performance, especially the initial latency you are experimenting.
If your application is a "TV-watching" type of app, I would recommend two solutions, depending on what is more important for you.

If you want to stay on Windows, drop TSDuck and write a pure DirectShow application. Develop your own filters if necessary. Fully adopt the DirectShow push model. The code will be strictly non-portable but optimal in the Windows context. This is the WinTV model, with better performance as you noticed.
If you prefer to stay with TSDuck and use its C++ code base for your application, move to Linux. The LinuxTV model is simple and straight. Tuning is just a few ioctl()'s followed by read() system calls directly to the drivers. It is much faster. And in terms of TSDuck coding, it is the same because the TSDuck API is system-agnostic.

If your application does not have too many dependencies on Windows, I would suggest to try it on Linux, just to compare the performances.
Additionally, real TVset or STB middleware have optimised channel switching strategies.

During the initial scanning, all PID's of all services are stored in an internal database. When switching to a new channel, the device immediately routes the audio and video PID's to the codecs. In parallel, the device classically listens to PAT, SDT, PMT to discover potential changes. In the rare cases of a PID reconfiguration, the first seconds or fractions of second are lost, the selected PID's are ineffective and the actual PID's are configured later (and updated in the database). But in 99.99% of the cases, the channel switching is much faster.
Modern receivers now implement "smart fast switching". The device requires multiple tuners, at least two for live channels. Based on the fact that most users are "zapping" using P+/P- keys on the remote, pre-switching is implemented. Based on the current channel LCN, the device assumes that the next channel to switch on will be LCN+1 (or LCN-1 if the current channel was selected using the P- key). Thus, the second tuner is pre-programmed to the frequency of channel LCN+1, the signalisation is acquired and all audio/video PID's are immediately ready to be injected in the codecs if the next channel is the anticipated one. Then, the only zapping time is the result of the parallel acquisition of the next I-frame and next ECM (plus ECM processing and descrambling setup). If you have dual codec and descrambler resources, you can even anticipate these, effectively reducing the zapping time to zero. But this is for high-end devices only.

Keep in mind that TSDuck is a toolkit to experiment transport stream processing and transformation. Implementing an optimised receiver uses a very different software architecture. Many frameworks already exist for that: DirectShow on Windows, gstreamer on Linux for instance. But almost none exist for head-end experimentations. This is why TSDuck exist. So, I am just wondering if TSDuck is simply not the right tool for what you wan to do. (I am not a sales guy, I do no try to over-sell inadequate stuff for the customer's need :))

So the project is a personal streaming server that can take input either from a video capture device (using either libs device dshow or custom Magewell Capture SDK) or from a tuner.
So you then configure various video quality profiles and channel favourites and the AV stream is then transmitted using SRT using a custom container format for the player to display.
It’s basically a higher quality customisable “Slingbox” style system that I wrote when I lived in Turkey and wanted near broadcast quality TV piped from the UK at full broadcast frame rate etc and using the best compression (h265 etc).
I chose TSDuck as it seemed like the only cross platform option and tbh I’m pretty happy with it. I could indeed battle myself with DirectShow directly but I would still like to keep the option open to use both Windows and Linux in future and at the moment there is no specific Windows dependency.
At the same time I don’t want to make it exclusively one platform or the other so Duck does suit.

At the same time I don’t want to make it exclusively one platform or the other so Duck does suit.

So, I would recommend to benchmark your issues on Linux, with same hardware and same tuner if possible to compare comparable configurations. And if the performances are better on Linux, well, just use Linux.
That won't be the first time a single software system would have very different performances on distinct systems.

Another option I guess would be to delve into the DShow stuff but still retain TSDuck for possible future Linux support but it doesn’t really appeal.

Yes, definitely. Don't use two completely different code bases for the two systems.

last time I looked that way the whole DShow model hurt my brain.....

I definitely sympathise. Interfacing tuners on Windows was one of my worst software experience. I would recommend anyone to fly away from DirectShow. Unfortunately, Microsoft made it a bottleneck to access tuners...

Thanks for the suggestions.
I've decided to take a bit of a detour and add HDHomeRun support to my project.
Rather than try and support these tuners when -as you clearly state - on Windows there simply isn't a reliable and properly-documented way to achieve this, it seems HDHR has a pretty simple cross-platform C API using sockets which should be straightforward to hook into.
I'll keep TSDuck in for the direct Tuner approach, as it almost works on Windows (depending on which way the wind is blowing), and would probably work well on Linux, but I don't think I want to spend a second more on trying to figure out why the DirectShow FilterGraph randomly delivers a stream containing corrupt packets to TSDuck (and it is, literally, depending on which way the wind is blowing, from what I can tell).

on Windows there simply isn't a reliable and properly-documented way to achieve this

This is entirely incorrect. It works reliably and it is used by dozens of applications without problems. The documentation is even better than in case of Linux. The only problem is that you need to be familiar with DirectShow first. That's what makes it hard to get started.

I don't think I want to spend a second more on trying to figure out why the DirectShow FilterGraph randomly delivers a stream containing corrupt packets to TSDuck (and it is, literally, depending on which way the wind is blowing, from what I can tell).

DirectShow does not deliver "corrupt packets". That's either a tuning parameters or a programming issue. Maybe not draining data fast enough. I have never seen such issues with tsDuck. I would suggest that you run a regular release of tsDuck and simply output data to a file. Then investigate that file and see whether your "wind blowing" theory still stands ;-)
Slower tuning times than with Linux: yes, that's true. From my experience, it takes about 0.5 to 2s longer. You would need to create your own DirectShow filters and make some other changes to avoid this, like @lelegard had stated. Not an easy task, though.

Well search my posts on these parts from a while back, i did a ton of testing and trial and error I made a long thread about it.
I couldn’t find any other logical explanation for what I was seeing than an issue with DirectShow, at least as implemented by TSDuck. Certainly TSDuck wasn’t introducing the corruption. You may be right about draining the buffer or something but again I couldn’t see how that was a possibility from the code, I tried everything.
I had hacked in code to grab the raw data from the DS filters before they were passed to TSDuck and saving them to file and they contained corruption. There is a comment on the TSDuck relating to it.
Then I’d start WinTV and get perfect reception. Switch back to TSDuck and I’d get corruption.
Just because you have not seen the issue doesn’t mean it doesn’t exist.

We have done tests with a long list of different tuner models and users from all over the world and never experienced such issues.
Though, my baseline version of tsDuck is about 2 years old and I have made a few modifications...

Sure, well it’s a known issue because @lelegard even has a comment in the code about it (where it loses synchronisation) and I’ve certainly seen it.
I have no idea how common or rare it is, only that I have seen the exact same thing as @lelegard’s code comment explicitly predicts and tries to mitigate.

Which comment are you referring to?

https://github.com/tsduck/tsduck/blob/master/src/libtsduck/dtv/windows/tsSinkFilter.cpp
Line 361.

Thanks. Hm, I haven't made any modifications there. I have modifications to graph building and tuning. Also, I'm streaming out via UDP, so in theory it could also be the disk-writing which is causing the streaming to stall in some way.
BTW, I have a HCW Solo as well. The only issue I'm seeing sometimes is that the device (or its driver) is getting into a state where it doesn't work anymore, not even after un- and re-plugging. Only a reboot helps. But others haven't seen this, so it might be something about my local system, like USB hw or driver.

From a quick look at the tsSinkFilter code, I see two things:

ReceiveCanBlock returns S_FALSE, even though there's a lock in the Receive method
This could have the effect that all available pool threads call into the Receive method and get stuck at the lock, which could in turn cause all processing at the side of upstream filter to cease and not process some input data.
The IMemInputPin implementation does not provide an allocator
That means that the samples are allocated elsewhere. That allocation might not be unlimited. The tsDuck code is putting the samples into a queue and calling ::AddRef, which means that they are not getting released. At some point, the source pin allocation might hit a limit (no more samples can be allocated) causing data to be dropped and never getting sent to the input pin

This is what I would look at.

Hi @oviano and @SS-FF,
First, I clearly admit that the TSDuck SinkFilter is far from good. I wrote it 10/12 years ago and never came back to it. At that time, the tuner interface of TSDuck was only working on Linux. I struggled quite a lot to port it on Windows and the discovery of DirectShow was a real nightmare. I do not agree on the fact that DirectShow is well documented. It is awful. There are a lot a small and probably auto-generated docs on each method. But there is absolutely nothing which gives a clear and detailed view of DirectShow in an open manner. You can find many examples of full DirectShow streaming applications. But it gives the idea that DirectShow is some independent world. You are in our out but there is no way to interface the two worlds, which was precisely what TSDuck wanted to do.
On Linux, the LinuxTV tuner API is clear and simple. It interfaces tuners, full stop. Then, if you want to develop some streaming application, you can use frameworks such as gstreamer which are equivalent to DirectShow at architectural level. But you can use LinuxTV tuners alone, without gstreamer.
On Windows, the concept of BDA driver is defined, but essentially on a device standpoint. The BDA documentation focuses on driver development. There is no complete and tuner-agnostic API for BDA drivers on the userland side. A tuner vendor usually develops the driver and the corresponding DirectShow filter. The only "clearly" defined API is inside DirectShow, forcing applications such as TSDuck to dig into DirectShow while we absolutely do not need a streaming framework.
I started to understand how DirectShow actually works when I read the parts of the VLC source code which interface tuners using DirectShow, but certainly not from the Microsoft documentation. I wrote the TSDuck SinkFilter, made it work, later made some modifications such as the corruption issue you mentioned and never modified it.
The corruption issues were not systematical. They occurred on some tuners only. I never really demonstrated if they came from the drivers of some devices, from DirectShow or from the way TSDuck uses it.
Note that DirectShow forces the application to have a push-oriented streaming architecture while LinuxTV is just an API which can be used in any architecture. A full DirectShow application which adopts the DirectShow push model will probably work better than applications such as TSDuck which use a push/pull adaptation layer.
According to my experience, working with tuners has always been more reliable on Linux. TSDuck was initially developed on Linux. Two years ago, it was ported to Windows, except the tuner interface. After a couple of years, I was more or less pushed to port the tuner part on Windows for one simple reason: my Linux machines with tuners were all desktop systems while my only laptop was a Windows office machine. I had to make tests on the terrestrial TV and moved to different regions to analyse local transmitters. So, I needed tuner support on my Windows laptop. This is the sad story of TSDuck on Windows... :)
My understanding of DirectShow improved a bit when you helped implementing ATSC and ISDB. If you have suggestions on how to improve the SinkFilter, please do. I must admit a have some reluctance to dig into it again and make lengthy tests and debug. Not to mention that I now got rid of any personal Windows machine. I just run a Windows VM on my Mac (USB redirection works pretty well for tuners with VMware Fusion). I still use a Windows laptop in my professional activity, but this is an independent domain.

Also, I'm streaming out via UDP, so in theory it could also be the disk-writing which is causing the streaming to stall in some way.

As mentioned, the corruption can appear before anything is done with the data such as writing to the disk.
My conclusion was that it was occurring higher up in the code before @lelegard's filter was getting hold of it.
I too have no current intention of diving into this code anytime soon tbh, if that changes then this would be a useful additional reference with your additional suggestions, thanks.

Also, I'm streaming out via UDP, so in theory it could also be the disk-writing which is causing the streaming to stall in some way.

As mentioned, the corruption can appear before anything is done with the data such as writing to the disk.

There is no "before" in a streaming chain that can block upstream data flow.

DirectShow forces the application to have a push-oriented streaming architecture

DirectShow supports both, push and pull models, even mixed in a single filter graph. Simple example for a pull-model is a file source filter connecting to an audio sink filter (e.g. default sound device). But in case of a BDA source - yes, the data is being pushed, which of course makes sense, because the data is received in realtime as the bytes come in and not when an application decides to consume them.
I think the best solution for the sink filter would be to have some memory stream implementation to which the Receive method can write data immediately without locking and avoiding to AddRef and store the actual IMediaSample objects.

Thanks, well it’s certainly useful to hear from someone else with knowledge in this area, I could’ve done with your input a couple of years ago when I was buried deep in this issue without much of a clue what I was doing 😊
Anyway thanks for the pointers. If I’m ever tempted to have another look then this will be very useful. Maybe @lelegard won’t be able to resist getting his hands dirty again, such is his love of DirectShow programming....

@oviano - may I ask which personal media server project you're working for?
(I know it's somewhat ignorant to ask, while I'm not allowed to disclose my project at this time..)

Of course, here is the website
emustream.tv
The site is a bit out-of-date for example where it mentions the thirdparty libs I’m using as I’ve changed a fair bit since then (for example the player no longer uses libVLC and I now use SRT for the stream) but the site explains the basic concept.
It’s basically a sort of customisable Slingbox-style system which I developed when I lived in Turkey and wanted as near-to-broadcast quality TV sent from the UK, ie proper high frame rate for sports viewing and using the best compression methods such as HEVC.
The code is currently on a private Gitea repo but I’ll probably put it out there publically at some point.

Thanks, looks like a nice project and luckily far enough away from mine.
I thought you might be interested in this:

The time between the 'Run' button click until you see the first picture includes starting up tsp, tuning to the channel, processing the stream (outside of tsduck), filtering out PES streams, creating HLS segments and playlist and starting MPV player to play the HLS stream.
PS: tsDuck was running in a Linux VM for this example.

Thanks for sharing, yes that looks nice and quick. Good to know!

I got the same with fmpeg video conversion adding only 0.5-1s additional delay.
It's been quite a way to get to that point and many details needed to get right. But one general advice that I can give is this:
Better use those components out-of-process instead of trying to plumb them together in a single application - especially for server-type applications where a certain level of reliability and error resiliency is required.

@lelegard - One thing I'm wondering about: Why isn't the DvbInputPlugin derived from PushInputPlugin? Wouldn't that make more sense?

I got the same with fmpeg video conversion adding only 0.5-1s additional delay.
It's been quite a way to get to that point and many details needed to get right. But one general advice that I can give is this:
Better use those components out-of-process instead of trying to plumb them together in a single application - especially for server-type applications where a certain level of reliability and error resiliency is required.

I started with that approach but as I became more and more familiar with the FFmpeg libraries I switched to doing exactly what you advise against and embedding the libraries directly in my framework.
I’ve never had a problem, and much prefer doing it this way and learning how the API works.

@lelegard - One thing I'm wondering about: Why isn't the DvbInputPlugin derived from PushInputPlugin? Wouldn't that make more sense?

It probably would if TSDuck was Windows-only.
But TSDuck is portable and, in principle, a tuner is "just" an input device like any other from which it is just natural to call some read() call. This is the case on Linux. And this would also be the case on Windows if there was a uniform userland API for BDA drivers. The actual and probably unique problem, as far as TSDuck is concerned, is that the uniform userland API was moved by Microsoft from BDA drivers to DirectShow filters. A very bad software design decision.
Anyway, the abstract class PushInputPlugin is just a buffering adaptation layer which converts the push model into a pull one, just as SinkFilter does. Using PushInputPlugin with dvb on Windows would simply move the push/pull adaptation layer into another class. This wouldn't solve the architectural problem. And we would loose the portability.
Interesting enough, PushInputPlugin was developed to cope with HTTP access where the problem is exactly the opposite: Windows uses a simple pull model where InternetReadFile() does a blocking read from the connection while libcurl uses a push model. There is one difference though : there are other ways than "curl easy" to use libcurl and there are other libraries too. But this one was the most widely spread and the most easy to use.

But TSDuck is portable and, in principle, a tuner is "just" an input device like any other from which it is just natural to call some read() call. This is the case on Linux.

Where is the data buffered on Linux? What happens when you don't read() fast enough? I mean, Linux can't infinitely buffer the data.

Anyway, the abstract class PushInputPlugin is just a buffering adaptation layer which converts the push model into a pull one, just as SinkFilter does. Using PushInputPlugin with dvb on Windows would simply move the push/pull adaptation layer into another class. This wouldn't solve the architectural problem.

I don't see an architectural problem actually. Just an implementation problem in the sink filter, that's all.

Using PushInputPlugin with dvb on Windows would simply move the push/pull adaptation layer into another class.

But that class would have a different implementation that does not block the DirectShow data flow.

And we would loose the portability.

Right. It's probably better to fix it in the Sink Filter directly.

@oviano

so that the HW decoded output frames are treated correctly on the various platforms

Sure. Client players is not what I meant, though.

treat FFmpeg as a blackbox

LOL. Not quite like that... I'm not running a regular version and I'm in fact very familiar with its code.

Where is the data buffered on Linux? What happens when you don't read() fast enough? I mean, Linux can't infinitely buffer the data.

All broadcast / multicast sources share the same principle: if you don't read fast enough, you loose data packets. This is equally true for IP multicast and DVB.

@oviano

so that the HW decoded output frames are treated correctly on the various platforms

Sure. Client players is not what I meant, though.

treat FFmpeg as a blackbox

LOL. Not quite like that... I'm not running a regular version and I'm in fact very familiar with its code.

Yes but client players are part of my project, and it's beneficial to share some of the code at least between server and client (demuxing, some parts of decoding etc). You offered advice, I'm explaining that my project is not the same as yours so my requirements are different and I have different priorities.
In terms of FFmpeg being a blackbox, I don't doubt you are very knowledgeable about those libraries, you shouldn't take what I wrote as some sort of slight! I'm merely explaining that my preference was to learn and understand the code; I started off by constructing all sorts of elaborate command line strings to pass to a launched FFmpeg process and in the end I decided I'd rather learn how to do it from the inside because for me, at that point, FFmpeg was a black box to me. In this sense, it's as much as personal hobby for me as anything else; similarly I started off with the player using SDL + libVLC, because this was an easy and quick way to get something up and running but over time this has evolved into using my own Windowing and input library, with sokol_gfx as the thin 3D API layer (at one stage I used bgfx, but recently switched to sokol - again because I bgfx was dragging in some extra dependencies and complication which I didn't need, whereas sokol_gfx is about as thin a layer as you can get for a 3D API wrapper), and then using libavcodec etc directly, rather than going through LibVLC.
In terms of stability, I assume you mean that it's better if FFmpeg code crashes if it's an external process, without bringing down your server, and that is a valid point. However, in my experience the code is pretty stable, and on the rare occasion it has crashed or hung I've usually figured out the reason. This process itself has often helped my understanding of the libraries and how everything fits together.
Error resilience is possibly a more valid concern, because ffmpeg.c (I think) contains a fair bit of logic for fixing broken timestamps and stuff like that, which you'd miss out on by linking directly, if you didn't re-implement it yourself but there is only a very limited subset of inputs to my project (essentially raw video captures, or broadcast mpegts) so it's not something that has been a problem for me. I can see if you were taking inputs from anywhere, files, a bigger variety of sources, then these might become bigger advantages.
Anyway, this is wildly off-topic, so I'll stop there. Good luck with your project 👍

I guess I need to apologize because I've been a bit too focused on my own requirements. Your guessing is right about the wide variety of input file, output requirements, hwa methods and platforms that I need to handle.
Another reason that caused me to identify with your efforts is, that when I had started into this a number of years ago, I had a very strong opinion against running ffmpeg in a separate process, as well. Today I'm still seeing user issues regularly where I think how glad I am that we haven't switched to an in-process model.
Apart from this, I agree to everything you said (your reply even includes my due apology-text).

Maybe @lelegard won’t be able to resist getting his hands dirty again, such is his love of DirectShow programming....

Argghhh, why are you doing this to me? 😢

I think the best solution for the sink filter would be to have some memory stream implementation to which the Receive method can write data immediately without locking and avoiding to AddRef and store the actual IMediaSample objects.

Some form of locking is needed anyway when you read/write a shared buffer. And AddRef is a negligible overhead.
The actual question is: is there is a fixed-size pool of MediaSample objects in a DirectShow graph? Or a maximum size for it? In that case, it would be beneficial to release the MediaSample objects early and buffer the TS packets in some other private internal pool. But if there is no limit to the number of allocated MediaSample objects (just the good will of everyone to release them when no longer necessary), I do not see the benefit of using another kind of buffer. Memory is just memory and using MediaSample objects may only add some negligible overhead.

Some form of locking is needed anyway when you read/write a shared buffer. And AddRef is a negligible overhead.

It's not the time overhead of calling AddRef. It's that this will prevent a sample from being re-used until it's released.

The actual question is: is there is a fixed-size pool of MediaSample objects in a DirectShow graph? Or a maximum size for it?

This depends on the IMemAllocator
IMemAllocator::GetProperties https://docs.microsoft.com/en-us/windows/win32/api/strmif/nf-strmif-imemallocator-getproperties
returns an ALLOCATOR_PROPERTIES structure:
https://docs.microsoft.com/en-us/windows/win32/api/strmif/ns-strmif-allocator_properties

These things are handled here:
https://github.com/tsduck/tsduck/blob/0f6a55f943a7f7b5852b868a67299b44b41e1886/src/libtsduck/dtv/windows/tsSinkFilter.cpp#L730-L746

I have just modified the No like this:
STDMETHODIMP ts::SinkPin::NotifyAllocator(::IMemAllocator* pAllocator, ::BOOL bReadOnly)
{
    TRACE(1, u"SinkPin::NotifyAllocator");

    ALLOCATOR_PROPERTIES props;
    pAllocator->GetProperties(&props);

    std::cerr << "SinkPin::NotifyAllocator - buffers: " << props.cBuffers << " buffer size: " << props.cbBuffer  << std::endl;

    return S_OK; // we don't care
}

The output I'm getting is this:
SinkPin::NotifyAllocator - buffers: 24 buffer size: 45120
while tsDuck uses:
static constexpr size_t DEFAULT_SINK_QUEUE_SIZE = 1000;  // media samples

That means that the sink filter may queue up to 1000 media samples while DirectShow won't supply more than 24 media samples (each having a max size of 45120 bytes)

That means that the sink filter may queue up to 1000 media samples while DirectShow won't supply more than 24 media samples (each having a max size of 45120 bytes)

Good to know, thanks! This may explain issues on Windows.
There are two options. The simplest one (for a DShow-hater like me), is to copy each MediaSample content into an intermediate object and queue those objects. But there is a copy overhead. The second option is to provide a custom "allocator" which allocates (or reuses) MediaSample objects. But I need to dive back into DShow 😱. Would you have a pointer to a sample MediaSmaple allocator, just to avoid some vomit on my side...
Note that there is already a buffer-copy overhead issue on Windows. tsp uses a large global TS packet buffer which is shared by all plugins. No copy ever occurs. On Linux, because LinuxTV is a simple API, packets are directly read from the tuner directly into the tsp global buffer. On Windows, because of the push model of DirectShow and its MediaSample objects, the packets are copied several times. This is why I would like to avoid having an additional level of buffering.

The simplest one (for a DShow-hater like me), is to copy each MediaSample content into an intermediate object and queue those objects.

A much more simple method might be to handle GetAllocatorRequirements and return a larger number of samples.
But I don't really want to suggest this.
Isn't there a way to immediately copy the data to the final "TSP packet buffer"?

Would you have a pointer to a sample MediaSmaple allocator, just to avoid some vomit on my side...

I don't think that it makes much sense to do a custom implementation of IMemAllocator.  I think that this would only make sense when it would be possible to return the final target memory range to which the data should be written to, but I don't think that this is possible at all. (samples can have variable lengths and need to be re-usable).
What's possible is to create the standard memory allocator with a custom set of allocator properties, but maybe this won't have a different result than just handling GetAllocatorRequirements.

Isn't there a way to immediately copy the data to the final "TSP packet buffer"?

The queue is the adapter between the push and pull models. We need at least some level of queuing when media samples are pushed and the TSDuck tuner thread is not currently trying to read from the tuner. When the queue is empty and the tuner thread is waiting for read, then it could be possible to copy the MediaSample content from SinkPin::Receive but this won't change the number of copy operations. Copying the TS packets directly from the tuner to the final tsp packet buffer would be possible only with a direct access to the BDA driver, as with Linux. The DirectShow push models imposes this additional level of data copy.

Why do we need to wait for the tuner thread in order to write to the packet buffer?

Why do we need to wait for the tuner thread in order to write to the packet buffer?

Because I do not want to break the complete design of TSDuck. For an input thread, waiting for and reading packets is a call to ts::Tuner::receive(). This method gets a buffer address and size where to receive packets. If no such call is currently in progress (ie. the input plugin thread is not currently reading), we do not know where to copy the packets.

OK, none of these things will work unfortunately.

GetAllocator does not get called
GetAllocatorRequirements does not get called

Both are optional:
https://docs.microsoft.com/en-us/windows/win32/directshow/negotiating-allocators

That means that the sink filter may queue up to 1000 media samples while DirectShow won't supply more than 24 media samples (each having a max size of 45120 bytes)

Each media sample can have up to 420 packets. The total buffering capability is 1 MB (1,082,880 bytes). Even with the fastest DVB-S2 stream, reading 10 times per second is enough. So, I do not really think that would cause packet loss.

GetAllocatorRequirements does not get called

Yes, the article you mention says:

Optionally, the output pin calls IMemInputPin::GetAllocatorRequirements.

So, based on the graph below, if the Tee filter does not negociate the allocator requirements, we have no way to resize the number of MediaSamples.
Tuner --> Receiver --> Tee -+-> SinkFiler (provided by TSDuck)
                            |
                            +-> MPEG-2 demux --> TIF

Each media sample can have up to 420 packets

45120 / 188 = 240

Each media sample can have up to 420 packets. The total buffering capability is 1 MB (1,082,880 bytes). Even with the fastest DVB-S2 stream, reading 10 times per second is enough. So, I do not really think that would cause packet loss.

Another idea: Maybe it's better to block in the receive method and wait until we can write directly to the TS Packet Buffer.
I think, anything is better than AddRefing the samples.

Each media sample can have up to 420 packets. The total buffering capability is 1 MB (1,082,880 bytes). Even with the fastest DVB-S2 stream, reading 10 times per second is enough. So, I do not really think that would cause packet loss.

Shouldn't this be viewed the other way round?
When the tuner thread doesn't poll for longer than 0.1s, the buffer is getting exhausted and packet loss occurs.

So, based on the graph below, if the Tee filter does not negociate the allocator requirements, we have no way to resize the number of MediaSamples.
Tuner --> Receiver --> Tee -+-> SinkFiler (provided by TSDuck)
                            |
                            +-> MPEG-2 demux --> TIF

Good point. For some reason I thought we would be behind the MPEG2 demux, but of course we're not.
In that case - we do not want to increase the sample count anyway. AND we must not block. All this can increase tuning time because the network provider communicates with the MPEG2 demux and the TIF.
Hmmm....could that even be the reason why tuning times are slower on Windows?`
When the Sink Filter blocks at the input side, this could block the Tee splitter in turn and prevent it from sending to the MPEG2 demux.

Each media sample can have up to 420 packets

45120 / 188 = 240

Yes, sorry, I have always been a bit dyslexic with the keyboard. I am sure you can find a few inverted letters in the doc, comments or anywhere the compiler doesn't spot such errors...

Another idea: Maybe it's better to block in the receive method and wait until we can write directly to the TS Packet Buffer.
I think, anything is better than AddRefing the samples.

I totally disagree.
In the Microsoft doc here about IMemInputPin::Receive method:

This method is synchronous and possibly blocking. The pin does one of the following:
Rejects the sample.
Returns immediately and processes the sample in a worker thread.
Processes the sample before returning.
In the last case, the method might block indefinitely. If this might happen, the IMemInputPin::ReceiveCanBlock method returns S_OK.
If the pin uses a worker thread to process the sample, it holds a reference count on the sample. In any case, the output pin cannot directly re-use this sample. It must call the IMemAllocator::GetBuffer method to obtain a new sample.

TSDuck is clearly in the case of a "worker thread" and "holds a reference count on the sample". This is much more efficient and "fluid" than blocking. Blocking would definitely increase the risk of packet loss.
In that case, it is totally normal to AddRef. The corresponding Release is performed in SinkFilter::Read, after the content of the media sample is returned to ts::Tuner::receive().

When the tuner thread doesn't poll for longer than 0.1s, the buffer is getting exhausted and packet loss occurs.

Yes, I meant the same thing, it was just formulated in another way. Given the way tsp is designed, there is zero delay between calls to ts::Tuner::receive(). Once a bunch of packets is received in the input plugin (whichever it is), the corresponding slice of the global buffer is logically passed to the next plugin and all the free space in the global buffer is immediately reused for the next ts::Tuner::receive(). See a diagram here.
So, in practice, the input thread will read packets much faster than 10 times per second. It is litterally reading packets all the time. The input plugin only blocks when waiting for packets (in ts::Tuner::receive() in the case of the dvb plugin).
If the input plugin blocks longer between two reads, this can be only because the global buffer is full (the thread then waits for free space in the buffer). In that case, we have much bigger problems anyway. That means that there is one or more plugin which "significantly" blocks the flow. I mean "significantly" because each plugin runs in its own thread, all of them, using all CPU cores in the system. This asynchronous architecture was designed from the beginning, precisely to avoid losing input packets. Since this design was adopted as early as 2005 to read from a Dektec ASI board up to 270 Mb/s on a Pentium IV at that time, I am rather confident that the input plugin will not be the bottleneck.

When the Sink Filter blocks at the input side, this could block the Tee splitter in turn and prevent it from sending to the MPEG2 demux.

For the same reason, I do not think that this explains the issue, the SinkFilter will likely not block, certainly not in a perceptible way as the obviously long zapping time.
Maybe I would have better implemented the SinkFilter that way:
Tuner --> Receiver -+-> SinkFiler --> MPEG-2 demux --> TIF
                    |
                    V
                    sink queue to ts::Tuner

The SinkFilter would have implemented the Tee itself, avoiding the fork in the graph. But, at that time, when I could not make it work and could not find any clear documentation on how to use DirectShow graphs, I used DirectShow Spy to view how proprietary applications such as WinTV built graphs. The TSDuck graph is exactly like those, with the SinkFilter replacing the rendering path.
In fact, functionally speaking, TSDuck does not need any demux or TIF because we just read all TS packets. But, in tha absence of demux and TIF, the graph does not start.
And honestly, today, I do not feel like redesigning the SinkFilter.

And honestly, today, I do not feel like redesigning the SinkFilter.

I don't either. I don't think it's time to start into developing something, because all we have right now is a couple of assumptions, but we don't know yet what's happening exactly.
So I would rather start with some logging that regularly prints out

the queue size
the number of threads waiting at the locks
the time span between subsequent Read calls (from the tuner thread)

We should also record and report the maximum values for these values.
Maybe not use the TRACE calls, as that might generate too much noise.
I can't help testing this, because I can't reproduce here.
Meanwhile, I'll try a different route: The plan is to create another DirectShow sink filter that will directly send out all received data via UDP. I hope this will allow me to gain some more insights into the latency issue.

Meanwhile, I'll try a different route: The plan is to create another DirectShow sink filter that will directly send out all received data via UDP. I hope this will allow me to gain some more insights into the latency issue.

This is good idea but I would be careful with this. This is a recipe for packet loss because the UDP reception kernel buffers are probably smaller than TS input. If is worth a try however.

UDP reception kernel buffers

Why "reception"?

Why "reception"?

Or better: what do you think would be different compared to using the IP output plugin from tsDuck?

I mean on the UDP receiver side. Sending is fine, but someone has to receive it. Receiving traditional UDP multicast (IPTV) means receiving small datagrams (typically 7 TS packets) on a regular basis, at the bitrate of an SPTS. Even if you are not fast enough to read each datagram on time, a few of them can be buffered. On the other hand, DVB MPTS streams have higher bitrates and media samples from the DirectShow filter are maybe larger than 7 TS packets (to be checked) since the buffer size is 240 packets (in the right order this time:)). If you send large UDP datagrams from the input pin's Receive() method, you should benchmark the effect on typical UDP receivers (including tsp, but not only). And if you cut the media sample is smaller pieces, either you send them all right away and your burst effect can be devastating, or you smooth them and you block in the input pin's Receive(), a method I did not recommend in a previous post.

Thanks for the explanation. I'm using UDP from the very first moment on to connect tsDuck to our application and it's working rock-solid.
But I had almost forgot how painful it was to get there.  I just looked it up and in fact, I'm adjusting the socket receive buffer, so you just hit the right point. (among others)

Today I have created a new UdpSinkFilter which - when used instead of the regular SinkFilter - sends out data via UDP directly.
This way, I was able to achieve the same startup times like on Linux. The delay of 1-2s that I have been experiencing on Windows vs. Linux doesn't exist anymore.
To be fair and transparent, here's some details on what I've compared:

Tuner, antenna and tuning parameters are always the same
I'm testing Linux in a virtual machine with the USB tuner being redirected from the Windows host
Received data is transmitted via UDP to a Windows application
Of course, the UDP Sink Filter is bypassing all TSP processing, while for Linux, the data is going through TSP processing and sent out via the IP output plugin
(I don't know how much delay the TSP processing itself is adding here)

This confirms what I have suspected above: This is not a DirectShow problem and not a BDA driver problem. It's an implementation problem in tsDuck.
I still need to find a nice way to keep TSP alive even when it doesn't receive anything from the tuner...

Interesting finding, nice test 👍
It is certainly surprising if the buffering is adding that much latency...

It's great that you found a solution for your application.
I would like to understand if the issue comes from the SinkFilter itself or from the graph setup and tuning. When you use your UDP sink filter, do you use the rest of TSDuck, the code that sets up the graph and sends the tuning request, or do you use a more traditional DirectShow application with only your UDP sink filter as specific piece?

It is certainly surprising if the buffering/locking/whatever-is-happening is adding that much latency...

You can play with options --max-input-packets and --max-flushed-packets to alter the buffering. But, with the dvb plugin, the default value is 1000 packets, which represent only 0.06 seconds of a 24 Mb/s DVB-T stream or 0.02 seconds of a 70 Mb/s DVB-S2 stream. Locking does not seem a realistic explanation either.

To add some more value rather than just make claims, I have done some more testing and recorded the log timings for the three cases to compare:
1. Normal TSP Run on Windows:
11:55:09.031 tsp: TSDuck - The MPEG Transport Stream Toolkit - version...
11:55:09.507 * dvb: tuned to transponder --frequency 370000000 --symbol-rate 6900000 --fec-inner none --spectral-inversion auto --modulation 256-QAM
11:55:09.507 * dvb: starting tuner reception
11:55:10.804 * dvb: signal locked: yes, strength: 100%, quality: 76%
11:55:12.610 All threads started.
11:55:12.640 FirstDataReceived

Total: 3.609s
2. Normal TSP Run on Linux VM:
11:38:21.720 tsp: TSDuck - The MPEG Transport Stream Toolkit - version...
11:38:21.896 * dvb: tuned to transponder --frequency 370000000 --symbol-rate 6900000 --fec-inner none --spectral-inversion auto --modulation 256-QAM
11:38:21.897 * dvb: starting tuner reception
11:38:22.148 * dvb: signal locked: yes, strength: 62%
11:38:23.399 All threads started.
11:38:23.421 FirstDataReceived

Total: 1.701s
3. TSP Run on Windows using UDP Sink Filter
11:43:44.297 tsp: TSDuck - The MPEG Transport Stream Toolkit - version...
11:43:44.820 * dvb: tuned to transponder --delivery-system DVB-C --frequency 370,000,000 --symbol-rate 6,900,000 --fec-inner none --modulation 256-QAM
11:43:44.820 * dvb: starting tuner reception
11:43:46.079 * dvb: tuner reception started
11:43:46.112 * dvb: signal locked: yes, strength: 100%, quality: 76%
11:43:46.156 FirstDataReceived

Total: 1.815s
Notes:

The results are very reproducible, I ran each variant multiple times and the variance was below 50ms
I have a local modification to always print out the version, so this can be considered as the TSP startup time
All lines are from TSP (as received by my test application), except FirstDataReceived, which indicates the first moment when data is received via UDP

When you use your UDP sink filter, do you use the rest of TSDuck, the code that sets up the graph and sends the tuning request, or do you use a more traditional DirectShow application with only your UDP sink filter as specific piece?

I've done this in a very simple way:

I've copied the SinkFilter
Removed the queuing stuff
Then I copied the relevant parts from the IP Output plugin (except the RTP stuff)
Changed the graph building to use the new UdpSinkFilter instead
Added 20s Sleep to the tuner::Read function
(to avoid TSP from exiting immediately)

PS: I can share this via PR, but it's just experimental code with the UDP target being hardcoded.

The "All threads started" message comes from this place (or rather the equivalent place in an earlier code):
https://github.com/SS-FF/tsduck/blob/4d28b68082d3f48d9d9b3874c4d136016cd321f0/src/libtsduck/plugin/tsTSProcessor.cpp#L241-L248
(before 'Create Control Server Thread')

Fasten your seat-belts, because I got something really weird to report:
My processing chain goes from tsDuck DVB+UDP to my custom stream processing, parsing all tables and descriptors, filtering program streams, parsing video parameters, rewriting PAT&PMT, detecting IDR, segmenting for HLS and serving this to the players, e.g. MPV.
What I have been observing quite often are small glitches in the video/audio streams which are causing errors being logged by MPV player. They are occurring during the first few seconds only after which they disappear.
I used to think that I might have some kind of bug in my code, but now - guess what...
It doesn't happen anymore when I'm directly streaming out the data with my new UDP sink filter.
Those issues occur both on Linux and Windows.
I haven't investigated at the stream level; but what I can say is that it's not about loosing sync (at least not when the data arrives at my application). It rather looks to me as if some packets would be omitted, but more like a single packet occasionally.
I had never considered tsDuck as a possible culprit for this, but now that it's "fixed", I'm starting to wonder..

Are you able to run both scenarios at the same time and capture both and see if the files differ?

Hi @SS-FF,
Thanks for reporting these time data. I tried to separate two durations: 1) from start to "signal locked" (tuning) and 2) from "signal locked" to "FirstDataReceived" (reception).

Windows
Linux
Win UDP SinkFilter

tuning
1.773
0.428
1.815

reception
1.836
1.273
0.077

The tuning time is similar in the two Windows cases, which is quite normal if you use the same code. It is 4 times faster on Linux. This is the difference between a simple driver API and a complex streaming system.
With the TSDuck standard code, the reception time is 50% higher on Windows compared to Linux but both are over 1 second. I can only blame the buffering. What are the buffering parameters? If you want to make sure, try tsp --debug=10 and search for "passPackets" messages. How many packets at a time are passed from the input plugin to the next one?
What is the "FirstDataReceived" event in the case of the UDP sink filter? Is this inside tsp? How get the packets reinjected in tsp?

as if some packets would be skipped, but more like a single packet occasionally, not a range of packets

This is not possible inside tsp because the code is deterministic and fully synchronised. This can occur before the input plugin (DVB reception, driver issue, dropped packets in SinkFilter because of queue overflow). This can also occur after the output plugin. How does your player read packets from tsp output? If this is using a pipe, then no packet loss can occur. If this is using UDP, then it is possible that the ip output plugin produces bursts which make the player miss some input packets.

With those timings - could it be that on Windows, the “tuning” also includes the time taken to buffer the first load of data, whereas on Linux it does not?

Thanks for reporting these time data. I tried to separate two durations: 1) from start to "signal locked" (tuning) and 2) from "signal locked" to "FirstDataReceived" (reception).
Windows 	Linux 	Win UDP SinkFilter
tuning 	1.773 	0.428 	1.815
reception 	1.836 	1.273 	0.077
The tuning time is similar in the two Windows cases, which is quite normal if you use the same code. It is 4 times faster on Linux. This is the difference between a simple driver API and a complex streaming system.

Nah, really - that doesn't make any sense. I think your "separation or durations" is not valid. It's probably more a result of where you have put the "signal locked" log statement in the Linux vs. the Windows implementation.
0.077 is not a realistic value for "reception" and "0.428s" is not a realistic value for tuning.

What is the "FirstDataReceived" event in the case of the UDP sink filter?

Please see above:

All lines are from TSP (as received by my test application), except FirstDataReceived, which indicates the first moment when data is received via UDP (by my application)

as if some packets would be skipped, but more like a single packet occasionally, not a range of packets

This is not possible inside tsp because the code is deterministic and fully synchronised. This can occur before the input plugin (DVB reception, driver issue, dropped packets in SinkFilter because of queue overflow).

No, because it happens both on Linux and Windows

This can also occur after the output plugin.

Probably yes...

How does your player read packets from tsp output? If this is using a pipe, then no packet loss can occur. If this is using UDP, then it is possible that the ip output plugin produces bursts which make the player miss some input packets.

I'm using a specific burst value and my application is adapted for this value. (also, I get an exception when the burst size is larger than what it should be). Further, those glitches occur during the first few seconds only.

Are you able to run both scenarios at the same time and capture both and see if the files differ?

I can do everything simultaneously at the receiving side:

But I suppose you mean to add both filters (Sink+UdpSink) inside tsDuck?

Nah, really - that doesn't make any sense. I think your "separation or durations" is not valid. It's probably more a result of where you have put the "signal locked" log statement in the Linux vs. the Windows implementation.
0.077 is not a realistic value for "reception" and "0.428s" is not a realistic value for tuning.

And it's just like I was suspecting:

ts::Tuner::start on Linux
Just polls the tuning status and exits once OK
ts::Tuner::start on Windows

Performs a sleep
Flushes the sink filter
Performs a blocking read on the sink filter

So, the criteria for reporting "signal locked" are totally different. On Linux it's just waiting for the tuner status, while on Windows, it waits until it actually has received some data (plus doing something that shouldn't be there).

I think the above is already the (or at least one of the) reason(s) for the delay that tsDuck exposes on Windows:
There is a delay of a few hundred milliseconds between the moment when a tuner has finished and locked the tuning and the moment when it starts to stream data.
On Linux, we exit the ts::Tuner::start method as soon as the tuner status indicates that it has locked the tuning.
That leaves enough time for the rest of the TSP setup to complete and start polling the tuner for data even before the first data arrives.
On Windows, we stay inside the ts::Tuner::start method much longer and when we exit there, the streaming is already in progress - but we haven't even completed the TSP setup and the pulling of data from the tuner is yet to be started.
As per our previous calculations, we need to read data at least every 100ms, but completing the TSP setup seems to take significantly longer (that's the time between "signal locked" and "all threads started") - always at least > 1 second.
In Summary:

We have two tasks that take time - let's say 1 second both:

Tuner time between signal lock and streaming start
TSP finishing setup after ts::Tuner::start method returns

On Linux, these two are happening in parallel
On Windows, these are performed sequentially
(because the ts::Tuner::start method waits for data to arrive)

I think this explains a large part of the delay. Another additional delay might be added by the blocking queue implementation of the sink filter.

On Linux, these two are happening in parallel
On Windows, these are performed sequentially

No, they are sequential in both cases. Everything happens in the input plugin thread where DVBInputPlugin::start() is called first and then multiple calls to DVBInputPlugin::receive(). So, I agree that the proportion of time in start and first receive may vary between Linux and Windows but the sum of the two should be still comparable. On Linux, we may exit faster from start() but we will wait for the first packet in receive().

No, they are sequential in both cases. Everything happens in the input plugin thread where DVBInputPlugin::start() is called first

On Windows, the start method doesn't exit before data is received.
On Linux, the start method exits as soon as the tuner status is locked.
Do you disagree to that?

I do not disagree. But we do not care about the start() time. We care about the time it takes to get a picture after deciding to tune. And, about this, we do not care if waiting for the first packet is done in start() or the first receive().

What's you explanation then, for why it takes 2 seconds longer with the Windows implementation?

Don't say "DirectShow" - I just proved that it's not.. 😄

Different question:
I'm not sure yet, but assuming I would stick to the approach with the UDP sink filter, what do you think would be the best approach to keep the TSP running?
Of course, it's calling into ts::Tuner::receive regularly and the process ends when I return zero. What should I do?
Block for long periods? Return lots of null packets?
Or is there a more elegant possibility?

I do not disagree. But we do not care about the start() time. We care about the time it takes to get a picture after deciding to tune. And, about this, we do not care if waiting for the first packet is done in start() or the first receive().

I have done further research and I need to admit that I was wrong. In fact, it doesn't matter a lot whether the ts::Tuner::start method is exited sooner or later - at least no with regards to the output delay.
To investigate this, I have done the following tests:
1. Resemble the Linux Functionality for Windows
It is in fact possible to do the same kind of polling for the "IsLocked" status like it's done for Linux:
bool ts::Tuner::start(Report& report)
{
    IBaseFilter* const sink = _guts->graph.sinkFilter();

    if (!_is_open || sink == nullptr) {
        report.error(u"tuner not open");
        return 0;
    }

    // Run the graph.
    if (!_guts->graph.run(report)) {
        return false;
    }

    // Wait for input signal locking if a non-zero timeout is specified.
    MilliSecond pollInterval = DEFAULT_SIGNAL_POLL;

    bool signal_ok = false;
    for (MilliSecond remain_ms = _signal_timeout; remain_ms > 0; remain_ms -= pollInterval) {

        if (!_is_open || !_guts->graph.isRunning())
        {
            report.log(Severity::Error, u"Tuner start failed.");
            return false;
        }

        // If the input signal is locked, cool...
        signal_ok = signalLocked(report);
        if (signal_ok) {
            break;
        }

        // Wait the polling time
        SleepThread(pollInterval < remain_ms ? pollInterval : remain_ms);
    }

    // If the timeout has expired, error
    if (!signal_ok) {
        report.log(_signal_timeout_silent ? Severity::Debug : Severity::Error,
                   u"no input signal lock after %d milliseconds", {_signal_timeout});
        return false;
    }

    return true;
}

That works in fact, but it doesn't change much:

Most of the time is spent waiting for the FilterGraph.Run() method to return
In summary, this doesn't provide any advantage - the ::Start method does not complete earlier

2. Asynchronously run the Filtegraph from a worker thread
That didn't change much either. Instead of waiting for the :Run method to complete, the code flow was eventually driven by the code above, checking for the lock property becoming set - actually exactly as intended, but it still didn't improve the startup time.
3. Don't run the filtergraph at all
For checking out the far edge, I also tried to immediately return true from ts::Tuner::start, which didn't help either.
The only interesting observation was, that it isn't required to have the filtergraph running in order to perform the tuning. When calling ::Run right after submitting a tuning request, the ::Run method can block for 0.5-1.5s until it completes (it's internally waiting for the tune operation to complete).
But when we process the locked-stated like in the code above, it's not required to have a filtergraph running. And when we call ::Run after a successful tuning operation, that method returns almost instantly with the graph being in the running state.
Conclusion
A few things learned, but the core issue is still an unsolved mystery...

But we're getting closer...
I had added two additional log statements before I did the tests above:

Output log line in the Sink Filter when the first data is received from the tuner
Output log line in the IP Output plugin when the first datagram is sent out

And there it gets interesting...
Example
03:28:24.690 tsp: TSDuck - The MPEG Transport Stream Toolkit - version...
03:28:24.720 * dvb:
03:28:25.198 SinkPin::ReceiveConnection - Enter
03:28:25.222 * dvb: tuned to transponder --delivery-system DVB-C --frequency 370,000,000 --symbol-rate 6,900,000 --fec-inner none --modulation 256-QAM
03:28:25.222 * dvb: starting tuner reception
03:28:26.459 * dvb: tuner reception started
03:28:26.467 SinkPin::Receive - First Data Received
03:28:26.490 * dvb: signal locked: yes, strength: 100%, quality: 75%
03:28:28.353 All threads started.
03:28:28.354 IPOutputPlugin::sendDatagram - First Data Sent
03:28:28.409 FirstDataReceived

What can be seen here is that there's a time span of 2 seconds between the first reception of tuner data and the first UDP packet being sent out.
And that was the one thing that remained constant (like between 1.5 and 2.5s)  during all the experiments I did.
Haven't come to find out the reason for this yet.,.

That is the buffering effect of the tsp pipeline. tsp is not only a streamer on live streams. It also processes offline files (very common in VoD or post-mortem problem analysis). So, since nothing is free, tsp has to find a balance between:

Lower CPU usage at the expense of higher latency, typically for batch processing of files. The multi-threading overhead is limited using a large global buffer and passing large chunks of packets between plugins.
Lower latency at the expense of higher CPU usage, typically for live streams. To remove the latency, each thread does not process many packets before passing them to the next plugin. The latency is reduced but the multi-threading overhead (and CPU usage) is higher.

The option tsp --realtime is used as a hint by all plugins to adopt suitable default values for streaming. This option is also forced by the presence of plugins such as dvb or dektec. But, better default values does not mean ideal in all cases. You may want to fine tune them using options such as --initial-input-packets, --max-input-packets or --max-flushed-packets or RTFM in general.

Hi @SS-FF,
As we all know, RTFM is the most important acronym ever in the history of computer science.
After writing my previous reply, I realized that I mentioned option --initial-input-packets:
  --initial-input-packets value
      Specify the number of packets to initially read in the buffer before
      starting the processing. The initial load is used to evaluate the bitrate
      so that all subsequent plugins can have a valid bitrate value from the
      beginning. The default initial load is half the size of the global
      buffer.

So, when tsp starts, it preloads half the buffer before starting the flow of packets. There are two reasons for this default.

Half of the buffer ensures an optimal usage of the global buffer, preventing most cases of plugin being temporarily blocked by underflow or overflow.
Half of the global buffer is usually enough to get an idea of the input bitrate, based on PCR's. Having a valid value of the bitrate from the beginning is useful to many plugins (the input bitrate is propagated all along the chain of plugins).

How much is the default, half of the global buffer?
  --buffer-size-mb value
      Specify the buffer size in mega-bytes. This is the size of the buffer
      between the input and output devices. The default is 16 MB.
      The value may include up to 6 meaningful decimal digits.

So, once tuned, tsp reads 8 MB before releasing the streaming. On a 24 Mb/s DVB-T stream, 8 MB represent 2.6 seconds. Sounds familiar?
Again, tsp is in the middle of a trade-off between offline and live processing. The default values are "average" but individual options shall be tuned according to each specific objective. Here, your objective is to speed up the start.
So, let's play with --initial-input-packets. Using the default value (half buffer, 8 MB) on a DVB-T stream, on Windows running in a VMware VM on a macOS host (I told you I have no longer any Wincrap machine):
$ tsp --timed-log --verbose -I dvb --uhf 22 -P until --seconds 10 -O drop
* 2020/12/06 15:37:48 - dvb: using AVerMedia TD310 BDA Filter (DVB-T, DVB-T2)
* 2020/12/06 15:37:48 - dvb: using default deliver system DVB-T
* 2020/12/06 15:37:48 - dvb: tuned to transponder --delivery-system DVB-T --frequency 482,000,000 --modulation 64-QAM --high-priority-fec auto --low-priority-fec auto --bandwidth 8-MHz --transmission-mode 8K --guard-interval 1/32 --hierarchy none
* 2020/12/06 15:37:50 - dvb: signal locked: yes, strength: 100%, quality: 80%
* 2020/12/06 15:37:53 - dvb: initial input bitrate is 24,882,395 b/s

Without modification, the time-stamps do not display subsecond precision. But it is sufficient to see roughly 3 seconds between "signal locked" (end of start) and "initial input bitrate" (first receive). The theoretical value was 2.6 seconds to fill 8 MB at a DVB-T bitrate.
Now with a very small --initial-input-packets:
$ tsp --timed-log --verbose --initial-input-packets 10 -I dvb --uhf 22 -P until --seconds 10 -O drop
* 2020/12/06 15:39:02 - dvb: using AVerMedia TD310 BDA Filter (DVB-T, DVB-T2)
* 2020/12/06 15:39:02 - dvb: using default deliver system DVB-T
* 2020/12/06 15:39:02 - dvb: tuned to transponder --delivery-system DVB-T --frequency 482,000,000 --modulation 64-QAM --high-priority-fec auto --low-priority-fec auto --bandwidth 8-MHz --transmission-mode 8K --guard-interval 1/32 --hierarchy none
* 2020/12/06 15:39:04 - dvb: signal locked: yes, strength: 100%, quality: 80%
* 2020/12/06 15:39:04 - dvb: unknown initial input bitrate

Here, the "signal locked" and "initial input bitrate" occur within the same second. The initial bitrate is unknown but we probably don't mind if we just want to grab a service. The input bitrate is anyway continuously evaluated and after a couple of seconds it will become known and propagated all along the chain of plugins. The only issue is that plugins won't be informed immediately of the bitrate. But you can't have everything at the same time.

@lelegard is any of the above relevant or configurable when using the API directly?
I'm basically doing:
m_tuner->tune(tuner_parameters, *m_report)
m_tuner->start(*m_report)
m_tuner->setReceiveTimeout(ts::MilliSecond(50), *m_report)
number_of_packets = m_tuner->receive(packets, PACKET_BUFFER_SIZE, nullptr, *m_report)
With a cursory RoTFM, I can see
setSinkQueueSize
setDemuxBufferSize
In my case, I pass the data straight onto the libavformat/libavcodec to demux and decode so I'm. not interested in any unnecessary analysis that TSDuck might be doing.

Ho @oviano

is any of the above relevant or configurable when using the API directly?

No, this is only the size of your initial receive(). In the case of tsp with the default options, it calls receive() again and again until 8 MB are read. Then, the tsp processing starts.
To get back to your initial post:

The slowest bit in the above, is tuner->start(). This takes 4-5 seconds. After that the TSDuck demux is more or less instant, and it's maybe 0.5s-1.0s for AVFormat to fully demux the retained packets in a format that AVCodec etc can use.

I am afraid that the initial start() is very slow on Windows. It's two seconds on the log of my previous post but it may vary with different tuner hardware, driver and DShow filter.

ts::Tuner::setSinkQueueSize

Windows-specific. This is the maximum size of the internal queue of media sample in the SinkFilter object. No relation.

ts::Tuner::setDemuxBufferSize

Linux-specific. This is a the size of the internal buffer of the /dev/dvb/adapterX/demuxX device. No relation either.

Great, thanks for that. All makes sense. It looks like I’m as fast as I can be then with this particular tuner/OS.

As we all know, RTFM is the most important acronym ever in the history of computer science.

"Read the f***in manual" is surely a valid answer in many cases.
But then you also shouldn't mind me responding
"When it's so obvious, why didn't you  f***in tell us about it 80 posts before?" 😆

I’m closing this a I think we have exhausted the various possibilities and with my experimental changes documented elsewhere (DirectShow push API) I don’t think there is anything further in the TSDuck code that might speed up channel changes in Windows.
I’ve at least now got my app only going through the whole tuning process on a channel change if it’s a new frequency, so channel changes within one frequency are now nice and quick.

Actually, we're far from having exhausted all possibilities yet.
There are at least three more ways (none of them related to what we have discussed above), but I haven't had enough time so far to explore these.


⚠ Bofrost.es has degraded performance

In 51c63a9, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 1743 ms

Resolved: Bofrost.es performance has improved in 060b4f0 after 7 minutes.


Use setex command with older versions of redis

My team's development environment is stuck with Redis 2.2 for now. 2.2 doesn't support the extended nx and px arguments for set, which came along in Redis 2.6. This PR tries using setex as a fallback if the server doesn't recognize those extra args. It will allow me to use redlock-py in development until we all get with the Redis 2.6+ program. Others in the same boat as me may find it useful as well.

This doesn't seem safe. Since acquiring the lock is now two operations and not atomic or locking in their own right we cannot guarantee that two clients didn't ask for the lock at the same time and received it. It's probably safe for development work, but I'm not comfortable pulling it in when it could be used in production.

You're right, glad we've got 2.6 in production! Thanks for taking a look.


Replacing our custom enum RustResult with Swift's Result enum

The Problem
Right now we define a Swift enum RustResult.
https://github.com/chinedufn/swift-bridge/blob/420ed791c0cef08f7cfff7f4024ee39cd13d31c2/crates/swift-bridge-build/src/generate_core/result_support.rs#L2-L5
We use RustResult instead of Swift's standard Result type since Swift's Result<T, E> requires the E to conform to Swift's Error trait.
In cases where the E conforms to Error we should use a Swift Result instead of a RustResult.
For example, given the following Swift code:
final class MyOpaqueError: Error {
    // ...
}

func downloadImage(callback: (Result<Vec<u8>, MyOpaqueError>) -> ()) {
    // ...
    callback(Result.success(bytes))
}

We want to be able to import the Swift downloadImage function into Rust with a bridge module along the lines of:
#[swift_bridge::bridge]
mod ffi {
    extern "Swift" {
        #[swift_bridge(Error)]
        type MyOpaqueError;

        #[swift_name = "downloadImage"]
        fn download_image(callback: Box<dyn FnOnce(Result<(), MyOpaqueError>)>;
    }
}

Today we have to use a RustResult on the Swift side, instead of the above Result.
// Note the `RustResult` here. We want it to be possible to use `Result` instead.
func downloadImage(callback: (RustResult<Vec<u8>, MyOpaqueError>) -> ()) {
    // ...
}

Potential Solution

[ ] We need to implement support for the #[swift_bridge(Error)] attribute shown above. We'd track whether or not this this Error attribute was present on the opaque type https://github.com/chinedufn/swift-bridge/blob/07a15d78782dc371a7d24301ac9625124e42f8d8/crates/swift-bridge-ir/src/parse/parse_extern_mod/opaque_type_attributes.rs#L17-L35

[ ]  When Error is present on an imported opaque Swift type it would emit Swift code along the lines of:
extension MyOpaqueError: __SwiftBridgeCheckIfError {}

protocol __SwiftBridgeCheckIfError: Error {} would get defined inside of an error_support module in swift-bridge-build https://github.com/chinedufn/swift-bridge/blob/e8d3859d2a68b5d5e0012185a87e88b3c811f853/crates/swift-bridge-build/src/generate_core.rs#L13-L14

[ ] During Result related code generation if there is an Error attribute then we can use Swift's Result instead of RustResult.

We can add a codegen test for this similar to https://github.com/chinedufn/swift-bridge/blob/2ab37cbb949be4a466848129e7f17ecbb3fecf5e/crates/swift-bridge-ir/src/codegen/codegen_tests/result_codegen_tests.rs#L67-L82
And an integration test similar to https://github.com/chinedufn/swift-bridge/blob/2ab37cbb949be4a466848129e7f17ecbb3fecf5e/crates/swift-integration-tests/src/result.rs#L10
and similar to https://github.com/chinedufn/swift-bridge/blob/2ab37cbb949be4a466848129e7f17ecbb3fecf5e/SwiftRustIntegrationTestRunner/SwiftRustIntegrationTestRunnerTests/ResultTests.swift#L11-L15

I want to try to address this this week.

Sounds great!

were you able to look into this?

@DegenTim , I haven't done it yet. If you are interested in this, I'll leave this to you.


Updated nnvm with fixed tvm submodule

Description
Follow up on missing tvm commit in nnvm repository: https://github.com/dmlc/nnvm/pull/525

This is not a proper fix as you're changing the nnvm package content.

@szha you're right, thank you for the fix: https://github.com/apache/incubator-mxnet/commit/ea0aec432f30f0a067ee695b946173d4e3940e4a


hidpp10: Parse G9 profile

This allows to read but not write profiles from the G9.
The dpi mode sturct is a little different as the mouse only supports
symetrical dpi for x/y. Given the current naming a suffix of _sym
was the best I could come up with.
The G9 profile has some additional unknown bytes compared to the
others. Unfortunately the content of them varies for the 4 first
profiles and the last. So adding a default value for the like with
the G700 is not possible. For this reason I dropped implementing
the write part for now.
Something else to address is that this mouse only has 10 buttons
compared to the usual 13 that we have in PROFILE_NUM_BUTTONS. For
now I just hardcoded it in the profile and hidpp10_get_profile.

The dpi mode sturct is a little different as the mouse only supports symetrical dpi for x/y. Given the current naming a suffix of _sym was the best I could come up with.

I think it might be better to rename the other with _dual than having this named _sym (or _single).
Anyway, not a big deal.

The G9 profile has some additional unknown bytes compared to the others. Unfortunately the content of them varies for the 4 first profiles and the last. So adding a default value for the like with the G700 is not possible. For this reason I dropped implementing the write part for now.

Might be easier to do what I did for the G500: just read the current values, and overwrite only what you know. This way, you keep the current data.

Something else to address is that this mouse only has 10 buttons compared to the usual 13 that we have in PROFILE_NUM_BUTTONS. For now I just hardcoded it in the profile and hidpp10_get_profile.

That's not very clean, but on the other hand... if the profile is that way... Could you do a follow up patch where you create a #define for it instead of having the plain value here and there?
I am pulling this in master. It seems pretty consistent an should not break any other hardware. Please send cleaning follow up patches :)


⚠ Bofrost.lu has degraded performance

In ff90de7, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 2115 ms

Resolved: Bofrost.lu performance has improved in 18fbc39 after 28 minutes.


Enhance derToPem to support XML pretty-print

When pulling this library to node-saml, it was discovered that there was a missing case in derToPem. This case is not being handled correctly and a test has been added for it.

@LoneRifle , this should be a simple change to review. It simply removes spaces from a pretty-printed XML payload so that the cert can be properly normalized.


Add versionning for DPR model

Problem description
Following #86 the DPR model was implemented without tracking the models versions. This is due to the fact that Haystack implementation of the DPR retriever allows only one parameter for model version to be passed although we really need two.
Solution description
Implement solution with two parameters for each models of the DPR.
Additional context / screeshot
see screenshot in #86

closed with #108


Update manage script to be more wizard-like

The manage script is meant to be a helper used to start the project, and should be as simple and automated as possible.
Some improvements to look into:

[ ] Prompt to decide whether to use a multi-tenant agent, or the default dedicated instance
[ ] If the multi-tenant configuration cannot be programmatically inferred, add prompts to collect it
[ ] Execute wallet_init.py when necessary
[ ] More? Assess what needed.

Possibly consider a note in the readme for Windows about WSL or Git Bash or any of the other ways to run a bash script, as if someone on Windows just pulls the repo and follows the instructions it won't run for them in powershell or cmd.

First pass is completed, closing issue. New enhancements to be logged as separate tasks.


Nuke - baking profiles reformat with a chain of multiple nodes

Is your feature request related to a problem? Please describe.
We support only a single reformat properties for baking profiles. Some productions are demanding to expand the feature so multiple reformats could be chained to achieve desired image geometry.
Describe the solution you'd like
Settings preset should be rewritten so it supports multiple nodes inputs the same way as prenodes in Creator Render plugins are. API should be able to build chain of repos nodes (reformat, transform, crop) from a preset.
[cuID:OP-4840]

any fix fir this yet?

@markandrewharris this issue was resolved by https://github.com/ynput/OpenPype/pull/4514


add Docker builder image

Hey!
I hope this Docker image will help other contributors to make developing great again.

Great, thanks!


Instance controller reconcile tests refactor

Description
This PR refactors the instance controller tests in order to use a simpler environment (use a simple kubernetes client without cache, by not using controller manager) that avoids the need to wait for the cache to propagate (which required the doesEventuallyExist function) and removes possible conflict which occasionally made tests fail.
Ref. #512

/merge


[WIP] 2FA refactor (followup)

Follow up to https://github.com/CenterForOpenScience/osf.io/pull/2007 (code review done in person)
for @HarryRybacki

This looks good. @HarryRybacki can you make the autofocus change as well? Once that's done, this is good to merge.


Alignment of buttons on home page

Currently the buttons are not aligned properly which is not giving a good user experience.
I want to work on this.
@Akanshamishra9582 @kashvi9 @YadavAkhileshh can you please assign me this issue? Also provide the gssoc and level label

I want to work on this , please assign it to me.


ci: fix noisy golangci-lint cache

Our Linting step in CI prints a lot of errors due to go files that already exist. I believe this is the issue reported in golangci/golangci-lint-action#23 + golangci/golangci-lint-action#244

https://discord.com/channels/877547706334199818/998918739561041940/1060579289633927258

That was fast!! Thanks!!

Let's wait until the Lint step runs to confirm + land


Problem building images

It seems that something in the process is still wrong:
1- CO-Manager request CO-Crane to create a new Context and waits until it is created. GOOD
2- After context ready, CO-Manager requests CO-Crane to build a new image (the first) and wait until it is created. GOOD
3- CO-Manager must do the previous step for each image, but It finishes the first one and goes to step 1 (in a infinite loop).  BAD
There are also some exceptions in certain queries to the database. Maybe the problem is related to that.
Execution log: http://pastebin.com/fSvxUyc4

Please check if the docker build executes correctly in the crane environment it seems that there are errors in the build process.
in the meantime I check the other parts of the code
Luca

The problem is in the call to the DockerService.isBuild method
after a fewcalls it break and does not work anymore this generate something I have not jet intercepted that causes the process to start again.
Maybe the task fails so this causes the reload
Can you check what happens on a call to the isBuild and activate some logging on crane debug will be easier

Something makes crane erase his datastore. I am making some changes to avoid that.

The new version I am testing right now solves the problem in the datastore. It will be released early next week.

Solved in https://github.com/CloudOpting/cloudopting-crane/releases/tag/0.8


Feat/hyp 119 rm cert manager n better query provider

name: Remove cert manager and better query provider wrapper
about: The removal of cert manager and better query provider wrapper
Summary
This is also in connection to the Jira ticket HYP-119.
This removes the CertificateManager component as it is not deeded anymore and since the <QueryClientProvider>..</QueryClientProvider> is added in multiple places making the code hard to read this places the QueryClientProvider at root level in App making the code more elegant.
NOTE: This should be closed as another PR with the same content has been already merged ( https://github.com/digicatapult/sqnc-hyproof-client/pull/57 )

Motivation
To make it easier to implement the design and improve readability

Describe alternatives you've considered
Not applicable

Additional context
OLD:
Router -> CertificateManager -> CertificateViewer
NEW:
Router -> CertificateViewer
AND:
QueryClientProvider wrapper at App.js root level.
Needs to be closed as per https://github.com/digicatapult/sqnc-hyproof-client/pull/57

Stale


How to sign using a hardware token?

Hello, is there some guide on how to use signxml using a hardware token?
-- I'm not sure if this is a feature request or just something I didn't find in the documentation -- I did spend a good time looking for it, but couldn't find it.
As a note, I actually can get to sign with the PyKCS11 library, but it doesn't really provide the other parts expected from XAdES (so, not sure if some integration would be welcome if it's something not supported).

Hello - there is no guide, because I've never heard of such a need before.
Can you provide some more details? What specifically are you trying to accomplish? What is the API for your token?

Well, my use case is signing a XML but using a certificate stored in hardware token (SafeNet eToken 5110 in my case).
I searched a bit more and ended up using a java library already (xades4j), so, it's not a huge priority for me.
Having it In Python would be nicer though (as all my other code to manage things is in Python).
I have some code which works for me to sign it (pasted below using the PyKCS11, which I know works when using my token here), but it lacks the support to save in the xades format.
# This is an experiment. It can sign, but doesn't make the correct
# format afterwards...

from PyKCS11 import *
from lxml import etree
import keyring

def sign(xml_file_path:str):
    # Define the XML file path

    # Load the XML file
    try:
        with open(xml_file_path, 'rb') as xml_file:
            xml_data = xml_file.read()
            xml_root = etree.fromstring(xml_data)
    except IOError:
        print("Error: Could not read the XML file.")
        exit(1)
    except etree.XMLSyntaxError:
        print("Error: Invalid XML format.")
        exit(1)

    # Define PKCS#11 library path and the slot where the token is present
    # pkcs11_lib_path = '/path/to/your/pkcs11/library.so'
    if sys.platform == 'win32':
        pkcs11_lib_path = 'c:\\windows\\system32\\eTpkcs11.dll'
    else:
        pkcs11_lib_path = '/usr/local/lib/libeTPKcs11.dylib'
    token_slot = 0  # Change this to the appropriate slot number

    # Initialize the PKCS#11 library
    pkcs11 = PyKCS11Lib()
    pkcs11.load(pkcs11_lib_path)

    # Find available slots and print their information
    slots = pkcs11.getSlotList()
    if len(slots) == 0:
        print("No available slots found.")
        exit(1)

    # Ensure the specified slot is valid
    if token_slot >= len(slots):
        print("Error: Invalid slot number.")
        exit(1)

    # Open session with the token
    session = pkcs11.openSession(slots[token_slot])

    # Could specify pin, but let's let it ask for it.
    pin = ''

    # Login to the token using the provided PIN
    try:
        session.login(keyring.get_password('etoken', 'username'))
    except PyKCS11Error as e:
        print(f"Error: Failed to login to the token. {e}")
        exit(1)

    # Get the private key for signing
    # Replace 'your_private_key_label' with the label of your private key
    private_key_label = 'your_private_key_label'

    private_key = session.findObjects([
        (CKA_CLASS, CKO_PRIVATE_KEY),
        # (CKA_LABEL, private_key_label)
    ])

    if len(private_key) == 0:
        print("Error: Private key not found.")
        exit(1)

    # Get the signing mechanism
    mechanism = Mechanism(CKM_SHA256_RSA_PKCS, None)

    # Canonicalize and sign the XML data
    try:
        canonicalized_xml = etree.tostring(xml_root, method="c14n", exclusive=True, with_comments=False)
        signature = session.sign(private_key[0], canonicalized_xml, mechanism)
    except PyKCS11Error as e:
        print(f"Error: Failed to sign the XML. {e}")
        session.logout()
        exit(1)

    # Convert the signature to base64 for inclusion in the XML
    import base64
    signature_base64 = base64.b64encode(bytes(signature)).decode('utf-8')

    # Embed the signature into the XML
    signature_element = etree.Element('Signature')
    signature_value_element = etree.SubElement(signature_element, 'SignatureValue')
    signature_value_element.text = signature_base64

    # Ok, we have signed it, now, we'd need to store it properly as needed by xades

    # Logout and close the session
    session.logout()
    session.closeSession()

Thanks, yeah I think support for PKCS11 is outside the scope of this library's design. If you need specific changes to the SignXML API to make it easier to subclass for your needs and isolate the signing functionality, let me know.

Opened #237 to track PKCS11 support.

Is there currently any way to sign XML with HSM and store it for example in specification like https://www.w3.org/TR/xmldsig-core/

@ii00 as I mentioned above, I've opened a separate issue to track PKCS11 support, so as long as your smart card HSM supports PKCS11, that work will help once it is completed.
The XML Signature specification you linked does not explicitly mention smart cards or HSMs. Can you elaborate on the specific part of the spec you were referring to, and the specific application that you need this for, with an example?

@kislyuk I have to authorize our medical app to our governmental National Health Information System:

GET request to auth endpoint. The response will contain XML as the example bellow.

<nhis:message xmlns:nhis="https://www.his.bg" schemaLocation="https://www.his.bg/api/v1/NHIS-S002.xsd">
  <nhis:contents>
    <nhis:challenge value="6iZcCqHy1zs1BtfQebrdQ8D_V1ZctaEKmD59QKwVOv8"/>
  </nhis:contents>
</nhis:message>

Sign the XML response with smart card HSM. The signature follows the specification at https://www.w3.org/TR/xmldsig-core/ where there are also examples. The schematic itself is available http://www.w3.org/2000/09/xmldsig#

POST request to same auth endpoint the signed XML

Get the XML response that contains the access token which our medical app will use to make further CRUD operations to  our National Health Information System.

Obviously I have to use library like PyKCS11 or python-pkcs11 to get the private key and certificate as @fabioz showed in his example.
Then use XAdESSigner() which takes data, key and cert. I can extract the cert and pass it in base64. But key parameter have to be string or RSAPrivateKey


uncaught exception not logged by http transport

Would love to jump in and help if possible, but a description is really necessary.
It seems an issue template is present to help with this :+1:
Please have a look at https://github.com/winstonjs/winston/blob/master/.github/ISSUE_TEMPLATE.md
This will be so much easier when someone is actively pursuing to help you with the problem experienced.
Have a great day :unicorn:

i forget to mention  handleExceptions: true after it started working fine


infographic data overload the server

handle infographic data GET request is for returning 502 HTTP  error

when a call for the news-flash API has been made to the ANYWAY server, a call for all relevant infographic data has been called at the background and saved to server cache. the execution of the infographic data  GET requests in a loop  results in a 502 HTTP  error for some of the data.


DF Settings Clarity: searching reveals extremely long text descriptions for every setting

Module
DF Settings Clarity v3.3.0
Describe the issue
When I enter a fuzzy keyword search, the results settings list loads a very long block of text that repeats for each setting in the same module. It makes reading the settings list quite difficult for modules with very long fields.
The long text remains even after the search field has been cleared.
To Reproduce
Steps to reproduce the behavior:

Go to Module Settings
Enter a search term in top left corner
Clear the search field
Browse settings

Expected behavior
A clear and concise description of the individual setting, rather than the massive wall of text.
Screenshots (Optional)
Before searching:

After searching:

After clearing the search input:

Other examples:

Seeing this too.  It seems to be a bad interaction with Tidy UI.  If either DF Settings Clarity or Tidy UI are disabled, then the problem goes away.

The workaround i've found is to do the search then re-open the settings window. It seems to reload the proper text descriptions.

Search for the setting as in the original post
Click on the right nav to open the settings window again

I've found the cause of this.  It's because Tidy UI adds a "module-settings-wrapper" element around all the settings for a module...

DF Settings finds .categories > .scrollable, then loops through the grandchildren of that element.  If Tidy UI is installed, then it finds the wrong elements, and ends up concatenating the notes of all the settings for a module into one big wall of text.
The solution is to find .form-groups in the children of .categories > .scrollable instead.


Faster range tests

Add tests and benchmarks reserved range testing, remove check for <IP_ADDRESS> since it is redundant.
This will resolve https://github.com/andrewthad/haskell-ip/issues/60 when completed.

Uh-oh. This is correct, but the performance is awful. The problem is that, rather than using a few registers like I would expect, GHC uses all the registers and then spills onto the stack. Here's the cmm:
[reserved#_entry() //  [R2]
         { info_tbls: [(c1aKA,
                        label: reserved#_info
                        rep: HeapRep static { Fun {arity: 1 fun_type: ArgSpec 4} }
                        srt: Nothing)]
           stack_info: arg_space: 8 updfr_space: Just 8
         }
     {offset
       c1aKA:
           (_c1aKF::I64) = call MO_Ctz W64(R2 &<PHONE_NUMBER> ^<PHONE_NUMBER>);
           (_c1aKS::I64) = call MO_Ctz W64(R2 &<PHONE_NUMBER> ^<PHONE_NUMBER>);
           (_c1aL5::I64) = call MO_Ctz W64(R2 &<PHONE_NUMBER> ^<PHONE_NUMBER>);
           (_c1aLi::I64) = call MO_Ctz W64(R2 &<PHONE_NUMBER> ^<PHONE_NUMBER>);
           (_c1aLv::I64) = call MO_Ctz W64(R2 &<PHONE_NUMBER> ^<PHONE_NUMBER>);
           (_c1aLI::I64) = call MO_Ctz W64(R2 &<PHONE_NUMBER> ^<PHONE_NUMBER>);
           (_c1aLV::I64) = call MO_Ctz W64(R2 &<PHONE_NUMBER> ^<PHONE_NUMBER>);
           (_c1aM8::I64) = call MO_Ctz W64(R2 &<PHONE_NUMBER> ^<PHONE_NUMBER>);
           (_c1aMl::I64) = call MO_Ctz W64(R2 &<PHONE_NUMBER> ^<PHONE_NUMBER>);
           (_c1aMy::I64) = call MO_Ctz W64(R2 &<PHONE_NUMBER> ^<PHONE_NUMBER>);
           (_c1aML::I64) = call MO_Ctz W64(R2 &<PHONE_NUMBER> ^<PHONE_NUMBER>);
           (_c1aMV::I64) = call MO_Ctz W64(R2 &<PHONE_NUMBER>);
           (_c1aN8::I64) = call MO_Ctz W64(R2 &<PHONE_NUMBER> ^ 167772160);
           (_c1aNl::I64) = call MO_Ctz W64(R2 &<PHONE_NUMBER> ^<PHONE_NUMBER>);
           (_c1aNy::I64) = call MO_Ctz W64(R2 &<PHONE_NUMBER> ^<PHONE_NUMBER>);
           R1 = _c1aNy::I64 | _c1aNl::I64 | _c1aN8::I64 | _c1aMV::I64 | _c1aML::I64 | _c1aMy::I64 | _c1aMl::I64 | _c1aM8::I64 | _c1aLV::I64 | _c1aLI::I64 | _c1aLv::I64 | _c1aLi::I64 | _c1aL5::I64 | _c1aKS::I64 | _c1aKF::I64 >> 5;
           call (P64[Sp])(R1) args: 8, res: 0, upd: 8;
     }
 },
 section ""data" . reserved#_closure" {
     reserved#_closure:
         const reserved#_info;
 }]

And here is the assembly, truncated after the stack spilling becomes clear:
_c1aKA:
        movl $3405803776,%eax
        movl $4294967040,%ebx
        movq %r14,%rcx
        andq %rbx,%rcx
        xorq %rax,%rcx
        bsf %rcx,%rax
        movl $64,%ebx
        cmovne %rax,%rbx
        movl $3325256704,%eax
        movl $4294967040,%ecx
        movq %r14,%rdx
        andq %rcx,%rdx
        xorq %rax,%rdx
        bsf %rdx,%rax
        movl $64,%ecx
        cmovne %rax,%rcx
        movl $3227017984,%eax
        movl $4294967040,%edx
        movq %r14,%rsi
        andq %rdx,%rsi
        xorq %rax,%rsi
        bsf %rsi,%rax
        movl $64,%edx
        cmovne %rax,%rdx
        movl $3221225984,%eax
        movl $4294967040,%esi
        movq %r14,%rdi
        andq %rsi,%rdi
        xorq %rax,%rdi
        bsf %rdi,%rax
        movl $64,%esi
        cmovne %rax,%rsi
        movl $3221225472,%eax
        movl $4294967040,%edi
        movq %r14,%r8
        andq %rdi,%r8
        xorq %rax,%r8
        bsf %r8,%rax
        movl $64,%edi
        cmovne %rax,%rdi
        movl $3232235520,%eax
        movl $4294901760,%r8d
        movq %r14,%r9
        andq %r8,%r9
        xorq %rax,%r9
        bsf %r9,%rax
        movl $64,%r8d
        cmovne %rax,%r8
        movl $2851995648,%eax
        movl $4294901760,%r9d
        movq %r14,%r10
        andq %r9,%r10
        xorq %rax,%r10
        bsf %r10,%rax
        movl $64,%r9d
        cmovne %rax,%r9
        movl $3323068416,%eax
        movl $4294836224,%r10d
        movq %r14,%r11
        andq %r10,%r11
        xorq %rax,%r11
        bsf %r11,%rax
        movl $64,%r10d
        cmovne %rax,%r10
        movl $2886729728,%eax
        movl $4293918720,%r11d
        movq %rbx,64(%rsp)
        movq %r14,%rbx
        andq %r11,%rbx
        xorq %rax,%rbx
        bsf %rbx,%rax
        movl $64,%ebx
        cmovne %rax,%rbx
        movl $4290772992,%eax
        movq %r14,%r11
        andq %rax,%r11
        xorq $1681915904,%r11
        bsf %r11,%rax
        movl $64,%r11d
        cmovne %rax,%r11
        movl $4278190080,%eax
        movq %rcx,72(%rsp)
        movq %r14,%rcx
        andq %rax,%rcx
        xorq $2130706432,%rcx
        bsf %rcx,%rax
        movl $64,%ecx
        cmovne %rax,%rcx
        movl $4278190080,%eax
        movq %rdx,80(%rsp)
        movq %r14,%rdx
        andq %rax,%rdx
        bsf %rdx,%rax
        movl $64,%edx
        cmovne %rax,%rdx
        movl $4278190080,%eax
        movq %rsi,88(%rsp)
        movq %r14,%rsi
        andq %rax,%rsi
        xorq $167772160,%rsi
        bsf %rsi,%rax
        movl $64,%esi
        cmovne %rax,%rsi
        movl $4026531840,%eax
        movq %rdi,96(%rsp)
        movl $4026531840,%edi
        movq %r8,104(%rsp)
        movq %r14,%r8
        andq %rdi,%r8
        ...

Darn.

Before changing the implementation of reserved:
benchmarking CIDR Inclusion/reserved
time                 551.4 ns   (549.1 ns .. 554.5 ns)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 553.1 ns   (549.1 ns .. 560.5 ns)
std dev              17.57 ns   (10.45 ns .. 28.96 ns)
variance introduced by outliers: 45% (moderately inflated)

After changing it by writing custom cmm (and turning on -msse4.2 and -mbmi2):
benchmarking CIDR Inclusion/reserved
time                 887.0 ns   (880.9 ns .. 893.6 ns)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 883.1 ns   (878.8 ns .. 892.1 ns)
std dev              19.85 ns   (12.08 ns .. 32.64 ns)
variance introduced by outliers: 28% (moderately inflated)

It appears that this attempted optimization does not pay its way.

dang. that sucks. that most recent benchmark includes the handwritten CMM?

They do include the handwritten cmm (which requires a patch to GHC to even build). It is unfortunate since I was expecting that a non-branching approach would do well, but at least in microbenchmarks, it looks like it just doesn't work out well. However, in https://github.com/andrewthad/haskell-ip/pull/63, I've come up with something that doubles the performance of this function and compiles without fancy hacks.
Closing this PR since this appears to be a dead end.