Tune Failed: Could Not Create Socket

Hi,
OK, this is so cool - really like how this all works together. Thanks!
Just having a bit of an issue with my Ceton card. I try to have Plex tune to a channel, but seeing this error (and Plex complains about it of course),
[2022-01-21 17:55:40,333] INFO - Client has requested stream for ceton channel 4 KDFW FOX KDFW FOX.
[2022-01-21 17:55:40,333] INFO - Selected Stream Parameters: method=direct duration=0 origin_quality=None transcode_quality=None.
[2022-01-21 17:55:40,334] INFO - Attempting to Select an available tuner for this stream.
[2022-01-21 17:55:40,334] INFO - Tuner #0 Acquired.
[2022-01-21 17:55:40,335] INFO - ceton Tuner #0 to be used for stream.
[2022-01-21 17:55:40,361] INFO - Selected Ceton tuner#: 0
[2022-01-21 17:55:40,362] INFO - Ceton tuner 0 to be started
[2022-01-21 17:55:40,579] INFO - Preparing Ceton tuner 0 on port: 43534
[2022-01-21 17:55:41,586] INFO - Initiate streaming channel 4 from Ceton tuner#: 0
[2022-01-21 17:55:41,587] INFO - Preparing Stream...
[2022-01-21 17:55:41,587] INFO - Attempting to create socket to listen on.
[2022-01-21 17:55:41,588] INFO - Created TCP socket at <IP_ADDRESS>:57467.
[2022-01-21 17:55:41,588] INFO - Created UDP socket at <IP_ADDRESS>:43412.
[2022-01-21 17:55:41,589] INFO - Closing UDP socket at <IP_ADDRESS>:43412.
[2022-01-21 17:55:41,590] INFO - Closing TCP socket at <IP_ADDRESS>:57467.
[2022-01-21 17:55:41,590] ERROR - TunerError: 806 - Tune Failed: Could Not Create Socket: 'username'
[2022-01-21 17:55:41,591] INFO - Tuner #0 Released.
[2022-01-21 17:55:41,595] INFO - <IP_ADDRESS> - - [2022-01-21 17:55:41] "GET /api/tuners?method=stream&channel=4&origin=ceton&stream_method=direct&accessed=http%3A//<IP_ADDRESS>%3A5006/hdhr/ceton/auto/v4 HTTP/1.1" 503 2424 1.265516
{'channel': '4', 'channel_name': 'KDFW FOX', 'channel_callsign': 'KDFW FOX', 'origin': 'ceton', 'method': 'direct', 'duration': 0, 'origin_quality': None, 'transcode_quality': None, 'accessed': 'http://<IP_ADDRESS>:5006/hdhr/ceton/auto/v4', 'base_url': 'http://<IP_ADDRESS>:5006', 'client': '<IP_ADDRESS>', 'client_id': 'e21ddcb0-eae3-4b25-a6a1-c014b2985fb2', 'stream_info': {'url': 'udp://<IP_ADDRESS>:43534', 'headers': None}, 'true_content_type': 'video/mpeg', 'content_type': 'video/mpeg'}

Absolutely could be me. I'm not sure that I need to set / define 'username' ... or do I?
And to clarify, this is for an InfiniTV 4, PCIe.
BTW, a bit of fiddling here, I was able to look at the Ceton web interface, manually get a channel to tune, and then right after that stream from the appropriate device to mplayer. Just to let you know, the card is working 🤣. Oh, and this is from Postman - used that to confirm the "API" is working.
Requests, channel change
import requests

url = "http://<IP_ADDRESS>/channel_request.cgi"

payload='instance_id=3&channel=593'
headers = {
  'Content-Type': 'application/x-www-form-urlencoded'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

Thanks!

Pinging @DanAustinGH as he's the master of this plugin.

Thanks! And a bit more digging - to try to help debug ... want to fix this myself if I can (and help out!), but I'm new to the code, wandering my way through 😜. It seems like I have to use ffmpeg for the ceton stream_method? I admit, not sure why. Thinking I want direct => let Plex see the stream directly, handle it there, no?
And not quite following the setting above, vs. stream_method in streaming, there it is set to direct. Again, still digging.
But if I go with ceton set to ffmpeg for now, a bit more progress ... but Plex still complains that it "could not tune channel" (which may not be entirely accurate. Updated info below. Will keep digging here!
[2022-01-22 07:39:20,665] INFO - Client has requested stream for ceton channel 8 WFAA ABC WFAA ABC.
[2022-01-22 07:39:20,665] INFO - Selected Stream Parameters: method=ffmpeg duration=0 origin_quality=None transcode_quality=None.
[2022-01-22 07:39:20,666] INFO - Attempting to Select an available tuner for this stream.
[2022-01-22 07:39:20,666] INFO - Tuner #0 Acquired.
[2022-01-22 07:39:20,667] INFO - ceton Tuner #0 to be used for stream.
[2022-01-22 07:39:20,740] INFO - Selected Ceton tuner#: 0
[2022-01-22 07:39:20,741] INFO - Ceton tuner 0 to be started
[2022-01-22 07:39:20,979] INFO - Preparing Ceton tuner 0 on port: 42262
[2022-01-22 07:39:21,926] INFO - Initiate streaming channel 8 from Ceton tuner#: 0
[2022-01-22 07:39:21,927] INFO - Preparing Stream...
[2022-01-22 07:39:21,928] INFO - Tuning Stream...
ffmpeg version N-105190-ga2372f904c Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.2.0-7ubuntu2)
  configuration: --enable-libnpp --enable-nonfree --enable-cuda --enable-libx264 --enable-gpl --enable-nvenc --enable-libx265 --enable-cuda-nvcc --enable-cuvid
  libavutil      57. 18.100 / 57. 18.100
  libavcodec     59. 20.100 / 59. 20.100
  libavformat    59. 17.100 / 59. 17.100
  libavdevice    59.  5.100 / 59.  5.100
  libavfilter     8. 25.100 /  8. 25.100
  libswscale      6.  5.100 /  6.  5.100
  libswresample   4.  4.100 /  4.  4.100
  libpostproc    56.  4.100 / 56.  4.100
[2022-01-22 07:39:25,641] INFO - <IP_ADDRESS> - - [2022-01-22 07:39:25] "GET /discover.json HTTP/1.1" 200 2693 0.005255
[2022-01-22 07:39:25,645] INFO - <IP_ADDRESS> - - [2022-01-22 07:39:25] "GET /lineup_status.json HTTP/1.1" 200 2464 0.002953
[2022-01-22 07:39:30,933] INFO - <IP_ADDRESS> - - [2022-01-22 07:39:30] "GET /discover.json HTTP/1.1" 200 2697 0.004641
[2022-01-22 07:39:30,939] INFO - <IP_ADDRESS> - - [2022-01-22 07:39:30] "GET /lineup_status.json HTTP/1.1" 200 2462 0.003997
[2022-01-22 07:39:36,237] INFO - <IP_ADDRESS> - - [2022-01-22 07:39:36] "GET /discover.json HTTP/1.1" 200 2694 0.004186
[2022-01-22 07:39:36,242] INFO - <IP_ADDRESS> - - [2022-01-22 07:39:36] "GET /lineup_status.json HTTP/1.1" 200 2462 0.003843
[2022-01-22 07:39:41,543] INFO - <IP_ADDRESS> - - [2022-01-22 07:39:41] "GET /discover.json HTTP/1.1" 200 2694 0.004087
[2022-01-22 07:39:41,548] INFO - <IP_ADDRESS> - - [2022-01-22 07:39:41] "GET /lineup_status.json HTTP/1.1" 200 2462 0.004081
[2022-01-22 07:39:45,245] INFO - <IP_ADDRESS> - - [2022-01-22 07:39:45] "GET /discover.json HTTP/1.1" 200 2695 0.004065
[2022-01-22 07:39:45,250] INFO - <IP_ADDRESS> - - [2022-01-22 07:39:45] "GET /lineup_status.json HTTP/1.1" 200 2465 0.003255

BTW, I did check, using the Ceton web interface - and the channel tune is working! Just the streaming back to Plex is failing? Thanks!
PS: anyone know of a way to access the Ceton web interface remotely? It's a (scope) local link, can't seem to figure out the route commands to get to it from elsewhere on my network. Minor, but just to make debug a bit easier 😆.

I know this plugin was created for the standalone ceton devices. there may have to be some poking on your end to see if it works the same.
One thing you can try, is from the web interface of fHDHR, grab the m3u file from the channel page, and open that in VLC or something. This should rule out fHDHR or Plex as the issue.
If the above works, remove the "dvr" from Plex, and restart Plex, and re-add.

Appreciate the suggestion! I tried like you said, no joy. It is interesting ... in tuners, it says it tunes to that channel, using the first available tuner (perfect!) ... but streaming (using cat) from the command line (to mplayer), I can see that the channel is actually not tuned.
Hmmm ... need to figure out how to get this running inside pyCharm - debugging that way would be much easier. Problem is that Flask seems to be multi-threaded, which is causing pyCharm grief. But I haven't given up 🤣.
Thanks again!

You are the first to report trying with a PCIe card.  I suspected it would work from what I saw documented about how it works.
You do need to use the FFMPEG method.  The Ceton sends the video in an UDP stream and neither Plex nor Emby currently can consume that.
What OS are you running fHDHR on?  This could be selinux related or another OS security feature preventing access to the UDP open to receive the Ceton stream.  After tuning a channel you can check to see if you have an ffmpeg process connected to the UDP port list on the log line 'INFO - Preparing Ceton tuner 0 on port:' using netstat -apn | grep ffmpeg .

And it could also be a host based firewall blocking UDP from the embedded Ceton IP and the host...

You are the first to report trying with a PCIe card. I suspected it would work from what I saw documented about how it works.

Agreed! It's looking just like a network device, so was thinking the same. I got the driver installed, and all working => can connect to the Ceton web interface, play back from the device directly (i.e. /dev/ceton/ctn91xx_mpeg0_# ... #=tuner)

You do need to use the FFMPEG method. The Ceton sends the video in an UDP stream and neither Plex nor Emby currently can consume that.

Got it, thanks! BTW, will look at this after the basics working, but it would be nice then to use HW transcoding inside ffmpeg, offload the CPU. Or, is it just copying the data (codec), no transcoding. In any case, will investigate here.

What OS are you running fHDHR on? This could be selinux related or another OS security feature preventing access to the UDP open to receive the Ceton stream. After tuning a channel you can check to see if you have an ffmpeg process connected to the UDP port list on the log line 'INFO - Preparing Ceton tuner 0 on port:' using netstat -apn | grep ffmpeg .

Running on Ubuntu, 21.10. Yep, was thinking the same! And I had already found those processes 😆. Output of the netstat command is,
udp        0      0 <IP_ADDRESS>:42617           <IP_ADDRESS>:*                           1220165/ffmpeg

Also, from ps -elF | grep ffmpeg
0 S user  1220165 1219768  0  80   0 - 47074 futex_ 17324   9 11:07 pts/0    00:00:00 /usr/local/bin/ffmpeg -i udp://<IP_ADDRESS>:42617 -reconnect 1 -reconnect_at_eof 1 -reconnect_streamed 1 -reconnect_delay_max 2 -c copy -f mpegts -loglevel info pipe:stdout

So seems to be running - but if I try vlc, or mplayer 'udp://<IP_ADDRESS>:42617' ... no go, I get,
MPlayer 1.4 (Debian), built with gcc-10 (C) 2000-2019 MPlayer Team
do_connect: could not connect to socket
connect: No such file or directory
Failed to open LIRC support. You will not be able to use your remote control.

Playing udp://<IP_ADDRESS>:42617.
STREAM_UDP, URL: udp://<IP_ADDRESS>:42617
Failed to connect to server
udp_streaming_start failed
No stream found to handle url udp://<IP_ADDRESS>:42617

So definitely not able to connect. Thoughts? I may be wrong, but it seems that the udp port there is the input (-i), not the output? Or am I just out to lunch? I may be looking the wrong place for the output.
Thanks!!
PS: minor side note, but makes debugging slower - I need to move the CableCARD back to my old tuner, to keep the family happy ... LOL. I have found folks saying that ClearQAM should work with this card - that would make debug easier (not swapping back and forth all the time, and I do have ClearQAM locals on the cable). Do you happen to know how to do this? Just to be able to work through issues here. No biggie if not, just figured I'd ask. Thanks!

Actually, it hit me as I walked away for a few minutes - the UDP stream may be "different", for an external vs. internal device. I can see the video by running (for example), cat /dev/ceton/ctn91xx_mpeg0_0 | mplayer -cache 8192 -
So do I need direct after all? From mplayer,
MPlayer 1.4 (Debian), built with gcc-10 (C) 2000-2019 MPlayer Team
do_connect: could not connect to socket
connect: No such file or directory
Failed to open LIRC support. You will not be able to use your remote control.

Playing -.
Reading from stdin...
Cache fill: 18.75% (1572864 bytes)

libavformat version 58.76.100 (external)
Mismatching header version 58.45.100
TS file format detected.
Cache empty, consider increasing -cache and/or -cache-min. [performance issue]
Cache empty, consider increasing -cache and/or -cache-min. [performance issue]
Cache empty, consider increasing -cache and/or -cache-min. [performance issue]
VIDEO MPEG2(pid=3031) AUDIO A52(pid=3032) NO SUBS (yet)!  PROGRAM N. 3303
VIDEO:  MPEG2  720x480  (aspect 2)  29.970 fps  15000.0 kbps (1875.0 kbyte/s)

Will dig into the code, but thinking (perhaps incorrectly!) that I need direct, and input coming from the device. Just not sure where it's supposed to go to ... LOL.
Thanks!

My cable co does not have an clearqam channels, so I cannot test that, but it should work.
Can you use the web interface to manually start a stream?  If you check the tuner status in the web, what do you see for the stream setup details?  Do you get a unique device name for each tuner under /dev/ceton ? If you do, the answer might be to add an option to the plugin to treat external and internal differently.  External uses <IP_ADDRESS>:port# and internal uses /dev/ceton/#devname
If you start a stream from Plex and then manually run:
ffmpeg -i /dev/ceton/{device used} test.ts
Do you get a video file that can be played?

I also want to point out I have not tried mplayer, so I do not know if it can read from an UDP socket and if it can if it will handle the video stream.  I was never able to get VLC to work, so I can only recommend ffmpeg for testing.  It makes the process a bit clunky since you need to output to a file and then check that file, but it works.

My cable co does not have an clearqam channels, so I cannot test that, but it should work.

Seems to work ... I can't quite figure out (yet) how to rescan. But not changing channels here, and all is OK for debug 😆.

Can you use the web interface to manually start a stream? If you check the tuner status in the web, what do you see for the stream setup details? Do you get a unique device name for each tuner under /dev/ceton ? If you do, the answer might be to add an option to the plugin to treat external and internal differently. External uses <IP_ADDRESS>:port# and internal uses /dev/ceton/#devname

Sorry, can't check the start / stop right now (CC moved to the old box), but pretty sure that was working before => channels were changing, all looked OK. I think the only real issue is the source - like you say, just an option for internal or external may be the fix.

If you start a stream from Plex and then manually run: ffmpeg -i /dev/ceton/{device used} test.ts Do you get a video file that can be played?

Yep! I also tried piping that device to ffmpeg as stdin, same result - all OK. I added an option to copy (codec), and can see the real bit rate, etc. All good, the .ts output is playable (mplayer, or vlc).

I also want to point out I have not tried mplayer, so I do not know if it can read from an UDP socket and if it can if it will handle the video stream. I was never able to get VLC to work, so I can only recommend ffmpeg for testing. It makes the process a bit clunky since you need to output to a file and then check that file, but it works.

No worries! And I like ffmpeg also - can see bitrate, etc. I do agree, it's a handy tool. And I have my own custom local build, to include HW accel for hevc 😄.
I can try to dig into the code, see where to set the internal or external option if you want? I admit, not real familiar with it yet, but more than happy to. Trying to help out if I can!
Thanks!

How many devices under /dev/ceton?
In plugins/fHDHR_plugin_origin_ceton/origin/init.py on line 197 you can change streamurl to point to one of the /dev/ceton/devices
That may fail due to core code validation functions that expect one of a few valid URL formats.  I can point that code out as well if you need it.
If changing the streamurl works, the code would need an option to indicate if the Ceton is PCIe or not,  If it is it needs to build a list of usable local devices, and use those instead of UDP ports.  Not too complex, but since I don't have a PCIe device I can only provide suggestions...

Silly question, but do you have ceton_ip set in your fhdhr.ini?  And address set as well in the [fhdhr] section?

How many devices under /dev/ceton?

Listing here,
lrwxrwxrwx  1 root root   15 Jan 22 06:40 ctn91xx_ctl0 -> ../ctn91xx_ctl0
lrwxrwxrwx  1 root root   20 Jan 22 06:40 ctn91xx_filter0_0 -> ../ctn91xx_filter0_0
lrwxrwxrwx  1 root root   20 Jan 22 06:40 ctn91xx_filter0_1 -> ../ctn91xx_filter0_1
lrwxrwxrwx  1 root root   20 Jan 22 06:40 ctn91xx_filter0_2 -> ../ctn91xx_filter0_2
lrwxrwxrwx  1 root root   20 Jan 22 06:40 ctn91xx_filter0_3 -> ../ctn91xx_filter0_3
lrwxrwxrwx  1 root root   20 Jan 22 06:40 ctn91xx_filter0_4 -> ../ctn91xx_filter0_4
lrwxrwxrwx  1 root root   20 Jan 22 06:40 ctn91xx_filter0_5 -> ../ctn91xx_filter0_5
lrwxrwxrwx  1 root root   18 Jan 22 06:40 ctn91xx_mpeg0_0 -> ../ctn91xx_mpeg0_0
lrwxrwxrwx  1 root root   18 Jan 22 06:40 ctn91xx_mpeg0_1 -> ../ctn91xx_mpeg0_1
lrwxrwxrwx  1 root root   18 Jan 22 06:40 ctn91xx_mpeg0_2 -> ../ctn91xx_mpeg0_2
lrwxrwxrwx  1 root root   18 Jan 22 06:40 ctn91xx_mpeg0_3 -> ../ctn91xx_mpeg0_3
lrwxrwxrwx  1 root root   18 Jan 22 06:40 ctn91xx_mpeg0_4 -> ../ctn91xx_mpeg0_4
lrwxrwxrwx  1 root root   18 Jan 22 06:40 ctn91xx_mpeg0_5 -> ../ctn91xx_mpeg0_5

I think the 6 is hard coded in the driver - will look at that after. For now, I stick to 0-3 (4 tuner version of the card).

In plugins/fHDHR_plugin_origin_ceton/origin/init.py on line 197 you can change streamurl to point to one of the /dev/ceton/devices

Changed just below that block, to force,  streamurl = "/dev/ceton/ctn91xx_mpeg0_0". Fails though, with a tuner error? This is likely because of no CC => trying to just skip that step for now 😄.
[2022-01-22 12:55:30,352] INFO - Client has requested stream for ceton channel 4 KDFW FOX KDFW FOX.
[2022-01-22 12:55:30,353] INFO - Selected Stream Parameters: method=ffmpeg duration=0 origin_quality=None transcode_quality=None.
[2022-01-22 12:55:30,353] INFO - Attempting to Select an available tuner for this stream.
[2022-01-22 12:55:30,353] INFO - Tuner #0 Acquired.
[2022-01-22 12:55:30,354] INFO - ceton Tuner #0 to be used for stream.
[2022-01-22 12:55:30,354] DEBUG - Attempting to gather stream information for ceton channel 4 KDFW FOX KDFW FOX.
[2022-01-22 12:55:30,383] INFO - Selected Ceton tuner#: 0
[2022-01-22 12:55:30,384] INFO - Ceton tuner 0 to be started
[2022-01-22 12:55:30,520] INFO - Preparing Ceton tuner 0 on port: 42943
[2022-01-22 12:55:30,621] INFO - A ceton stream request for ffmpeg channel 4 was rejected due to TunerError: 806 - Tune Failed
[2022-01-22 12:55:30,622] ERROR - TunerError: 806 - Tune Failed
[2022-01-22 12:55:30,623] INFO - Tuner #0 Released.
[2022-01-22 12:55:30,624] DEBUG - Client GET requested /api/tuners Closing

If changing the streamurl works, the code would need an option to indicate if the Ceton is PCIe or not, If it is it needs to build a list of usable local devices, and use those instead of UDP ports. Not too complex, but since I don't have a PCIe device I can only provide suggestions...

No worries! And agreed. Trying to "hack" this first, prove it gets working => then clean up, enumerate, etc. I see a PR or two in my future 🤣.

Silly question, but do you have ceton_ip set in your fhdhr.ini? And address set as well in the [fhdhr] section?

Never silly! You know me already ... LOL. Yep, both set,
[ceton]
ceton_ip = <IP_ADDRESS>
stream_method = ffmpeg

[fhdhr]
address = <IP_ADDRESS>
port = 5006
discovery_address = <IP_ADDRESS>
tuner_count = 4

Thanks!

Silly question 2.  Are you running all this on the base host, or using a container (docker/kubernetes)?
The code should have the Ceton start sending an UDP stream to <IP_ADDRESS>:%port% and the streaming code uses localhost to open and connect to the UDP port.
The web tuner status should confirm the destination IP and port are what was configured by fHDHR.  I am not seeing a lot of docs, but is appears that the PCIe should be capable of doing both local device and UDP/RTP.  If it is not setting up the RTP stream properly, it might be that the current code is not passing a setup argument that the PCIe card needs.
The tune error in the log is related to the core code refusing URLs that do not match an expected pattern.  Look in
fHDHR/device/tuners/init.py
fHDHR/device/tuners/stream/init.py
for udp:// and add "/dev/ceton/ctn91" to the tuple list  This should allow you get past the 806 error

And before diving deep in the code, although I think the result might be worth it, do you have firewalld running?  If so you might need add a rule that allows all UDP traffic from <IP_ADDRESS>/32
The stream from the Ceton appears to be unsolicited and would be blocked by default if you have any firewall running

Silly question 2. Are you running all this on the base host, or using a container (docker/kubernetes)?

You know you only get 3, right?!?! LOL! Base host - directly running the code in the OS (Linux).

The tune error in the log is related to the core code refusing URLs that do not match an expected pattern. Look in fHDHR/device/tuners/init.py fHDHR/device/tuners/stream/init.py for udp:// and add "/dev/ceton/ctn91" to the tuple list This should allow you get past the 806 error

OK, a tad scary, like you are watching my screen 😉. I was right in that section of the code, looking for the error. I added to that tuple list (/dev/ceton) ... like you say, let's me past then, and ... video in Plex! I think that addition is valid, agreed? Meaning - playback from a device is also acceptable. If you agree, I'll submit a PR, to get that update in.
Then - just need to handle the internal / external on the ceton side of the code, agreed?
Thanks!!

I am curious about the firewall if it is in the way, but in general the following would be needed-
The tuple list is in DBD's code so he will have to be OK with it.
In the Ceton plugin code we need an option, say Ceton_PCI=T|F (add to ceton_conf.json default False)
If Ceton_PCI = Y then the streamurl is set to /dev/ceton/ctn91xx_mpeg{cardnum}_{tunernum}
Cardnum can be assumed to be the first(0).  Right now I do not think the framework allows for multiple Ceton devices, but double bonus points if you can add the code in a way to handle more than one.
And other than tweaking the logging to reflect local devices instead of IP:port that should be all of it

More than happy to try to dig into the firewall question - but I'm relatively sure I don't have one running. Then again, I was wrong once before 😜. Any particular things you want checked?
Agreed on the tuple list. And really, thinking /dev would make more sense, not limit to /dev/ceton at the top level?
Will work on the code updates, get you something to look at (will get it working, then submit a PR that we can work through ... OK?). And yes, logging as part of that. I really wish I could get pyCharm working with this, but I think Flask and multi-threading is causing it grief 😞. Just makes debugging more difficult without it.
Thanks!

Right now the only local device that fHDHR could read from is Ceton, so starting out specific makes sense, but DBD can adjust if he has other local devices he might want to open up access to.
I am not an Ubuntu user, so I had to dig.  Looks like for that distro the firewall is ufw and off by default.  I figured if it was as simple as a firewall, I could add it to the docs to look for.  the code you are looking to add may get more folks interested in the project.  I am only aware of 3~4 Ceton plugin users.
I see a few PCI Cetons on ebay.  I may order one just to play with...

Sure, will make it more specific, no issue here!
And yes, agreed - ufw, and off. Just wasn't sure if you knew of something else.
Will code it up, get it to you! And definitley ... eBay - I found a 4 tuner one for $25. Pretty nice for a 4 tuner card 😆
Thanks!

Hmm ... OK, I have code to "count" (determine) the number of PCIe cards. Allocating between them can come later 😆. And I have it selectively setting the url (internal or external). But a bit stuck - the state never seems to change from STOPPED (using stream_request.cgi). Not quite sure why (yet), but it prevents me from getting past using the first tuner ... it always thinks it's available.

OK, figured that one out - from the log on the Ceton web UI,
Jan  2 11:38:43 ocur[21]: ocur: ERROR: Unable to open presentation file "/stream_request.cgi" (No such file or directory)
Jan  2 11:38:43 ocur[21]: ocur: ERROR: unknown write handle

So seems that page doesn't exist 😞. I did find the attached - really need to all that complexity? It does set up rtsp, but seems to limit to being on the same machine (Ceton, fHDHR)? Need to look a bit more. I also find it doesn't exit very gracefully, seems to keep the device hung.
Thanks!

You do not need that code, although I never found that package, so there might be something interesting in there to review.
In the web interface do you have a top level menu with an entry for each of the tuners?  If you do under each is there a status and tuner tab/option?  If you have a tuner tab, is there section near the bottom labeled 'Streaming Setup'?
If all of the above is true, you can view source in the page to see what the required form data is and the url to submit the API as.

Too funny - yet again, similar thoughts at the same time 😆. I was looking at that, and at /api/tuners?method=status. That seems to include status I could use to decide which tuner to use (and then don't need to check STOPPED, as it doesn't change for the internal device).
Just trying to figure out how to get that status from the tuner api (in the code I mean).
Thanks!

In the web interface do you have a top level menu with an entry for each of the tuners? If you do under each is there a status and tuner tab/option? If you have a tuner tab, is there section near the bottom labeled 'Streaming Setup'?

Sorry, didn't answer this one. Nope, no streaming setup. That's one thing I noticed in the Ceton web UI ... no place to enable streaming. Or I may be missing it.

What firmware is running on that card?  I was under the impression that both the PCIe and standalone used an identical web interface.  I have <IP_ADDRESS> on my eth6.  If you do have a tuner tab, are there any buttons on it?  Some of the code in the plugin is based on what I found using view source in the web interface...

What is retuned if you call this- http://<IP_ADDRESS>/get_var?i=0&s=av&v=TransportState (i={tunernum})

Oh and you really do want to get the cards tuner state.  Ideally it should always match the core tuner state, but they can get out of sync.

What firmware is running on that card?

<IP_ADDRESS>. Wow, that's quite a difference. I do find this. Thoughts? Try this? Don't want to bake my card of course.

If you do have a tuner tab, are there any buttons on it? Some of the code in the plugin is based on what I found using view source in the web interface...

Agreed! Was thinking the same thing. And agree - I also looked at that code, as well in the browser devtools.

What is returned if you call this- http://<IP_ADDRESS>/get_var?i=0&s=av&v=TransportState (i={tunernum})

STOPPED ... no matter what the state. That's why I was thinking about using the core tuner state. Thoughts?
Thanks!

Looks like the version numbers are quite different, 4 vs. 6
http://cetoncorp.com/infinitv_support/linux_drivers/

You need an update.  This page has links to various versions.  I think you will have to do multiple updates since your version is so old.  Further it is documented that older firmware (like yours) does not stream.  Getting the latest firmware on your device will likely allow it to work with no code mods (but they might still be worthwhile)

Perfect! Sorry, which page? Or did I just miss the link?

Failed to paste it- https://githubhelp.com/craigmox/cetonproxy
Folks on that page have that app working with a PCIe 4 tuner.  It uses the same http API calls that the plugin uses, so I take that as a positive sign.

Oh and while all Ceton owned pages are MIA, it appears that they wen from code 1.2.* to 13.*
No idea why but there does not appear to have been a dozen in between releases

Thanks! Giving it a shot 🥁

OK, up to 14 so far ... and now those buttons show up. LOL!
Let me get to 15, then will go back to fHDHR, give it a shot.

Well, firmware is current, no error messages now - but still fails to playback. ffmpeg is running, so very odd. Not sure if I need to reboot, then remove and re-add in Plex perhaps?
Thanks!

The status page should show 'PLAYING' once a tuner is tuned.  The tuner page streaming section should confirm the host IP and the port number that the plugin requested.  You can now use the fhdhr web interface and check the tuners view and confirm that the desired channel is tuned and both downloaded and served are increasing.  I doubt you need a reboot or changes in Plex, but they won't hurt.

The status page should show 'PLAYING' once a tuner is tuned.

Yep!

The tuner page streaming section should confirm the host IP and the port number that the plugin requested.

Yes, but ... the destination IP seems to be my Ceton IP (<IP_ADDRESS>), and the port is 8000? Not sure that is right, and may be why the video isn't getting to Plex? Otherwise it all seems happy. Will dig more here!

Hmmm, from the Ceton log - wondering if this is the issue?
Jan  1 00:24:10 ocur[21]: ocur: [0] stream request <IP_ADDRESS>:45673 start:1 prot: 0
Jan  1 00:24:10 ocur[21]: upnp: Event(cm[0]): CurrentConnectionIDs, "0"
Jan  1 00:24:10 ocur[21]: ocur: WARNING: failed to get hwaddr for remote rtp addr

Thanks!!

@DanAustinGH let's make sure we document tested firmware versions that this plugin works for

Others have documented the minimum as <IP_ADDRESS> For my use I just went to the latest.
The streaming page defaults to 192.168.200.X:800X where X is the tuner numbers and the tuner number, so that is not super helpful.  If the tuner status is still showing Playing with that error, you could connect to the hardware device to matches that tuner.  Another option would be to add a loopback network adapter to your system and assign it to <IP_ADDRESS>.
Can you do a view source on the tuner page, zip it and post it here?  It is possible that the PCIe card has more options on the page than the standalone version.

@DanAustinGH let's make sure we document tested firmware versions that this plugin works for

Agreed! I took a note to myself this morning actually - good info to capture, knowing that a specific (minimum) version is needed. One of the (slight) gotchas here.

Can you do a view source on the tuner page, zip it and post it here? It is possible that the PCIe card has more options on the page than the standalone version.

Sure, NP! Attached - just yell if you need anything else. BTW, this is when not running - but I can update if you want.
http _<IP_ADDRESS>_Services_1_Tuner.zip
OK, and I tried one thing here ... made this slight (temporary) change to the code, in get_channel_stream()
streamurl = "/dev/ceton/ctn91xx_mpeg0_%s" % instance

Works great, even tried with 3 tuners running, all good! Not saying this is the final answer, but it really all is working except the streaming.  I admit, a bit confused by some of the address settings, perhaps I have one wrong? Here is what I have - noting that my LAN IP is <IP_ADDRESS>, so thinking that's what I need to use for fHDHR also? But perhaps I have misunderstood.
[ceton]
ceton_ip = <IP_ADDRESS>
[fhdhr]
address = <IP_ADDRESS>
port = 5006
discovery_address = <IP_ADDRESS>

Also, what I do note is, from ip route
<IP_ADDRESS>/24 dev ctn0 proto kernel scope link src <IP_ADDRESS> metric 100

Scope is local? FYI, I did try from another machine (after adding a route to <IP_ADDRESS>/24) ... and I can ping <IP_ADDRESS>, but not <IP_ADDRESS>. Expected? And needed? Does it mean that fHDHR needs to be running on the machine where the Ceton card is? It is in my case, but perhaps part of the issue?
Sorry if this is rambling thoughts - trying to figure out the IP config here, as this may be part (or all) of it.
Thanks!

Your understanding that you want to set the fhdhr address to the LAN IP is  correct.  The Ceton plugin assumes that the Ceton device will send the stream to that IP.
What I suspect is the issue is that the Ceton internally does not have a default gateway so can only communicate with 192.168.200.X.  The IP route line seems to support this.  can you run ip addr and confirm that the ctn0 interface is assigned <IP_ADDRESS> ?  If it is, we might be able to use '<IP_ADDRESS>' in dest_ip in function startstop_ceton_tuner and keep the streamurl as <IP_ADDRESS>
BUT since the stream is going over the PCI bus in either case, UDP stream or direct device access, it might by very slightly more efficient to get the stream from the hardware device.  We will use the http control API to tune channels and start the tuner.
I can think of a few hacky ways to try to run fhdhr on a separate system, but for the PCI card, I think it should be on the same box to keep things simple.

What I suspect is the issue is that the Ceton internally does not have a default gateway so can only communicate with 192.168.200.X. The IP route line seems to support this. can you run ip addr and confirm that the ctn0 interface is assigned <IP_ADDRESS> ? If it is, we might be able to use '<IP_ADDRESS>' in dest_ip in function startstop_ceton_tuner and keep the streamurl as <IP_ADDRESS>

Yep, confirmed ... ran ip -c addr, result is,
4: ctn0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    link/ether 00:22:2c:ff:ff:ff brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/24 brd <IP_ADDRESS> scope global dynamic noprefixroute ctn0
       valid_lft 847482sec preferred_lft 847482sec

And I was thinking the same - no default GW. I went to the Ceton GUI, was going to try to manually add it ... but even if not running DHCP, you cannot set the GW. So I'm with you, thinking it's not used / available => limited to that subnet only. And the log output (above) supports that.

BUT since the stream is going over the PCI bus in either case, UDP stream or direct device access, it might by very slightly more efficient to get the stream from the hardware device. We will use the http control API to tune channels and start the tuner.

Agreed! Was thinking exactly the same thing 😄. The tuning is all working, as is PLAYING / STOPPED (i.e. status) ... so like you are saying - don't use the RTP stream, just get it directly from the device. I hacked in the code above, and it does work ... I can go back to adding the variable we talked about (for PCIe True/False), and then set the url based on that. Work for you? And perhaps for now, just limit to 1 PCIe card? I can put back the code I had locally added though also, that does display in the log the number of cards. For later 😜.

I can think of a few hacky ways to try to run fhdhr on a separate system, but for the PCI card, I think it should be on the same box to keep things simple.

Agreed!

I support everything you propose.  Multiple cards would be nice, but would require changing the config system to support at least a list of card IP, or a dict that maps card number to card IP and then tracking which channels are tuned to which card/tuner (we already track tuner).
Since you are the first to use a PCI card, I would guess the demand for multiple is not very high.
You'll end up with one PR for changes inside the Ceton plugin and one for the tuple changes, unless @deathbybandaid wants to make those for you.

Agreed! And yes, a lot of complexity for multiple cards - could be an addition later, but perhaps not critical right now 😆.
And agreed as well on 2 PR's - was thinking the same. FYI, code already modified here, and it's working - now to beta test on the family 😜. It's not super complex, and I do at least log the number of Ceton PCIe cards.
Just one other minor question, thinking efficiency, and not adding load if not needed. As the stream now is not RTP / UDP, but rather the "direct" tuner traffic ... wondering if it could be moved from ffmpeg to direct? Ya, I tried it, didn't work out of the box - no biggie, but wondering if I should dig more, or if that's just not possible. Also, if it is ffmpeg, wondering about HW accel ... just to offload the CPU. Or both of these - future enhancements?
Thanks for all the help, and putting up with my wandering path - really appreciated! I do like how this works, hope you can find a PCIe card on eBay!

Whatdo you mean by tuple changes?
If needbe, the ceton plugin can have a custom ffmpeg streaming method.

The card is using a video stream format based on RTSP over UDP.  Emby/Plex and others want to see HLS over TCP or HTTP chunks over TCP.  True direct is not going to work.  I looked into a few Python projects to convert RTSP to HLS or HTTP chunks.  I never got anything to work, and after rethinking it, I just don't see how a Python implementation would be better than FFMPEG.  It would eliminate a dependency, but would likely be slower and far less able to handle bad streams.
Earlier you mentioned hardware acceleration.  I tried that.  Turns out to be not needed.  A 10 year old CPU has more than enough power to handle the conversion in realtime with very little CPU.

Earlier you mentioned hardware acceleration. I tried that. Turns out to be not needed. A 10 year old CPU has more than enough power to handle the conversion in realtime with very little CPU.

Makes sense ... it's just copy, no transcoding (at least if set that way), right?
NP, this is the easier solution 😉.
OK, about to put it to the test ... but missing some channels (that show up in the Ceton GUI). Hmmm ... forced a channel update, still missing. Let me have the Ceton clear, reload the channels, see if that does it.
Thanks!

I wish this forum numbered posts.  The core only allows stream URLs to start with rtp, rtsp and udp.  We want to also allow direct device input into ffmpeg.

For missing channels, make sure you have not set favorites or disabled them in the fhdr web.

Right! This is my update,
        if stream_args["stream_info"]["url"].startswith(tuple(["rtp://", "rtsp://", "udp://", "/dev/ceton"])):

OK with you @deathbybandaid?

I think if /dev/ would be more universal for potential other hardware devices.
I try to keep the core code so it doesn't have anything plugin specific.

Yup.  We discussed that above and I thought you might want to consider other devices, but did not want to make that call, so I suggested more specific.

I'll do they change for core myself

This is the update I pushed to core
https://github.com/fHDHR/fHDHR/pull/212
I've also opened up the idea of possibly coding in some handling for the built-in method of reading the hardware device directly without ffmpeg.

Thanks! Will pull it in here. FYI, I missed this in the logs, but my channel issue is due to,
  File "fHDHR/plugins/fHDHR_plugin_origin_ceton/origin/__init__.py", line 136, in get_channels
    count = int(count.group(0))
AttributeError: 'NoneType' object has no attribute 'group'

Will fix it, PR for it.

OK, I take that back 😆. I rescanned (after card remove + insert), I see this ... huh? Odd - the different counts (687 vs. 832).
[2022-01-23 12:15:33,323] INFO - Tuner #1 Performing Channel Scan for ceton origin.
[2022-01-23 12:15:33,326] INFO - Performing Channel Scan for ceton.
[2022-01-23 12:15:33,330] INFO - <IP_ADDRESS> - - [2022-01-23 12:15:33] "GET /api/tuners?method=scan&origin=ceton&redirect=/channels?origin=ceton HTTP/1.1" 302 2835 0.009086
[2022-01-23 12:15:33,451] INFO - <IP_ADDRESS> - - [2022-01-23 12:15:33] "GET /channels?origin=ceton&retmessage=scan%20Success HTTP/1.1" 200 423311 0.116107
[2022-01-23 12:15:33,544] INFO - <IP_ADDRESS> - - [2022-01-23 12:15:33] "GET /style.css HTTP/1.1" 200 5364 0.004337
[2022-01-23 12:15:33,553] INFO - <IP_ADDRESS> - - [2022-01-23 12:15:33] "GET /brython.js HTTP/1.1" 304 2447 0.003119
[2022-01-23 12:15:33,557] INFO - <IP_ADDRESS> - - [2022-01-23 12:15:33] "GET /brython_stdlib.js HTTP/1.1" 304 2454 0.002996
[2022-01-23 12:15:33,593] INFO - <IP_ADDRESS> - - [2022-01-23 12:15:33] "GET /favicon.ico HTTP/1.1" 200 17955 0.004499
[2022-01-23 12:15:34,599] INFO - <IP_ADDRESS> - - [2022-01-23 12:15:34] "GET /brython.bry?1642961734572 HTTP/1.1" 200 8477 0.004601
[2022-01-23 12:15:34,618] INFO - <IP_ADDRESS> - - [2022-01-23 12:15:34] "GET /favicon.ico HTTP/1.1" 200 17953 0.004419
[2022-01-23 12:15:35,893] INFO - Found 687 channels for ceton.
[2022-01-23 12:15:35,894] INFO - Performing ceton Channel Import, This can take some time, Please wait.
[2022-01-23 12:15:36,934] INFO - ceton Channel Import took 1 seconds
[2022-01-23 12:15:36,934] INFO - Found no NEW channels for ceton.
[2022-01-23 12:15:36,935] INFO - Total ceton Channel Count: 832
[2022-01-23 12:15:36,946] INFO - Requested Channel Scan for ceton origin Complete.
[2022-01-23 12:15:36,946] INFO - Tuner #1 Released.

Another thought, does the Ceton webpage function correctly?
If so, is there any additional information to pull fromthe ceton api that would be useful for others?
Maybe pull hardware type and firmware version to display here?

I like that idea! Would be good to pull that in - at least IMHO.

And BTW, a bit of hacking around, but if I set page=0 when getting channels ... I get them all at once, no need to page through them. Do you want me to try to make that change? Simplifies the code a bit I think.
Thanks!

OK, this one is odd ... LMAO. I checked, import is correct (number, count, etc.). But the channel list then in the fHDHR web interface is not. Weird.
Digging into it!
Oh, and I have it running here on a single page - we can include or not, your call!

The Ceton will only deliver 1026 channels in one go.  The page code was to get additional channels above that.  Will review to see if something got reverted during a merge. if you go to http://<IP_ADDRESS>/view_channel_map.cgi? is there a line that looks like 1 to 50 of 832 ?
There is quite a bit of extra details that we could pull.  Most stuck me as uninteresting from an fhdhr point of view and could be retrieved directly from the device.  BUT my case has the device web interface is available from any system on my network.
@deathbybandaid thanks for those changes.

Yep, I do see 1 of 50 - didn't know about the 1026 limit. Thanks!
And agreed, thanks @deathbybandaid - much appreciated!!

Also agree on the items to fHDHR - as I can only see the Ceton details on a single machine 😞

It looks like the code is failing to get the total count.  In the while block I start with page 0 and if there are more than 1024 entries, I jump to page 21 and collect the rest.  One possibility is your local channel list was damaged, which can happen if you stop fhdhr during a channel import.  You might try renaming data/cache/fhdhr.db with the app stopped and then restart without the page=0 changes to see if you get a clean import

I think plex also has a limit for channel count

I see a better way to get the channel count if you want to try it.
In get_channels replace
count_url = 'http://%s/view_channel_map.cgi?page=1' % self.ceton_ip
with
count_url = 'http://%s/get_var?i=0&s=diag&v=NumChannels' % self.ceton_ip
then delete:
count = re.search('(?<=1 to 50 of )\w+', countReq.text)
Finally replace:
count = int(count.group(0))
with
count = int(countReq.text)

One possibility is your local channel list was damaged, which can happen if you stop fhdhr during a channel import. You might try renaming data/cache/fhdhr.db with the app stopped and then restart without the page=0 changes to see if you get a clean import

That did it, thanks!
Will give that code try a change, get back to you!

BTW, looking at the output (log), should this one be changed to DEBUG level?
[2022-01-23 16:03:24,604] INFO - Serving Chunk #12734: size 1152033
frame=217716 fps= 30 q=-1.0 size=14328328kB time=02:01:05.02 bitrate=16156.5kbits/s speed=   1x
[2022-01-23 16:03:25,189] INFO - Serving Chunk #12735: size 1152033
frame=217731 fps= 30 q=-1.0 size=14329450kB time=02:01:05.52 bitrate=16156.7kbits/s speed=   1x
[2022-01-23 16:03:25,565] INFO - Serving Chunk #12736: size 1152033
frame=217748 fps= 30 q=-1.0 size=14330361kB time=02:01:06.09 bitrate=16156.5kbits/s speed=   1x
[2022-01-23 16:03:26,128] INFO - Serving Chunk #12737: size 1152033
frame=217765 fps= 30 q=-1.0 size=14331451kB time=02:01:06.66 bitrate=16156.4kbits/s speed=   1x
[2022-01-23 16:03:26,724] INFO - Serving Chunk #12738: size 1152033
frame=217781 fps= 30 q=-1.0 size=14332537kB time=02:01:07.19 bitrate=16156.5kbits/s speed=   1x
frame=217794 fps= 30 q=-1.0 size=14333459kB time=02:01:07.62 bitrate=16156.5kbits/s speed=   1x

How's this?
https://github.com/fHDHR/fHDHR/pull/213

I like it!

Agreed, and dang that was quick. Thanks!!

Please check this out, let me know your thoughts. Thanks!
https://github.com/fHDHR/fHDHR_plugin_origin_ceton/pull/17

And let me apologize, as I'm not trying to turn this item into a bunch of other things ... rather the opposite, that's why I'd like your thoughts. We discussed above about trying to stream from HW directly, and I know I'd like to set the script up to run as a service on my Linux box (for that I also need to dump all the console output to a file). All things that aren't critical or gating, but also that I don't want to forget. Thoughts on how to capture them?
My initial thought was to capture them as Issues - they aren't bugs or problems, not at all, rather potential enhancements. But just looking for a good, organized, way to capture items, and work through them. Heck, assign the service to me, I'll get it worked out😄. But really, want to know your thoughts on it. If you don't like it - NP! I'm not trying to redirect, not at all. I think what is here really is awesome. But also thinking a couple of the things I'd like to add to my setup - well, if I can share to help repay in a small part for all the work here, I'd like to.
Thanks again!

I do not have much I plan to add, so if you want to open an issue for the items you would like to add, I can help with them.  I do have a systemd unit file for fhdhr, which I never thought about adding to the package, but it would be worth it I think.

@arrmo once #17 is merged, we'll close this thread. Feel free to open additional threads for each individual idea you might have, to open them for discussion.
If you have ideas you'd like to attempt to tackle for more origin plugins, you can open those in the fHDHR/fHDHR repo.

Sorry, catching up - work getting in the way of fun 😆

I do not have much I plan to add, so if you want to open an issue for the items you would like to add, I can help with them. I do have a systemd unit file for fhdhr, which I never thought about adding to the package, but it would be worth it I think.

Yes, was thinking to capture the info for folks - and it's handy in general. Was going to do exactly the same thing ... systemd.

@arrmo once #17 is merged, we'll close this thread. Feel free to open additional threads for each individual idea you might have, to open them for discussion.

Will do, thanks! Some folks / projects don't like "Issues" raised, if not bugs - but seems like you're OK with it. Just wanted to make sure first.

I've always felt that github should copy what they do with the issues tab and make a feature requests tab

Just a thought, but ... add Labels? Could be for bug states also (confirmed, working, etc.) ... and "Feature Request"? Just a thought.

whenever I open feature requests on repos, I usually do "[Feature Request] blahblah"
I do add labels to stuff sometimes

Awesome, thanks! Will take a look. And my apologies, I'll be a bit slower during the week. Dang work! 🤣

Went ahead and made a method for file://
https://github.com/fHDHR/fHDHR/pull/218

Will take a look, thanks! Need to understand it yet of course 😆. Thinking the intention is to send the "raw" packets from /dev to the Plex client ... do I have that right?

yes, and preferably I'd find a way to determine the content type properly instead of assuming mpegts

Thinking out loud, and complete unproven 😜 ... but, use ffprobe on that (file) stream, to get the content type? Just a thought.

Just a thought, tune a channel using Plex then do a cat /dev/ceton/ctn91xx_mpeg{tuner} > test.raw then ffprobe test.raw
If we are lucky the direct device read is not using the UDP/RTP packet structure, and in a format that does not need converting before streaming.
@arrmo I put this in the PR, but can you check what this call outputs on your card-
http://<IP_ADDRESS>/get_var?i=0&s=diag&v=Host_Connection
That might be the easiest call to use to figure out if it is PCI or standalone.

You posted in the PR as I posted here.  Instead of the config option, we can use that call to detect device type. and switch input methods.

Any potential "direct" python method in core can't rely on external dependencies like ffmpeg/ffprobe

I'm suggesting using ffprode to document the packaging/format that we get from reading directly from the tuner device.  It is unlikely that changes, but we can check an SD channel and bot a 720 and 1080 HD channel.  If the packaging/stream format is consistent, we can then determine if it can easily be direct streamed or converted without VLC or ffpmeg to be direct streamable.

You posted in the PR as I posted here. Instead of the config option, we can use that call to detect device type. and switch input methods.

I like it! Will remove the variable, auto-detect. To confirm ... in your case, is it "Ethernet" (including case, so not ethernet), right?

Nope, just lowercase ethernet.  You might just check to see if PCI or pci are in the string...

if "pci" in returned_string.lower()

if "pci" in returned_string.lower()

Yep, that's my plan - agreed! 😄

FYI, did a very quick check here (not near the machines right now), and looks like the video is (always?) mpeg2video, .ts ... so do we assume that?

mpegts is the format that Plex would expect to see from a real HDHR device. However, fHDHR is able to tell Plex what kind of video to expect.
https://github.com/fHDHR/fHDHR/blob/e04e0b0d5ff63fa904160cc6490308b51c7e4648/fHDHR_web/api/tuners.py#L200

With the changes you already have for reading from the hardware, you might also be able to use direct streaming.  worth a try...

Yep! Plan to give it a roll tonight, will let you know. Just not quite sure which of the changes are meant to be included for this - few updates above, sorry.

I'd include these-
-  Device type detection
- Conditionally changing streamurl based on device type
- If your tests confirm direct stream works, update Readme to state that method works for PCI cards only

I wish we could make the direct_rtp method actually work for this as well someday

I'll take another look at that soon.  I had a very similar implementation last year.  It would process a handful of packets and then crash...

I recently made changes to the tuner buffer to re-attempt a stream 3 times before failing out, all while the buffered chunks still play in a client.

Hmmm ... OK, I have device detection updated (in a new local testing branch), and streamurl ... but when I try to merge in the latest from upstream/main, it says I'm already up to date (but clearly I'm not, when looking at the code.
Any thoughts?

 I was in plugins (ceton) ... different upstream. LOL!

Close! 😄. OK, if I leave stream_method = ffmpeg, it works fine. With stream_method = direct, it's trying, but I get the error,
[2022-01-24 16:28:23,334] INFO - Client has requested stream for ceton channel 505 KXAS NBC HD KXAS NBC HD.
[2022-01-24 16:28:23,335] INFO - Selected Stream Parameters: method=direct duration=0 origin_quality=None transcode_quality=None.
[2022-01-24 16:28:23,335] INFO - Attempting to Select an available tuner for this stream.
[2022-01-24 16:28:23,336] INFO - Tuner #0 Acquired.
[2022-01-24 16:28:23,336] INFO - ceton Tuner #0 to be used for stream.
[2022-01-24 16:28:23,362] INFO - Selected Ceton tuner#: 0
[2022-01-24 16:28:23,363] INFO - Ceton tuner 0 to be started
[2022-01-24 16:28:23,508] INFO - Preparing Ceton tuner 0 on port: 46361
[2022-01-24 16:28:24,526] INFO - Initiate PCIe direct streaming, channel 505 from Ceton tuner#: 0
[2022-01-24 16:28:24,527] INFO - Preparing Stream...
[2022-01-24 16:28:24,527] INFO - Stream Method Detected as a /dev/ hardware device.
[2022-01-24 16:28:24,528] ERROR - TunerError: 806 - Tune Failed: /dev/ceton/ctn91xx_mpeg0_0 PATH does not seem to exist
[2022-01-24 16:28:24,528] INFO - Tuner #0 Released.

I did check, and yes ... that path is correct, and does exist. Thoughts?
Thanks!

The new direct for hardware is untested and may need some work, as right now I'm doing a check if the path exists as if it was a file.

https://github.com/fHDHR/fHDHR/blob/main/fHDHR/device/tuners/stream/direct_hardware_stream.py

Perhaps try os.path.exists instead of os.path.isfile the former should for for character, block, directory and file, while the second is not documented as working for character or block devices (most common type in /dev)

Exactly what I did :rofl. Now looking at this,
[2022-01-24 16:37:35,675] INFO - Direct Hardware Stream from device: /dev/ceton/ctn91xx_mpeg0_0
[2022-01-24 16:37:36,414] ERROR - Chunk #1 unable to process: 'utf-8' codec can't decode byte 0xbc in position 5: invalid start byte

Checking it!

Sounds like progress

Absolutely! And pulling chunks seems to be OK ... looking now why it seems to think it can't handle them. Thanks!

@arrmo https://github.com/fHDHR/fHDHR/pull/223

you might want to remove the try/except for testing and write more explicit Exception handling

Yep. I made a slight change here,
with open(self.stream_args["stream_info"]["url"], 'rb') as device_stream

=> data is binary, not text (agreed?). I have video in Plex now!
But it's a bit "herky-jerky" ... technical term. LOL! Not quite sure why (yet).

Maybe try tuning bytes per read in your config.ini, double, cut in half to see if either works better and repeat

Agreed! Was messing with it here in streaming.json (directly), and it does seem to have an impact. In config.ini ... not sure which section to use?
[streaming]
[direct_stream]

And I guess not in [ceton]?

The code says it gets that value from [streaming]

You can also set it in the UI.  Not sure if it requires a restart,  If not it will make testing go much quicker...

https://github.com/fHDHR/fHDHR/pull/224

Could be - testing in pyCharm, I stop and restart each time, to make sure. No biggie. Was thinking it may be pyCharm causing it, so I tried from bash ... the same. Changing up / down does have an impact, but non seem quite happy. Wondering about buffering (and samples lost perhaps?)?
BTW, one pain ... in bash, I can't seem to Ctrl-C to get execution to stop, Seems odd.

I've noticed the Ctrl-C, but I rarely run it manually.  Ctrl-Z; kill %1 works for me
You might try multiples of 188 bytes, assuming we are getting good/correct mpegts packets from the card

Tried 8k * 188 ... perhaps a bit better, but still some glitches.
Wondering if somehow the input needs to be aligned to packets / frames? Will dig into this some. I'm thinking it may make sense to dump out the outgoing packets (just a couple, to compare), in the ffmpeg vs. direct cases.

Looking at a few HD programs I have recorded from my Ceton.  It looks like 8K TS packets is just over 1 second.   Losing a fraction of one should not be noticeable.  A much larger read would mean a bigger delay during initial tuning, but would also greatly reduced the error%, assuming that when we open the device the first byte is not the 1st in the packet, we would be losing the 1st and last in the read.  Does going up to 16k or 32 make any difference?  Or dropping to 4k?
And out of curiosity are the /dev/ceton/ctn* character or block devices (or other)?

Just curious, but what happens if you replace /dev/ceton/ctn91xx_mpeg0_%s with /dev/ceton/ctn91xx_filter0_%s
I read through the device driver driver code and the 188 bytes per packet is on the filter device and the mpeg device use a 3948 bytes per page/packet  (no idea where this comes from)

Just dropping a couple links for ideas....
https://github.com/fHDHR/fHDHR/blob/9aabbc0641b65c65e63630b81048ff328958b43c/data/internal_config/streaming.json#L10
https://github.com/fHDHR/fHDHR/blob/9aabbc0641b65c65e63630b81048ff328958b43c/fHDHR_web/api/tuners.py#L131
https://github.com/fHDHR/fHDHR_plugin_origin_tvheadend/blob/main/origin/__init__.py#L80-L84
I will probably look at moving some of these universal streaming settings to being origin specific.  Keep in mind, some config settings are from before fHDHR supported multiple origins.
HOWEVER, I share these links as there is a method for selecting quality profiles. With that in mind, if there's any way to select quality from the origin by changing the /dev/ path, that would be an extra nice addition.

I like when people actually collaborate, because then it pushes me to make cool updates to the core code. I've taken most of the streaming settings and made them origin specific (at least from the config file), I need to implement a better way of making some of the extra imposed settings more available from the web interface.
https://github.com/fHDHR/fHDHR/pull/225

Looking at a few HD programs I have recorded from my Ceton. It looks like 8K TS packets is just over 1 second. Losing a fraction of one should not be noticeable. A much larger read would mean a bigger delay during initial tuning, but would also greatly reduced the error%, assuming that when we open the device the first byte is not the 1st in the packet, we would be losing the 1st and last in the read. Does going up to 16k or 32 make any difference? Or dropping to 4k?

This does help! I checked, and my input is ~ 16 Mbps (video) ... so ~ 2 MB/s. If I shoot for ~ 2 MB packets => 1 per second, don't overload the loop. Wondering also about spawning it to another thread?

And out of curiosity are the /dev/ceton/ctn* character or block devices (or other)?

Not sure, sorry! I tried using https://docs.python.org/3/library/pathlib.html#pathlib.Path.is_block_device (and char_device) ... False to both?!?!

Just curious, but what happens if you replace /dev/ceton/ctn91xx_mpeg0_%s with /dev/ceton/ctn91xx_filter0_%s

No output at all - confirmed with mplayer also.

I read through the device driver driver code and the 188 bytes per packet is on the filter device and the mpeg device use a 3948 bytes per page/packet (no idea where this comes from)

That helps! OK, so I went with a fixed number of (3948 byte) pages / packets, 2021376 bytes. Used this, and also did buffering, to avoid lost bytes while processing them,
with open(self.stream_args["stream_info"]["url"], 'rb', buffering=self.bytes_per_read) as device_stream:

MUCH cleaner - almost perfect (just not 100% sure, that's all). But working, for sure. If I go down 2x ... a bit iffy => too fast for the loop to handle?
FYI, I did a stream to file (cat output), and then opened the file in Wireshark (it recognizes mpeg2ts!) ... packets starting right at the beginning of my file => so thinking it's a block (packet) based device?
Thoughts?

@deathbybandaid I agree! And we're aligned. Let me pull your code, drop the buffering I added 😆

FYI, only part way through, but I like this so far
https://tsduck.io/download/docs/mpegts-introduction.pdf

OK, I take it back (a little) about the video - I was getting fooled a bit, as Plex was transcoding. Arrgh. Once I went to Direct Stream from Plex, I could see it few more artifacts.
Odd to me, where / how it's getting dropped. Thinking buffering should help!

Might need to increase the buffer size,,,
Defaults to 3 chunks

To clarify, the stream code itself is a giant buffer layer on top of the layer that pulls the chunks of video
https://github.com/fHDHR/fHDHR/blob/main/fHDHR/device/tuners/stream/__init__.py#L123

And with my new code push, this should work per origin or the universal streaming setting

Meaning that the buffer (bytes_per_read) can be set for [streaming] or [ceton]? Just to understand.

Correct

Because in theory you might run multiple origins with different needs

Makes complete sense. Still odd to me that something is getting dropped somewhere. Thinking out loud, but for debug ... in addition to sending to Plex, dump the input (from Ceton) and output (to Plex) to a file? Then can try to analyze that?
Thoughts?

From the fhdhr interface, you can grab m3u to play in vlc and cut plex out of the picture

From one f the "play" links

Perfect! That works ... so I tried it, ffmpeg => no lost (video) frames. But direct - yep. That's good actually, helps to debug, in a non-eyeball way 😜
This means it's on the "input" capture side, right?
Thanks!

As I have no way to test any of this, and the direct_hardware method is brand new, you might just have to poke at the code to see how to make it work.
On the flipside, a partial stream is progress over no stream at all

As I have no way to test any of this, and the direct_hardware method is brand new, you might just have to poke at the code to see how to make it work.

Yep, agreed! And that's my plan. Just wanted to make sure I was looking the right place (i.e. input).

On the flipside, a partial stream is progress over no stream at all

Couldn't agree more! Thinking this is great progress - very close.

The main goal of fHDHR is to be able to pull any video origin, and output however wanted
The hdhr plugin simply does a 302 redirect to the /api/tuners endpoint. It just lets plex believe it's a real HDHR device. You could realistically write any output you wanted, including being available by dlna

That is so cool! And it really does work very well. This is a minor issue now, great progress IMHO.
Thanks!

FYI, with both input and output buffers set (to ~ 2 MB, but a fixed number of 3984 packets) ... so far, no dropped video frames (out of 10k). So progress. But odd ... when playing back an m3u, I lose the fHDHR web interface (it becomes non-responsive)? Comes back as soon as I stop the m3u.
I tried playing Plex in parallel (with m3u), but it tried to grab the same tuner => wouldn't let me do it.
Overall, looking quite good, just a few minor things. Thanks!

Oh, and in the code, I set buffering=0 (file open) ... as you have higher level buffering, correct?

fHDHR has it's own buffering system, but it's an amount of chunks and not a video duration.
As for the web page responsiveness, multi-threading with a web interface in python is not great.

Understood. It's odd - as soon as I stop the m3u stream, the page comes back.
And I need to take a look - seems only 1 tuner working now (but I know more were yesterday). Even with ffmpeg. Hmm. Enough for tonight ... LOL!

This morning I realized I needed to update the ffmpeg and vlc alt_stream plugins with the updated bytes_per_read logic
https://github.com/fHDHR/fHDHR_plugin_stream_vlc/pull/8
https://github.com/fHDHR/fHDHR_plugin_stream_ffmpeg/pull/6

wasted this morning playing with opencv/cv2 as a possible alt_stream plugin, and turns out it doesn't play well with handling in-memory byte objects, so disk IO would be a nightmare

OK, I need to dig more - but just to give you a heads up. I found that the "reason" for the non-responsive web UI, and it's also what is blocking the 2nd tuner ... as it goes into la-la land, and never responds. So for grins, I went back a day or so, when it was working,

fHDHR, git checkout 59ac29f8d832a08a7efc391d86d8725338081f23
fHDHR_plugin_stream_ffmpeg (needed, given 1 above), git checkout 239037da19abec53969ab1aedaeea8e3a9ef5a23

Then multi-tuner works again! So I still need to dig, but it's something changed since those commits ... which is actually good, narrows it down.
Thanks!

I'm on the most recent commit, and not seeing the web interface issue
Testing using 2 origins even

Yes, very odd. Thinking I pull the latest code for all 3, right? Meaning fHDHR, ffmpeg plugin, ceton plugin.
Thanks!

You gents have been busy this AM.  @arrmo The entries in /dev/ceton are actually links to the real devices in /dev
If you use 'ls -l /dev/ctn*' you should see the same 12 device entries.  Is the permissions starts with a 'c' it is a character device, 'b' block, 'l' link, 'd' directory, '-' normal file.  I found another post on the web with the ls -l output and it showed they are character devices.  I doubt the device type made a difference, but wanted to research to see if it did.
One point on buffering.  Most of the origins have the ability to stream faster than realtime.  Ceton cannot, so any extra buffer is not likely to add a lot of value.

https://github.com/fHDHR/fHDHR/pull/226
New update is an improved method of applying attributes to origin plugins, as well as creating easier web configuration of these items:

And went ahead and upgraded the EPG system with similar logic, and now those config settings are exposed
https://github.com/fHDHR/fHDHR/pull/227

Will give this a try, thanks! But not thinking I set 0 for bytes_per_read? Let me check out the code.
And I have to apologize - I think (99%) sure that the hanging issue was related to pyCharm, and you have to manually "Reload from disk" if pulling from git. I'd like to blame git, but I think the issue is between the keyboard and chair 😜
Thanks!

Oh, and I need to check on the new counting in the (stderr) output? 😉

Counting only seems to be there in ffmpeg. Also - frozen UI is back ... only with direct.

@arrmo doublecheck that you pull the newest push I did, there was a debug print line in there by mistake

Just a thought, but I am finding file reads can block all threads in python.  Perhaps add a time.sleep(.5) right before the read call?  The value of the sleep needs to be large enough to allow other work to complete and small enough that we do not miss packets in the device.  if you are reading about 1S of packets, .5 seems a reasonable start.  you might get away with as much as .9

I don't think you can sleep for decimal points in python

Agreed. That's actually why I was asking about putting that action (read, then write) in a separate (async) thread ... thoughts? And/or, can we have it configured to run automatically (kinda like DMA, of sorts) => cat /dev/xxx directly to the pipe going out?

I've toyed with the concept of tuning the stream in a separate thread, but it simply didn't work well.
My goal was to be able to serve multiple clients from the same source stream.

I've seen it documented both ways only > 1 and sub-second allowed.  If this is the issue an async thread might suffer the same way.  The thread is async, but the read is not.  The code is capable of reading far faster than realtime, so the generate function may be looping too tightly, keeping the device open until it has gotten a full read.
It should be an easy test to validate or disprove the theory, then the solution might look very different.

Wondering about async / await- will dig a bit

If you use 'ls -l /dev/ctn*' you should see the same 12 device entries. Is the permissions starts with a 'c' it is a character device, 'b' block, 'l' link, 'd' directory, '-' normal file. I found another post on the web with the ls -l output and it showed they are character devices. I doubt the device type made a difference, but wanted to research to see if it did.

Sorry, forgot to get back to you on this one. The output,
crw-rw-rw- 1 root root 231,  0 Jan 25 07:45 /dev/ctn91xx_ctl0
crw-rw-rw- 1 root root 231,  7 Jan 25 07:45 /dev/ctn91xx_filter0_0
crw-rw-rw- 1 root root 231,  8 Jan 25 07:45 /dev/ctn91xx_filter0_1
crw-rw-rw- 1 root root 231,  9 Jan 25 07:45 /dev/ctn91xx_filter0_2
crw-rw-rw- 1 root root 231, 10 Jan 25 07:45 /dev/ctn91xx_filter0_3
crw-rw-rw- 1 root root 231, 11 Jan 25 07:45 /dev/ctn91xx_filter0_4
crw-rw-rw- 1 root root 231, 12 Jan 25 07:45 /dev/ctn91xx_filter0_5
crw-rw-rw- 1 root root 231,  1 Jan 25 07:45 /dev/ctn91xx_mpeg0_0
crw-rw-rw- 1 root root 231,  2 Jan 25 07:45 /dev/ctn91xx_mpeg0_1
crw-rw-rw- 1 root root 231,  3 Jan 25 07:45 /dev/ctn91xx_mpeg0_2
crw-rw-rw- 1 root root 231,  4 Jan 25 07:45 /dev/ctn91xx_mpeg0_3
crw-rw-rw- 1 root root 231,  5 Jan 25 07:45 /dev/ctn91xx_mpeg0_4
crw-rw-rw- 1 root root 231,  6 Jan 25 07:45 /dev/ctn91xx_mpeg0_5

OK, doing some poking around ... it really seems that file access is blocking. So trying to find a way around it. Came across a couple interesting links, this one in particular.
https://stackoverflow.com/questions/39948588/non-blocking-file-read/39948643
I modified the code - to below, but still seems to be blocking (sleep may be blocking?),
            try:
                self.fhdhr.logger.info("Setting up file executor ...")
                executor = concurrent.futures.ThreadPoolExecutor(1)
                future_file = executor.submit(read_file, self.stream_args["stream_info"]["url"])
                while self.tuner.tuner_lock.locked():
                    if future_file.done():
                        chunk_counter += 1
                        self.fhdhr.logger.debug("Pulling Chunk #%s" % chunk_counter)
                        chunk = future_file.result()

                        if not chunk:
                            break

                        yield chunk

                        self.fhdhr.logger.info("Setting up file executor ...")
                        executor = concurrent.futures.ThreadPoolExecutor(1)
                        future_file = executor.submit(read_file, self.stream_args["stream_info"]["url"])

                    else:
                        time.sleep(0.1)

BTW, this is working, but still blocking. Pulling my 👨🦲 out! LOL.
Seems we really want a FIFO of sorts - agreed? Thinking the stream (from /dev) needs to go in to a buffer, and "just interrupt me when it's ready". The FIFO could (should) be longer than what we read, so we never lose data ... and just go get it when the desired amount of data is ready. But to do that, I'm not sure we want to just cycle in a loop, do we? That's my confusion - I can use asyncio / await (I think), but we stay in this loop from what I can tell? Or may be me misunderstanding!
A thought though ... do I push my latest changes - we have this problem fixed, using ffmpeg and /dev, all good. Then like we discussed, open some new "Issues" (features), for ...

Direct streaming from ceton (/dev)
Add more ceton info (like firmware version, etc.)
systemd setup (and document it)
Others you can think of?

Thoughts? Thanks!

FYI, also looked at .get(), thinking I could call it like,
https://thispointer.com/python-how-to-create-a-thread-to-run-a-function-in-parallel/
But the loop for read / write isn't really set up for async / callback, agreed? Or am I misunderstanding.
Other option then is to figure out how to read in a non-blocking way?

Excuse the made up math here.  We expect to get 2MB of data every second.  If we had the bytes_per_read set to exactly 1 seconds worth of data, how fast can we get that off the PCI bus?  Maybe 1ms?  Assume moving through the code and putting it out the TCP stack is another 20ms.  We would be back at the read and be blocking for 980ms until the device has another 1 sec of data ready to read.  To be honest, both the time to read and for the rest of the code might be on the high side...
I'm curious if you revert the concurrent.futures changes and add a time.sleep(0.5) right after the yield chunk line.  The amount of time to sleep should be proportional to bytes_per_read.  I do not know the ratio, but I suspect we could get away with a sleep of 0.9 for every 1s worth of bytes.

No worries, I like the made up math 😄. And it may make sense, but let me propose a WA idea if you don't mind, slightly different direction.
Trying to stick with the general flow of the code (which I'm gradually starting to understand) ... what if when the Tuner is "started", we kick off a process (separate thread), that is then non-blocking, and just reads the data, sticks it in a FIFO (Queue?). Then, read / pull from there, also non-blocking. Make sense so far?  Sort of mimics the read, but lets it happen in parallel, and non blocking. Or - am I on glue?!?! 😜
The other thing rolling around in my mind - what buffering exists on the device (/dev) itself? It can't be 1 byte ... must be some buffer there? It comes back to how fast to read from the stream.
Thoughts?

The issue with threading is memory object not being available to the main thread.

Even pi-hole can kill an SD card on a raspberry pi over time

Hmmm ... OK, gotcha. I admit, not sure I have a lot more ideas. The read is blocking ... and thinking about it, we are likely spending most of the time there, just waiting for the buffer to fill. Will keep thinking about it though!

Another concern is if the additional layers take too long, the client will possibly time out waiting.
The client could be plex, jellyfin, emby, vlc,,, and each has their own tolerance limits

I spent a lot of time down the rabbit hole trying to serve multiple clients from a single source stream, and never could make it work

Yep. And just as I hit send, and idea 😆.
Looking at ffmpeg, it's running a sub-process, and we grab stdout. What if we did the same ... but the direct (new) subprocess just does a device cat to stdout. Then do the same, grab stdout.
Thoughts? Just modeling after ffmpeg, but in that subprocess, just read, write to stdout.

I even still have code that allows for multiple clients registered to a tuner
https://github.com/fHDHR/fHDHR/blob/3fa34d8db61cbac93bb42f592e618a1cbff3702f/fHDHR/device/tuners/tuner.py#L157

Yep, makes sense! And that's perfect, no issue here. But to the note above - why not "behave" like ffmpeg ... start a subprocess, which is cat instead of ffmpeg, output to stdout? Or perhaps I'm missing the point?
Thanks!

Here's an example of what I've seen for using subprocesses with another python script
https://github.com/morpheus65535/bazarr/blob/master/bazarr.py#L52

I'm also still wrapping my head around some of the code, so we might be equally lost.  My take is that the generate() function is supposed to be the very thread you describe.  Read some data, put it in the buffer, yield to other threads, repeat.
The device cannot be fed data faster than real-time (2MB/s+/-) and the rest of the code can consume that amount of data in an instant.  Forcing the while loop to yield longer between reads gives other threads more time to handle their business.  Of course I too may may have a glue issue...
In the concurrent.futures version, is the sleep even firing?
I knew this felt familiar.  I had played around with a watchdog thread and one of the early versions had silly short sleep and it too locked the UI.  Turns out the watchdog was not needed, so it is no longer in the code.
I have looked for similar uses of python reading from a device, and have not come up with much.  I did have the idea to see if there are any projects that handle input from an audio device.  That seems more likely that a video and might provide clues on non-blocking device reads.
And you gents type too fast.  Been working on this for 20 minutes....

At one point I was using multiprocessing.Process, but the threading wasn't what I wanted, and i lost support for Windows.
One of the other goals of fHDHR is that it should be able to run on anything that python can install on

Agreed (Windows)! @deathbybandaid, you OK with the subprocess approach, similar to ffmpeg? I can give it a go, if OK with you. Thanks!

You can play with it in a branch on your own fork, but Idk if it will have any real benefit, as it still reads the same speed, and stdout seems like an extra step

Yes, understand - and will try on my branch, let you know 😆.
The reason is to try to avoid the blocking (as it limits me to 1 tuner), and the UI is also blocked.
Thanks!

It works! Ahh, life is good ... LOL. I have 3 tuners running - 2 to Plex, 1 to VLC. UI still fully responsive, and ... 0 dropped frames! Checking in VLC, up to 25000 frames so far, not a single one missing. Will clean up, and push if OK with you. But here is the "new" Direct_HardWare_Stream get() function. Thoughts? And yes, I pretty much just copied it from ffmpeg, minor edits - so very much consistent I think?
    def get(self):
        """
        Produce chunks of video data.
        """

        self.fhdhr.logger.info("Direct Hardware Stream from device: %s" % (self.stream_args["stream_info"]["url"]))
        self.directstr_commmand = 'cat %s' % self.stream_args["stream_info"]["url"]
        directstr_proc = subprocess.Popen(self.directstr_commmand.split(' '), stdout=subprocess.PIPE)

        def generate():
            chunk_counter = 0
            try:
                while self.tuner.tuner_lock.locked():
                    chunk = directstr_proc.stdout.read(self.stream_args["bytes_per_read"])
                    chunk_counter += 1
                    self.fhdhr.logger.debug("Pulling Chunk #%s" % chunk_counter)
                    if not chunk:
                        break
                    yield chunk

            except Exception as err:
                self.fhdhr.logger.error("Chunk #%s unable to process: %s" % (chunk_counter, err))

            finally:
                directstr_proc.terminate()
                directstr_proc.communicate()
                directstr_proc.kill()

        return generate()

Thanks!

Sorry, catching up here too!

The device cannot be fed data faster than real-time (2MB/s+/-) and the rest of the code can consume that amount of data in an instant.

You had me noodling on this one before - but I think you mean ... it's a real-time running stream, so you can't run ahead of it, right?

And you gents type too fast. Been working on this for 20 minutes....

I was struggling to keep up with you guys yesterday! 🤣

Next thought is windows compatibility

https://github.com/fHDHR/fHDHR/blob/main/fHDHR/versions/__init__.py#L167
I already register operating system if there's something different to be done

Next thought is windows compatibility

Agreed, but I admit - not quite sure how to handle that one, as there are no Windows drivers for the Ceton PCIe ... are there? Meaning - what device does it show up as, or is it even possible?
Thanks!

Actually, thinking you mean more generic than Ceton - just figured that out ... LOL.
Have to noodle a bit on that one. Basically, take a device, stream the data to stdout, right? That's really what / all this is doing.
I'll open a new Issue for the direct stream item, OK? Will mark it as New Feature (Label)?

You had me noodling on this one before - but I think you mean ... it's a real-time running stream, so you can't run ahead of it, right?
Yes and a bit more.  What happens when you try to read 1000 bytes from a file handle when it has only 500 bytes?  With a standard file you get 500 bytes, encounter the EOF marker and return.  Now when that handle is a I/O device that is being fed bytes by the hardware?  I suspect you will read 500 bytes quickly and the last 500 bytes as they are being delivered (in realtime), and blocking all other processes until it completes.
Placing a print time.time() before and after the read would instrument this.  The difference between the 1st and 2nd output should be small, maybe .002 of a second, then the gap between the 2nd and 3rd might be .02 of a second (while yielding the  chunk), I expect the 3rd to the 4th (during the 2nd read call) might get close to a full second, then the pattern would repeat .02, close to a second, .02, close to a second...
We would only need a few loops to see this.  The 1st stream is going to play fine as 1 second of data will be produced, collected and delivered in right around 1 second, but the vast majority of that second was in a blocking read call.
If we have a time.sleep(0.8) after the yield chunk call the loop will not return to the read call until the buffer is almost full, assuming the read is of 1 second worth of data.  And during that 0.8 seconds all other threads get a few CPU cycles to do things like service the UI, or another stream...
I made these numbers up, but based on PCIe bandwidth and the ~2MB/s we are consuming I would not be surprised if these are close to real.
If the theory proves to hold, then we'd want to figure out how to calculate a 'good enough' sleep value based on bytes_to_read.  But first it needs validation.

@arrmo you van always make it part of the ceton plugin, as an alt_stream method called ceton_direct

No clue, honestly, this is all new to me

No worries! I'll give a try - get back to you (after I open a new Issue, track it there 🤣). Just wanting to make sure to test it out.
@DanAustinGH, are you running on Windows or Linux? Just asking as once I have this ready, perhaps you can test it? I'm on Linux, hence the question.
Thanks!

@arrmo
opersystem  = self.fhdhr.versions.dict["Operating System"]["version"]

if opersystem in ["Linux", "Darwin"]:

    # cat

elif opersystem in ["Windows"]:

    # type

If we can't test it on Windows, it can always be looked into further in the future if it doesn't work.
So I'd say we implement this for the core direct_hardware method with the opersystem logic, and revisit if/when the need arises

Will do! And the logic above makes complete sense. Will capture / include this (this evening), get you a PR to look at.
Thanks!

I use Linux for the media server.  In fact I came to this project as the last Ceton HDHR emulator I used was Windows only and not a good fit for my use.

I thought about using a save ts file.  But unless you want to write another to slowly write to that file, it will have all the chunks available for reading not matter how fast you loop on it.

You might be able to use s bytesio object to store bytes in memory

Created PR #233 - please let me know your thoughts. Thanks!

And @DanAustinGH - updated the push to ceton as well, please comment. I made use of your get_ceton_getvar function, just added one more option 😄.
Thanks!

Merged.  And adding to get_ceton_getvar was the right way to do that.

If you have time it would be sweet to compare system load with ffmpeg vs direct as well as how quickly a channel starts from plex with each method.  Precise measurements may be hard, but if noticeably quicker in either case would be interesting.

If you have time it would be sweet to compare system load with ffmpeg vs direct as well as how quickly a channel starts from plex with each method. Precise measurements may be hard, but if noticeably quicker in either case would be interesting.

Will see if I can, NP at all! I checked real roughly yesterday, and the load due to cat was something like 0.3% (from top).


AQMesh read

We need an AQMesh processing function to process the live data for the dashboard.

See #147


fix binary predicate result should compare with 0 (#39474)

pick from master #39474

run buildall

run feut


[Autocomplete] fix paper slotProps

@vadimka123 Thanks for working on this – would you mind linking to/creating an issue that briefly explains the changeset?

@mj12albert https://github.com/mui/material-ui/issues/38870


Can cluster profile deploy a kustomization ref to only a set number of clusters in a cluster label

In the example provided for deploying a kutomization, https://projectsveltos.github.io/sveltos/addons/kustomize/ it will deploy the app, config map and secrets to all the clusters with the label env=fv .
In case i have two clusters with label env=fv, it will deploy it to both the clusters.
Does cluster profile have any mechanism to restrict the number of cluster to which the kustomization is deployed in when there are multiple clusters in a label?
ex:
i have two custers with label prod,
kubectl get sveltosclusters -n projectsveltos --show-labels
NAME                         READY   VERSION   LABELS
clusterapi-workload-dev      true    v1.27.1   env=dev,sveltos-agent=present
clusterapi-workload-prod     true    v1.27.1   env=prod,sveltos-agent=present
clusterapi-workload-prod-1   true    v1.27.1   env=prod,sveltos-agent=present
When i deploy the sample kustomization ref, it gets deployed to both the prod labled clusters:
sveltosctl/bin/sveltosctl show addons
+-------------------------------------------+-----------------+-------------+----------------------+---------+-------------------------------+-----------------------------------+
|                  CLUSTER                  |  RESOURCE TYPE  |  NAMESPACE  |         NAME         | VERSION |             TIME              |             PROFILES              |
+-------------------------------------------+-----------------+-------------+----------------------+---------+-------------------------------+-----------------------------------+
| projectsveltos/clusterapi-workload-dev    | helm chart      | vela-system | kubevela-core-latest | 1.9.9   | 2024-02-08 15:49:19 -0500 EST | ClusterProfile/kubevela-core      |
| projectsveltos/clusterapi-workload-prod   | helm chart      | vela-system | kubevela-core-latest | 1.9.6   | 2024-02-08 15:52:03 -0500 EST | ClusterProfile/kubevela-core-prod |
| projectsveltos/clusterapi-workload-prod   | apps:Deployment | eng         | the-deployment       | N/A     | 2024-02-08 17:31:56 -0500 EST | ClusterProfile/flux-system        |
| projectsveltos/clusterapi-workload-prod   | :Service        | eng         | the-service          | N/A     | 2024-02-08 17:31:56 -0500 EST | ClusterProfile/flux-system        |
| projectsveltos/clusterapi-workload-prod   | :ConfigMap      | eng         | the-map              | N/A     | 2024-02-08 17:31:56 -0500 EST | ClusterProfile/flux-system        |
| projectsveltos/clusterapi-workload-prod-1 | helm chart      | vela-system | kubevela-core-latest | 1.9.6   | 2024-02-08 17:01:49 -0500 EST | ClusterProfile/kubevela-core-prod |
| projectsveltos/clusterapi-workload-prod-1 | apps:Deployment | eng         | the-deployment       | N/A     | 2024-02-08 17:31:56 -0500 EST | ClusterProfile/flux-system        |
| projectsveltos/clusterapi-workload-prod-1 | :Service        | eng         | the-service          | N/A     | 2024-02-08 17:31:57 -0500 EST | ClusterProfile/flux-system        |
| projectsveltos/clusterapi-workload-prod-1 | :ConfigMap      | eng         | the-map              | N/A     | 2024-02-08 17:31:57 -0500 EST | ClusterProfile/flux-system        |
+-------------------------------------------+-----------------+-------------+----------------------+---------+-------------------------------+-----------------------------------+
Can we restrict it to deploy the cluster to only one of the prod clusters.

Thank you @ramekris3163
Have you considered using Profile? Profiles are like ClusterProfile but namespace scoped.
So if you have:

cluster1 (label env=fv) in namespace foo
cluster2 (label env=fv) in namespace bar

You can create this profile in namespace foo and Sveltos will deploy only in cluster1
apiVersion: config.projectsveltos.io/v1alpha1
kind: Profile
metadata:
  name: flux-system
  namespace: foo
spec:
  clusterSelector: env=fv
  syncMode: Continuous
  kustomizationRefs:
  - namespace: flux-system
    name: flux-system
    kind: GitRepository
    path: ./helloWorld/
    targetNamespace: eng

Another option available, is to set Spec.Paused to true.
So assuming you have two clusters:

cluster1 (labels: env=dev)
cluster2(labels: env=dev)

A single ClusterProfile can be created with clusterSelector: env=dev,  so it matches both.
Assuming we want Sveltos to reconcile only cluster1 (and not cluster2), set cluster2.Spec.Paused = true
Sveltos won't reconcile any cluster set to Paused.
In order for Sveltos to start reconciling add-ons and applications on cluster2,  reset cluster2.Spec.Paused = false

ClusterSet/Set might also help.
https://github.com/projectsveltos/addon-controller/commit/b842ac302cf671b5ef2e3262e7dd4d5507763eac

@ramekris3163 are you OK if we close this now that ClusterSet/Set have been introduced?


Feign REST Client - Converts GET requests to POST

When implementing a Feign client  using the following interface all GET requests to "getAllPlayers" are actually being attempted as POST requests resulting in an Unsupported Operation exception being thrown.
Interface;
import feign.Headers;
import io.swagger.annotations.ApiOperation;
import io.swagger.annotations.ApiResponse;
import io.swagger.annotations.ApiResponses;
import org.springframework.cloud.netflix.feign.FeignClient;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;

import static org.springframework.http.MediaType.APPLICATION_JSON_VALUE;

@FeignClient(name ="mgm",url = "http://localhost:6175")
@Headers({"Accept: application/json","Content-Type: application/json", "message_id: 818691693","authorization: cGxheXN0dWRpb3M6UEBzc3dvcmQ="})

@RequestMapping(value = "/api/ccas/v1/Patrons/{id}", produces = {APPLICATION_JSON_VALUE},method = RequestMethod.GET)
public interface IntegrationMGMServiceClient {

    @ApiOperation(value = "Returns a list of players. If no parameters are passed, a limited set of players are returned.", notes = "A GET operation on the Players endpoint returns information about the players. The response includes an array of players including name and primary bankroll. Historical data is not returned.", response = CasinoPlayer.class, responseContainer = "List")
    @ApiResponses(value = {
            @ApiResponse(code = 200, message = "An array of players"),
            @ApiResponse(code = 400, message = "Bad request, check syntax and parameters."),
            @ApiResponse(code = 500, message = "Internal server error."),
            @ApiResponse(code = 503, message = "Sevice not available.")})
    @RequestMapping(value = "",
            method = RequestMethod.GET)
    public PatronProfileResponse getAllPlayers(
            @PathVariable("id") Integer id);

Exception;
Caused by: feign.FeignException: status 405 reading IntegrationMGMServiceClient#getAllPlayers(Integer); content:

To add on , Instead of passing "id" as PathVariable, if I specify it directly in url, it works perfectly fine.

feign annotations are not supported by default. You need to use @RequestHeader


Allow drags while holding shift if boxZoom is disabled

Holding the shift key disables the ability to move around Draggables, to enable the use of boxZoom. When boxZoom is disabled, shift-drags of Draggables should be allowed.
Our use case is modifying the result of a drag based on whether the shift key is pressed.

Can this pls fixed

@Falke-Design
See #7088


Polishing before splitting out service provider framework

Code changes that remove excessive dependencies of classes required to be moved or be available after the Service Provider framework is modularized.
Required for

https://github.com/gradle/gradle/pull/29622

@bot-gradle test this

I've triggered the following builds for you. Click here to see all build failures.

PullRequestFeedback build

Sorry I don't understand what you said.
Currently, the following commands are supported:
@bot-gradle test <BuildTrigger1> <BuildTrigger2> ... <BuildTriggerN> [without PTS]
Add without PTS to run the build with full test coverage.

A trigger is a special build for this PR on TeamCity, common triggers are:

SanityCheck/CompileAll/QuickFeedbackLinux/QuickFeedback/PullRequestFeedback/ReadyForNightly/ReadyForRelease
Shortcuts: SC/CA/QFL/QF/PRF/RFN/RFR
Specific builds:

BD: BuildDistributions/BuildDocs so that you can preview the generated docs/distribution.
PT: PerformanceTest, all performance tests for Ready For Nightly stage.
APT: AllPerformanceTest, all performance tests, including slow performance tests.
AST: AllSmokeTestsPullRequestFeedback
AFT: AllFunctionalTestsPullRequestFeedback
ASB: AllSpecificBuildsPullRequestFeedback
ACC: AllConfigCacheTestsPullRequestFeedback
ACT: AllCrossVersionTestsReadyForNightly
AFTN: AllFunctionalTestsReadyForNightly
ACTR: AllCrossVersionTestsReadyForRelease
AFTR: AllFunctionalTestsReadyForRelease

@bot-gradle squash

Squash the current pull request into a single commit. The squash message will come from the pull request body, with:

All Signed-off-by: lines kept.
All authors being Co-Authored-By:

@bot-gradle merge

Enqueue this PR into GitHub merge queue for merging.

GitHub will create a merge commit from your PR branch HEAD and the target branch
A GitHub Merge Queue Check Pass build will be triggered on the merge commit
When the build passes, the target branch will be fast-forwarded to this merge commit (i.e. merge the PR)

@bot-gradle squash and merge

Squash the current pull request into a single commit as described in squash command above,
then enqueue this PR into GitHub merge queue as described in merge command above.

@bot-gradle cherrypick to <branch>

Cherrypicks the current PR to another branch.
A new PR will be created and a build will triggered automatically if there is no conflict.

@bot-gradle cancel

cancel a running build in GitHub merge queue and remove the PR from the queue

@bot-gradle clean

clear the conversation history

@bot-gradle help

display this message

To run a command, simply submit a comment.  For detailed instructions see here.

@bot-gradle test

I've triggered the following builds for you. Click here to see all build failures.

PullRequestFeedback build


Add Universidad Metropolitana para la Educación y el Trabajo - ARG

Official website: https://umet.edu.ar/
Bachelor of Computer Science: https://umet.edu.ar/course/informatica/

@ballesterosd lib/domains/ar/edu/soflab.umet.txt: this request has an incorrect file path format, please ensure that you are familiar with     this repository rules located at the bottom of this page and change     your request accordingly. If you don't know how to change request, close it and open a new one.     If you are not able to make the pull request according the instructions, please send us an email to<EMAIL_ADDRESS>Please mention this request URL in the message.

@philipto
Hi. I just submit a fix.
Sorry for this mistake.
Thanks for your atention.

@ballesterosd I have to close this request without merging, because you requested to add a non-existent domain to this repository: Host soflab.umet.edu.ar not found: 3(NXDOMAIN) Please ensure that you are familiar with this repository rules located at the bottom of this page and submit a new request with the correct information. Thank you for your understanding!