"Digital Divices" 8 chan modulator PCI card compatibility

I found a 8 channels DVB modulator PCI card with windows drivers and attractive price regarding to Hides and Dectek products (www.digital-devices.eu).
Do you think there is a chance to be compatible with tsduck ?
Also, FYI we are testing the new Hides PT-120 (4 channels DVB mod PCI board) in replacement of PT-100
and all seems good with new Hides driver and tshides plugin (I am interested with all feedback)
Fred

Hi @fredags

Do you think there is a chance to be compatible with tsduck ?

No chance at all, sorry.
There is a big difference between modulators and tuners in terms of software.
Tuners are consumer electronics, mostly used to watch TV on a computer. Consequently, there is a standard software API to interface tuners on Linux (LinuxTV) and another one on Windows (DirectShow). Each tuner comes with its own driver which acts as a proxy between the standard API of the operating system and the hardware. In the dvb plugin, TSDuck uses these standard API for TV reception and it works with all tuners (except potential specific driver defects).
Modulators, on the other hand, are professional equipments. Very few models exist. There is no standard API. Each modulator comes with its own driver API and the application layer (ie. output plugin for TSDuck) must be completely redesigned for each vendor. Currently, there are two dedicated plugins for modulators: dektec and hides. Any other modulator vendor would require another specific plugin.
The specific driver API of the modulator vendor is sometimes not even documented (that seems to be the case for Digital Devices as far as I can see on their site). That means that the modulator can only be used with their software. Writing a TSDuck plugin would not even be possible because of the lack of documentation. That's the reason why you cannot use those devices from TSDuck.
There will be soon a third TSDuck output plugin for modulators using a specific modulator chip vendor. I cannot disclose which one yet. Modulator products using this chip will be announced soon. The chip vendor develops its own TSDuck output plugin, so TSDuck support will be guaranteed from day one. This will be documented when available.


module 'zmq' has no attribute '0'

When I use NetGear, I got this error:   module 'zmq' has no attribute '0'
I install  pyzmp==17.1.2
numpy ==1.15.4
OpenCV-contirb-python==<IP_ADDRESS>
vidgear==0.1.5
And I run this server in container base ubuntu16.04
here is my code:
# import libraries
from vidgear.gears import NetGear
import cv2

stream = cv2.VideoCapture('hamilton_clip.mp4') #Open any video stream

options = {'flag': '0', 'copy': False, 'track': False}
#change following IP address '192.168.1.xxx' with yours
server = NetGear(address = '<IP_ADDRESS>', port = '5454', protocol = 'tcp',  pattern = 0, receive_mode = False, logging = True, **options) #Define netgear server at your system IP address.

# infinite loop until [Ctrl+C] is pressed
while True:
	try:
		(grabbed, frame) = stream.read()
		# read frames

		# check if frame is not grabbed
		if not grabbed:
			#if True break the infinite loop
			break

		# do something with frame here

		# send frame to server
		server.send(frame)

	except KeyboardInterrupt:
		#break the infinite loop
		break

# safely close video stream
stream.release()
# safely close server
writer.close()

it come from netgear demo

@Julius-ZCJ Thanks for reporting this bug. As temporary workaround Kindly remove flag attribute for now (as it's useless in current scenario) from **options dict which then becomes:
options = {'copy': False, 'track': False} //without `flag`

Also, I'm push fix for this bug in development branch shortly. Stay tuned.

Thank you for your reply. I modify it now, but I meet some other errors:
1).when I use NetGear frist demo, using sever and client in same container, I got this error:
AttributeError: 'NetGear' object has no attribute 'id' (this error raise in client)
2).when I use NetGear second demo, using server and client in differ container, using tcp connect, I got follow error:
zmq.error.ZMQError: Cannot assign requested address
ValueError: Failed to bind address: tcp://192.168.1.xxx: xxxx and pattern: 0! Kindly recheck all parameters.(this also raise in client)
it tell me my address is error, I can't connect my server ,what can I do to solve this problem?

1).when I use NetGear frist demo, using sever and client in same container, I got this error:
AttributeError: 'NetGear' object has no attribute 'id' (this error raise in client)

@Julius-ZCJ This bug is already been fixed in the development branch, I'm posting related instructions shortly. As a workaround, kindly enable logging=True in Netgear Class as follows:
client = NetGear(receive_mode = True, logging = True)

2).when I use NetGear second demo, using server and client in differ container, using tcp connect, I got follow error:
zmq.error.ZMQError: Cannot assign requested address
ValueError: Failed to bind address: tcp://192.168.1.xxx: xxxx and pattern: 0! Kindly recheck all parameters.(this also raise in client)

@Julius-ZCJ Kindly change IP address parameter to your Client's IP Address, you can use : hostname -I terminal command to know client's IP.

@abhiTronix I have successed in first demo ，but I'm failed in second demo.
I run server in container, which ip is <IP_ADDRESS>. And the container host ip is <IP_ADDRESS>,
I mount port is 5454->5454, and my pc ip is <IP_ADDRESS>
I run client in my pc, my pc ip is <IP_ADDRESS>. So, what ip i could used in server, what ip i could used in client? I first use <IP_ADDRESS>, and run client in same container with same ip, and I successed.
But when I run server in container with ip <IP_ADDRESS>, and run client in my pc with ip <IP_ADDRESS>
I always failed.  I have tried to using container ip in client but it also failed. what can I do to make container can connect my pc?

but I'm failed in second demo.
I run server in container, which ip is <IP_ADDRESS>. And the container host ip is <IP_ADDRESS>,
I mount port is 5454->5454, and my pc ip is <IP_ADDRESS>
I run client in my pc, my pc ip is <IP_ADDRESS>. So, what ip i could used in server, what ip i could used in client? I first use <IP_ADDRESS>, and run client in same container with same ip, and I successed.
But when I run server in container with ip <IP_ADDRESS>, and run client in my pc with ip <IP_ADDRESS>
I always failed. I have tried to using container ip in client but it also failed. what can I do to make container can connect my pc?

You have to use the Client's IP address on both client and server, i.e <IP_ADDRESS> in your case since you are using this pc as a client.

@Julius-ZCJ Kindly test testing branch by strictly following steps:

Firstly, Kindly clone and install the development branch as follows:

git clone git clone https://github.com/abhiTronix/vidgear.git
cd testing
git checkout development
sudo pip install .
cd

Then rerun the following updated algorithm for testing, but remember to change the IP address and port address with yours :

A. Server End:
Open the terminal on System-1 (Server) and execute the following python code. **Also, remember to write down the IP-address of Server by executing the command: 'hostname -I' and replace it in the following code:
# import libraries
from vidgear.gears import NetGear
import cv2

stream = cv2.VideoCapture('test.mp4') #Open any video stream

options = {flag : 0, copy : False, track : False}

#change following IP address '192.168.1.xxx' with yours
server = NetGear(address = '192.168.1.xxx', port = '5454', protocol = 'tcp',  pattern = 0, receive_mode = False, logging = True, **options) #Define netgear server at your system IP address.

# infinite loop until [Ctrl+C] is pressed
while True:
	try:
		(grabbed, frame) = stream.read()
		# read frames

		# check if frame is not grabbed
		if not grabbed:
			#if True break the infinite loop
			break

		# do something with frame here

		# send frame to server
		server.send(frame)

	except KeyboardInterrupt:
		#break the infinite loop
		break

# safely close video stream
stream.release()
# safely close server
writer.close()

B. Client End:
Then open a terminal on the System-2(Client) and replace the IP address with Server's IP address and execute the following python code and see the output:
# import libraries
from vidgear.gears import NetGear
import cv2

options = {flag : 0, copy : False, track : False}

#change following IP address '192.168.1.xxx' with yours
client = NetGear(address = '192.168.1.xxx', port = '5454', protocol = 'tcp',  pattern = 0, receive_mode = True, logging = True, **options) #Define netgear client at Server IP address.

# infinite loop
while True:
	# receive frames from network
	frame = client.recv()

	# check if frame is None
	if frame is None:
		#if True break the infinite loop
		break

	# do something with frame here

	# Show output window
	cv2.imshow("Output Frame", frame)

	key = cv2.waitKey(1) & 0xFF
	# check for 'q' key-press
	if key == ord("q"):
		#if 'q' key-pressed break out
		break

# close output window
cv2.destroyAllWindows()
# safely close client
client.close()

Successfully Resolved & Merged in commit: https://github.com/abhiTronix/vidgear/commit/48c6a69d46e8b02683e4760c0e9488f58dfc2f47

Successfully Resolved & Merged in commit: https://github.com/abhiTronix/vidgear/commit/48c6a69d46e8b02683e4760c0e9488f58dfc2f47

@abhiTronix  Thank you vary much, I have solved this problem


Tokemak deposit

Note: I haven't added in the abstraction for depositing, I have created a ticket to handle that later here https://github.com/shapeshift/foxy/issues/17

Got rid of the mock FOX and used a mainnet fork to interact with TOKE directly
Removed the test deploys since we aren't using mock FOX anymore
Added mainnet env variable and use that when instantiating a local node
Added TOKE deposit to the stake function
Included the FOX balance in TOKE to the contractSupply to give correct rewards

On top of the review changes, I added the tokePool address to the deploy script


Only run test from CASSANDRA-11038 on 2.2+

Since 2.1 is critical fixes only, the patch to ensure correct behaviour for 11038 was only added from 2.2 onwards, so it's probably wise to only run it's test on those versions.
(see also: CASSANDRA-12056)

+1


Over right phone_number field as CharField in ShippingAddress model but its giving <phonenumber_field.modelfields.PhoneNumberDescriptor object at 0x7f8a95a6c460>

class ShippingAddress(AbstractShippingAddress):
place = models.TextField(blank=True, default="")
place_id = models.CharField(max_length=255, blank=True, default="")
location = PointField(srid=4326, null=True)
country = models.ForeignKey(
"address.Country",
on_delete=models.CASCADE,
verbose_name="Country",
default=get_default_country,
)
phone_number = models.CharField(max_length=100, null=True, blank=True)
Technical details

Python version: 3.8.5
Django version: Django==3.0
Oscar version: django-oscar==3.0.2

Hi there. We use Github only for managing bugs and feature requests, and not for support. We recommend that you ask for help on Stack Overflow using the django-oscar tag, or joining our slack workspace or mailing list and asking a question there. Thanks.


Initial opening of the CB takes a long time

This task is automatically imported from the old Task Issue Board and it was originally created by James Dunkerley.
Original issue is here.

The first time we open the component browser it takes a long time to get the values.
Looking at the logs on the network of the IDE it appears it takes 13-15s to get the suggestion database loaded.
Comments:
(James Dunkerley - Feb 1, 2023)

Performance issue for @wdanilo.  (Michael Mauderer - Feb 1, 2023)

It looked like it was taking the time in the language server getting the initial suggestionDB sent over so possibly one for @dmitry_bushev  (James Dunkerley - Feb 1, 2023)

The initial processing of suggestions takes significant time of the LS startup, and can be definitely improved as discussed here https://discord.com/channels/401396655599124480/1055451457576509501/1055477146367295488  (Dmitry Bushev - Feb 1, 2023)

Sheduled #5068 and #5570 should speedup the delivery of initial suggestions. This issue should be re-evaluated after the linked optimizations are implemented.

Lets review the state post completion of #5068 and see what is still needed.

Closing this ticket as the performance has now been significantly improved.


Set button types as "button" for modal open and close buttons

Hi there !
Default button type is "submit" in most browsers, this has fired "submit" events if the modal is embedded in a form for example.
Setting the type to "button" seems more in line with the intent in this case.
What do you think ?

YES!
This totaly makes sence, buttons are submit by default when they have no callback but in this case it should absolutely be a button thank you for figuring this out!


Uncaught TypeError: Cannot read property 'element' of undefined

[Enter steps to reproduce:]

...
...

Atom: 1.16.0 x64
Electron: 1.3.13
OS: Unknown Windows version
Thrown From: open-files package 0.5.2
Stack Trace
Uncaught TypeError: Cannot read property 'element' of undefined
At C:\Users\Utilisateur externe\.atom\packages\open-files\lib\open-files-pane-view.js:222

TypeError: Cannot read property 'element' of undefined
    at OpenFilesPaneView.updateModifiedState (/packages/open-files/lib/open-files-pane-view.js:222:15)
    at /packages/open-files/lib/open-files-pane-view.js:177:18
    at Function.module.exports.Emitter.simpleDispatch (/app.asar/node_modules/event-kit/lib/emitter.js:25:14)
    at Emitter.module.exports.Emitter.emit (/app.asar/node_modules/event-kit/lib/emitter.js:141:28)
    at TextBuffer.module.exports.TextBuffer.emitModifiedStatusChanged (/app.asar/node_modules/text-buffer/lib/text-buffer.js:1407:27)
    at /app.asar/node_modules/text-buffer/lib/text-buffer.js:1396:24

Commands
  7x -6:22.2.0 core:backspace (input.hidden-input)
undefined -6:16.2.0 intentions:highlight (input.hidden-input)
     -6:11.4.0 editor:select-to-end-of-word (input.hidden-input)
     -6:10.7.0 intentions:highlight (input.hidden-input)
     -6:10.5.0 core:copy (input.hidden-input)
     -6:10.3.0 core:move-left (input.hidden-input)
     -5:26.7.0 core:backspace (input.hidden-input)
     -3:06.7.0 intentions:highlight (input.hidden-input)
  2x -3:06.5.0 core:undo (input.hidden-input)
     -3:02.9.0 core:backspace (input.hidden-input)
     -3:01.9.0 intentions:highlight (input.hidden-input)
     -3:01.7.0 core:save (input.hidden-input)
     -1:01.6.0 tree-view:remove (span.name.icon.icon-file-text)
     -0:14 editor:newline (input.hidden-input)
     -0:13.2.0 intentions:highlight (input.hidden-input)
     -0:13 core:paste (input.hidden-input)

Non-Core Packages
busy-signal 1.4.0
compare-files 0.8.0
highlight-selected 0.13.1
intentions 1.1.2
linter 2.1.4
linter-ui-default 1.2.3
markdown-preview-opener 0.1.1
minimap 4.27.1
minimap-highlight-selected 4.6.1
minimap-pigments 0.2.2
open-files 0.5.2
pigments 0.39.1
svn 0.0.13
todo-show 1.11.0
tortoise-svn 0.5.0

I can't reproduce the issue. Am I correct that you were just deleting a word in a file when the error message appeared? Can you tell me what kind of file you were editing when this happened?

Hi, yes when a deleting a word in a file, the message appears and also when I save it. I work with Embedded RuBy (with double file extension example_file.html.erb)

I added all of your packages to my Atom installation and tried with an .html.erb file and I still can't reproduce the problem. It all works fine for me. I wonder if this is a Windows problem (I use OS X and I don't have access to a Windows system, so I can't try it there). Does the problem file appear in the 'Open Files' list before you try to erase and save, or does it not even appear there? Does that happen with other .html.erb files? Does it happen with other types of files at all?

I get the same error with c files.
When deleting a line I get the same error:
TypeError: Cannot read property 'element' of undefined
at OpenFilesPaneView.updateModifiedState (/local/home/frq07517/.atom/packages/open-files/lib/open-files-pane-view.js:221:15)
at /local/home/frq07517/.atom/packages/open-files/lib/open-files-pane-view.js:176:18
at Function.module.exports.Emitter.simpleDispatch (/opt/atom/resources/app.asar/node_modules/event-kit/lib/emitter.js:25:14)
at Emitter.module.exports.Emitter.emit (/opt/atom/resources/app.asar/node_modules/event-kit/lib/emitter.js:129:28)
at TextBuffer.module.exports.TextBuffer.emitModifiedStatusChanged (/opt/atom/resources/app.asar/node_modules/text-buffer/lib/text-buffer.js:1386:27)
at /opt/atom/resources/app.asar/node_modules/text-buffer/lib/text-buffer.js:1375:24
It is not systematic though, and I can't identify a pattern to reproduce it.

I still can't reproduce, but I made some changes that might take care of this. Can you please let me know if the problem is still there? Thanks!

I also found this problem but only when I had two or more files open and split them between panes.

Damn. I was hoping this would be fixed. I tried to split files between panes and I still can't reproduce the bug. @J-Mills , could you please give me more details about your setup, and how exactly you got the problem to appear? Thanks!

Really sorry I didn't get back to you sooner! I didn't get a notification for your reply.
[Enter steps to reproduce:]

...
...

Atom: 1.17.0 x64
Electron: 1.3.15
OS: Microsoft Windows 10 Education
Thrown From: open-files package 0.5.5
Stack Trace
Uncaught TypeError: Cannot read property 'element' of undefined
At C:\Users\Jack\.atom\packages\open-files\lib\open-files-pane-view.js:230

TypeError: Cannot read property 'element' of undefined
    at OpenFilesPaneView.updateModifiedState (/packages/open-files/lib/open-files-pane-view.js:230:15)
    at /packages/open-files/lib/open-files-pane-view.js:181:20
    at Function.module.exports.Emitter.simpleDispatch (~/AppData/Local/atom/app-1.17.0/resources/app/node_modules/event-kit/lib/emitter.js:25:1)
    at Emitter.module.exports.Emitter.emit (~/AppData/Local/atom/app-1.17.0/resources/app/node_modules/event-kit/lib/emitter.js:141:1)
    at TextBuffer.module.exports.TextBuffer.emitModifiedStatusChanged (~/AppData/Local/atom/app-1.17.0/resources/app/node_modules/text-buffer/lib/text-buffer.js:1457:1)
    at ~/AppData/Local/atom/app-1.17.0/resources/app/node_modules/text-buffer/lib/text-buffer.js:1446:1

Commands
     -2:44.3.0 tree-view:show (atom-workspace.workspace.scrollbars-visible-always)
     -2:21.2.0 settings-view:open (div.tool-panel.tree-view)
     -0:15.7.0 docblockr:parse-enter (input.hidden-input)
     -0:15.7.0 editor:newline (input.hidden-input)
  5x -0:14.7.0 core:backspace (input.hidden-input)

Non-Core Packages
atom-beautify 0.29.24
autoclose-html 0.23.0
autocomplete-paths 1.0.5
busy-signal 1.4.1
color-picker 2.2.5
docblockr 0.9.4
file-icons 2.1.6
flex-tool-bar 0.12.0
git-plus 7.8.1
highlight-line 0.12.0
highlight-selected 0.13.1
intentions 1.1.2
language-babel 2.61.0
language-swift 0.5.0
linter 2.1.4
linter-csslint 1.3.4
linter-htmlhint 1.3.3
linter-javac 1.9.4
linter-js-standard 3.9.1
linter-jsonlint 1.3.0
linter-phpmd 1.5.3
linter-shellcheck 1.4.4
linter-ui-default 1.5.4
minimap 4.28.2
minimap-highlight-selected 4.6.1
minimap-pigments 0.2.2
oceanic-next 1.0.0
open-files 0.5.5
pigments 0.39.1
script 3.14.1
set-syntax 0.3.2
sort-lines 0.14.0
sync-settings 0.8.1
todo-show 1.11.0
tool-bar 1.1.0
tree-view-open-files 0.3.0

Every time I drag a file to one half of the screen to split them, I get the error as soon as I change anything in that file.

Thanks for the info, @J-Mills. I was able to reproduce and work on the bug. I think it should be fixed now, but please let me know if you're still encountering the problem

@oriolmirosa It's fixed!! Thank you. Everything seems to work fine now. Great package by the way - the only one I've found that actually works.


fix #1837, Use java.net.preferIPv4Stack to force IPv4

Reason:

Be consistent with hadoop.
Avoid confusion in configuring hostname and master address.

+1.


Find out how to get FB share data.

Apparently not:
https://developers.facebook.com/docs/graph-api/reference/v20.0/insights


Reliving Memory is not showing copy of entered text in Scrapbook Chat Window

Describe the bug
When I relive a memory and add new details utilizing the Scrapbook bot it doesn't display a copy of my entered text like it does when I organically use the Biography or Scrapbook bots. Not having a way to see the text I've entered and having it disappear immediately makes me concerned. Additionally upon pressing the NEXT button the text just disappears and it appears as though nothing is happening. There is no indication that something is happening behind the scenes and the inputted text is being looked at.  In the video provided the entered text doesn't seem to even be absorbed by MyLife so what I typed was just gone.  In previous attempts in the same bot I did make new entries that appeared to add on to my memory bank however I didn't make mention to other key words from already banked memory so perhaps an inability to draw a correlation seemed to prevent a new memory from being recorded and feedback provided by the bot.
To Reproduce
Steps to reproduce the behavior:
Watch video
Expected behavior

I expect to see a copy of my entered text.
I expect the information to show with at least paragraphs formatting.
I expect to see some type of visual indicator that after pressing Next button that information is being processed.  Not having this feedback could lead to people refreshing the page or something else leading to poor  experience.  In the video it takes approximately 17 seconds from the point that the Next button is pressed and my text disappears for it to reappear indicating that the text has been parsed and possibly something has happened.

Screenshots
(https://somup.com/cZjjFVGaFZ)
Desktop (please complete the following information):

OS: Windows 11 64 bit
Browser Chrome
Version 127 64 Bit

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
I am doing this entry to record how much time after I click the next button that it begins to show a response. I am also curious as to why the text I have entered doesn’t show in the chat window like I am able to see with using the bottom text entry field. It seems to only put a copy of the text less the formatting when using the Primary chat field.

This solution will come from the same vein as chageTitle

in avatar memory (upon callback from LLM-provider) present frontend instructions to force the correct update to the frontend

Really fun to fix this one @Whammo22 - it might not be perfect, but it's dynamically incorporating and updating the story content summary as well


Delete Books As Owner

Sometimes you need to delete negativity right out of your life... This will let you do that

WHEN YOU REVIEW THIS MAKE SURE YOU DO THIS

Damn, this looks like it was more work than anticipated. Much more than a 1, probably at least a 3. But nice job!

That looks sliiiiiiiiiick


Improve channel behavior in shutdown situations

-- expose ShutdownToken (of type CancellationToken) property on Channel to simplify detection of channel being shutdown (and possibly register some behavior associated with that). Fixes #6311.
-- channel.State now returns ChannelState.FatalFailure after the channel has been shutdown instead of throwing. Fixes #6426.
-- added a few more tests and fine tuned channel behavior related to shutdown

CC @jskeet

LGTM


not saving shrec_net.pth after dynamic training

After dynamic training the shrec_net.pth is not saved in models directory. and also the user_net.pth is not as a zip file?

Hi @kishorkuttan , it appears you started using this repo just before I could fully migrate the changes :) Could you get the latest commits and see if the issue persists?
To answer your question, .pth files are not zip files. .pth is the convention used to denote a saved PyTorch model.

Its working great. Trained it on torch 1.4.0 and torchvision 0.5.0. Now its working. You can close the thread

Awesome :)


Adds admin controller

Adds /admin/users and /admin/users/* route to get admin details.
Also adds is_admin to user table to create admins to authorize them
inside the application
I will work on adding more stuff :star2:
cheers!

I used the sqlite3 cli to mark myself as an admin. I've also created a function for the same to create admins in the User.pm schema.
@kyzn how should we mark the admin initially?
also, please review this PR and the README one as well.
Thanks :smiley:

Hey @gabru-md, thanks a lot for your PRs! I will review them as soon as I get a chance.

sure sir!
:smile:

review @kyzn ? :sweat_smile:

Will do tomorrow night, once the week is over! Thanks for your patience @gabru-md  🙂

Hey @gabru-md, are you still interested in these PRs?

yes I'll be updating them.
sorry was busy with my university exams for a long time

No worries -- and good luck with your exams 🎉


API Request - add a section to see if domaine is still valid

I was wondering if you could add to the API to possibility of being able to see if the somaine is still valid:
This is the list of all disposable domain that was up and running:

wifi-map.net
convoitucpa.com
mantutimaison.com
shhongshuhan.com
snasu.info
phonestlebuka.com
hungtaoteile.com
victorysvg.com
piribet100.com
oanhxintv.com
yongshuhan.com
toanciamobile.com
pfmretire.com
mantutivi.com
mamonsuka.com
hansgu.com
usbvap.com
hacktoy.com
ladz.site
cerapht.site
suprj.site
screalian.site
bookez.site
rtfat.site
syndonation.site
howtb.site
kaspecism.site
pdfrj.site
literc.site
azwen.site
foundents.site
refap.site
rtfie.site
qtxm.us
azwee.site
pubiy.site
bookue.site
rustracker.site
refal.site
omilk.site
pubis.site
devoi.site
how1z.site
coduh.site
zipzx.site
filezw.site
readoe.site
devou.site
rollercover.us
prcea.site
betration.site
bookug.site
1ki.co
supoz.site
zipea.site
txtee.site
codur.site
lahi.me
docxu.site
manau.site
readyx.site
pdfro.site
csht.team
suprd.site
plainst.site
devib.site
sproces.shop

Last check:
August 26, 2022

@Daniel3356 this is useful. How did you create that list?

@Daniel3356 this is useful. How did you create that list?

You can look at google domain (bulk registration) and see what domain can you buy. All the one that you can buy are no longer valid.
from a software approach, you can still look at an A and MX record, if none of them are found you are 50% sure. Probably Google, NameCheap, Godady have an api for bulk domain registrations.

That's a good idea. I have a similar problem where I want to see if an email is really valid. Is the MX lookup accurate? Can anyone update the DNS and add fake registers or there is a way to validate that?

The idea of the JSON formats firstseen and lastseen properties are to cleanup domains (https://github.com/7c/fakefilter/issues/6) - what do you guys think about this concept ? Everyone can have different concepts how to calculate a valid domain.
We want to introduce new json format next weeks/months because single domain can be seen in multiple providers. As an example:'pfmretire.com' which has been seen at 4 different providers: 'mail-temp.com', 'generator.email','emailfake.com','email-fake.com'

the cleanup was implemented and detailed at #6 - thanks!


Appstudio update test-custom-default-branch-lswp

Pipelines as Code configuration proposal
To start the PipelineRun, add a new comment with content /ok-to-test
For more detailed information about running a PipelineRun, please refer to Pipelines as Code documentation Running the PipelineRun
To customize the proposed PipelineRuns after merge, please refer to Build Pipeline customization

Pipelines as Code CI has failed.There was an error creating the PipelineRun: test-annotations-yrqk-on-pull-request-
creating pipelinerun test-annotations-yrqk-on-pull-request- in namespace build-e2e-gcnu-tenant has failed.
Tekton Controller has reported this error: the namespace of the provided object does not match the namespace sent on the request

Pipelines as Code CI has failed.There was an error creating the PipelineRun: test-simple-build-mcmm-on-pull-request-
creating pipelinerun test-simple-build-mcmm-on-pull-request- in namespace build-e2e-gcnu-tenant has failed.
Tekton Controller has reported this error: the namespace of the provided object does not match the namespace sent on the request

Pipelines as Code CI has failed.There was an error creating the PipelineRun: test-simple-build-nwtb-on-pull-request-
creating pipelinerun test-simple-build-nwtb-on-pull-request- in namespace build-e2e-gcnu-tenant has failed.
Tekton Controller has reported this error: the namespace of the provided object does not match the namespace sent on the request

Pipelines as Code CI has failed.There was an error creating the PipelineRun: test-simple-build-imul-on-pull-request-
creating pipelinerun test-simple-build-imul-on-pull-request- in namespace build-e2e-gcnu-tenant has failed.
Tekton Controller has reported this error: the namespace of the provided object does not match the namespace sent on the request

Pipelines as Code CI has failed.There was an error creating the PipelineRun: test-simple-build-uqzm-on-pull-request-
creating pipelinerun test-simple-build-uqzm-on-pull-request- in namespace build-e2e-gcnu-tenant has failed.
Tekton Controller has reported this error: the namespace of the provided object does not match the namespace sent on the request

Pipelines as Code CI has failed.There was an error creating the PipelineRun: test-simple-build-wadk-on-pull-request-
creating pipelinerun test-simple-build-wadk-on-pull-request- in namespace build-e2e-gcnu-tenant has failed.
Tekton Controller has reported this error: the namespace of the provided object does not match the namespace sent on the request

Pipelines as Code CI has failed.There was an error creating the PipelineRun: test-simple-build-lqle-on-pull-request-
creating pipelinerun test-simple-build-lqle-on-pull-request- in namespace build-e2e-gcnu-tenant has failed.
Tekton Controller has reported this error: the namespace of the provided object does not match the namespace sent on the request

Pipelines as Code CI has failed.There was an error creating the PipelineRun: test-simple-build-ufgi-on-pull-request-
creating pipelinerun test-simple-build-ufgi-on-pull-request- in namespace build-e2e-gcnu-tenant has failed.
Tekton Controller has reported this error: the namespace of the provided object does not match the namespace sent on the request


INVALID OUTPUT FROM UNEXISTING COMMAND

The issue:
-> lsjdf | wc
-> minishell : lsjdf: command not found
->   0   1    4

this command line has spotlighted a real issue in our code. In case of a pipeline, we go through pipex() creating a child process to execute the current command meanwhile this command is invalid, so we check its validity inside the child process, by making sure that the cmd->path == NULL and cmd->builtin == -1, at this stage we had already set the data streams, then we output to STDERR the error message and we exit the child proc.  Question : From where we get this data that wc processes and output as 0  1  4 knowing that it reads nothing??
Using the read() syscall we could see the data that wc receives and those are 4 characters 27 91 48 109.
The Fix:
By Checking the validity of a command before creating the child process that we'll use for its execution, we were able to prevent this
issue and get the correct output, so that
-> lsjdf | wc
-> minishell : lsjdf: command not found
->   0   0    0

We still don't know the answer for the question above.

This was fixed in  9c2848cfccb68ee4b888c3e23097b1e3b46cd5ea


DXOC-661 Updated Objective-C/Swift codegen

Both actions are failed:

ERROR request to https://data-api.staging.amplitude.com/graphql failed, reason: connect ETIMEDOUT <IP_ADDRESS>:443
swift tests

Not sure how to fix them.

Thank you @falconandy. Changes look good but I am going to wait until the tests are working to merge.
The https://data-api.staging.amplitude.com/graphql error is a known issue. I am working with the security team to resolve.
I was able to fix the Swift tests (locally) in my other branch. Hoping to fix ObjC soon.


Added Hebrew translation

Full translation to Hebrew.

It will be easier to manage along time if you'll use a localization platform such as Weblate or Damned Lies.
Thank you :)

@yarons https://hosted.weblate.org/engage/Lock/ (361021982de8d36c4c6babb34d26818f08f348cd)
:)


NAT-2295 - Adds getRecordingStatus method.

Nevermind this. Was trying to merge into our forks master branch... Didn't read everything carefully enough.


Compiler error when using user32.LockWorkStation

VS is throwing an error during compilation because of a backend file that I cannot see.  Here are the details that I can see
CS0122: 'BOOL' is inaccessible due to its protection level

File: PInvoke.User32.cs, Line 36

The method endpoint is generated properly, Pinvoke.LockWorkStation() is available and no errors appear in my app code.  "LockWorkStation" is the only method in my NativeMethods.txt

a backend file that I cannot see

Tip: if you're using VS 16.9 (in Preview), all generated files will be discoverable via Solution Explorer:

a backend file that I cannot see

Tip: if you're using VS 16.9 (in Preview), all generated files will be discoverable via Solution Explorer:

The first tip that you showed about finding the generated files does work for me, thanks (but it has a bug, see footnote)
I made a new discovery that might help.  The project structure is a bit bigger than I first said, and the extra complexity is key to the issue.
Project structure:
Plywood
    |__Plywood.Client
Plywood.Client references Plywood.  Plywood and Plywood.Client both use CsWin32 generators.  I have added the NativeMethods.json modifier to  both projects

Right now, if both projects have populated NativeMethods.txt files, then Plywood will compile but Plywood.Client will fail.  If I remove the NativeMethod.txt from Plywood, then Plywood.Client will compile.  So this BOOL error is happening when there is a parent project that also uses CsWin32
Here is the NativeMethods.cs file from Plywood.Client if both generators are enabled:
Plywood.Client.NativeMethods_Broken.txt
Here is the Here is the NativeMethods.cs file from Plywood.Client if the parent generator is disabled:
Plywood.Client.NativeMethods.txt
When the parent generator is active, the child does not generate its own BOOL.  This is likely because BOOL is already generated by the parent, but its marked as internal so the child cant access it.  Maybe the generators run in a shared context even though they target different projects?  Or possibly related to work done for #3?

Now for a bonus bug: This is related to viewing the generator output code in VS.  In the screenshot, I show two projects with generated NativeMethods.cs files.  Both of those file icons open the same physical file.  So I cannot see the actual contents of Plywood.NativeMethods.cs, it just opens the Plywood.Client.NativeMethods.cs file
Also, if both projects have active generators, then F12 will also show the same file when used from either project

The first tip that you showed about finding the generated files does work for me, thanks (but it has a bug, see footnote)
I made a new discovery that might help.  The project structure is a bit bigger than I first said, and the extra complexity is key to the issue.
Project structure:
Plywood
    |__Plywood.Client
Plywood.Client references Plywood.  Plywood and Plywood.Client both use CsWin32 generators.  I have added the NativeMethods.json modifier to  both projects

Right now, if both projects have populated NativeMethods.txt files, then Plywood will compile but Plywood.Client will fail.  If I remove the NativeMethod.txt from Plywood, then Plywood.Client will compile.  So this BOOL error is happening when there is a parent project that also uses CsWin32
Here is the NativeMethods.cs file from Plywood.Client if both generators are enabled:
Plywood.Client.NativeMethods_Broken.txt
Here is the Here is the NativeMethods.cs file from Plywood.Client if the parent generator is disabled:
Plywood.Client.NativeMethods.txt
When the parent generator is active, the child does not generate its own BOOL.  This is likely because BOOL is already generated by the parent, but its marked as internal so the child cant access it.  Maybe the generators run in a shared context even though they target different projects?  Or possibly related to work done for #3?

Now for a bonus bug: This is related to viewing the generator output code in VS.  In the screenshot, I show two projects with generated NativeMethods.cs files.  Both of those file icons open the same physical file.  So I cannot see the actual contents of Plywood.NativeMethods.cs, it just opens the Plywood.Client.NativeMethods.cs file
Also, if both projects have active generators, then F12 will also show the same file when used from either project

Thanks. I'll see if I can repro the BOOL bug and get a fix.
For the bonus bug, can you please file that against the dotnet/roslyn repo, as they own that?

Thanks. I'll see if I can repro the BOOL bug and get a fix.
For the bonus bug, can you please file that against the dotnet/roslyn repo, as they own that?

Yup, this is biting me too.

Looks like the "bonus bug" has a fix in the pipeline: https://github.com/dotnet/roslyn/issues/50714


add typescript example for basic mutation

Looking at the documentation I couldn't find an example of using useMutation hook. so I have put together a simple example showing the usage of the hook in a React/TypeScript app

we have this, which does cover useMutation, typescript and optimistic updates: https://react-query.tanstack.com/examples/optimistic-updates-typescript

To keep the maintenance burden as low as possible, I think we should instead direct our efforts to improving the existing optimistic updates typescript example (if needed at all).


A Stored XSS vulnerability exists

When the administrator logs in

Click Click on the Site_settings module to use XSS POC and save

Then visit Preview Page

I have submitted CSRF vulnerabilities before and think it can be exploited in combination with CSRF.
The following is the cause of the vulnerability:
Source
\ignitedcms\application\controllers\admin\Site_settings.php

Thank you for highlighting this issue. The intention was to allow the site administrator to write their own custom javascript as necessary. But there is a bigger issue with CSRF that has been highlighted. I haven't had the time yet to address this as the project is still old sitting on older PHP versions, but I may look at it in the future by enabling the CSRF token to be set to true in the config file.

got it
------------------ Original ------------------
From: ignitedcms @.>
Date: Thu,Mar 3,2022 10:33 PM
To: ignitedcms/ignitedcms @.>
Cc: Silence D @.>, Author @.>
Subject: Re: [ignitedcms/ignitedcms] A Stored XSS vulnerability exists (Issue#8)

Closing as this is intended for admin use.

Just letting you guys know, an enhanced beta version is going to be released sometime next month, still a WIP
https://www.ignitedcms.com/


feature request: device support

Add support for HUAWEI mobile router B311B-853

HUAWEI Mobile Router is now supported after v0.1.0


[UNR-3626] 4.25 Support

Description
4.25 things.
Associated engine PR: https://github.com/improbableio/UnrealEngine/pull/580.
Release note
Ah, I forgot.
Tests
A lot of testing, but will need a lot more.
Documentation
Will need the support matrix updated.
Primary reviewers
@jessicafalk @m-samiec @improbable-valentyn

A lot of testing, but will need a lot more.

Have we updated the example project or test gyms? Would be good to give the NFRs a run asap.

TestGyms and Example project did not need updating, only some deprecation warning fixes that Valentyn already merged in, I think, and they are #if-guarded.


Bluetooth Controller - MAC Address Appears as AA:AA:AA:AA:AA:AA

A user is reporting that on a CM4, Bluetooth is generating a MAC Address of AA:AA:AA:AA:AA:AA (and sometimes does not come up at all).
User is on balenaOS 2.85.2+rev3.  User reports that on Raspbian, it works properly.
root@artsy-sea:~# bluetoothctl
Agent registered
[CHG] Controller AA:AA:AA:AA:AA:AA Pairable: yes
AdvertisementMonitor path registered
[bluetooth]#

@markcorbinuk tested and got similar results on his Pi, and AlexG asked for an Issue.  Mark also linked to these threads:
https://forums.raspberrypi.com/viewtopic.php?t=304000&start=25
https://forums.raspberrypi.com/viewtopic.php?t=272448

Hi, I'm the user that reported the issue. I've been digging into it today, posting some notes below in case they are useful.
With regards to the device coming up all the time, if I set dtparam=krnbt=on- Bluetooth comes up consistently and without fail.
With regards to the mac address, I can see that I can set it manually using hcitool:
# hcitool cmd 0x3f 0x001 0x01 0x02 0x03 0x06 0x05 0x06
< HCI Command: ogf 0x3f, ocf 0x0001, plen 6
  01 02 03 06 05 06
> HCI Event: 0x0e plen 4
  01 01 FC 00

This led me to dig into how RPi sets it when using krnbt=on.
They use a script /usr/bin/bthelper but it doesn't seem to work for me when I run it manually, it just claims that the address it already set.
/usr/bin/bthelper hci0
Raspberry Pi BDADDR already set

Maybe there's something here?

Indeed @mhazley , looks like the switch back to '\s43:4[35]:' in upstream pi-bluetooth causes this, the old check for \s(B8:27:EB:|DC:A6:32:) works fine on the cm4-ioboard. I've added a patch in the PR above for this.

@mhazley I'll wait a bit before merging this, so the upstream maintainer of pi-bluetooth can confirm if this is the right approach

Hi @mhazley , a new image v2.88.4 is now available for the CM4 in the dashboard. We haven't observed any issue with BT coming up, but if you need to set krnbt you can do that from the dashboard, in the device configuration page.


Convert string to char array in function parameter

Hi @mingodad , this change fixes compile error for qscm.c:
https://github.com/tyfkda/xcc/issues/92#issuecomment-1233114026

Yes it seems to fix https://github.com/tyfkda/xcc/issues/92#issuecomment-1233114026 , thank you again !


Mac Unopt is 6% flaky on step test: Host Tests for host_debug_unopt

There are 6 post-submit flakes in recent 100 runs.
Example flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac Unopt/6173/overview
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac Unopt/6165/overview
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac Unopt/6161/overview
Error logs:
[356/360] HashCombineTest.CanHash (41 ms)
[357/360] StringRangeSanitizationTest.HandlesInvalidRanges (43 ms)
[358/360] StringRangeSanitizationTest.HandlesInvalidRanges (38 ms)
[359/360] AutoResetWaitableEventTest.MultipleWaiters (4233 ms)
[360/360] AutoResetWaitableEventTest.MultipleWaiters (4400 ms)
FAILED TESTS (1/360):
     100 ms: /opt/s/w/ir/cache/builder/src/out/host_debug_unopt/fml_unittests TaskRunnerCheckerTests.FailsTheCheckIfOnDifferentTaskRunner (try #1)
SUMMARY:
  /opt/s/w/ir/cache/builder/src/out/host_debug_unopt/fml_unittests TaskRunnerCheckerTests.FailsTheCheckIfOnDifferentTaskRunner passed 1 / 2 times.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Failed Command:

python3 /opt/s/w/ir/cache/builder/src/third_party/gtest-parallel/gtest-parallel /opt/s/w/ir/cache/builder/src/out/host_debug_unopt/fml_unittests --gtest_filter=-*TimeSensitiveTest* --repeat=2

Exit Code: 245

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Traceback (most recent call last):
  File "/opt/s/w/ir/cache/builder/src/flutter/testing/run_tests.py", line 1071, in <module>
    sys.exit(main())
  File "/opt/s/w/ir/cache/builder/src/flutter/testing/run_tests.py", line 1021, in main
    build_dir, engine_filter, args.coverage, args.engine_capture_core_dump
  File "/opt/s/w/ir/cache/builder/src/flutter/testing/run_tests.py", line 397, in RunCCTests
    gtest=True
  File "/opt/s/w/ir/cache/builder/src/flutter/testing/run_tests.py", line 225, in RunEngineExecutable
    env=env
  File "/opt/s/w/ir/cache/builder/src/flutter/testing/run_tests.py", line 80, in RunCmd
    (command_string, process.returncode)
Exception: Command "python3 /opt/s/w/ir/cache/builder/src/third_party/gtest-parallel/gtest-parallel /opt/s/w/ir/cache/builder/src/out/host_debug_unopt/fml_unittests --gtest_filter=-*TimeSensitiveTest* --repeat=2" exited with code 245.

The three linked flakes are not the same. The second one is FlutterMenuPluginTest.TestSetMenu which is a known bad test. The first one, we will need to investigate. It is in the Darwin message loop termination. The test has also been stable forever. So not sure if this could be because of platform specific issues.
@keyonghan How are these failures aggregated? We may need to go chase down the other other unrelated failure in FlutterMenuPluginTest.TestSetMenu.

Do we have separate issues for the separate flakes listed above? Which one is this issue specifically about?

Here is the bug: https://github.com/flutter/flutter/issues/106747. I will try cycling back on this issue.

For flaky builds, here are the list for recent 100 commits (since 07/07/2022)
6394
6408
6428
6432

Two of these are SIGSEGV crashers in TaskRunnerCheckerTests.FailsTheCheckIfOnDifferentTaskRunner in the call to MessageLoopDarwin::Terminate() from here:
https://github.com/flutter/engine/blob/2962099077b36704071802ea5595f4a016a1c214/fml/memory/task_runner_checker_unittest.cc#L40
Looking at the git blame, it looks like @cyanglaz may be familiar with this area. Any idea why this test might be crashing?

For the one issue where we have a backtrace, we can try to scribble to TLS slot on terminate to see if the look is getting terminated by someone else. But other than that, the test is extremely simple so I am not able to tell how this could fail. We will need to reproduce the error locally.

The linked PR should fix this.


The component is unstable

It sometimes crops out the submit button, it loads only half the picture sometimes, and recently just I run the streamlit file and the label studio front end didn't even load (the screen was blank)

hey @Usif-A
I am working with the package too, would love to collaborate with you !

dear Fathimanourin
oh sorry I can't help. I have too many commitments right now. I worked
around the issue by having my code open label studio in another tab instead
of using the package.
another solution I looked into was having an iframe in the streamlit app.
have a great day,
Youssif Ahmed
On Wed, 16 Jun 2021 at 16:11, fathimanourin @.***>
wrote:

hey @Usif-A https://github.com/Usif-A
I am working with the package too, would love to collaborate with you !
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/deneland/streamlit-labelstudio/issues/4#issuecomment-862414668,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ATXQ7WCSFY7IBENJECLHHW3TTCWIHANCNFSM46TKO46A
.

oh sure, I'll try these, thanks anyways !

@deneland @w0ut0 Could you add the above enhancement and update the pip package please?

Did this update ever happen? I want to run two instances of this component so need to pass the key argument.


main_v2 - Refactor Baseboard layer

Summary:

Refactor Baseboard layer code following main_v2

Dependency: #214 & #224


Consider moving to GitHub Actrion

As title, the Travis CI dot org will be deprecated soon and the Travis CI dot com is not friendly for the open source project.
And I think we can consider moving to the GitHub Action.

Agree. Could you make a PR ?

Thanks for your reply. I will do that.


Responsive Tweaks

[x] Home Screen

The Title and image are in a row format and must be stacked.

The Heading section is illegible. When completing the other ticket, the legibility should be fixed here with that, but it is still not setup for mobile. The size of the 'paragraph' should be max 300px. The title heading seems to be centered, but Business Hours and Location does not (that could do with 'business hours' being longer so maybe do a space-around so it focuses on the edges. Unsure, but play with it. Worse case scenario could be be having them stacked when it is on mobile view).
On the contact info section:

Capitalized contact info

[x] Business Details & FAQ

There is a gap between header and 'business details'. Please make that much smaller in this mobile screen.
Please capitalize all headers
The business details section seems off-centered.

Based on the example website: https://www.shampoochgahanna.com/about it seems we are missing some styles. Can you please make sure you imitate the style. I know there is no disclaimers and we are adding a FAQ at the bottom, but the font seems different, horizional lines are missing, and I am unsure how you are going to add business hours location and phone information. Use fake data so we can see what it looks like.
The FAQ questions in mobile form is okay until we click on the questions. That width is too small.

It might be best to change them to modals when clicked to avoid this issue || The width should take a wider space on the mobile screen and not the width of the question itself.
Again the navbar here is not mobile friendly.

[x] Photo Gallery

The navbar (check all pages for this)
The modal is not mobile friendly

[x] Events

Navbar only
Will see what client says about this.

went through all four tasks and completed them, the gallery modal was a little harder to figure out due to the picture you provided not happening on the dev server but I think I figured it out to a width that needs to be unset in mobile view and the navbar was I think was part of it as well


[UI] Part of pop-up content is hidden in Meshery Dashboard

Current Behavior
When not in full screen, part of the text on pop-up is hidden underneath the top bar. To reproduce: login to meshery playground on your browser via Dashboard | Meshery
Desired Behavior
User should be able to clearly view the message on the pop-up whether in full screen or not.
Screenshots/Logs
This is how it looks when not in full screen

How it should look (whether in full screen or not)

Environment

Browser: Microsoft Edge
Host OS: Windows

Contributor Guides and Handbook

🎨 Wireframes and designs for Meshery UI in Figma (open invite)
🖥 Contributing to Meshery UI
🙋🏾🙋🏼 Questions: Discussion Forum and Community Slack

@nwanduka I would like to work on this issue

@nwanduka Can i work on this?

@prabhjeevnijjar  its yours

@nwanduka
I have fixed the issue mentioned above locally.
I observed an inconsistency in UI:
When I get this popup as unregistered user.
The text says "Get early access to MeshMap!"
The close modal button shifts to the left side along with the text.
This is a small bug should I fix it right now or do I create a new bug report?
PFA SS

Thanks for working on this issue @prabhjeevnijjar. Nice job 👍🏽. And the UI issue you raised seems valid. Let's discuss it in the Websites call on Monday. Check out the schedule and add to your calendar if you haven't already done so.
Then add this issue as an agenda item for the call.

@nwanduka
I have fixed the issue mentioned above locally.
I observed an inconsistency in UI:
When I get this popup as unregistered user.
The text says "Get early access to MeshMap!"
The close modal button shifts to the left side along with the text.
This is a small bug should I fix it right now or do I create a new bug report?
PFA SS

Did you raise a PR on this?

@nwanduka
I have fixed the issue mentioned above locally.
I observed an inconsistency in UI:
When I get this popup as unregistered user.
The text says "Get early access to MeshMap!"
The close modal button shifts to the left side along with the text.
This is a small bug should I fix it right now or do I create a new bug report?
PFA SS

Did you raise a PR on this?

Will raise PR tommorow.

Created PR @nwanduka

Awesome. Please jump on the Websites call on Monday so you can get feedback on your PR, and also share the UI issue you raised earlier. Let's get this merged as soon as possible.


added versioned docs for example page

Copied example.md to versioned docs 0.7 so that we can have different versions, similar to other pages.
Added appkind: deployment in chaosengine.yaml to make it compatible with 0.7
Litmus operator yaml file link is changed to include version number (similar to getting started page)
Modified link to getting started page to use relative path instead of absolute url (as specified in https://docusaurus.io/docs/en/navigation

No other changes in md file
fixes litmuschaos/litmus#905
Signed-off-by: Jayadeep KM<EMAIL_ADDRESS>

@ksatchit It seems like appkind is only required in next release and not in 0.7. I have tested with 0.7 operator without appkind and it worked. Can you please confirm this?

@kmjayadeep The example file is not required in 0.7 can you just update the example page link in getstarted page

@kmjayadeep The example file is not required in 0.7 can you just update the example page link in getstarted page

Wouldn't that be a problem when we release new docs for 0.8?
I think it is better to keep a uniform structure for all files

@kmjayadeep The example file is not required in 0.7 can you just update the example page link in getstarted page

Wouldn't that be a problem when we release new docs for 0.8?
I think it is better to keep a uniform structure for all file

@kmjayadeep The example file is not required in 0.7 can you just update the example page link in getstarted page

Wouldn't that be a problem when we release new docs for 0.8?
I think it is better to keep a uniform structure for all files

The file adding in versioned docs is done by docusaurus, it's not the manual step, File is added to the released version only when the file contains the changes from previous release.
Closing this PR as this is not required.

@chandankumar4 Thanks for the explanation.
One additional note: The litmus operator yaml file in the doc is still pointing to latest version (post 0.7) instead of 0.7 itself. So, https://github.com/litmuschaos/litmus/issues/905 is still open!

Yes, Thanks for noticing, Can you raise a fix for that?


Visual Studio 2017 Crashes with Default ASP.NET Core Web API Project

Steps to Reproduce
Create new ASP.NET Core Web Application (.NET Core)
Start Debugging (F5)
Windows 10, Version	10.0.10240 Build 10240
Microsoft Visual Studio Enterprise 2017
Version 15.0.26228.9 D15RTWSVC
Microsoft .NET Framework
Version 4.6.01055

Delete all the Nuget package cache (%UserProfile%.nuget\packages) and then rebuild the project.  Still crash when debug.

Solved.
It turns out that it is cause by the VPN software Astrill, after uninstalling Astrill’s LSP.
Things are back to normal again.


HnwsGraph creates disconnected components

Description
I work for Amazon Retail Product search and we are using Lucene KNN for
semantic search of products.
We recently noticed that the hnsw graphs generated are not always strongly
connected and in worst case scenario some products may be undiscoverable.
Connectedness of Hierarchical graph can be complicated, so below I am
mentioning my experiment details.
Experiment:
I took the Lucene indexes from our production servers and for each segment
(hnsw graph) I did following test.
At each level graph I took the same entry point, the entry point of HNSW
graph, checked how many nodes are reachable from this entrypoint. Note that
connectedness at each level was checked independently of other levels.
Sample code attached. My observations are as below.

Observation :

Of all the graphs across all the segments, across 100s of indexes that I
considered, one graph for each "level" of HNSW, almost 18% of the graphs
had some disconnectedness.
Disconnectedness was observed at all the levels of HNSW graphs. I saw we have
at most 3 to 4 levels in HNSW graphs.
percentage disconnectedness ranged from small fractions 0.000386% (1
disconnected out of 259342)  to 3.7% (eg. 87 disconnected out of 2308).
In some extreme case the entry-point(EP) in zeroth level graph was disconnected
from rest of the graph making the %age disconnected as high as 99.9% (only 65
reachable nodes from EP out of 252275). But this does not necessarily mean
that the 99.9% of nodes were not discoverable, it just means that if
unluckily we end up on EP in the 0th level graph for a query, there can at
max be 65 nodes that can be reached. But had we diverted our path from EP
to some other node in the upper level graphs then may be more nodes be
discoverable via that node.

Reproduciability
Although it is fairly easy to be reproduced with Amazon's internal data, but I found it really hard to reproduce it with random vectors. I will attached a simple main class of my test and the parameters that I tried. I only got some disconnectedness as listed in below table. For higher number of vectors, the execution becomes painfully slow because of hnsw graph creation. I will continue to work to find some  parameters for reproducibilty and also explore open source datasets for the same.
DIS-L0 means: %age of disconnected nodes in graph at level-0 of HNSW graph.

randomness seed
dimension
no of doc vectors
Max-conn
Beam-width
dis-l0
dis-l1
dis-l2
dis-l3
dis-l4

-219343918
256
100_000
16
100
0
0.0158 (1 node disconnected)
0
0
0

-245556271
256
100_000
16
100
0
0.0158 (1 node disconnected)
0
0
0

-766887769
256
1_000_000
16
100
0.0003
0.04166
0
0
0

No direct solution comes to my mind but as per my discussion in this email thread. I will look into the part where some connections are removed to maintain the Max-Conn property (diversity check etc.).
Version and environment details
Lucene version : 9.7

I was able to run tests on wiki dataset using the luceneutils package. The results shows that even with a single segment index and no updates, around 1% nodes gets disconnected for about 1M vectors. It would be great if someone else can have a look at the CheckHNSWConnectedness for correctness.
This may or may not be an issue for different system given that this is 'approximate' nearest neighbour search. But in my opinion it is worth exploring more and if possible some fix.
Next I will try to reproduce with multiple segments and try to find the cause and fix for it.

I'm thinking of something like this: first build a fully-connected graph (no link removal, no enforcement of maxconns). Then apply a global pruning algorithm of some sort that attempts to preserve connectivity. For example, you could iterate over all the vectors in the graph, searching for each one using a consistent entry point (node 0 in the graph say, since it will probably be maximally-connected), and "coloring" the visited nodes and edges during the searches. Stop once every node has been visited. Some of the edges will not have been visited, and we could prune those from nodes with large numbers of edges.

Thanks @msokolov : These are really good suggestions. I will try to incorporate these ideas in solutions. I think in the end there can be multiple ways to allow more connectivity.
I was also worried about how the connections are made and pruned. I checked these lines in our algorithm. Here we connect a node to the neighbour list only if is not closer to any of the current neighbour. Think of a situaion where for some nodes (say a) if the first neighbour connected (say b) has a good score but the rest of the candidate nodes have higher score with b than a. Then the rest of the candidates will NOT get connected to a AT ALL. Later if because of some other connection of b, say the non diverse connection of b -> a is removed then the a will get completely disconnected from graph. Basically we should have more nodes connected in the begining itself so that if some nodes are removed we do not run danger of disconnected graphs.
To check this I added lot of code to collect the events (like add, connect, disconnect) and was able to reproduce it with just 10K graph size itself with max-conn 16. Example below. But the same is not reproduced with max-conn 32 or max-conn 64. Although, increasing max-conn in this case helped but the basic algorithm for connections does not seem very good to me.
To fix this I checked that the original paper proposes the concept of keepPrunedConnections. I implemented that in my local and ran tests. But something is not right, I will keep checking if I did something wrong in the implementation etc. Will submit "draft" PR soon for others comments.
I think there is another issue in current implementation. Look at this code where we finally connected edge from b ---> a and now if b has more connections then max-conn then we are removing one of the non diverse connection of b Say we remove the b ---> c. So we are removing this half of the undirected connection but I don't think we are removing the other half c ---> b anywhere. This will leave inconsistent Graph. I tried fixing this as well, but I am getting some errors. I will try to fix and submit the draft PR.
Note that adding the concept of keepPrunedConnections will not just impact these disconnected nodes but in general the connections of all the nodes will increase and will become close to 'max-conn'. I think this is a good property without much additional cost at the time of graph creation. But the search time may increase a bit since there will be more connections to explore for the same max-conn. But I think this will also increase the Recall with ExactKNN. I will run the performance tests once code complete to find those numbers.

Real example of disconnectedness
In one of the 10K vector runs, I found 3 nodes were disconnected. namely 297, 298, 5601
I checked that 297 was initially connected to 293. After that it was compared to a lot of candidates but was not connected to any because their score with 293 was higher than their score with 297. For example here 297 was compared with 43 and 197 but was NOT connected to either because their score with 293 was higher. In this case 297 was left with just one connection (with 293). Later on the connection of 293 ---> 297 also got deleted because of diversity checked triggered for other nodes connection with 293.

level=0,node=297,type=CONNECT,source=297,dest=293,otherdata=score=0.9578
level=0,node=297,type=COMPARE,source=297,dest=43,otherdata=score=0.9541
level=0,node=297,type=COMPARE,source=43,dest=293,otherdata=neighborSimilarity=0.9816,  score=0.9541
level=0,node=297,type=COMPARE,source=297,dest=197,otherdata=score=0.9539
level=0,node=297,type=COMPARE,source=197,dest=293,otherdata=neighborSimilarity=0.9868,  score=0.9539
......
level=0,node=298,type=CONNECT,source=298,dest=297,otherdata=score=0.9854
....

Final connections were, full 2-way connection 297 <---> 298 and one sided connection of 297---> 293 (left over connection).
But there was no path to 297 and 298 from any other node as the other way connection from 293 ---> 297 got deleted.

So we are removing this half of the undirected connection but I don't think we are removing the other half c ---> b anywhere. This will leave inconsistent Graph

This is by design; the resultant graph is not expected to be symmetric (ie undirected, bidirectional). It is allowed to have a->b with no corresponding b->a

Hi @msokolov ! Thanks for clarifying. But I think it can help to remove the 'less important' edge from both sides, since it frees up a degree of "other" node to accept a new connection without indulging into diversity check. Most of the time the problem that I saw with diversity check is that, the most recently added connection to the node itself turns out to be non-diverse and gets removed immediately. Because of this the new incoming node does not even get enough connections to begin with in the graph. Even in the case where I removed the diversity check while connecting incoming node (a) to neighbours (say b,c) and allow full max-conn number of connections, most of the connections from neighbours (b->a) to the nodes are removed because of diversity check, leaving mostly half connections from (a->b). Now when a new node say x tries to connect to a even that is kicked off by a because of diversity check. So keeping half connections sometimes acts against the connectedness nature of hnsw.
I have come up with a new huristic which seems to work well against disconnectedness, but I am yet to optimize it and conduct performance test on it. Will still update the algorithm here and post numbers later.

I was able to conduct some perf-test as well. I would like to propose following changes
With example of adding node a to 'b, c, d'

node
neighbours

b
c, d, e

c
b, d

e
e, x, y, z

Proposed heuristic

At the time of adding a node always add all (upto maxConn) edges to highest scoring neighbours (disregarding the diversity check) eg. adding edges a-->b, a-->c etc. Currently we check for diverse connections at this time also.
At the time of adding reverse edges from neighbours to the node, if the given neighbour has more than maxConn number of connections then use new diversity check formula to calculate the edge that can be removed. For example at the time of adding edge b --> a if we see that b has more than maxConn connections then apply new hiuristic and try to remove one edge.
Check for diversity only between the common neighbours of candidate node and the node. Example, if b has more than maxConn connections then we iterate over b's neighbours (which now includes 'a' as well) considering each one as a possible candidate for disconnection. When deciding if we should disconnect an edge say b-->c or not. We will consider the only the common neighbours of c and b for diversity check, so we will compare the score(c,b) with only score(c,d) and not with score(c,e) as only d is the common neighbour of c and b. If we find a non diverse node we return that, otherwise
we return a node which has highest number of common neighbours with b, otherwise
we return a node with maximum number of edges (eg. e in above case).
If none of the above condition exists, then we return -1, meaning we should not remove any connection.
If we receive a valid node for removal, we remove that and the other half of the connection i.e. we remove both bi-directional connections. so if c is returned as non-diverse node, then we remove both b---> c and c ---> b connections.

Performance number:

Branch
recall
avgCpuTime
numDocs
reindexTimeMsec
fanout
maxConn
beamWidth
totalVisited
selectivity
prefilter

Baseline
0.451
17.78
1000000
370916
0
16
100
10
1.00
post-filter

Candidate
0.599 (33% increase)
21.63 (22% increase)
1000000
624797 (68% increase)
0
16
100
10
1.00
post-filter

Since now we are increasing the number of connections per node, we also see a good increase in the recall for same max-conn. For the same reason we do expect the avgCpuTime to increase at query time. I can work a little bit on improving the indexing time as the my current implementation is very naive and finds common nodes by two for loops and some other places recalculates the scores where not needed. But overall the indexing time will increase for sure.
Another data that I will post next is the number of (histogram) number of edges each node has. Expect it to increase from lower number to max-conn on average.
I will upload my (rough) draft PR now.
How I reached this heuristic:
Before coming up with this heuristic I tried quite some simpler ones, but in those cases I was either getting much much higher disconnections than the mainline or if very less and even just 1 disconnection, but some disconnection was happening. I am open for removing or adding any other condition which is considered for edge removal.
Initially, I tried with just (1) and left the rest of the algorithm same as in mainline. This "increased" the disconnections by large factor instead of decreasing it. The issue was that say 0 to 32 nodes are added with star kind of formation, now if 33rd node comes in, it will form connections with 1-32 nodes and each may find 0 as the non diverse node, so each one removes 0 leaving it disconnected from the graph. Other than this extreme case there were others too.
That is where (3) came in handy where we don't check diverseness of an edge unless we are sure that there is an alternate path to the disconnecting node via a neighbouring node. This coupled with (7) reduced the disconnectedness from few thousand nodes to just couple of nodes at level-1 and only occassional disconnected node at level-0.
Then to achieve no disconnection at all, I initially tried (1, 2, 3) with (6) to remove even slightest chance of creating disconnectedness, but with this we end up not removing any connection a large number of times and the number of connections per node increased from max-conn of 32 at level-0 to upwards of 250 for more than 5% of nodes.
To control the number of connections I added (4) and (5) mostly to remove at least some connection without affecting the graph geometry by much.
One might wonder if (7) is actually required, so I will run some test without (7). But I think it plays an important by freeing up some of the edges and allowing those nodes to accept un-conditional connections from other incoming node.

@benwtrent : I have added draft PR. The code is not at all optimized right now for performance and I am hoping to fix some obvious stuff and will post perf results here.

have you explored those to see how they can be modified or added here?

No, I haven't checked those. Thanks for pointing it out. I will check and revert.

@nitirajrathore @msokolov I had an idea around this, and it will cost an extra 4bytes per node on each layer its a member (maybe we only need this on the bottom layer...)
What if we added an "incoming connection" count for every node? Then when disconnecting non-diverse neighbors, we enforce that we can never disconnect the last connection to a node.

Is the problem primarily to do with single isolated nodes or do we also see disconnected subgraphs containing multiple nodes? I think this idea would prevent the isolated nodes, but not fix the other case.

@msokolov good point.
It seems to me we would only fully disconnect a sub-graph only if its very clustered. Is there a way to detect this in the diversity selection?
One other thing I just found that confuses me: https://github.com/apache/lucene/pull/12235
This slightly changed the diversity connection in a way to to attempt to improve performance. The key issue is here:
https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphBuilder.java#L386-L407
If we have "checked the indices", we always just remove the furthest one, which doesn't seem correct to me. We should check for diversity starting at the furthest one, not always remove it?
@nitirajrathore for your connection checker, are you testing on a Lucene > 9.7.0?

@nitirajrathore could you add something to KnnGraphTester that is a test for connectedness?
If you can't get to it, I can try. It would be good to have it along side all our other performance testing as another metric to record and optimize for.

What if we added an "incoming connection" count for every node?

&

I think this idea would prevent the isolated nodes, but not fix the other case.

I will write a small code to check the type of disconnectedness. Clustered vs single

but then we (I think?) decided that wasn't the approach advocated by the academic papers we were emulating, and implemented the diversity check to every added neighbor

Paper proposes 3 different heuristics.

check diversity before adding any node. -- Standard in paper.  (getNeighborsByHeuristic2() in the code)
check diversity before adding any node, but if the nodes added are lesser than the max-conn then add atleast max-conn nodes. keepPrunedConnections in paper (getNeighborsByHeuristic1() in the code)
add all the neighbours of neighbours in the candidate list before finding the diverse nodes as in (1) and optionally apply keepPrunedConnections as well. Called extendCandidates in paper. ( Not sure but, I guess this is the getNeighborsByHeuristic3() in the code. but need to understand it a bit.)

Our current implementation is like (1) with some optimizations for performance. My proposed approach is similar to (2) with more strict conditions by checking diversity only for nodes which are surely connected via their neighbours.
I earlier tried to implement (2) just by keeping the pruned connections till max-conn. As I mentioned, this increased the disconnected nodes by a factor instead of decreasing it. The reason I understood was that now there was more competition between nodes as every node now had max-conn number of connections all the time, triggering the findWorstNonDiverse() to be called every time for every neighbour, and most of the time the just now newly formed edge will be non-diverse and will get dropped. In this I think there might be something wrong in the implementation of checked and unchecked checking that we are doing in findWorstNonDiverse(). I am not able to get my head around that checked-unchecked code completely, but I think it give green signal to all the current neighbours as diverse incorrectly and checking just the 'unchecked' connections triggers the diversity check only for the newly added edge which it founds to be non-diverse and removes it.
I think there is huge difference between the number of connections between heuristic (1) and (2), and by inference between the current implementation and my proposed implementation, so I will post numbers around that. But I don't think that is necessarily bad, if the number of avg connections are increasing it is also increasing the recall with ExactKNN, meaning that we can get same recall for a smaller max-conn value now. I will also try to get some number around same recall with current implementation with lower max-conn and then check the search latency. This may also reduce the indexing time, lets see by how much.

@nitirajrathore could you add something to KnnGraphTester that is a test for connectedness?

I have separate scripts for those, I will post the links here and will try to get the code in the luceneutil, although compilation is a issue in that package ( which we are trying to fix separately ). We can think of adding that as optional check in the KnnGraphTester as well.

meaning that we can get same recall for a smaller max-conn value now.

I ran some tests with with max-conn 16 and max-conn = 8 and it seems like with my proposal, even max-conn=8 is better as compared to max-conn=16 of mainline. I will also add more stats.
I diverged from main branch at commit f64bb19697708bfd91e05ff4314976c991f60cbc (15 Oct). I haven't merged back from there. I think this is Lucene > 9.7.0. But will confirm.

main : Commit ID : f64bb19697708bfd91e05ff4314976c991f60cbc with max-conn = 16

recall
avgCpuTime
numDocs
fanout
maxConn
beamWidth
totalVisited
reindexTimeMsec
selectivity
prefilter

0.451
17.22
1000000
0
16
100
10
406166
1.00
post-filter

candidate with max conn = 16

recall
avgCpuTime
numDocs
fanout
maxConn
beamWidth
totalVisited
reindexTimeMsec
selectivity
prefilter

0.595 (+32%)
24.19 (+40%)
1000000
0
16
100
10
581090 (+43%)
1.00
post-filter

candidate with max conn = 8

recall
avgCpuTime
numDocs
fanout
maxConn
beamWidth
totalVisited
reindexTimeMsec
selectivity
prefilter

0.465 (+3%)
16.35 (- 5%)
1000000
0
8
100
10
325321 (-20%)
1.00
post-filter

Interesting fact: simple implementation of using 2 for loops to find the common neighbours works better than using HashSet or IntIntHashMap(). As I think the major contribution to indexing time is because of increased number of connections.
But as shown above, the indexing time decreases drastically by decreasing max-conn, while still maintaining or slightly improving the recall and search avgCpuTIme.
I will update with more scripts + info/stats and some code improvements next.
Also, I am thinking I should do a 1-1 comparison of the 3 heuristics as mentioned in the paper, with the level of disconnecteness in each approach.

@nitirajrathore very interesting findings.
This makes me wonder if the heuristic should take a middle ground and instead of keeping all pruned connections, keep half.
I am interested in seeing the connective-ness between the heuristics.

yes, this is a promising avenue to explore! One note of caution: we should avoid drawing strong inferences from a single dataset. I'm especially wary of GloVe because I've noticed it seems to have poor numerical properties. We especially should not be testing with random vectors. Ideally we would try several datasets, but if I had to pick one I'd recommend the minilm (384-dim) vectors we computed from wikipedia, or some internal Amazon dataset, or I know Elastic folks have been testing with a Cohere dataset? You can download the minilm data from sftp @home.apache.org; cd /home/sokolov/public_html if you have an apache login. You can also regenerate using infer_vectors.py in luceneutil, but it takes a little while

I +1 this issue and would like to try the gains in Amazon codebase. I think with lowered max-conn this can give some latency gains.
I also feel that there is a use case for indexes with less number of updates where the disconnected nodes might never be connected again. Looking forward to the results from the latest benchmark run.

OK, I added https://github.com/mikemccand/luceneutil/pull/253
Doing some local benchmarking. It seems that the more merges occur, the worse we can get.
Sometimes I get good graphs like this:
Leaf 5 has 5 layers
Leaf 5 has 137196 documents
Graph level=4 size=2, Fanout min=1, mean=1.00, max=1
%   0  10  20  30  40  50  60  70  80  90 100
    0   1   1   1   1   1   1   1   1   1   1   1   1   1   1
Graph level=3 size=34, Fanout min=4, mean=7.82, max=12
%   0  10  20  30  40  50  60  70  80  90 100
    0   5   6   7   7   8   8   9   9  10  12
Graph level=2 size=545, Fanout min=13, mean=15.98, max=16
%   0  10  20  30  40  50  60  70  80  90 100
    0  16  16  16  16  16  16  16  16  16  16
Graph level=1 size=8693, Fanout min=16, mean=16.00, max=16
%   0  10  20  30  40  50  60  70  80  90 100
    0  16  16  16  16  16  16  16  16  16  16
Graph level=0 size=137196, Fanout min=3, mean=24.57, max=32
%   0  10  20  30  40  50  60  70  80  90 100
    0  14  17  19  22  26  31  32  32  32  32
Graph level=4 size=2, connectedness=1.00
Graph level=3 size=34, connectedness=1.00
Graph level=2 size=545, connectedness=1.00
Graph level=1 size=8693, connectedness=1.00
Graph level=0 size=137196, connectedness=1.00

Other times I get graphs that are pretty abysmal:
Leaf 7 has 4 layers
Leaf 7 has 39628 documents
Graph level=3 size=8, Fanout min=1, mean=1.75, max=7
%   0  10  20  30  40  50  60  70  80  90 100
    0   1   1   1   1   1   1   1   1   1   7   7
Graph level=2 size=153, Fanout min=1, mean=1.10, max=16
%   0  10  20  30  40  50  60  70  80  90 100
    0   1   1   1   1   1   1   1   1   1  16
Graph level=1 size=2503, Fanout min=1, mean=1.01, max=16
%   0  10  20  30  40  50  60  70  80  90 100
    0   1   1   1   1   1   1   1   1   1  16
Graph level=0 size=39628, Fanout min=1, mean=1.00, max=32
%   0  10  20  30  40  50  60  70  80  90 100
    0   1   1   1   1   1   1   1   1   1  32
Graph level=3 size=8, connectedness=1.00
Graph level=2 size=153, connectedness=0.12
Graph level=1 size=2503, connectedness=0.01
Graph level=0 size=39628, connectedness=0.00

The numbers are so bad, I almost think this is a bug in my measurements, but it isn't clear to me where it would be.
I am going to validate older versions of Lucene to see if this changes.

I have done this test back in Lucene 9.4, and we still end up every once in a while a graph where the mean number of connections hovers around 1 and whose connectedness is very low. It may be that these particular docs are tightly clustered, but even then, it seems weird that it should end up being so bad.

OK, I did some more digging, and it seems like my data is garbage, or at least how I am reading it in. I looked at one of these extremely disconnected graphs and found that all the float32 vectors were exactly the same. So, this appears to be a bug in my code.

Hi @benwtrent,
I left Amazon but I was able to run some tests with open dataset and also with Amazon dataset before leaving. I cannot share whole lot of detail about Amazon dataset but some aggregated results are shown. Rest assured I am picking this issue on priority now.
I was able to run tests trying out different heuristics. Basically, I tried the Lucene default diversity check, which generates disconnected even with 50K nodes of minilm dataset. I further tried different variations as suggested in HNSW paper. The extended candidates approach reduced the disconnectedness but did not eliminate it completely and it increased the indexing time by many times. Next, keepPruned candidates approach with keeping max-conn connections increased the disconnected nodes. So I tried keepPruned candidates with keeping pruned connections only till max-conn/2. This also did not show any improvement. Next, I tried the new proposed hueristic but without the component of removeing the two way connections, so basically the new hueristic with just removing one side of the duplex connection in the graph. Interestingly, this also did not change the disconnectedness. This was a bit surprising to me. Then I tried new heuristic with remove-otherhalf, basically removing the two way connections completely. This completely removed disconnecteness and number of disconnected nodes at all levels was zero. But unfortunately this means that the edges at some nodes can grow beyond the max-conn. I did not get chance to find the counts and distribution of total connections at each node which goes beyond max-conn, but I assume this is something that we may not want in lucene. So I thought may be the removing duplex edges (i.e. the remove-otherhalf) is the key behind decreasing disconnectedness. So I tried default algo and keep prunned both with the remove-otherhalf. Interestingly, those two experiments also did not decrease number of disconnected nodes.
Now, I was left with new heuristic with remove other half as the best option. To make sure that total connections per node do not grow beyond max-conn, I modified the algo to remove some random node in case it is not able to find the best node to remove (new heuristic with remove otherhalf and honour max-conn). This did help to keep the overall disconnectedness to zero, but it did show some nodes at level1 to be disconnected still (see comments) for minilm dataset. I tried with max-conn=8, max-conn=16, num_docs=50K and num_docs=100k. All gave zero overall disconnectedness and zero disconnected nodes at level0 but there were some disconnected nodes at level1 graph.
Anyway, I also did the experiment using Amazon  dataset for new heuristic with remove otherhalf and honour max-conn. I had to do code changes again to adopt it to Lucene 9.7 version. I saw similar results. Here also default algorithm gave lot of disconnected nodes but the new heuristic gave zero disconnected nodes at level0 and zero overall disconnected nodes. But at level1 and sometimes at level2 also there were some nodes that were disconnected.
I am more confident of the new heuristic now, but won't mind to run more tests and perf tests.
PR with all heuristics : https://github.com/apache/lucene/pull/12783/commits

Algo
no. of Disconnected Nodes at zeroth level
%age overall disconnected (nodes)
Comments
index time

Baseline Equivalent
90
0.1740 (87 nodes)
Same as baseline but with some extra parameters to allow more experiments
33s

Baseline
90
0.1740 (87)
Exactly the code as in production

Extend Candidates
39
0.0760 (38)

280s

keep-pruned till max-conn
154
0.2880  (144)
disconnected at level1=4 and at level2=1
36s

keep-pruned till half of max-conn
97
0.1860 (93)

34s

new heuristic without remove-otherhalf
119
0.2240 (112)

35s

new heuristic with remove-otherhalf
0
0
fully connnected at both 50K docs and 100K docs but there were errors at max-conn=8 as the size of neighbour array cannot grow and this algo allows more than max-conn connections.
33s

baseline with remove-otherhalf
91
0.1720 (86)
remove-otherhalf does not give zero disconnectedness with mainline algo

keep-half-pruned with remove-otherhalf
90
0.1740 (87)
no effect of remove-otherhalf with keep-half pruned
33s

new heuristic with remove otherhalf and honour max-conn
0
0
but for max-conn=8 and docs=100k I was 35 disconnected nodes at level1. There were no disconnected nodes at level0 even with max-conn=8
36s

Amazon data set baseline (lucene-9.7 release)
(75 - 338) in each segment
0.26 - 0.48 %
docs=7.5M, segments=13

Amazon data new heuristic with remove otherhalf and honour max-conn
77 in 3 out of 10 segments
0% overall
docs=7.5M, segments=10, Indexing time 4% increase, 6% increase in red-line queries/sec, 26% decrease in hnsw nodes visisted but 8% increase in avg query latency.

@nitirajrathore very interesting results. This sort of indicates to me that no matter the heuristic, we just need a second pass over the graph to ensure connectedness and fix it up :/
Even though the graph is different, the idea is the same here: https://github.com/jbellis/jvector/blob/main/jvector-base/src/main/java/io/github/jbellis/jvector/graph/GraphIndexBuilder.java
I will see about testing your new "new heuristic with remove otherhalf and honour max-conn" as if I am reading this correctly, seems the most promising. But, I suspect, even in extreme scenarios, we will still need a second pass over the graph to ensure graph connectedness.

Oh!, I see we added some tooling in https://github.com/mikemccand/luceneutil/pull/253 as part of KnnGraphTester. Maybe we can migrate some of this to lucene's test-framework

@msokolov I think adding a "reachable" test to Lucene would be nice. The main goal of such a test would be ensuring that every node is eventually reachable on every layer. The tricky part is I am sure such a test would be noisy until we have a "connectedness check", especially for random vectors.
I do like having something in Lucene-util though as I have found the larger statistics useful when debugging why a particular graph or change breaks.
So, something in both places would be good.

I struggle to make this work though since the changes to make everything more typesafe have also made the interesting bits inaccessible. EG I thought of adding something like this:
        for (LeafReaderContext ctx : reader.leaves()) {
          CodecReader codecReader = (CodecReader) ctx.reader();
          HnswGraph graph = ((HnswGraphProvider) codecReader.getVectorReader()).getGraph(vectorField);
          if (HnswTestUtil.isFullyConnected(graph) == false) {
            // for now just bail out.
            return;
          }
        }

to BaseVector...QueryTest however the VectorReader returned from that is hiding its HnswGraph behind a few layers of abstraction:
class org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsReader cannot be cast to class org.apache.lucene.codecs.HnswGraphProvider (org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsReader and org.apache.lucene.codecs.HnswGraphProvider are in unnamed module of loader 'app')
not sure what the approved way to go about this would be

@msokolov in the HNSW codec, we do something like this already when gathering the underlying graph.
I would do something like:
 if (FilterLeafReader.unwrap(ctx.reader()) instanceof CodecReader codecReader) {
   KnnVectorsReader currKnnVectorsReader = codecReader.getVectorReader();
   if (currKnnVectorsReader instanceof PerFieldKnnVectorsFormat.FieldsReader perFields) {
     currKnnVectorsReader = perFields.get(vectorField);
   }
   if (currKnnVectorsReader instanceof HnswGraphProvider graphProvider) {
     if (HnswTestUtil.isFullyConnected(graphProvider.getGraph(vectorField)) == false) {
        // for now just bail out.
        return;
     }
   }
 }

Is there a clean separation between:

HNSW ANN (e.g. clean room implementation) is known to have issues with unreachable nodes for certain configurations.
and
The Lucene implementation of HNSW ANN has additional issues that other implementations do not have
?

If this is mostly about the first, then it's a known issue with known mitigations.
If it's mostly about the second, doesn't that point to some specific defect in the implementation, a defect which could/should be fixed (in contrast, e.g., to two-pass approaches that may address both 1 and 2) ?

I'm not aware of any Lucene-specific issue here. We see this mostly in unit tests, but it has also been replicated with production data. Although one can speculate this might be more of an issue in Lucene because of its segmented multi-graph structure; I'm not sure since I don't have all that much experience of HNSW in other settings. I would be interested to hear about the known mitigations you mentioned.

@msokolov I am not sure what particular mitigations @CloudMarc is talking about, but I know of a couple of options outlined here: https://github.com/apache/lucene/issues/12627#issuecomment-1814046690
Though I would prefer not having yet another configuration item for HNSW. I think we should do a second "clean up pass" over HNSW (like JVector does for Vamana), to ensure we don't have orphaned nodes or clusters.
A second pass over the layers wouldn't be too bad performance wise (we wouldn't have to do many, if any at all, vector comparisons in the second pass), and it would be mostly for adding a couple of backlinks that were originally removed.

Re "known mitigations" - I simply meant common HNSW/ANN configuration changes like increasing MaxConnections and BeamWidth.
Thanks for mentioning precedence for the two-pass approach.  This seems useful:  https://weaviate.io/blog/ann-algorithms-vamana-vs-hnsw#vamana-implementation-details
I'm +1 for the 2-pass approach, fwiw.

One might hypothesize that this issue is due to adapting HNWS to Lucene's approach to segmentation.
This seems undercut by these observations:  https://github.com/apache/lucene/issues/12627#issuecomment-1767662289
Key question:  Does single-segment Lucene have the same rate of dis-connectedness as a stock or reference HNSW implementation, or more?  If it's significantly more, that may indicate an implementation defect.

Hi @benwtrent,
I left Amazon but I was able to run some tests with open dataset and also with Amazon dataset before leaving. I cannot share whole lot of detail about Amazon dataset but some aggregated results are shown. Rest assured I am picking this issue on priority now.
I was able to run tests trying out different heuristics. Basically, I tried the Lucene default diversity check, which generates disconnected even with 50K nodes of minilm dataset. I further tried different variations as suggested in HNSW paper. The extended candidates approach reduced the disconnectedness but did not eliminate it completely and it increased the indexing time by many times. Next, keepPruned candidates approach with keeping max-conn connections increased the disconnected nodes. So I tried keepPruned candidates with keeping pruned connections only till max-conn/2. This also did not show any improvement. Next, I tried the new proposed hueristic but without the component of removeing the two way connections, so basically the new hueristic with just removing one side of the duplex connection in the graph. Interestingly, this also did not change the disconnectedness. This was a bit surprising to me. Then I tried new heuristic with remove-otherhalf, basically removing the two way connections completely. This completely removed disconnecteness and number of disconnected nodes at all levels was zero. But unfortunately this means that the edges at some nodes can grow beyond the max-conn. I did not get chance to find the counts and distribution of total connections at each node which goes beyond max-conn, but I assume this is something that we may not want in lucene. So I thought may be the removing duplex edges (i.e. the remove-otherhalf) is the key behind decreasing disconnectedness. So I tried default algo and keep prunned both with the remove-otherhalf. Interestingly, those two experiments also did not decrease number of disconnected nodes. Now, I was left with new heuristic with remove other half as the best option. To make sure that total connections per node do not grow beyond max-conn, I modified the algo to remove some random node in case it is not able to find the best node to remove (new heuristic with remove otherhalf and honour max-conn). This did help to keep the overall disconnectedness to zero, but it did show some nodes at level1 to be disconnected still (see comments) for minilm dataset. I tried with max-conn=8, max-conn=16, num_docs=50K and num_docs=100k. All gave zero overall disconnectedness and zero disconnected nodes at level0 but there were some disconnected nodes at level1 graph.
Anyway, I also did the experiment using Amazon dataset for new heuristic with remove otherhalf and honour max-conn. I had to do code changes again to adopt it to Lucene 9.7 version. I saw similar results. Here also default algorithm gave lot of disconnected nodes but the new heuristic gave zero disconnected nodes at level0 and zero overall disconnected nodes. But at level1 and sometimes at level2 also there were some nodes that were disconnected. I am more confident of the new heuristic now, but won't mind to run more tests and perf tests.
PR with all heuristics : https://github.com/apache/lucene/pull/12783/commits
max-conn=16 & num_docs = 50K (unless specified)
Algo	no. of Disconnected Nodes at zeroth level	%age overall disconnected (nodes)	Comments	index time
Baseline Equivalent	90	0.1740 (87 nodes)	Same as baseline but with some extra parameters to allow more experiments	33s
Baseline	90	0.1740 (87)	Exactly the code as in production
Extend Candidates	39	0.0760 (38)		280s
keep-pruned till max-conn	154	0.2880 (144)	disconnected at level1=4 and at level2=1	36s
keep-pruned till half of max-conn	97	0.1860 (93)		34s
new heuristic without remove-otherhalf	119	0.2240 (112)		35s
new heuristic with remove-otherhalf	0	0	fully connnected at both 50K docs and 100K docs but there were errors at max-conn=8 as the size of neighbour array cannot grow and this algo allows more than max-conn connections.	33s
baseline with remove-otherhalf	91	0.1720 (86)	remove-otherhalf does not give zero disconnectedness with mainline algo
keep-half-pruned with remove-otherhalf	90	0.1740 (87)	no effect of remove-otherhalf with keep-half pruned	33s
new heuristic with remove otherhalf and honour max-conn	0	0	but for max-conn=8 and docs=100k I saw 35 disconnected nodes at level1. There were no disconnected nodes at level0 even with max-conn=8	36s
Amazon data set baseline (lucene-9.7 release)	(75 - 338) in each segment	0.26 - 0.48 %	docs=7.5M, segments=13
Amazon data new heuristic with remove otherhalf and honour max-conn	77 in 3 out of 10 segments	0% overall	docs=7.5M, segments=10, Indexing time 4% increase, 6% increase in red-line queries/sec,
26% decrease in hnsw nodes visisted but 8% increase in avg query latency.

Hello, benwtrent! I came across your discussion regarding various tests you conducted with the "minilm" dataset and found it particularly interesting. I'm currently engaged in similar research and am keen on exploring this further. Could you please share how I might be able to access the "minilm" dataset, or point me to any resources where it might be available? It seems like it is a open dataset. Any guidance you could provide would be greatly appreciated.

I'd like to take a stab at the "second pass" idea for patching up disconnected graph components. As a first step I think we ought to add state to the HnswGraphBuilder in order to clearly indicate that all nodes have been added and we are now engaged in finalizing the graph. My plan is to add a getCompletedHnswGraph() method that can be called when flushing, leaving the existing getHnswGraph() method that allows callers to observe the graph during construction.

also backported to 9x


Control plane loadbalancer

How is the control plane HA? I don't see a loadbalancer for the control plane nodes. E.g. the agents connect to the first node, what happens if the first node goes offline?

The cluster becomes HA when the control planes nodes number is equal or superior to 3, and always odd numbers as advised by Rancher.
When you have 3 control plane nodes, you can take any of them offline, and the cluster will continue to work. That is what happens when the automatic upgrades of both OS and k3s happen.
Now, if you are talking of using kubectl in an HA manner, that is not implemented. We use the IP of the first control plane in the kubeconfig.yaml. However, if it is not reachable, you can swipe it for other IPs of other control planes manually. But normally, control plane nodes go offline for a few seconds max for a reboot.
If you have a better solution for that, please propose it.

Ah that clears some things up. I thought that you needed a loadbalancer to make the control plane HA. Didn't know that under the hood the agents themselves talk to all 3 control plane nodes.
Kubectl HA is not a priority for me now but if somebody else wants to fix it, here are some general pointers:

Hetzner loadbalancer pointing to all control plane nodes
Kubeconfig must contain loadbalancer public ip instead of control plane ip
Control plane nodes need to be initialized with the following extra flags:

--tls-san ${lb_private_ip}
--tls-san ${lb_public_ip}

@mysticaltech
You might want to check this example from TechnoTim his Youtube channel.
He solves the controlplane HA with Kubevip and MetalLB. Don't know if this is something easy to introduce in your setup, but this takes care of auto-updating the IP
https://www.youtube.com/watch?v=CbkEWcUZ7zM
Another solution would be to add KeepAlived with 2 extra nodes and a floating IP which in return route traffic to 1 of the 3 control nodes that are healthy. If they go down, it routes to 2nd or 3rd one.
This method does add 2 extra node as cost

I think those articles about Keepalived are because in that time Hetzner did not have loadbalancers yet. This problem can be solved using their loadbalancers IMO

Exactly... Maybe something like that would work. But really, it's almost a nonissue. If you ever were to try to kubectl into a node that is being rebooted or down, you would change the IP in the kubeconfig.yaml file manually to the next control plane node.
So unless you folk need this, we can close the issue.