IoTHubDeviceClient connect()/disconnect(): Exception when used in function. (Synch method)

Package Name:  azure.iot.device
Package Version:  ?
Operating System: Windows 10
Python Version:  3.7.4

Describe the bug
Unsing the synch method, a simple connect() and disconnect() of a device client (IoTHubDeviceClient) gives an error if it is done in a function. (Without the function ist works.) The exception message after about 10 seconds is:
Exception caught in background thread.  Unable to handle.
["AttributeError: 'NoneType' object has no attribute '_on_connected'\n"]
To Reproduce
Just run the following script with a valid connection string, the error occurs after about 10 seconds:
import os
from azure.iot.device import IoTHubDeviceClient
import time

def f():
    conn_str = 'YourConnectionString'
    device_client = IoTHubDeviceClient.create_from_connection_string(conn_str)
    # connect the client.
    device_client.connect()
    # usually do something but this dies not matter
    # Finally, disconnect
    device_client.disconnect()

# using the function
f()

# wait - after about 10 seconds the error occurs
for i in range(20):
    print(i)
    time.sleep(1)

Expected behavior
No exception (and therefore no exception message)

Issue moved to Azure/azure-iot-sdk-python #534 via ZenHub

Hi @HoLue
I'm moving your issue to the repo of the right team, thanks!


Fix mongo specs & re-enable cucumber

Failures were due to changes in the mongodb format of listCollections. Bumping the mongo gem from 1.11.1 to 1.12.0 fixed the issue, and now we have all our specs and cucumber features running again.

Spurious spec failure is unrelated, and fixed by #535

:green_heart:


Added mongoose configuration in appConfig

Great work @Freedisch . I will work on the docker image part.


CORE-321 - Slack Regression Issues

Gate removal has been causing a few issues...

Conditional on membership dashboard always showing as not eligible.
Email invite never being sent.

If someone doesn't have a slack_id, they should be eligible. Simple!

I am asuming this means that slack is now meant to be open to all VATSIM members and not just UK/V/T members?

I am asuming this means that slack is now meant to be open to all VATSIM members and not just UK/V/T members?

Yep, we opened it up in 432036303de93bf2ae0740d18f647b7c47592c78

:tada: This PR is included in version 4.3.8 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


.spf-toolsrc ignored?

Hello
I created a file /root/.spf-toolsrc:
DOMAIN=our-domain.eu
ORIG_SPF=spf-orig.our-domain.eu
DESPF_SKIP_DOMAINS=
DNS_TIMEOUT=5
DNS_SERVER=<IP_ADDRESS>
TOKEN=cloudflare_token_here
<EMAIL_ADDRESS>
If I run this from /opt/spf-tools, where I checked out your repository:
./despf.sh | ./normalize.sh | ./simplify.sh | ./iprange.sh | ./mkblocks.sh | ./cloudflare.sh
Absolutely nothing happens our appears.
It only works a bit if I specify it like this:
./despf.sh spf-orig.our-domain.eu | ./normalize.sh | ./simplify.sh | ./iprange.sh | ./mkblocks.sh our-domain.eu
I get the idea that the config file isn't loaded or ignored or something. If I add | ./cloudflare.sh at the end, it also results in just nothing.

Hi! Let me have a look. I have not used the cloudflare.sh script for a while...

@MaartenUreel Please try now. The patches are on master branch already.

And thank you for reporting this!

If it works for you, please close the issue.

I just tried and I can confirm that it works. Thank you very much for responding so quickly and fixing this!

Hmm, there seems to be something still off:
root@appserver:/opt/spf-tools# ./despf.sh | ./normalize.sh | ./simplify.sh | ./iprange.sh | ./mkblocks.sh | ./mkzoneent.sh | ./cloudflare.sh
jq is /usr/bin/jq
awk is /usr/bin/awk
sed is /bin/sed
grep is /bin/grep
Getting spf1.our-domain.eu.eu
jq: error: Cannot iterate over null
Changing jasan.tk with id ... OK

It did something with the correct domain since it fetched spf1.our-domain.eu, but then in the last phase it went to the default jasan.tk after all.
If I specify the domain to start off:
root@appserver:/opt/spf-tools# ./despf.sh spf-orig.telsmart.eu | ./normalize.sh | ./simplify.sh | ./iprange.sh | ./mkblocks.sh | ./mkzoneent.sh | ./cloudflare.sh
jq is /usr/bin/jq
awk is /usr/bin/awk
sed is /bin/sed
grep is /bin/grep
Getting spf.protection.outlook.com
Getting spfa.protection.outlook.com
Getting spfb.protection.outlook.com
Getting spf.smtp2go.com
Getting spf.icontroller.eu
Getting email.freshdesk.com
Getting sendgrid.net
Getting mailgun.org
Getting spf1.mailgun.org
Getting spf2.mailgun.org
Getting _spf.exactonline.be
Getting spf.flowmailer.net
Getting spf.emailsignatures365.com
jq: error: Cannot iterate over null
Changing spf8.jasan.tk with id ... OK
Changing spf7.jasan.tk with id ... OK
Changing spf6.jasan.tk with id ... OK
Changing spf5.jasan.tk with id ... OK
Changing spf4.jasan.tk with id ... OK
Changing spf3.jasan.tk with id ... OK
Changing spf2.jasan.tk with id ... OK
Changing spf1.jasan.tk with id ... OK
Changing jasan.tk with id ... OK

@MaartenUreel yes, my fault. I moved the line that reads the spf-toolsrc, but then the defaults rode over the values from the rc file.

@MaartenUreel please try dabbb75

Nope:
root@appserver:/opt/spf-tools# ./despf.sh spf-orig.our-domain.eu | ./normalize.sh | ./simplify.sh | ./iprange.sh | ./mkblocks.sh | ./cloudflare.sh
jq is /usr/bin/jq
awk is /usr/bin/awk
sed is /bin/sed
grep is /bin/grep
Getting spf.protection.outlook.com
Getting spfa.protection.outlook.com
Getting spfb.protection.outlook.com
Getting spf.smtp2go.com
Getting spf.icontroller.eu
Getting email.freshdesk.com
Getting sendgrid.net
Getting mailgun.org
Getting spf1.mailgun.org
Getting spf2.mailgun.org
Getting _spf.exactonline.be
Getting spf.flowmailer.net
Getting spf.emailsignatures365.com
Changing jasan.tk^"v=spf1 with id ... OK
Changing spf1.jasan.tk^"v=spf1 with id ... OK
Changing spf2.jasan.tk^"v=spf1 with id ... OK
Changing spf3.jasan.tk^"v=spf1 with id ... OK
Changing spf4.jasan.tk^"v=spf1 with id ... OK
Changing spf5.jasan.tk^"v=spf1 with id ... OK
Changing spf6.jasan.tk^"v=spf1 with id ... OK
Changing spf7.jasan.tk^"v=spf1 with id ... OK
Changing spf8.jasan.tk^"v=spf1 with id ... OK

Also if I run with the mkzoneent, it also gives the wrong domain:
root@appserver:/opt/spf-tools# ./despf.sh spf-orig.telsmart.eu | ./normalize.sh | ./simplify.sh | ./iprange.sh | ./mkblocks.sh | ./mkzoneent.sh
Getting spf.protection.outlook.com
Getting spfa.protection.outlook.com
Getting spfb.protection.outlook.com
Getting spf.smtp2go.com
Getting spf.icontroller.eu
Getting email.freshdesk.com
Getting sendgrid.net
Getting mailgun.org
Getting spf1.mailgun.org
Getting spf2.mailgun.org
Getting _spf.exactonline.be
Getting spf.flowmailer.net
Getting spf.emailsignatures365.com
spf8.jasan.tk. 1800 IN TXT "v=spf1 ip4:<IP_ADDRESS> ip4:<IP_ADDRESS> ip4:<IP_ADDRESS> ip4:<IP_ADDRESS> ip4:<IP_ADDRESS>/26 ip4:<IP_ADDRESS> ip6:2001:489a:2202::/48 ip6:2a01:111:f400::/48 ~all"
[..snip..]
jasan.tk. 1800 IN TXT "v=spf1 ip4:<IP_ADDRESS>/22 ip4:<IP_ADDRESS>/22 ip4:<IP_ADDRESS>/22 ip4:<IP_ADDRESS>/23 ip4:<IP_ADDRESS>/28 ip4:<IP_ADDRESS>/17 ip4:<IP_ADDRESS> ip4:<IP_ADDRESS>/26 ip4:<IP_ADDRESS>/26 ip4:<IP_ADDRESS>/26 ip4:<IP_ADDRESS>/27 include:spf1.jasan.tk ~all"

@MaartenUreel please check 64b980d
Seems you are the first one to really run this full chain of scripts :-)

OK, the mkzoneent now returns the correct domains.
However, I still have to pass my spf-orig.our-domain.eu as a parameter to despf.sh while it is also in the config file.
The records are not being created / updated on CloudFlare neither though.. :)

@MaartenUreel See 9811f7b

@MaartenUreel See a5a2fa7 please

@MaartenUreel See 99c683d

lomidrevo:~/src/spf-tools$ ./despf.sh | ./normalize.sh | ./simplify.sh | ./ipran
ge.sh | ./mkblocks.sh | ./cloudflare.sh
jq is /usr/bin/jq
awk is /usr/bin/awk
sed is /bin/sed
grep is /bin/grep
Getting mailgun.org
Getting spf1.mailgun.org
Getting spf2.mailgun.org
Getting _spf.google.com
Getting _netblocks.google.com
Getting _netblocks2.google.com
Getting _netblocks3.google.com
Changing jasan.tk with id a2f0c894e7a71759508ea034772cc26b... OK
Changing spf1.jasan.tk with id 940b5ae32e24350407b208ff3ba68b03... OK
Changing spf2.jasan.tk with id 7f6c6730a9ffdfe637d1d6a563f9e17d... OK
Changing spf3.jasan.tk with id 0e9af188245dd693617a7a91f0ab71ae... OK
lomidrevo:~/src/spf-tools$ export USE_UPSTREAM=1
lomidrevo:~/src/spf-tools$ ./compare.sh
Getting spf1.jasan.tk
Getting spf2.jasan.tk
Getting spf3.jasan.tk
Getting mailgun.org
Getting spf1.mailgun.org
Getting spf2.mailgun.org
Getting _spf.google.com
Getting _netblocks.google.com
Getting _netblocks2.google.com
Getting _netblocks3.google.com
Everything OK

@MaartenUreel Thank you very much for bringing this up! It really needed some care.

root@appserver:/opt/spf-tools# ./despf.sh spf-orig.telsmart.eu | ./normalize.sh | ./simplify.sh | ./iprange.sh | ./mkblocks.sh | ./mkzoneent.sh | ./cloudflare.sh
jq is /usr/bin/jq
awk is /usr/bin/awk
sed is /bin/sed
grep is /bin/grep
Getting spf.protection.outlook.com
Getting spfa.protection.outlook.com
Getting spfb.protection.outlook.com
Getting spf.smtp2go.com
Getting spf.icontroller.eu
Getting email.freshdesk.com
Getting sendgrid.net
Getting mailgun.org
Getting spf1.mailgun.org
Getting spf2.mailgun.org
Getting _spf.exactonline.be
Getting spf.flowmailer.net
Getting spf.emailsignatures365.com
./cloudflare.sh: 54: shift: can't shift that many
rm: cannot remove ‘/tmp/cloudflare-zone-9QQY-data’: No such file or directory

Please try c1d1d46
Now it's too late. I will have a look tomorrow.

You also do not need to supply the domain name on the command line of despf.sh anymore (given it is set either in ~/spf-toolsrc or in environment variable DOMAIN).

I can indeed drop the domain now, that part works. Cloudflare part not yet:
root@appserver:/opt/spf-tools# ./despf.sh |  ./normalize.sh | ./simplify.sh | ./iprange.sh | ./mkblocks.sh | ./cloudflare.sh
jq is /usr/bin/jq
awk is /usr/bin/awk
sed is /bin/sed
grep is /bin/grep
Getting spf.protection.outlook.com
Getting spfa.protection.outlook.com
Getting spfb.protection.outlook.com
Getting spf.smtp2go.com
Getting spf.icontroller.eu
Getting email.freshdesk.com
Getting sendgrid.net
Getting mailgun.org
Getting spf1.mailgun.org
Getting spf2.mailgun.org
Getting _spf.exactonline.be
Getting spf.flowmailer.net
Getting spf.emailsignatures365.com
./cloudflare.sh: 54: shift: can't shift that many
rm: cannot remove ‘/tmp/cloudflare-zone-Pnn4-data’: No such file or directory

Btw, do you need to pipe through mkzoneent.sh before Cloudflare, or not? What format does it expect?

Are you planning to do any work on this? Just to know, because otherwise I'll have to write some python scripts to take care of this :)

@MaartenUreel excuse me for silence. I do not know how to reproduce the error since it works for me. I can have a look but please provide me with more details.
 There should be DEBUG environment variable used in most scripts so first try to export DEBUG=1 and run the same way as you did before. If it doesn't help, add sh -x before the last (cloudflare) script.

@MaartenUreel please send me output of following:
ls -l /bin/sh

And to answer your question, piping through mkzoneent.sh is not needed. That script outputs the zone entries in format for ISC BIND DNS server.

@MaartenUreel This works for me:
lomidrevo:~/src/spf-tools$ ./despf.sh | ./normalize.sh | ./simplify.sh | ./ipran
ge.sh | ./mkblocks.sh | ./cloudflare.sh
jq is /usr/bin/jq
awk is /usr/bin/awk
sed is /bin/sed
grep is /bin/grep
Getting mailgun.org
Getting spf1.mailgun.org
Getting spf2.mailgun.org
Getting _spf.google.com
Getting _netblocks.google.com
Getting _netblocks2.google.com
Getting _netblocks3.google.com
Changing jasan.tk with id a2f0c894e7a71759508ea034772cc26b... OK
Changing spf1.jasan.tk with id 940b5ae32e24350407b208ff3ba68b03... OK
Changing spf2.jasan.tk with id 7f6c6730a9ffdfe637d1d6a563f9e17d... OK
Changing spf3.jasan.tk with id 0e9af188245dd693617a7a91f0ab71ae... OK
lomidrevo:~/src/spf-tools$ cat ~/.spf-toolsrc
DOMAIN=jasan.tk
ORIG_SPF=spf-orig.jasan.tk
DESPF_SKIP_DOMAINS=
DNS_TIMEOUT=5
DNS_SERVER=<IP_ADDRESS>
TOKEN=<mytoken>
<EMAIL_ADDRESS>

Oops. Reopening the automatic close and waiting for feed-back.

@MaartenUreel how does it work for you? May I close this issue, please?


Add foundry tests in CI workflow

Something similar is done in the flow-evm-bridge repository.

what kind of tests would we add? because I'm not sure this tests are really relevant to evm gateway? more to evm core if anything, but please explain in more detail the issue.

Almost all these commands accept an --rpc-url option, so they interact with the EVM Gateway.


Detect whether cursor is within closed fold?

Does this feature exist in Emacs orgmode core?
N/A
Orgmode link
No response
Feature value
A typical problem is this: you load a new file. Vim remembers where your last cursor position was and puts the cursor there. Say the cursor position is within the body of some headline. But when a new file is loaded, the headline is folded, so the cursor appears to be at the headline itself, because the cursor is on the line indicating that there's a fold. Now when you do some org command like creating a new headline, you will be surprised that you might create some kind of intermediate headline, because you don't know that "in reality" your cursor is not on the headline, but somewhere in its body.
Hopefully the following makes it clear: https://asciinema.org/a/b9zg7fiHCl35E2xgsJYzz0Iu7
My proposal is this: if it's possible to detect that the cursor is on a fold, then org should "pretend" that the cursor is on the headline itself, not somewhere in the folded body. This way the user won't be surprised because she doesn't know where the cursor really is.
Additional context
No response

@Maltimore how do you load a new file and end up inside a fold? When I :e <file> the cursor will jump to the top of the file

I agree, your point is valid. However, zc not changing the line number until any vertical movement happens is default vim behavior and I'm not sure if we want to change that. Another option that comes to mind is to use <tab> for toggling the current fold, whether one is inside the fold or not

Yep! I also don't think we should move the cursor when closing the fold, that would go against the vim way. My suggestion is only that if the user does some org-action, like changing the TODO state or creating another headline, we should pretend that the cursor is on the headline line which determines the fold. So if the user changes TODO state, it changes the TODO state of the headline determining the fold, and not any other headline within the fold.

Sorry, I misunderstood you then. Thanks for clarifying. If there is a way to check whether the current line is within a fold, your suggestion sounds sensible. We would then need to find out which headline is the one responsible for the fold we're currently in, as we can fold up to the top-level with zC. Just thinking out loud here

When headline is opened from agenda, we do zv (see here). I'm not sure how we would handle a case when file is opened outside of org (directly). This autocmd should do the trick, but I'm not sure if we should add it to the plugin, or let user decide if they want it or not:
vim.api.nvim_create_autocmd('BufReadPost', {
  pattern = '*.org',
  callback = function ()
    -- vim.schedule needed to make sure org is fully loaded
    vim.schedule(function()
      vim.cmd[[norm! zv]]
    end)
  end
})

Ah! This is really nice and really simple! Thanks a lot.
I would be in favor of including this in the plugin. But if not, we could add it to some sort of "Tips and Tricks" section in the docs.

In the interest of keeping the bug tracker lean and mean (not full of open issues), I am going to close this as it has been 95% fixed for me with the suggested workaround. The only remaining issue is the part where you close a fold but the cursor is still somewhere inside, and then you perform an org-action. This doesn't really happen that often. Once we work on some kind of "tips and tricks" section in the docs or on the website, I'll suggest the workaround there.


Rework file structure

@ArwynFr after merging all remaining fixes and features from dev to master, shouldn't we change the way that Orion is hosted on Github on the same model than Tembelan is ?

Yes, it would allow not to embed some useless files such as .github directory content

Do you mean, embedding in the PBO file ? It should be on the exclusion list by default, at least for the integrated builder. I have no idea on how Mikero behave on this point.

It's easier to have mission content in a directory and non mission content outside, rather than maintaining a blacklist of files.
BTW, I'm relying on armake2 to rapify the mission and pack to pbo.

'kay then I'm waiting for your feedback on the current dev version (whether it is loading properly or not), and if all is okay we will merge dev and reorganize this repo


TF 2.0: Spyder auto complete and go to definition not working. Please fix it!

Due to the nature of the "lazy import" spyder IDE is not able to resolve the package tree and fails both "auto complete" and "go to function definition".
Spyder guys told me TF should implemeny .pyi files to overcome this.
https://github.com/spyder-ide/spyder/issues/11553

Which version of TF are you using? Can you please fill in the issue template?
.pyi files already exist on master as @annarev is working on exactly this.

Interesting. I added a init.pyi file under keras/ in https://github.com/tensorflow/tensorflow/commit/69f6601034942b018081715b31f6dd40111250af but looks like that doesn't help in Spyder. It still doesn't autocomplete for from tensorflow.keras import metrics

Retested with correct environment now. Now autocomplete works for from tensorflow.keras import metrics, but jump to definition doesn't work for from tensorflow.keras.layers import Conv2D

Retested with correct environment now. Now autocomplete works for from tensorflow.keras import metrics, but jump to definition doesn't work for from tensorflow.keras.layers import Conv2D

Exactly, I also tested with Kite engine installed. I dunno if this is the beauty of the Kite engine but Autocomplete now works smooth though go to definition still fails.

Could you please refer to the new import mechanism of keras here Related changes: https://github.com/keras-team/keras/blob/master/pip_build.py  and the Related post which should solve the auto complete issue.
For more details and discussion on this issue, please refer the issue here https://github.com/tensorflow/tensorflow/issues/56231


BluetoothEndPoint does not contain a constructor that takes 2 arguments

On version 4.0.4 when I code the following line in a Visual Studio 2017 .NET Framework 4.6.1 Windows Forms application:
BluetoothEndPoint blep = new BluetoothEndPoint(BluetoothAddress.Parse("001122334455"), BluetoothService.SerialPort);
I get the error "BluetoothEndPoint does not contain a constructor that takes 2 arguments"
As per the documentation BluetoothEndPoint has constructor that takes 2 or 3 arguments.
Note: I am using 4.0.4 version because 4.0.5 nuget package fails to install as per #74
Thanks!

In the current release the BluetoothEndPoint is not emphasised as the BluetoothClient.Connect(address, service) provides a simpler approach for cross platform use. I'm looking at reintroducing BluetoothEndPoint as a supported alternative but will have to assess how it will be implemented on platform APIs which are not Socket oriented.

What I am missing from BluetoothClient.Connect(address, service) that BluetoothEndPoint has is the possibility of specifying a port number for the connection BluetoothEndPoint(BluetoothAddress, Guid, Int32).
I am having some issues connecting to a HeaadSet (HFP) for getting the battery level and I want to see if specifying the port numbre will help me.

Specifying a port number may be useful, especially for reconnecting to a previously used device but using the service uuid will ensure you're using the correct port for an exposed service. If you can't connect using the Headset profile it may be because the device is already connected or that it uses the Handsfree profile instead.

Ok, so using the guid exposed by the device on BluetoothDeviceInfo.InstalledServices is equivalent to using a port number.
In my case I am able to connect to the device with BluetoothClient.Connect(address, service) but once connected I try to send and/or receive but there is no data on the socket stream, I thought that specifying a port number would help, but now I see that it is pointless.
This is my code:
BluetoothDevicePicker picker = new BluetoothDevicePicker();
BluetoothDeviceInfo device = await picker.PickSingleDeviceAsync();

BluetoothClient cli = new BluetoothClient();
cli.Connect(device.DeviceAddress, /*BluetoothService.SerialPort*/ device.InstalledServices.ElementAt(0));
if (cli.Connected) MessageBox.Show("Connected");

//cli.Authenticate = true;
//cli.Encrypt = true;

NetworkStream stream = cli.GetStream();
stream.Write(System.Text.Encoding.ASCII.GetBytes("\r\n\r\n"), 0, 4);

if (stream.CanRead)
{
    byte[] buff = new byte[1024];
    int n = 0;
    StringBuilder str = new StringBuilder();
    while (stream.DataAvailable)
    {
        n = stream.Read(buff, 0, buff.Length);
        str.AppendFormat("{0}", Encoding.ASCII.GetString(buff, 0, n));
    }
    MessageBox.Show(str.ToString());
}

I am trying to use the RFCOMM service in order to get the battery information from the device (Headset with Bluetooth 4.1).

I have logged the bluetooth communications between and Android device (that can get the battery value from the headset) and the headset. I have opened with WIreShark and it seems that AT commands that inform about the battery are not sent trough RFCOMM protocol, they are sent trough HFP protocol. How can I use the HFP protocol to send/get AT commands with this library?

Handsfree Profile runs over an RFComm connection. All of it's control commands/responses are text based AT commands. To connect to a handsfree device you open an RFComm socket to the BluetoothService.Handsfree service uuid. There is a set of required commands which the handsfree device will send to the phone (Audio Gateway) and you must respond to before you can get indicator values. Both Handsfree and Headset are RFComm based but Handsfree has a lot more functionality and has been updated more recently. You may find however that if the handsfree device is connected and active on the Android device you can't open another connection as the handsfree device may only support a single device at once.

Thanks @peterfoot for your support!
I have extracted the bluetooth log from my Android phone (which is able to get the battery level) to be sure that the headset is using HFP to send AT command with the battery information (AT+IPHONEACCEV), this is one of the packets extracted from the log (with wireshark):
Frame 1122: 38 bytes on wire (304 bits), 38 bytes captured (304 bits)
Encapsulation type: Bluetooth H4 with linux header (99)
Arrival Time: Jul 16, 2020 12:18:16.449232000 Romance Daylight Time
[Time shift for this packet: 0.000000000 seconds]
Epoch Time:<PHONE_NUMBER>.449232000 seconds
[Time delta from previous captured frame: 0.001885000 seconds]
[Time delta from previous displayed frame: 0.018263000 seconds]
[Time since reference or first frame: 21.363811000 seconds]
Frame Number: 1122
Frame Length: 38 bytes (304 bits)
Capture Length: 38 bytes (304 bits)
[Frame is marked: False]
[Frame is ignored: False]
Point-to-Point Direction: Received (1)
[Protocols in frame: bluetooth:hci_h4:bthci_acl:btl2cap:btrfcomm:bthfp]
Bluetooth
    [Source: GNAudio_xxxxx (AA:BB:CC:DD:EE:FF)]
    [Destination: SamsungE_xxxxx (FF:EE:DD:CC:BB:AA)]
Bluetooth HCI H4
    [Direction: Rcvd (0x01)]
    HCI Packet Type: ACL Data (0x02)
Bluetooth HCI ACL Packet
Bluetooth L2CAP Protocol
Bluetooth RFCOMM Protocol
    Address: E/A flag: 1, C/R flag: 0, Direction: 0, Channel: 2
    Control: Frame type: Unnumbered Information with Header check (UIH) (0xef), P/F flag: 0
    Payload length: 25
    Frame Check Sequence: 0xbf
Bluetooth HFP Profile
    [Role: HS - Headset (2)]
    AT Stream: AT+IPHONEACCEV=2,1,5,2,0\r
    Command 0: +IPHONEACCEV
        Command Line Prefix: AT
        Command: +IPHONEACCEV (Apple Bluetooth Headset Battery Level Indication)
        Type: Action Command (0x003d)
        Parameters
            Count: 2
            Key: Battery Level (1)
            Value: 5
            Key: Dock State (2)
            Value: 0

And this is the code I am using to get this AT commands from the headset in my Windows C# application using the 32feet (InTheHand.Net.Bluetooth) library. To do so I try to connect to the headset using BluetoothService.Handsfree:
BluetoothDevicePicker picker = new BluetoothDevicePicker();
BluetoothDeviceInfo device = await picker.PickSingleDeviceAsync();
BluetoothClient cli = new BluetoothClient();

//device.SetServiceState(BluetoothService.Handsfree, true);

cli.Connect(device.DeviceAddress, BluetoothService.Handsfree);
if (cli.Connected) MessageBox.Show("OK");

NetworkStream stream;

stream = cli.GetStream();
if (stream.CanRead)
{
    byte[] buff = new byte[1024];
    int n = 0;
    StringBuilder str = new StringBuilder();
    while (stream.DataAvailable)
    {
        n = await stream.ReadAsync(buff, 0, buff.Length);
        str.AppendFormat("{0}", Encoding.ASCII.GetString(buff, 0, n));
    }
    MessageBox.Show(str.ToString());
}

When executing the code it gives the following error in cli.Connect(device.DeviceAddress, BluetoothService.Handsfree):
System.Net.Sockets.SocketException: 'A socket operation failed because the destination host was down'
With cli.Connect(device.DeviceAddress, BluetoothService.SerialPort) it connects (so the host it is not down) but i do not get any reply from the stream.ReadAsync call. I have checked that my PC is the only one connected to the headset.

The headset is definitely paired to the PC? You could try checking the BluetoothDeviceInfo InstalledServices array for the Handsfree uuid but this only returns services which Windows knows about and so on older versions of Windows this won't include Handsfree.
Secondly the command you are using is an Apple specific extension to the Handsfree service so while it will work on many newer devices it's not universally supported. Also you should not simply connect and send this command - there is an expected sequence of commands and responses when the handsfree device and audio gateway (your PC in this case) go through when first connected. This allows both sides to negotiate supported features. Then not only can you get indicator values but also receive notifications when these change. You'll have to refer to the HFP spec which has a flowchart describing this.

The headset is definitely paired to the PC? You could try checking the BluetoothDeviceInfo InstalledServices array for the Handsfree uuid but this only returns services which Windows knows about and so on older versions of Windows this won't include Handsfree.

I tried to connect using each one of the 5 InstalledServices I can get from device:
InstalledServices[0] {00001101-0000-1000-8000-00805f9b34fb}
InstalledServices[1] {0000110b-0000-1000-8000-00805f9b34fb}
InstalledServices[2] {0000110c-0000-1000-8000-00805f9b34fb}
InstalledServices[3] {0000110e-0000-1000-8000-00805f9b34fb}
InstalledServices[4] {0000111e-0000-1000-8000-00805f9b34fb}

The only one that is connecting is the first one:
cli.Connect(device.DeviceAddress, device.InstalledServices.ElementAt(0))
But still I cannot get any commands from the device when I execute:
NetworkStream stream;

stream = cli.GetStream();
if (stream.CanRead)
{
    byte[] buff = new byte[1024];
    int n = 0;
    StringBuilder str = new StringBuilder();
    n = await stream.ReadAsync(buff, 0, buff.Length);
    str.AppendFormat("{0}", Encoding.ASCII.GetString(buff, 0, n));

    MessageBox.Show(str.ToString());
}

The execution gets stuck in ReadAsync call.

Secondly the command you are using is an Apple specific extension to the Handsfree service so while it will work on many newer devices it's not universally supported. Also you should not simply connect and send this command - there is an expected sequence of commands and responses when the handsfree device and audio gateway (your PC in this case) go through when first connected. This allows both sides to negotiate supported features. Then not only can you get indicator values but also receive notifications when these change. You'll have to refer to the HFP spec which has a flowchart describing this.

The IPHONEACCEV command is sent from the headset to de Android smartphone. Yes you are right that I need to send some AT commands before in order configure the communciation, but I least I excpected to receive any command once I connect to the Headset, but as you can see in my code, the ReadAsync function which is called just after cli.Connect gets stuck with no data received from the headset.

Check the IsConnected property of the BluetoothDeviceInfo returned from your picker. If Windows is connected to the Handsfree you won't also be able to connect to the service. There isn't a documented API to get the battery level in this case but clearly the value is there as Windows 10 surfaces it through the Settings > Devices page.

Yes, Connected property of the device is true. Should I disconnect it before calling cli.Connect() ? How to do it from code?

Ok, I have made some progress /thanks to you :) )
If I call device.SetServiceState(BluetoothService.Handsfree, false) before calling cli.Connect(device.DeviceAddress, BluetoothService.Handsfree) now I am able to connect to the headset and I get an AT command.
The only thing here is that when I call this SetServiceState my computer is not able to use the headset for calls. I would like to do both thing in parallel, is it possible? (my headset allow 2 connections in parallel)

They sometimes support multiple devices but it's unlikely to support multiple connections from the same device. If you need to take a snapshot you could disable the service, connect and retrieve the battery and then disconnect and re-enable the windows service. I'm trying to find if there is another API which will expose the handsfree battery when it is connected to Windows...

So the process now is as follows:
BluetoothDevicePicker picker = new BluetoothDevicePicker();
BluetoothDeviceInfo device = await picker.PickSingleDeviceAsync();

device.SetServiceState(BluetoothService.Handsfree, false);

BluetoothClient cli = new BluetoothClient();
cli.Connect(device.DeviceAddress, BluetoothService.Handsfree);
if (cli.Connected) MessageBox.Show("YES");

NetworkStream stream;
stream = cli.GetStream();
// Stream read/write operations to get the battery level
stream.Close();
cli.Close();
device.SetServiceState(BluetoothService.Handsfree, true);

The only issue I am facing now is that when i reconnect the headset to windows with device.SetServiceState(BluetoothService.Handsfree, true) the sound quality is really bad. I checked in the audio properties and the Headset is now using 8000Hz for Handsfree. If I disconnect the headset from Windows and connect it again, it gets 16000hz back again. Is any chance to force by code to use 16000hz?

Hey @sturla78 you're right, method could be different for JBL, but I haven't tested it. In my case both Sony and Firebolt headphones worked with just sending an "OK" to whatever was incoming until I got the IPHONEACCEV

I am trying to get battery level from headset too, glad i've found this thread. I tried both 32feet api and approach based on @SpartanX1 code with winrt apis, both have requirement that device must be disconnected prior to establishing socket connection. Ok, but we send/read AT commands, at the same time in Windows Settings, headset becomes "connected" and battery level also gets displayed there! So windows surely does something else: our manual socket connection does not prevent it from reading battery level. Maybe it shares existing socket? I tried looking at BluetoothClient and StreamSocket and don't see any options to modify socket options (if there are any at all). Also there's a closed source app called "bluetoothgoodies" that claims to "just work"...

I also have a similar problem. Is there any official example code for your reference? It has been bothering me for several days, thank you

I disabled Headset driver on my BT headphone. Windows can show only battery and can't connect Headset because it is off.
@Rast1234 May be Windows use cached data? I did not try to wait while battery information changed.

App from bluetoothgoodies used custom driver to get battery. I do not know what it does, I checked all my headphones and does not see that this driver was used.

UPD: If you connect to any service first, after it Windows will be able to connect to this service without any trouble. So looks like Windows block socket for yourself and does not allow connect other application.

@steam3d that's exactly why i think it could be possible to work with the socket at a lower level, but no idea how. idk about caching, but windows UI sometimes shows out-of-date battery level after my device is disconnected so yeah, maybe it has an internal cache

@Rast1234 Or Windows use HID device to collect battery information.  Windows is mapped Bluetooth headphones also as a HID device that I suppose control the headphone buttons.  HID device can provide battery information or at least event when battery is low.
https://www.bluetooth.com/specifications/specs/human-interface-device-profile-1-1-1/

4.3.1 Low Battery Notifications
Battery-powered Bluetooth HID devices may utilize the available HID controls in HID reports to inform the Bluetooth HID Host of a low battery condition. There are standard usages available for power status and battery reporting in the HID Specification; see USB HID Usage Tables [4], Generic Device Controls page.

We connect to RFcomm service, that lays on top of L2CAP.

https://www.bluetooth.com/specifications/specs/human-interface-device-profile-1-1-1/

The Bluetooth HID Protocol (HIDP) operates over the Bluetooth L2CAP layer and is an adaptation of the USB HID Protocol. The USB HID specification [5] defines Control and Interrupt channels which are mapped to USB Control and Interrupt pipes. This profile defines analogous logical channels which are mapped to L2CAP channels. See section
5.2 for details on how the L2CAP channels are opened and configured.

So Window can just operate of data on L2CAP layer and because of it does not have any trouble to get Battery information even if socket is busy.
Anyway I tried to find HID device to get more info but without success

Well I made lot of test what I figure out:

Windows caches battery value somewhere. When you connect headphones it will show battery from previews session and then update it to actual one. (I could not find where It can storage)
When you connect to RFCOMM, Windows can't access to this service or stream. My headphones changed battery information and notify through RFCOMM, but Windows did not see this changes.

If Windows is connected to the Handsfree service then you won't be able to open a socket - and vice-versa - Windows won't get battery updates if you've disconnected and connected to the service yourself. If only the Windows cached value was accessible we'd be able to read this without interfering with the connection.

@peterfoot Do you have any idea where Windows can storage cached value? I checked everything from devices properties to regedit.exe. Maybe I something miss

Hi there. I have a problem with the subject of this thread. Guess someone could help me. I am trying to get the battery level of Bluetooth headphones via Handsfree service. I have two devices for tests: AirPods and Logitech G735. Windows can get battery level only for Airpods. And MacOS only for Logitech. This is strange for me but also we know that we can get battery level for both devices. So I'm using this piece of code but getting only one AT Command in response AT+BRSF=671 for AirPods and AT+BRSF=147 for Logitech. What am I doing wrong? Thank you in advance
private static async Task<byte> GetBatteryLevelAsync(Headphones headphones)
        {
            return await Task.Run(() =>
            {
                using (var bluetoothClient = new BluetoothClient())
                {
                    var device = bluetoothClient.PairedDevices.FirstOrDefault(d => d.DeviceName == headphones.Name);
                    if (device != null)
                    {
                        device.SetServiceState(BluetoothService.Handsfree, false);
                        bluetoothClient.Connect(device.DeviceAddress, BluetoothService.Handsfree);

                        var stream = bluetoothClient.GetStream();
                        if (stream != null && stream.CanRead && stream.CanWrite)
                        {
                            while (true)
                            {
                                if (!stream.DataAvailable)
                                {
                                    break;
                                }

                                byte[] responseBytes = new byte[1024];
                                int bytesRead = stream.Read(responseBytes, 0, responseBytes.Length);
                                string response = Encoding.ASCII.GetString(responseBytes, 0, bytesRead);

                                byte[] commandBytes = Encoding.ASCII.GetBytes(response);
                                stream.Write(commandBytes, 0, commandBytes.Length);
                            }
                        }

                        if (bluetoothClient.Connected)
                        {
                            bluetoothClient.Close();
                        }

                        device.SetServiceState(BluetoothService.Handsfree, true);
                    }
                }

                return Task.FromResult((byte)1);
            });
        }
      ```

@m1adow What did you mean windows can get? Your code does nothing or what? What does the output console say?

@m1adow What did you mean windows can get? Your code does nothing or what? What does the output console say?

I mean that in the Windows Bluetooth & Devices section, I can see headphones and their battery level.

My code is returning only this one line

@m1adow I suppose mac can't get battery because it is Fake AirPods.
You code does nothing because headphone waiting for reponce. You need read about AT+ commands.  You must reponce on each request from headphones.

@steam3d Hello again! I successfully managed to get the battery from the code thanks to you. But for some reason between checking data availability from the stream, I need to add a small delay. If I run code without delay then in Output I will get "false". Is there a workaround to avoid this?
while (true)
{
    await Task.Delay(50); // how to avoid waiting for data?
    Debug.WriteLine(stream.DataAvailable);
    if (!stream.DataAvailable)
    {
        break;
    }

    byte[] buffer = new byte[1024];
    int bytesRead = await stream.ReadAsync(buffer);
    string receivedData = Encoding.ASCII.GetString(buffer, 0, bytesRead);
    recievedResponse.Append(receivedData);
    Debug.WriteLine(receivedData);

    await stream.WriteAsync(responseBytes);
}

@m1adow Check Microsoft Rfcomm example on their GitHub. You do not need use while(true).


ECR support

Hello,
I tried using ORAS with ECR support and I get Error: unexpected response: 404 Not Found
I know the project is still in its infancy but I wanted to hack on it in AWS and I was wondering if anyone got it to work. Anything I can do to help test this? :)

@zmarouf I don't think ECR supports arbitrary content types at this time. Maybe you can send feature request to their support?
Alternatively, you can run a custom registry on AWS EC2 node. container, etc:
docker run -it --rm -p 5000:5000 registry

All config options here

@zmarouf I've just opened https://github.com/aws/containers-roadmap/issues/308

Awesome! Thank you so much for confirming this and opening a ticket chez AWS's appropriate repo.
I'll play with the custom registry for now! :)

This is now official supported in ECR, so all that's left is ECR auth right?


[Major Update] Detail transfer / Reinforce FG and more

This PR is based on #23.
@Haoming02 did amazing improvement on the IC-Light extension.
Here is the list of actual changes made in Haoming's fork:

Reorganize UIs
Add explanations on how each mode works
Load models from ic-light folder with arbitrary filenames
Use pre-built rembg package instead
Important: Swap the way how img2img works: Now the input image is the subject while the extension takes lighting conditions.

This just makes more sense to me...

New: Implement Difference of Gaussians to reintroduce some details after the processing

My changes made on top of Haoming's fork:

Revert the change that swaps input_fg and img2img input. img2img input can be used by other extensions such as ControlNet. We should not swap it mid way, which can cause confusion.
Moved some scripts from /script folder to /libiclight folder. The general guideline is that you don't want to put a file with generic name in scripts folder as the namespace is shared with other extensions.
New: Add reinforce fg checkbox which can help preserve the base color of fg object. This option essentially does the same processing as https://github.com/huchenlei/ComfyUI-IC-Light-Native/blob/main/examples/ic_light_preserve_color.json

Breaking change:

You need to move IC-Light models from models/unet to models/ic-light

Updated How to use section:
How to use
For best result, it is recommended to use low CFG and strong denosing strength.
Given FG, Generate BG and relight [Txt2Img][HR available]

Infotext:
sunshine from window, a beautiful girl, absurdres, highres, (masterpiece:1.2), (best quality, highest quality),
Negative prompt: (lowres, low quality, worst quality:1.2), (text:1.2), watermark, (frame:1.2), deformed, ugly, deformed eyes, blur, out of focus, blurry, deformed cat, deformed, photo, anthropomorphic cat, monochrome, photo, pet collar, gun, weapon, blue, 3d, drones, drone, buildings in background, green
Steps: 25, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 2, Seed:<PHONE_NUMBER>, Size: 512x768, Model hash: e6415c4892, Model: realisticVisionV20_v20, Clip skip: 2, Version: v1.9.3-13-g8e355f08

Given FG and light map, Genereate BG and relight [Img2Img]
Img2Img input image is lightmap. After you select value from the radio, the img2img input will automatically be set by the extension. You can also upload your own lightmap by selecting Custom.

Infotext:
sunshine from window, a beautiful girl, absurdres, highres, (masterpiece:1.2), (best quality, highest quality),
Negative prompt: (lowres, low quality, worst quality:1.2), (text:1.2), watermark, (frame:1.2), deformed, ugly, deformed eyes, blur, out of focus, blurry, deformed cat, deformed, photo, anthropomorphic cat, monochrome, photo, pet collar, gun, weapon, blue, 3d, drones, drone, buildings in background, green
Steps: 20, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 2, Seed:<PHONE_NUMBER>, Size: 512x768, Model hash: e6415c4892, Model: realisticVisionV20_v20, Denoising strength: 0.96, Clip skip: 2, Version: v1.9.3-13-g8e355f08

Given FG and BG, Put FG on BG and relight [Txt2Img][HR available]

Infotext:
a beautiful girl, absurdres, highres, (masterpiece:1.2), (best quality, highest quality),
Negative prompt: (lowres, low quality, worst quality:1.2), (text:1.2), watermark, (frame:1.2), deformed, ugly, deformed eyes, blur, out of focus, blurry, deformed cat, deformed, photo, anthropomorphic cat, monochrome, photo, pet collar, gun, weapon, blue, 3d, drones, drone, buildings in background, green
Steps: 25, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 2, Seed:<PHONE_NUMBER>, Size: 512x768, Model hash: e6415c4892, Model: realisticVisionV20_v20, Clip skip: 2, Version: v1.9.3-13-g8e355f08

[2024-06-06] Restore Detail
Detail transfer was originally implemented in https://github.com/kijai/ComfyUI-IC-Light. It captures high frequency details, e.g. text, in the input fg image and reapplys them to the output image.

Original output:

After detail restore:

Infotext:
A bottle of oyster sauce, kitchen counter, absurdres, highres, (masterpiece:1.2), (best quality, highest quality),
Negative prompt: (lowres, low quality, worst quality:1.2), (text:1.2), watermark, (frame:1.2), deformed, ugly, deformed eyes, blur, out of focus, blurry, deformed cat, deformed, photo, anthropomorphic cat, monochrome, photo, pet collar, gun, weapon, blue, 3d, drones, drone, buildings in background, green
Steps: 20, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 2, Seed:<PHONE_NUMBER>, Size: 768x768, Model hash: e6415c4892, Model: realisticVisionV20_v20, Denoising strength: 0.96, Clip skip: 2, Version: v1.9.3-13-g8e355f08

[2024-06-06] Reinforce FG
A big problem of IC-Light is that it often alters the FG object's base color too much. By adding the fg image on top of the lightmap, this issue can be alleviated. This essentially implements this comfyui workflow.
Here is a comparison:
Without reinforce-fg: You can observe that the fg object almost looks transparent.

With reinforce-fg (Same seed): The fg object no longer look transparent.

Skip remove bg
If the default remove bg cannot achieve your desired effect, you can use other tools to create an RGBA image and uncheck the remove bg checkbox. Image with grey background can also be used the same way as RGBA image.

img2img input can be used by other extensions such as ControlNet

Hence why doesn't it make more sense when the img2img input is your subject?
What use is sending a black and white gradient to ControlNet anyway?

img2img input can be used by other extensions such as ControlNet

Hence why doesn't it make more sense when the img2img input is your subject?
What use is sending a black and white gradient to ControlNet anyway?

If you make people use img2img input as fg input, they might think the fg image will be passed to ControlNet when "use independent control" checkbox is not checked.
I think swapping the images is just too risky. It also introduces more mental effort when the user compares the ComfyUI impl and Forge/A1111 impl.

the fg image will be passed to ControlNet when "use independent control" checkbox is not checked

Yes. That's my intention. The img2img input, is, well, the input. The lightmaps is used by the Extension, so it belongs to the Extension instead.
When you generate a portrait in txt2img, and click Send to img2img, where does it send the image to? That's right, the img2img input.
Again, this just makes more sense imho
And I revert your revert again in my own fork

swapping the images is just too risky

Honestly, the only reason why I did the swapping approach is cause I don't know how the original model works. So I just swap them early to continue with your implementaion. If IC Light can take the images directly, then swapping wouldn't be needed.

swapping the images is just too risky

Honestly, the only reason why I did the swapping approach is cause I don't know how the original model works. So I just swap them early to continue with your implementaion. If IC Light can take the images directly, then swapping wouldn't be needed.

I think the txt2img/img2img design is the core problem. A1111 does not allow you launch img2img task without an input image, so the lightmap upload cannot be in the extension accordion.
In ComfyUI it is much more clear that txt2img is just setting initial latent as random noise.

the fg image will be passed to ControlNet when "use independent control" checkbox is not checked

Yes. That's my intention. The img2img input, is, well, the input. The lightmaps is used by the Extension, so it belongs to the Extension instead.
When you generate a portrait in txt2img, and click Send to img2img, where does it send the image to? That's right, the img2img input.
Again, this just makes more sense imho
~And I revert your revert again in my own fork~

How about we make it a checkbox in the settings so that users can choose which behaviour they want? This should also free you maintaining a separate fork.

Yeah, that would probably be the better idea

Installed the latest version of ic-light, the model selection box cannot be opened, the model cannot be selected, the production map is invalid.


MC move refactoring

We need to figure out a good balance between between OOPing and code readability/extensibility.
Currently, we have a pretty significant hierarchy of inherited classes in terms of the MCMC moves.
The basic MetropolisDisplacementMove inherits from MetropolizedMove which inherits from MCMove, which inherits from StateUpdateMove .  While some hierarchy is definitely going to be necessary, this might be a bit excessive from a code readability standpoint (even if it will ultimately reduce the amount of code needed for each move).
The MCMove class seems to sketch out a reasonably clear structure (I'll note isn't necessarily followed in the moves that build upon the  MetropolizedMove class), as a starting point.
In MCMove there are a few key functions already in there:

apply_move
compute_acceptance_probability
accept_or_reject

I'll note the MCMCSampler class expects moves to have a run function, so adding that into the MCMove class would likely be a good plan (where run doesn't necessarily change much between different routines, as it essentially is calling a function that performs a the move n times).
In terms of a generally applicable and easy to extended structure...lets start from the run function for an MC displacement move.

run : this will call an _apply function (which actually performs an MC move).  This should be general for the MCMoves, effectively just calling _apply the number of times moves specified.
_apply_move: this can likely be made generic for the MC moves.  As a basic pseudocode sketch:

make a copy of the original state and the calculate the reduced potential of the original state
call a _generate_new_state function  (this will need to be an abstract function defined in each move)
calculate  the reduced potential of the new state
call compute_acceptance_probability (again, an abstract function to be defined in each move...this will need to accept the original and proposed sampler_state and associated reduced potentials)
call accept_or_reject based on computed acceptance probability

This approach would only require a user to define compute_acceptance_probability  and _generate_new_state.  It would make the acceptance criteria much clearer (it should not require many logic statements with regards to checking the thermodynamic state variables, unlike the current apply function in the MetropolizedMove function).
New MCMove key functions:

run -- this is the key public function that should be generic for all MC move routines
_apply_move -- this should be generic for all MC move routines
_generate_new_state -- this will need to be defined for each move
_compute_acceptance_probability -- this will need to be defined for each move
_accept_or_reject -- this should be generic and work for each move
statistics -- stores number of accepted and proposed moves

For functions where we want to allow dynamics updates to parameters, such as the maximum volume scaling or maximum displacement to achieved a target acceptance probability, this likely can happen within the _generate_new_state function (before the state is actually generated).
I think best course of action will be to do this in a separate PR after #14 and #19 are merged.

Current plan will be do refactoring in #21 (as it includes new reporter changes) and skip merging of #14 .

Effectively, we want to implement  this for each MCMCMove:
(proposed_sampler_state, log_proposal_ratio,  proposed_thermodynamic_state) = self._propose(current_sampler_state, current_thermodynamic_state)  # log proposal ratio + proposal sampler state
current_reduced_pot = current_thermodynamic_state.get_reduced_potential(current_sampler_state)
proposed_reduced_pot = proposed_thermodynamic_state.get_reduced_potential(proposed_sampler_state)
decicion = self._accept_or_reject(current_reduced_pot, proposed_reduced_pot, log_proposal_ratio, method="metropolis",  # including the log acceptance ratio
if decicion:
     self._replace_states(proposed_sampler_state, proposed_thermodynamic_state)

Below I sketched out things a little more in detail.  I think we agreed to name the main public function update and refer to each step as _step (but please correct me if I'm wrong).  This will still need some work to streamline space operations (neighbor list updating and wrapping functions) but that can be fleshed out as code is written
A few questions:

should _step return a new sampler/thermo state or just update the ones passed?  I think it either is fine, but I think if we update the ones that are passed, we should make sure to make a copy of the initial states that are passed, in case we want to just, say, reject the entire move (that is, if we run for 500 displacement steps, reject the whole trajectory).

we should probably define MCMCMove as an ABC, so we can ensure things such as _propose are abstract methods (and similarly, update as an abstract method in MCMCMove).  I don't think this API is so complex that it would be hard to enforce the API without defining it as ABC, but probably still in the interest of best practices and will make it really clear what needs to be defined to make things work well with the Sampler.

class MCMove(MCMCMove):
    def __init__(self,
        atom_subset: Optional[List[int]] = None,
        nr_of_moves: int = 100,
        reporter: Optional[_SimulationReporter] = None,
        report_frequency: int = 1,
        method: str = "metropolis",
        )
        """
        Initialize the move  stats

        Parameters
        ----------
        atom_subset : List[int], optional, default=None
            List of atom indices to apply the move to. If None, the move will be applied to all atoms.
        nr_of_moves : int, optional, default=100
            Number of times to apply the move.
        reporter : _SimulationReporter, optional, default=None
            Reporter to save the simulation data.
        report_frequency : int, optional, default=1
            Frequency with which to report the simulation data.
        method : str, optional, default="metropolis"
            Methodology to use for accepting or rejecting the proposed state.
        """
        self.nr_of_moves = nr_of_moves
        self.method = method

    def update(self, sampler_state: SamplerState, thermodynamic_state: ThermodynamicState, nbr_list: Optional[PairsBase] = None):
        """
        Perform the defined move and update the state.
        """

        calculate_current_potential = True
        for i in range(self.nr_of_moves):
            self._step(self.sampler_state, self.thermodynamic_state, calculate_current_potential)
            calculate_current_potential = False

    def _step(self, current_sampler_state, current_thermodynamic_state, calculate_current_potential):

        # if this is the first time we are calling this,
        # we will need to recalculate the reduced potential for the current state
        if calculate_current_potential:
            current_reduced_pot = current_thermodynamic_state.get_reduced_potential(current_sampler_state)
        else:
            current_reduced_pot = self._current_reduced_pot

        # propose a new state and calculate the log proposal ratio
        # this will be specific to the type of move
        # in addition to the sampler_state, this will require/return the thermodynamic state
        # for systems that e.g., make changes to particle identity.
        proposed_sampler_state, log_proposal_ratio,  proposed_thermodynamic_state = (
            self._propose(current_sampler_state, current_thermodynamic_state))

        # calculate the reduced potential for the proposed state
        proposed_reduced_pot = proposed_thermodynamic_state.get_reduced_potential(proposed_sampler_state)

        # accept or reject the proposed state
        decision = self._accept_or_reject(current_reduced_pot, proposed_reduced_pot, log_proposal_ratio, method=self.method)

        if decision:
            # stash the reduced potential of the accepted state so we don't have to recalculate it the next iteration
            self._current_reduced_pot = proposed_reduced_pot

            # replace the current state with the proposed state
            # not sure this needs to be a separate function but for simplicity in outlining the code it is fine
            # or should this return the new sampler_state and thermodynamic_state?
            self._replace_states(current_sampler_state, proposed_sampler_state, current_thermodynamic_state, proposed_thermodynamic_state)

        # a function that will update the statistics for the move
        self._update_statistics(decision)

    def _propose(self, current_sampler_state, current_thermodynamic_state):
        """
        Propose a new state and calculate the log proposal ratio.

        This will need to be defined for each move

        Parameters
        ----------
        current_sampler_state : SamplerState, required
            Current sampler state.
        current_thermodynamic_state : ThermodynamicState, required

        Returns
        -------
        proposed_sampler_state : SamplerState
            Proposed sampler state.
        log_proposal_ratio : float
            Log proposal ratio.
        proposed_thermodynamic_state : ThermodynamicState
            Proposed thermodynamic state.

        """
        pass

    def _replace_states(self, current_sampler_state, proposed_sampler_state, current_thermodynamic_state, proposed_thermodynamic_state):
        """
        Replace the current state with the proposed state.
        """
        # define the code to copy the proposed state to the current state

    def _accept_or_reject(self, current_reduced_pot, proposed_reduced_pot, log_proposal_ratio, method=method):
        """
        Accept or reject the proposed state with a given methodology.
        """
        # define the acceptance probability

Since _propose will return the log_proposal_ratio, we do not need a separate call to get the proposed reduced potential.  That should be within _propose.  Similarly, this should return the proposed_reduced_potential and accept the current_reduced_potential (again, keeping track of these is simply to avoid having to calculate again). Also, as noted in #24 we should have this return sampler/thermodynamic states, rather than modifying in place.
def _step(self, current_sampler_state, current_thermodynamic_state, calculate_current_potential):

        # if this is the first time we are calling this,
        # we will need to recalculate the reduced potential for the current state
         if calculate_current_potential:
            current_reduced_pot = current_thermodynamic_state.get_reduced_potential(current_sampler_state)
        else:
            current_reduced_pot = self._current_reduced_pot

        # propose a new state and calculate the log proposal ratio
        # this will be specific to the type of move
        # in addition to the sampler_state, this will require/return the thermodynamic state
        # for systems that e.g., make changes to particle identity.
        proposed_sampler_state, log_proposal_ratio,  proposed_thermodynamic_state, proposed_reduce_potential = (
            self._propose(current_sampler_state, current_thermodynamic_state, current_reduced_potential))

        # accept or reject the proposed state
        decision = self._accept_or_reject(current_reduced_pot, proposed_reduced_pot, log_proposal_ratio, method=self.method)

        if decision:
            # stash the reduced potential of the accepted state so we don't have to recalculate it the next iteration
            self._current_reduced_pot = proposed_reduced_pot

            return proposed_sampler_state,  proposed_thermodynamic_state

In refactoring, a few other things have come up.  New moves will only require users to define three functions:

_propose  - proposes the change to the sampler or thermodynamic state.  this is required for each move and thus gets the abstract method decorator in the MCMCMove class
_report - This will prepare quantities and pass them to the reporter.  Right now I've only put this as an abstract method in the MCMove (which is a child of MCMCMove);  the Langevin code actually does have a _report function but the constructor is different given the way dynamics moves are implemented as compared to defining a _step in the MC moves.
Having this defined in each move will  ensure the appropriate quantities are reported, as what we are interested will depend on the type of move.  Again, this is just for individual moves that will mostly be used for debugging, so I don't want to stress too much about figuring out a more generic solution (and I'll note, being in a separate function will help ensure _propose function is a little cleaner, as it won't have extra code related to logging).
_adapt_move_parameters - I don't think this is the best name, but the general idea is this is the code that will adapt, e.g., step size in a displacement move or maximum volume scaling to achieve a target probability.   This will be an abstract method as part of the MCMove class.  If an mc move has no criteria to update to achieve a given acceptance probably, it can just be defined with debug warning that it does nothing.  This is not yet implemented so the exact details may change.

should _step return a new sampler/thermo state or just update the ones passed? I think it either is fine, but I think if we update the ones that are passed, we should make sure to make a copy of the initial states that are passed, in case we want to just, say, reject the entire move (that is, if we run for 500 displacement steps, reject the whole trajectory).

As we mentioned in #24 and #23 , I think the idea will be to have functions accept and return sampler and thermodynamic states (and any other associated parameters we might want).
Two place where we don't follow this:
1- current the MCMCSampler itself just updates in place.  This should probably be changed, too.  This will require a little refactoring, since here the constructor takes the sampler/thermo state, as opposed to the run function.
2- Neighborlist is modified in place.  I think we already discussed the idea of moving this into the sampler state to minimize the amount of stuff we need to pass around.  This might be more efficient for some of the MC moves, because if we reject a move, it would basically automatically revert back, even if we had to rebuild the neighbor list in the proposed_state.  I just need to think about a bit what is the easiest way to implement this (since neighbor list depends on the sampler state itself) (can probably wrap this so we don't have to explicitly pass the sampler state, just like sampler_state.build_nbr_list() and sampler_state.check() )

I changed the code around such that the neighbor list is returned along with sampler states and thermodynamic state.  One place where this might be useful is in the case of a rejected MC move, since we can just step back to the prior neighbor list state, rather than have to do another check to make sure that the neighbor list modified by the proposed state still works.
I left the MCMC Sampler in place, as we will need to discuss that in the context of the multistate sampling.
The PR should be ready to go.


test: add uts for cni add function

Reason for Change:

Adds unit tests to test that endpoint info struct creation is as expected based on args, cns/ipam invoker response, and network configuration for the following scenarios:

swiftv2 linux ipam response (infra + delegated) -> endpoint info
swiftv2 windows ipam response (infra + delegated) -> endpoint info
swiftv1 windows multitenancy dual nic ipam response  (infra + infra) -> endpoint info
swiftv1 linux multitenancy (infra) ipam response -> endpoint info

Each scenario tests the Add method from ipam response until we call an EndpointCreate, given args, ipam response, and network configuration and ensure the endpoint info struct generated exactly matches what is expected (so all fields are checked) except for the interface name and endpoint id as they are nondeterministic identifiers generated during runtime.
Adds a check to ensure each endpoint info's Option and Data fields are separate.
Adds a check to ensure swiftv2 linux's endpoint info interface name is overwritten with the correct interface name.
Issue Fixed:

Requirements:

[X] uses conventional commit messages

[ ] includes documentation
[X] adds unit tests
[X] relevant PR labels added

Notes:
This does not add tests for swiftv2 ib. It also does not explicitly add tests for swiftv1 non multitenancy scenarios as they are mostly covered by the multitenancy ones (singletenancy is just a single ipam response). This does not test anything that happens inside EndpointCreate.

/azp run Azure Container Networking PR


SimpleMemoryBackend() is counter-intuitive

The current implementation of SimpleMemoryBackend() always returns a reference to the same class when callers might reasonably expect that it creates a new cache each time.   This confusion has lead to a bug in one of our libraries that uses SimpleMemoryBackend.  It would be great if it were possible to instead create multiple caches with the constructor.

So I've been thinking about this for a while. The original design of this library was to provide a caching framework with an interface that allows you to swap backend if needed so you expect the functionality to be the same regardless if you are using memcached, redis or memory. The rationale I followed when I did this for the memory backend is to treat it also as a remote server where you would be storing the data and if you need to differentiate you would use the namespace arg like you do with Redis and Memcached. So I prefer to keep this behavior. I'll add some explanation in the docstring of the class and will leave this open in case someone wants to chime in for more discussion on the topic


Cannot  from vim (either to vim or to tmux) when TERM is tmux-256color

I have tmux-256 color set up so I can have italics but it breaks this :(

Hey @RatanRSur, sorry for the delayed response here! Can you confirm if you've tried the neovim <C-h> fix in the readme?
Beyond that, I don't know if there is anything we can do. I personally run with screen-256color as my default-terminal setting, and assuming we can't resolve this, we may just need to add a note making that explicit.


Hide zoho button on drops page, better handle joining agreement, show error if empty agreement

:tada: This PR is included in version 1.28.0-dev.38 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.28.1-dev.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.28.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Old PCL version detected. Please update PCL to 1.11 or later

My ROS version is Noetic. Even though I have specified PCL version 1.14 in the CMakeLists, I am still encountering an error with RegistrationPCL stating "Old PCL version detected. Please update PCL to 1.11 or later."

Sorry, my bad. I installed PCL in the default directory.


Fetch ReDoc js file when building hab package rather than rely on CDN.

Fix for #2855.
I had originally planned to use redoc-cli to package the docs, but that had a few downsides:

The combined HTML + js file didn't work with hugo the way we're doing things currently, and I don't understand hugo enough to rework things to get this to work.
This would have added npm to the docs development process and package building, which seems a bit onerous.
While trying to get this going there was work done that would have conflicted and needed to be redone (#2902).
Some other smaller concerns that can basically be summed up as redoc-cli doesn't do much to justify depending on it.

This still relies on the CDN, but would move potential failures to package build time rather than when a customer views the docs, which seems like a much better trade off.

What impact does this have on the loading time for the api docs, if any?

I haven't measured, but it should be slightly faster not having to contact the CDN every time the docs loads.

I just tried loading the API docs at chef.automate.io with the firefox inspector open and downloading redoc took 12ms, so this probably won't have much effect.

If it isn't slower, then it must be faster :-D

This means one fewer network request, so it definitely should not be slower.


feat: Add Azure Key Vault Variable Provider

This PR adds Azure Key Vault as Variable provider.
Users can provide a runtime config using necessary information for Client Credential Flow:
[[config_provider]]
type = "azure_key_vault"
vault_url = "https://some.vault.azure.net/"
client_id = "11111111-1111-1111-1111-111111111111"
client_secret = "11111111-1111-1111-1111-111111111111"
tenant_id = "11111111-1111-1111-1111-111111111111"
# authority_host = "AzurePublicCloud"

If not specified, authority_host will default to AzurePublicCloud.
Unfortunately, I wasn't able to run make lint and make test on my machine. Both commands ran into errors shown below.
Am I missing something on my machine?
Output from make lint
    Checking spin-expressions v2.5.0-pre0 (/Users/thorsten/dev/thorstenhans/spin/crates/expressions)
error: you are explicitly cloning with `.map()`
  --> crates/plugins/src/badger/store.rs:89:26
   |
89 |             badgered_to: to.iter().map(|v| <semver::Version>::clone(v)).collect(),
   |                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: consider calling the dedicated `cloned` method: `to.iter().cloned()`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#map_clone
   = note: `-D clippy::map-clone` implied by `-D warnings`
   = help: to override `-D warnings` add `#[allow(clippy::map_clone)]`

    Checking spin-outbound-networking v2.5.0-pre0 (/Users/thorsten/dev/thorstenhans/spin/crates/outbound-networking)
    Checking spin-doctor v2.5.0-pre0 (/Users/thorsten/dev/thorstenhans/spin/crates/doctor)
    Checking spin-build v2.5.0-pre0 (/Users/thorsten/dev/thorstenhans/spin/crates/build)
    Checking spin-templates v2.5.0-pre0 (/Users/thorsten/dev/thorstenhans/spin/crates/templates)
error: could not compile `spin-plugins` (lib) due to 1 previous error
warning: build failed, waiting for other jobs to finish...
error: could not compile `spin-plugins` (lib test) due to 1 previous error
make: *** [lint] Error 101

Output from make test
Checking spin-key-value v2.5.0-pre0 (/Users/thorsten/dev/thorstenhans/spin/crates/key-value)
error: you are explicitly cloning with `.map()`
  --> crates/plugins/src/badger/store.rs:89:26
   |
89 |             badgered_to: to.iter().map(|v| <semver::Version>::clone(v)).collect(),
   |                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: consider calling the dedicated `cloned` method: `to.iter().cloned()`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#map_clone
   = note: `-D clippy::map-clone` implied by `-D warnings`
   = help: to override `-D warnings` add `#[allow(clippy::map_clone)]`

    Checking outbound-http v2.5.0-pre0 (/Users/thorsten/dev/thorstenhans/spin/crates/outbound-http)
error: could not compile `spin-plugins` (lib) due to 1 previous error
warning: build failed, waiting for other jobs to finish...
error: could not compile `spin-plugins` (lib test) due to 1 previous error
error: field `0` is never read
   --> crates/templates/src/template.rs:649:21
    |
649 |     struct TempFile(tempfile::TempDir, PathBuf);
    |            -------- ^^^^^^^^^^^^^^^^^
    |            |
    |            field in this struct
    |
    = note: `-D dead-code` implied by `-D warnings`
    = help: to override `-D warnings` add `#[allow(dead_code)]`
help: consider changing the field to be of unit type to suppress this warning while preserving the field numbering, or remove the field
    |
649 |     struct TempFile((), PathBuf);
    |                     ~~

error: could not compile `spin-templates` (lib test) due to 1 previous error
make: *** [lint] Error 101

If you rebase against the latest main branch, the errors you are seeing should be fixed. They were address here.

Looks great overall! I did not actually run the code, but it looks pretty straight forward to me. I think we can merge this. The only points I had were nitpicks.

Awesome feedback @rylev I refactored the PR as part of 0a3e792
When implementing FromStr, I ended-up using bail! to fail early. Is that considered a valid practice here, or is there a better alternative?


feat: add session sigs formatter/reader

Description
Log formatted session sigs out to check expiration dates for the outer & inner auth sigs.

❤ This is nice


Scala 3.5.0 support

Description

Added artifacts for Scala 3.5.0 version
Updated tests that are dependent on the chosen scala version

(Merge after PR #1603)
Motivation
There are newer versions of Scala that can be supported.

@liucijus @simuons could you take a look at this? Thank you :)

@liucijus conflicts are resolved now :)

@liucijus can you please help with merging & maybe releasing this PR?


Disable RC4 in TLS per RFC7465 (chacha20 possible replacement) !!!

This document requires that Transport Layer Security (TLS) clients and servers never negotiate the use of RC4 cipher suites when they establish connections.  This applies to all TLS versions.
An alternate cipher for low powered/cost hw is chacha20-poly1305 that could provide up to 2x performance for any x64 capable device.

Modify files:
/src/Cedar/Server.c
Line 132:
#define SERVER_DEFAULT_CIPHER_NAME "AES128-GCM-SHA256"
/src/Mayaqua/Network.c
Line 236:
static char *cipher_list = "AES128-GCM-SHA256 AES128-SHA256 AES256-GCM-SHA384 AES256-SHA256 DHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES128-SHA256 DHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES256-SHA256 ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES256-GCM-SHA384 ECDHE-RSA-AES256-SHA384";
Recompile, instructions in BUILD_UNIX.TXT
Install:
cp bin/vpnserver/hamcore.se2 /opt/vpnserver
cp bin/vpnserver/vpnserver /opt/vpnserver
Good luck

Solved by #343.


Implement ES template literals

It would be great to support this syntax in AskScript:
`This is a ${variable} text.`

More on template literals: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals

I'd vote for implementing just the template literals, without tags (see the link to MDN).
Thoughts?

@czerwinskilukasz1 yeah, we can add tags functionality later if we need them


Describe the order of hooks

The order is probably as listed here but would be nice to read it explicitly.

in what order do field vs collection vs global hooks run?
https://payloadcms.com/docs/hooks/overview

Hey @thgh thanks for the request! Docs are actually written into Payload core here. We retrieve these files from GitHub and render them onto the website. But to answer your question, hooks are fired in the order they appear in the docs, which is:

beforeOperation
beforeValidate
beforeChange
afterChange
beforeRead
afterRead
beforeDelete
afterDelete

If you want to make a contribution to the docs, feel free to open a PR here.


Issue with bash aliases and variants

Environment

Linux Centos 7.9
2.114.0
Rez python version: Python 3.9.6

To Reproduce

Add Python variants to your package.py with bounds i.e. variants = [["python-2.7+<3"], ["python-3.7+<4"]]
Add an alias to a binary script i.e. alias("do-something", "{root}/bin/do_something.py")

Expected behavior
Running do-something will resolve and run the do_something.py script.
Actual behavior
Error of the form:
bash: 4/bin/do_something.py: No such file or directory

When seeing what function Rez defines:
> declare -f do-something
do-something ()
{
    python /path/to/some_package/0.1.0/python-3.7+ "$@" < 4/bin/do_something.py
}

So it appears that the resolved script path is not correctly escaped.

Thanks for reporting this! This kind of problem can be fixed by using hashed variants. Right now it has to be done on a per package basis but there is plans to add a global config setting and make it the default.

But I think this particular issue you are reporting would warrant some more some investigation. It feels kind of weird that the path was split in half and "@" was inserted in the middle... I would have expected
python /path/to/some_package/0.1.0/python-3.7+<4/bin/do_something.py "$@"

which would have still been problematic , but would be more "correct".


chore: added SwiftPM support

Very useful, thank you!

No description provided.

Very useful, thank you!


fix composer classname collisions with already published migrations

if you already published the twill migrations in an existing twill project, you will get composer classname collisions.
PHP Fatal error:  Cannot declare class AddLocaleColumnToTwillMediables, because the name is already in use
this PR applies TwillDefault to the class and filename.

Can we please have this merged? The bug this is supposed to fix is currently breaking deployments, requiring ugly hacks in production.

Fixed in https://github.com/area17/twill/commit/74173acd0b750243c2d3bab382ed842f36efdce9.


Release v3.0.3

Suggested changes are in this PR: https://github.com/VirtusLab/git-machete-intellij-plugin/pull/1197


Add support for Python 3.7

closes #226

Hey @pdecol this is awesome, thank you!  The test failure is because Travis CI's trusty distro doesn't know 3.7 is out of development; if you remove this line from the .travis.yml file, it should work:
https://github.com/iotaledger/iota.lib.py/blob/master/.travis.yml#L1

Thanks! All tests passed

Awesome!  I'll have a look through the changes this week!

Sure. It is good to have things consistent and up to date. I updated the pull request. Thanks for your review!

Looks good!  Thanks @pdecol!


Add support for new acl types.

Without this patch you get lots of warning from atmos.vipronline.com as it seems there are new
permission bits implemented in newer versions of the atmos protocol.

mvwieringen - Are you seeing these warning through the use of the rest client or the irods plugin?
Code inspection looks good. I just want to verify that the enumerated permissions are correct.
Thanks for your contributions

I see it using the check_atmos check program and with our storage backend driver. As irods doesn't
ring a bell I guess it must be the rest client.


Smash apart the workspace into more focused crates

TODO:

[x] Ensure each commit builds, updates lockfile correctly and is formatted
[x] Extract test suite into dedicated crate
[x] Extract a tokio-tasks crate for the Tasks and tokio_ext stuff
[x] Extract xtras crate
[x] Write up document explaining crate structure and incentives?

Can only comment because it is my PR but consider this an ACK!

bors r+

bors r-
There are fixups in the history and they are not detected :(

bors r+


Virtual WAN P2S Multi App for different user groups

This document describes exactly what we needed, to give different user groups access to VPN (preferrably access to different resources on VPN as well). We imagined this could be done using multi app and assigning different client ip ranges for the different user groups, and use firewall rules to grant user groups different access.
However after looking more into this I'm really unsure how this is meant to actually work. That it is possible to define multiple applications for different user groups is all great, but one gateway can only use one app, and one hub can only have one gateway in a region.. So to actually use different apps for different user groups we have to create multiple hubs in different regions to assign them different apps? That sounds very unpractical.. So feature to create a separate app for a specific user group works, but I don't see how to use multiple apps without creating a new gateway in a new region.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 3bf25f7f-2cb4-6699-b31c-6ee5b5784910
Version Independent ID: c6f3348a-06ff-fcaa-e173-9f801b853478
Content: Virtual WAN: Azure AD tenant for different user groups: Azure AD authentication
Content Source: articles/virtual-wan/openvpn-azure-ad-tenant-multi-app.md
Service: virtual-wan
GitHub Login: @cherylmc
Microsoft Alias: alzam

@tesharp Thanks for the feedback! Unfortunately this is a current limit with Azure WAN and AAD authentication. If you would like us to add more applications per gateway in the future, please leave your feedback at Azure Networking's feedback.azure.com page. This will allow the community to upvote and for the product team to include into their plans.
We also have the "Give Product Feedback" button in our documents now to take you directly to the appropriate feedback page.
We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.


Bugfix for issue #153

https://github.com/django-silk/silk/issues/153
cProfiler now located within thread local storage.

@Drache91 Thanks for this contribution!


Avoid collapsible body sliding down for elements into the collapsible header

Is it possible to do this, currently?
I have a button into the collapsible header which opens a modal, but it also opens its respective collapsible body. How can I avoid this behaviour and only open the modal?
Thanks in advance

https://api.jquery.com/event.stoppropagation/


MLPack chooses params that violate boundary conditions

Minimal code example:
import xacc
import numpy as np
lower_bounds = np.zeros(2)
upper_bounds = np.ones(2)
# Creating a set of initial parameters that
 # *should* be in violation of the boundary conditions
# provided above, expecting the opt. to throw an error.
initial_params = np.ones(2) * -1.
# Using the adam algorithm for this example, but
# bug occurs for other mlpack optimizers as well
algoName = 'adam'
 m_options = {
"algorithm": algoName,
"lower-bounds": lower_bounds.tolist(),
"upper-bounds": upper_bounds.tolist(),
"initial-parameters": initial_params.tolist(),
"maxeval": 500}
optimizer = xacc.getOptimizer("mlpack", m_options)
# Creating a fake function that returns (1 - iterable)
  # and a gradient of (1-iterable)**2 to simulate fake
 # data returns. We'd *expect* this optimization to terminate
# before even running due to the violation of boundary conditions
# with our initial set of parameters. However, it keeps pushing
 # the values of x well lower than the min. boundary of 0.0.
  itr = 0.
 def foo(x):
global itr
print(x)
itr += 0.1
return (1. - itr), [(1. - itr)*(1. - itr),(1. - itr)*(1. - itr)]
optimizer.optimize(foo, 2)

I've quickly checked this and indeed we don't support generic lower-bounds and upper-bounds in mlpack.
One difficulty is that it looks like not all optimizers in mlpack support boundary conditions.
e.g. the Adam optimizer doesn't have any parameters for boundary conditions:
https://ensmallen.org/docs.html#adam
@amccaskey what do you think?


Pipelines API review issues

[x] Readonliness of ReadableBuffer
[x] Use IMemoryList
[ ] Making pipes implementation of MemoryPool hidden
[ ] Move Write* extension methods to a single class, change them to take ReadOnly*
[ ] Move ReadableBuffer/Cursor->Position to BufferPrimitives
[x] Should we keep ReadableBuffer.Preserve? No
[x] Should we keep WriteableBuffer.Append? No, it includes lifetime management.
[ ] Operators on cursors/positions - Remove
[x] Return default + false from ISequence and remove End?.
[x] Change IOutput.Span to GetSpan (as it can be expensive)

Things that are to be moved to BCL:

[x] ValueAwaiter
[x] IScheduller

This is not the full list, it would be added to during upcoming API reviews
/cc @davidfowl @KrzysztofCwalina @halter73 @muratg

cc @joshfree

Here are some others I would like to add:
[ ] I would make the enumerators nested types in Readable/Writable Buffers. Today, the enumerator can be shared between these two types, but once ReadableBuffer gives out only ReadOnlyMemory, the buffers will need separate enumerators.
[ ] Why do we need IPipe, IPipeConnection, IReadableBufferAwaiter, IWritableBufferAwaiter interfaces? Nothing consumes them. Can we just have Pipe class?
[ ] I find Advance with two cursors confusing. We talked about having simplified API where if consumed is empty, then examined is all and if consumed is not empty, then examined is the same as consumed. Remind me what scenarios are not covered by this?
[ ] MaximumSizeHigh and MaximumSizeLow are not very self explanatory. Can we find better names?
[ ] Do we really need preserved buffer? It's a whole new type (not polymorphic with ReadableBuffer) just so we can dispose it?
[ ] IsSingleSpan should be renamed to IsSingleSegment.
[ ] We should use ReadOnly as a prefix for ReadableBuffer. We use ReadOnly all over the BCL
[ ] We should remove RB.Create(byte[], int, int). We should use Memory or Span for those.
[ ] Should all the awaitables be moved to .Threading subnamespace?
[ ] Cursor needs to be unified with Position
[ ] When we move ReadableBuffer to System.Memory, it should have a create method that takes Memory (and fish out OwnedMemory or byte[])

I find Advance with two cursors confusing. We talked about having simplified API where if consumed is empty, then examined is all and if consumed is not empty, then examined is the same as consumed.
Remind me what scenarios are not covered by this?

Header parser only consumes header when it is completely received but examines everything.
So for the following incomplete data:
                      v Consumed
Host: http://localhost
Accep
     ^Examined

@KrzysztofCwalina
I would also like to remove CopyTo from IMemoryList, I think it's not very useful in current state of things when no-one would get access to IML directly.

Is there a simple basic sample how to set up the pipeline and use it for basic (even inefficient) scenario? Then for advanced scenarios? I assume it will be needed for the API review anyway ;)

Before we go deleting all of the abstractions there was this very early plan to figure out how to expose the pipe interface on our existing body streams so that we could expose them in ASP.NET:
https://github.com/aspnet/AspLabs/issues/2
I'd like to see a new plan for this with the suggested API changes here (if we remove the abstractions)

@davidfowl if we implement await pipe.Writer.ReadStarted we can just run loop that pipes bytes from body stream to body pipe every time someone sets the body stream without wrapping anything.

@davidfowl if we implement await pipe.Writer.ReadStarted we can just run loop that pipes bytes from body stream to body pipe every time someone sets the body stream without wrapping anything.

I'm not concerned about ReadStarted. I'm concerned about implementing IPipeReader on the request body stream and IPipeWriter on the response body stream. How do we fix that after we remove the interfaces? Expose new properties that don't work well with the existing ones?

We don't, we copy from stream to pipe if someone reads from pipe.

That defeats much of the purpose.

@davidfowl isn't it temporary?

Has there been any consideration about bringing back WritableBuffer.BytesWritten as mentioned in #1838?
Being able to support the scenario described in that issue is important for a lot of protocols.

Agreed. I think we should bring it back

WritableBufferWriter => BufferWriter that its writable is inferred, what other type of buffer could it be writing to?

Also please suffix all the awaitable methods with Async

@kasper3 which ones aren't?

@KrzysztofCwalina @pakrym We should rename BufferReader.Skip -> Advance

Some open issues regarding pipelines

I had created an issue about changing the MaximumSize{Low,Heigh} semantics: https://github.com/dotnet/corefxlab/issues/1355#issuecomment-295141102

I also wonder what is the proper place to deal with sending part of files using OS sendfile system call: https://github.com/aspnet/KestrelHttpServer/issues/1673.

Is there a way to allow foreign buffers in the pipeline (i.e. not allocated using the pipe's memorypool), to avoid copy like: https://github.com/aspnet/KestrelHttpServer/blob/73a37363e1a44e221f070c530a5b987e118cc7a7/src/Kestrel.Core/Internal/Http/Http1OutputProducer.cs#L166-L188

I had created an issue about changing the MaximumSize{Low,Heigh} semantics: #1355 (comment)

We can look at this again but I'm not sure it can't be added later.

I also wonder what is the proper place to deal with sending part of files using OS sendfile system call: aspnet/KestrelHttpServer#1673.

Not really pipe specific. This is about the core assembly (its at the stream level).

Is there a way to allow foreign buffers in the pipeline (i.e. not allocated using the pipe's memorypool), to avoid copy like: https://github.com/aspnet/KestrelHttpServer/blob/73a37363e1a44e221f070c530a5b987e118cc7a7/src/Kestrel.Core/Internal/Http/Http1OutputProducer.cs#L166-L188

We've removed append so far but the foreign buffer would likely need to be an OwnedBuffer or something similar since buffers can outlive the call to WriteAsync (in the specific example you provided).

@KrzysztofCwalina I'm going to close this out.

@davidfowl Any more info yet regarding https://github.com/dotnet/corefxlab/issues/1972#issuecomment-353739809?

No there's an issue filed for it but I'd rather not turn this issue into a bunch of feature requests. Once the main API is in then we'll review and add individual features.

@davidfowl, sounds good.


Implicitly typed parameters of function properties of generic parameters break autocompletion

Bug Report

🔎 Search Terms

generic parameter autocomplete
extends autocomplete
parameter breaks autocomplete
implicit parameter breaks autocomplete
🕗 Version & Regression Information

This is the behavior in every version I tried, and I reviewed the FAQ for entries about (almost?) everything
I was unable to test this on prior versions because TS v3.3.3 acts function parameters as implicitly any, while 3.5.1 and higher versions(the ones on the playground) correctly get the type as a string.

⏯ Playground Link

Playground link with relevant code
💻 Code

// (Simplified)
interface StepDefinition {
  title: string;
  enabled: true | ((foo: string) => boolean);
}

// To avoid TS from narrowing the keys to 'string' and definitions to just StepDefinition
function defineSteps<T extends Record<keyof T, StepDefinition>>(steps: T) {
  return steps;
}

const STEPS = defineSteps({
  test: {
    title: 'Test',
    enabled: true,
  },
  bar: {
    title: 'Foo Bar',
    enabled: (foo: string) => foo.length > 0, // When this is enabled, baz: { ... } gets auto-completion
    // enabled: () => 'foo'.length > 0, // When this is enabled, baz: { ... } gets auto-completion
    // enabled: (str) => str.length > 0 // When this is enabled, baz: { ... } does NOT get auto-completion
  },
  //baz: {

    // enabled: () => [].concat([ 1 ]).length > 0, // When this is enabled, auto-completion here still works
    // enabled: (foo) => !!foo, // When this is enabled, auto-completion here stops working
  //}
});

// The code below is for context
export type Steps = typeof STEPS;

// The aim is getting the step names and other types while also having IntelliSense when defining the steps
export type StepNames = keyof Steps;
export type ConditionalSteps = {
  [S in keyof Steps]: Steps[S]['enabled'] extends true ? never : S;
}[keyof Steps];

// ...

🙁 Actual behavior

No autocomplete inside defineSteps({ ... }) after defining a function with an implicitly typed parameter in any of enabled properties. TS can't even get the parameter type implicitly on v3.3.3, so my guess is some stuff got improved which helped to infer the type, but there is something slightly wrong under the carpet, which causes this.
🙂 Expected behavior

Getting autocomplete inside defineSteps({ ... }), even when I define a function with an implicitly typed parameter.

Note for future investigators: The magic concept at play here is the "context-sensitive" expression
// enabled: (str) => str.length > 0

which is somehow defeating our logic that gets completions in the face of an incomplete generic call

There are different angles to this issue. I'm not sure if it's worth fixing this but perhaps a fix to this would improve some other, more compelling, scenarios.
First of all, this signature doesn't make much sense to me. The type parameter is defined as T extends Record<keyof T, StepDefinition> but I can't find any reason to actually use such a constraint. If I'm not mistaken this is essentially the same as T extends Record<PropertyKey, StepDefinition> and that works just OK when it comes to property completions within that incomplete object.
While investigating this case we can learn that, in a case without context-sensitive functions, completions are provided based on the signature computed by getCandidateForOverloadFailure. The computed signature is (steps: Record<"test" | "bar" | "baz", StepDefinition>): Record<"test" | "bar" | "baz", StepDefinition>.
So why do context-sensitive functions break this? They make all objects that contain them (recursively, up to the root object) non-inferrable. So when getCandidateForOverloadFailure gets called no inference candidate can be gathered for T. This makes getInferredType return the T's constraint while instantiating T within it to... unknown. keyof unknown is just never so the whole instantiatedConstraint is Record<never, StepDefinition> and the returned candidate for the overload failure is (steps: Record<never, StepDefinition>): Record<never, StepDefinition>.
That is later used to obtain the getApparentTypeOfContextualType for the object in which we want to get completions and getTypeOfPropertyOfContextualType just can't return anything useful. It's essentially trying to do Record<never, StepDefinition>["baz"] and that's an invalid indexed access.
So what could potentially be done about this?

how non-inferrable types could be reworked, it's essentially an implementation detail and this could be changed at anytime. I don't think that computing Record<never, StepDefinition> here for the constraint is the best that can be done. It's rather tricky to improve the situation this way though and that's definitely over my head
when requesting completions you could allow this indexed access - the worst case scenario is that some completions would get returned to the user when they shouldn't. That's rather unlikely but either way - it's usually better to give some completions than none.
typeChecker.getContextualType (maybe only when called with ContextFlags.Completions) could also, somehow, collect contextual types used within checkExpressionWithContextualType that happens while choosing regular overloads. The current problem is that the contextual type is obtained from the resolved signature that failed to resolve correctly. I'm not sure how that would actually play out with overloads and stuff but potentially it actually would be the improvement because right now overloads are sort of ignored in completions requests that use typeChecker.getContextualType. Some completions requests (for "direct" arguments) are using checker.getResolvedSignatureForStringLiteralCompletions today and that call doesn't ignore~ overloads. So in a sense, there is already a discrepancy between what completions might be returned, and it's solely based on the positions within the function's argument.

I'm thinking about this more and more and I have a hunch that there might be something in that third idea. I imagine that each checkExpressionWithContextualType in inferTypeArguments within a completion request could create a new stack of contextual types and pushCachedContextualType could push into it (without popping). Then completions request could query all of those stacks (1 stack per 1 explored overload), find the closest contextual type there, and compute its own contextual type based on the object path from that closes contextual type to the node that the completion request is interested in.
cc @andrewbranch maybe you'd be interested in this... am I completely delusional here or does this sound plausible? 😅


Can create instance and specify settings for it. This allow use different settings in same project.

Sorry, it has bug now, I will fix it.


In .NET Framework 4.5.2, X509Certificate2 does implement IDisposable

I have code that iterates an array of objects that dispose objects that implement IDisposable.   In .net 4.5.2, the certificates do in fact get disposed and all references become invalid.  I did not check prior versions.  The remarks for this class are wrong/misleading and need to be updated.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 28d79da2-23c7-034a-6edd-5968af41c7fc
Version Independent ID: 4ad593fd-ef7c-bea2-bd50-7b569554e66d
Content: X509Certificate2 Class (System.Security.Cryptography.X509Certificates)
Content Source: xml/System.Security.Cryptography.X509Certificates/X509Certificate2.xml
Product: dotnet-api
GitHub Login: @mairaw
Microsoft Alias: mairaw

@michaelmcdaniel, thanks for your feedback. @bartonjs, the .NET Framework 4 documentation on docs.microsoft.com (but not on MSDN) includes Dispose as inherited from X509Certificate. Can you confirm when IDisposable was first added?

IDisposable was added to X509Certificate for .NET Framework 4.6.
Since it's an in-place upgrade with prior 4.x releases, a runtime check will say that it's IDisposable, but at compile-time it can only be used in a using statement when targeting 4.6 or higher.
So, the existing comments are correct, from a compile-time perspective; but a runtime as cares about what version was installed, not what version the compile target was.


In version 0.2.0, file not found

In version 0.2.0, files not found at
https://github.elemecdn.com/highlightjs/
https://github.elemecdn.com/codemirror/CodeMirror

I upgrade latest version fix it


Feature: gravity and alignment

Hi,
Is there a way to specify gravity or alignment in this library? LTR or RTL, gravity center or left etc...
Thanks

Hi,
Did you find a work around for this? I need to control the flow of items, to be from right to left.
Thanks


Upgrade highlight.js to version 9.0.0

Addresses issue https://github.com/jdan/cleaver/issues/110. This change adds syntax highlighting for many languages. See the highlight.js changelog for a full list.
This pull request also updates the github.css theme to the latest version. This is required as the css class names have changed.

Excellent, thank you @omarkhan!
I was working with hljs in another app and was confused why my github.css wasn't working, so thanks for fixing that as well.