device client register method fails with CredentialError caused by UnauthorizedError when attempting to connect to a device on iot central using python sdk.

Context

OS and version used: Linux 4.19.0-xilinx-v2019.2 on PicoZed
Python version: Python 3.7.8
pip version: pip 19.2.3
list of installed packages: azure-iot-device 2.4.0
cloned repo: NA

Description of the issue
We are attempting to check if we can push telemetry data onto a device created in Azure IoT Central by following the tutorial:
https://docs.microsoft.com/en-us/azure/iot-central/core/tutorial-connect-device-python
On running the environmental_sensor.py that we created, we were hit with the following message:
# python3 env_sensor.py
 Device could not connect

We then modified the script to be a sync version of the same using:
https://github.com/Azure/azure-iot-sdk-python/blob/master/azure-iot-device/samples/sync-samples/provision_symmetric_key.py
as reference.
Upon running the environmental_sensor.py, we encountered the following error:
azure.iot.device.exceptions.CredentialError: CredentialError('Credentials invalid, could not connect') caused by UnauthorizedError('Connection Refused: not authorised.')

Code sample exhibiting the issue
Here's the modified script that we used. All it does is register, connect and attempt to send telemetry every 5s - synchronously.
sync_env_sensor.txt
Console log of the issue
~# python3 sync_env_sensor.py
 Traceback (most recent call last):
   File "/usr/lib/python3.7/site-packages/azure/iot/device/provisioning/provisioning_device_client.py", line 25, in handle_result
     return callback.wait_for_completion()
   File "/usr/lib/python3.7/site-packages/azure/iot/device/common/evented_callback.py", line 70, in wait_for_completion
     raise self.exception
 azure.iot.device.common.transport_exceptions.UnauthorizedError: UnauthorizedError('Connection Refused: not authorised.')

 The above exception was the direct cause of the following exception:

 Traceback (most recent call last):
   File "sync_env_sensor.py", line 25, in <module>
     registration_result = provisioning_device_client.register()
   File "/usr/lib/python3.7/site-packages/azure/iot/device/provisioning/provisioning_device_client.py", line 73, in register
     self._enable_responses()
   File "/usr/lib/python3.7/site-packages/azure/iot/device/provisioning/provisioning_device_client.py", line 96, in _enable_responses
     handle_result(subscription_complete)
   File "/usr/lib/python3.7/site-packages/azure/iot/device/provisioning/provisioning_device_client.py", line 35, in handle_result
     raise exceptions.CredentialError(message="Credentials invalid, could not connect", cause=e)
 azure.iot.device.exceptions.CredentialError: CredentialError('Credentials invalid, could not connect') caused by UnauthorizedError('Connection Refused: not authorised.')

This issue has also been raised @ https://docs.microsoft.com/en-us/answers/questions/154410/device-client-register-method-fails-with-credentia.html
Any help will be greatly appreciated.
AB#8828384

The error message that you have received seems to be pointing to one of the 4 values to be wrong.  T
    provisioning_host = 'global.azure-devices-provisioning.net'
    id_scope = '{your Scope ID}'
    registration_id = '{your registration id}'
    symmetric_key = '{your Primary Key}'

if you are creating your own device identity on the portal (which you can to test it out to see if that works) please follow this
page
The registration id is by default the device id unless you have chosen those 2 to be different. So please make sure that you are using the correct registration id.

@olivakar, I have used the appropriate values for the registration and connection - with registration_id being the unique device ID as seen in the device connection screen below:

I still am facing the same issue.
PS: How does one go about increasing the logging level to DEBUG?

In your python code :-
logging.basicConfig(level=logging.DEBUG)

Additionally if you want to save to a file also add filename=example.log
Did you try creating a symmetric key individual enrollment record directly on the portal and then trying out the provisioning sample ? I just did with a new enrollment record and everything was fine.
If you are able to do that successfully using the same sync sample you should see your enrollment record like this.

Thanks @olivakar. As I mentioned earlier, we are creating/managing devices on Azure IoT Central. Refer:
https://docs.microsoft.com/en-us/azure/iot-central/core/tutorial-connect-device-python
We currently don't make use of the Azure IoT Hub service (directly, although I believe Azure IoT Central has it integrated).
Here's what we have done - created a device template and then a corresponding device. Applied the device connection information into our python script (provisioning_host,  id_scope, registration_id, symmetric_key) and then run the script to be able to register with and connect to the device instance created to push telemetry data.
The only difference between the sample app in the reference and what we have used is that we have modified it to be synchronous using an example available in the azure-iot-sdk-python specifically this.
Here's the console log after logging level at DEBUG (based on the changes suggested by you to this effect):
# python3 sync_env_sensor.py
DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Starting run_op in pipeline thread
DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Creating pipeline executor
DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:Scheduling SAS Token renewal for 3480 seconds in the future
DEBUG:azure.iot.device.common.pipeline.pipeline_stages_mqtt:Gateway Hostname not present. Setting Hostname to: global.azure-devices-provisioning.net
DEBUG:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage(InitializePipelineOperation): got connection args
DEBUG:azure.iot.device.common.mqtt_transport:creating mqtt client
INFO:azure.iot.device.common.mqtt_transport:Creating client for connecting using MQTT over TCP
DEBUG:azure.iot.device.common.mqtt_transport:creating a SSL context
DEBUG:azure.iot.device.common.mqtt_transport:configuring SSL context with default certs
DEBUG:azure.iot.device.common.mqtt_transport:Created MQTT protocol client, assigned callbacks
DEBUG:azure.iot.device.common.pipeline.pipeline_ops_base:InitializePipelineOperation: completing without error
DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Starting <azure.iot.device.common.evented_callback.EventedCallback object at 0xb5f2c5f0> in callback thread
DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Creating callback executor
DEBUG:azure.iot.device.common.evented_callback:Callback completed with result None
INFO:azure.iot.device.provisioning.provisioning_device_client:Registering with Provisioning Service...
INFO:azure.iot.device.provisioning.provisioning_device_client:Enabling reception of response from Device Provisioning Service...
DEBUG:azure.iot.device.provisioning.pipeline.mqtt_pipeline:enable_responses called
DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Starting run_op in pipeline thread
DEBUG:azure.iot.device.common.pipeline.pipeline_ops_base:EnableFeatureOperation: creating worker op of type MQTTSubscribeOperation
DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:AutoConnectStage(MQTTSubscribeOperation): Op needs connection.  Queueing this op and starting a ConnectionOperation
DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:AutoConnectStage(MQTTSubscribeOperation): calling down with Connect operation
INFO:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage(ConnectOperation): State changes LOGICALLY_DISCONNECTED->LOGICALLY_CONNECTED.  Adding to wait list and sending new connect op down
DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage: sending new connect op down
DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ConnectionLockStage(ConnectOperation): blocking
DEBUG:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage(ConnectOperation): connecting
DEBUG:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage(ConnectOperation): Starting watchdog
DEBUG:azure.iot.device.common.mqtt_transport:connecting to mqtt broker
INFO:azure.iot.device.common.mqtt_transport:Connect using port 8883 (TCP)
DEBUG:paho:Sending CONNECT (u1, p1, wr0, wq0, wf0, c0, k60) client_id=b'1uwj4ak5wit'
DEBUG:azure.iot.device.common.mqtt_transport:_mqtt_client.connect returned rc=0
DEBUG:paho:Received CONNACK (0, 5)
INFO:azure.iot.device.common.mqtt_transport:connected with result code: 5
DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Starting _on_mqtt_connection_failure in pipeline thread
INFO:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage: _on_mqtt_connection_failure called: UnauthorizedError('Connection Refused: not authorised.')
DEBUG:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage: failing connect op
DEBUG:azure.iot.device.common.pipeline.pipeline_stages_mqtt:MQTTTransportStage(ConnectOperation): cancelling watchdog
DEBUG:azure.iot.device.common.pipeline.pipeline_ops_base:ConnectOperation: completing with error UnauthorizedError('Connection Refused: not authorised.')
DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ConnectionLockStage(ConnectOperation): op failed.  Unblocking queue with error: UnauthorizedError('Connection Refused: not authorised.')
DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ConnectionLockStage(ConnectOperation): unblocking and releasing queued ops.
DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ConnectionLockStage(ConnectOperation): processing 0 items in queue for error=UnauthorizedError('Connection Refused: not authorised.')
DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage(ConnectOperation): on_connect_complete error=UnauthorizedError('Connection Refused: not authorised.') state=LOGICALLY_CONNECTED never_connected=True connected=False
DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:ReconnectStage: completing waiting ops with error=UnauthorizedError('Connection Refused: not authorised.')
DEBUG:azure.iot.device.common.pipeline.pipeline_ops_base:ConnectOperation: completing with error UnauthorizedError('Connection Refused: not authorised.')
DEBUG:azure.iot.device.common.pipeline.pipeline_stages_base:AutoConnectStage(MQTTSubscribeOperation): Connection failed.  Completing with failure because of connection failure: UnauthorizedError('Connection Refused: not authorised.')
DEBUG:azure.iot.device.common.pipeline.pipeline_ops_base:MQTTSubscribeOperation: completing with error UnauthorizedError('Connection Refused: not authorised.')
DEBUG:azure.iot.device.common.pipeline.pipeline_ops_base:EnableFeatureOperation: Worker op (MQTTSubscribeOperation) has been completed
DEBUG:azure.iot.device.common.pipeline.pipeline_ops_base:EnableFeatureOperation: completing with error UnauthorizedError('Connection Refused: not authorised.')
DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Starting pipeline_callback in callback thread
INFO:azure.iot.device.common.mqtt_transport:disconnected with result code: 5
INFO:azure.iot.device.common.evented_callback:Callback completed with error UnauthorizedError('Connection Refused: not authorised.')
INFO:azure.iot.device.common.evented_callback:NoneType: None

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/azure/iot/device/provisioning/provisioning_device_client.py", line 25, in handle_result
    return callback.wait_for_completion()
  File "/usr/lib/python3.7/site-packages/azure/iot/device/common/evented_callback.py", line 70, in wait_for_completion
    raise self.exception
azure.iot.device.common.transport_exceptions.UnauthorizedError: UnauthorizedError('Connection Refused: not authorised.')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "sync_env_sensor.py", line 29, in <module>
    registration_result = provisioning_device_client.register()
  File "/usr/lib/python3.7/site-packages/azure/iot/device/provisioning/provisioning_device_client.py", line 73, in register
    self._enable_responses()
  File "/usr/lib/python3.7/site-packages/azure/iot/device/provisioning/provisioning_device_client.py", line 96, in _enable_responses
    handle_result(subscription_complete)
  File "/usr/lib/python3.7/site-packages/azure/iot/device/provisioning/provisioning_device_client.py", line 35, in handle_result
    raise exceptions.CredentialError(message="Credentials invalid, could not connect", cause=e)
azure.iot.device.exceptions.CredentialError: CredentialError('Credentials invalid, could not connect') caused by UnauthorizedError('Connection Refused: not authorised.')
DEBUG:azure.iot.device.common.mqtt_transport:  File "/usr/lib/python3.7/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.7/site-packages/paho/mqtt/client.py", line 2913, in _thread_main
    self.loop_forever(retry_first_connection=True)
  File "/usr/lib/python3.7/site-packages/paho/mqtt/client.py", line 1578, in loop_forever
    rc = self.loop(timeout, max_packets)
  File "/usr/lib/python3.7/site-packages/paho/mqtt/client.py", line 1072, in loop
    rc = self.loop_read(max_packets)
  File "/usr/lib/python3.7/site-packages/paho/mqtt/client.py", line 1376, in loop_read
    return self._loop_rc_handle(rc)
  File "/usr/lib/python3.7/site-packages/paho/mqtt/client.py", line 1991, in _loop_rc_handle
    self.on_disconnect(self, self._userdata, rc)
  File "/usr/lib/python3.7/site-packages/azure/iot/device/common/mqtt_transport.py", line 210, in on_disconnect
    logger.debug("".join(traceback.format_stack()))

INFO:azure.iot.device.common.mqtt_transport:Forcing paho disconnect to prevent it from automatically reconnecting
DEBUG:azure.iot.device.common.mqtt_transport:in paho thread.  nulling _thread
DEBUG:azure.iot.device.common.mqtt_transport:Done forcing paho disconnect
DEBUG:azure.iot.device.common.pipeline.pipeline_thread:Starting _on_mqtt_disconnected in pipeline thread
ERROR:azure.iot.device.common.mqtt_transport:Unexpected error calling on_mqtt_disconnected_handler

I was able to follow the steps in the tutorial and create device, provision device and also send telemetry. I can't be sure what the problem is at your end. Other than advising you to. verify your values for the 4 variables.
What. is the value of your provisioning_host ?

The provisioning host is set to 'global.azure-devices-provisioning.net' as suggested in the tutorial.
I have verified the 4 variables necessary for registration/connection - we are setting them to the values provided in the device connection screen (in Azure IoT Central).

Hi @avwater , unfortunately we are unable to repro this issue with the tutorial. We will work with Satish (from your Microsoft Q&A post) to communicate with you and find a root cause. For now, let's use that channel. We'll leave this open in the meantime and return to it when we have a solution.

Closing this issue as it was resolved on a separate thread. Please reopen if there are still issues, thank you!

Closing this issue as it was resolved on a separate thread. Please reopen if there are still issues, thank you!

@avwater, thank you for your contribution to our open-sourced project! Please help us improve by filling out this 2-minute customer satisfaction survey

@avwater, thank you for your contribution to our open-sourced project! Please help us improve by filling out this 2-minute customer satisfaction survey

Closing this issue as it was resolved on a separate thread. Please reopen if there are still issues, thank you!

Can you please post the resolution or a link to the thread? I've just followed the tutorial (running python3.7 in WSL) and am having the same issue.

Closing this issue as it was resolved on a separate thread. Please reopen if there are still issues, thank you!

Can you please post the resolution or a link to the thread? I've just followed the tutorial (running python3.7 in WSL) and am having the same issue.

running python3.7 in WSL

For anyone else - the exact same code (with hardcoded parameters) works in Windows command prompt, and not Ubuntu WSL. Should I file this as a bug or is it a know issue @olivakar ?

running python3.7 in WSL

For anyone else - the exact same code (with hardcoded parameters) works in Windows command prompt, and not Ubuntu WSL. Should I file this as a bug or is it a know issue @olivakar ?

@kodonnell , thanks for following up on this! Could you please file the issue with WSL? that is not a known bug to my knowledge, and we will coordinate with the IoT Central team. The initial resolution was having the appropriate roles set (need to be an application Administrator), but it sounds like your issue may be different.
Here's a verbatim explanation for this issue:
_To access and use the Administration section, you must be in the Administrator role for an Azure IoT Central application. If you create an Azure IoT Central application, you're automatically added to the Administrator role for that application. For more information, see Manage users and roles in Azure IoT Central application.
You can either create your own IoT Central application for testing and to run the tutorial, in which case you'll be in Administrator role automatically. Or, you can ask someone who is in the Administrator role for your application to retrieve the group SAS key for you._

@kodonnell , thanks for following up on this! Could you please file the issue with WSL? that is not a known bug to my knowledge, and we will coordinate with the IoT Central team. The initial resolution was having the appropriate roles set (need to be an application Administrator), but it sounds like your issue may be different.
Here's a verbatim explanation for this issue:
_To access and use the Administration section, you must be in the Administrator role for an Azure IoT Central application. If you create an Azure IoT Central application, you're automatically added to the Administrator role for that application. For more information, see Manage users and roles in Azure IoT Central application.
You can either create your own IoT Central application for testing and to run the tutorial, in which case you'll be in Administrator role automatically. Or, you can ask someone who is in the Administrator role for your application to retrieve the group SAS key for you._


feature: Add CSS class for the dragged grid item animating back to it's placeholder location

Similar to the CDK drag-drop library's cdk-drag-animating class...
Classes should be added to the grid and grid items for each of their different "states" - one of these being the dragged item animating back to it's placeholder location after a pointer up event.
CDK drag/drop source for inspiration:
https://github.com/angular/components/blob/6eff9386b8145e4b3d50c59d353787d103d2fa8d/src/cdk/drag-drop/drag-ref.ts#L1011

I agree we should have this feature to correctly solve the bug: https://github.com/katoid/angular-grid-layout/issues/6
The code from angular material is totally fine IMO.
As I said in #6 , do you want to do a MR with the solution? Tell me if you need some kind of information.

Closed in https://github.com/katoid/angular-grid-layout/pull/102


Binary releases and Homebrew Cask

Dear Samuel,
it's very good to distribute Glance via App Store but some people might prefer installing binary releases themself or via Homebrew Cask.
Can you supply Binaries for Glance for every GitHub release?
If you do I will take care of having it added to Homebrew.
Some people also might not be able to install Glance because of restrictions on their work devices like this:

Done: v1.0.0
If you could manage the formula, that would be great 😊
Paths for the zap stanza:

~/Library/Application\ Scripts/com.samuelmeuli.Glance/
~/Library/Application\ Scripts/com.samuelmeuli.Glance.QLPlugin/
~/Library/Containers/com.samuelmeuli.Glance/
~/Library/Containers/com.samuelmeuli.Glance.QLPlugin/
~/Library/Group\ Containers/group.com.samuelmeuli.glance/

@samuelmeuli Thank you very much. I will take care of the formula later today and report here.

@samuelmeuli I am placing the Glance.app in /Applications/ and executing it but that doesn't seem to be enough. Will I have to copy the stanzas in the appropriate folders myself or how is that done?

That should be enough. The Quick Look plugin is registered when the app is opened for the first time.
Are you on macOS 10.15?

@samuelmeuli You can try the cask yourself when you create it with brew cask create glance, then replace the template with this:
cask 'glance' do
  version '1.0.0'
  sha256 '8ef7ba4a5130955e213fdfcc2bf43cba47c742d5f02e0e9c62405acaf19626b5'

  url "https://github.com/samuelmeuli/glance/releases/download/v#{version}/Glance.dmg"
  appcast 'https://github.com/samuelmeuli/glance/releases.atom'
  name 'Glance'
  homepage 'https://github.com/samuelmeuli/glance'
  depends_on macos: '>= :catalina'

  app 'Glance.app'
  installer script: {
    executable: "Glance.app/Contents/MacOS/Glance",
  }

  zap trash: [
    '~/Library/Application\ Scripts/com.samuelmeuli.Glance/',
    '~/Library/Application\ Scripts/com.samuelmeuli.Glance.QLPlugin/',
    '~/Library/Containers/com.samuelmeuli.Glance/',
    '~/Library/Containers/com.samuelmeuli.Glance.QLPlugin/',
    '~/Library/Group\ Containers/group.com.samuelmeuli.glance/',
  ]
end

then brew cask install glance.

installer script: {
  executable: "Glance.app/Contents/MacOS/Glance",
}

This part is breaking the installation for me. Without it it's working fine.
I suspect this is because when the app is opened by the script, it's not yet in the /Applications/ directory. And this seems to be a requirement for Quick Look plugins to be registered by macOS.

@samuelmeuli Thank you for trying. I made some progress. I moved the execution of Glance.app to postflight section. It now executes the app but the terminal output hangs at the following until you quit the app manually. After quitting it Glance is working as expected.
Now I could set a small timer and quit the application in the postflight script but I think the cleaner way would be to set a --setup or --quiet argument when calling the binary, then you do the setup magic, then the app quits with return code zero by itself, preferably without showing a dock icon.  How do you think about it?
This is where the cask installation hangs:
==> Installing Cask glance
==> Moving App 'Glance.app' to '/Applications/Glance.app'.

I think the user should open the app by themselves. Automatically executing a binary that was downloaded from the internet seems a bit problematic to me. Maybe a note in caveats is enough?

@samuelmeuli My opinion differs a bit but I respect yours and therefore added it to the caveats and created a merge request. Since it seems to be the first QL-plugin of its kind to enter the Caskroom I suspect there will be some discussion so feel free to chime in at https://github.com/Homebrew/homebrew-cask/pull/82816

Great, I'm curious to hear what other people think. Thanks a lot for working on this, @crunchtime-ali! 😊

@samuelmeuli Can you maybe chime in on vitorgalvaos' question https://github.com/Homebrew/homebrew-cask/pull/82816#discussion_r426924354?
I looked at the source code but feel you are more qualified to answer the question. As I understand it the Quick Look plugin resides in /Applications/Glance.app/Contents/PlugIns/QLPlugin.appex

Glance is now available on Homebrew! 🎉
brew cask install glance

Many thanks to @crunchtime-ali!


How to exclude a particular country from list of all country

I try to find how to exclude some countries in CountryPickerView, Please suggest the correct way to achieve it. Thanks

You could just set the preferred countries and remove the ones you don't want to show.
extension DemoViewController: CountryPickerViewDataSource {
    func preferredCountries(in countryPickerView: CountryPickerView) -> [Country] {
        return countryPickerView.countries.filter {
            // Filter the countries you want to show.
        }
    }

    func sectionTitleForPreferredCountries(in countryPickerView: CountryPickerView) -> String? {
        return "Preferred title"
    }

    func showOnlyPreferredSection(in countryPickerView: CountryPickerView) -> Bool {
        return true
    }
}

@kizitonwose As you suggested to use preferredCountries function I already used that for showing user current country. See in image https://www.dropbox.com/s/o3cl0rotwekzkux/Simulator%20Screen%20Shot%20-%20iPhone%2011%20Pro%20Max%20-%202020-06-24%20at%2010.04.56.png?dl=0
I would like to remove a country from rest of the list and from searching for example Afghanistan So how will this achieve.
One solution I have made in Pod folder I removed the json object of that country, remove related flag resource, and from Copy and bundle resources I removed the link of flag icon. That works for me but that is temporary solution. as I update the pod next time changes will undo. And need to do it again.
Do you have any alternate way of doing this. Please suggest.

Will add this feature in an upcoming version.

Added in 3.1.3


Made every test exclusive by flushing the DB changes after every test

Made every test exclusive by flushing the DB changes after every test

Checking on why tests are failing.
Those are passing in my local. 😢


Add multi-cluster app support

Address issue: https://github.com/rancher/rancher/issues/17147

@dramich Addressed comments.

@dramich Updated

LGTM

Squashed

@gitlawr : there were some RBAC related changes that got merged to rancher just recently: https://github.com/rancher/rancher/pull/17721. Some MCA updates got moved into actions vs being accepted as PUT. @mrajashree is going to follow up with you on the details. We can either incorporate the changes to this pr, or to the follow up one.

Follow up: will merge the pr as it is for now. Addressing RBAC changes are tracked in https://github.com/rancher/rancher/issues/17981 and https://github.com/rancher/rancher/issues/17982


Accessing FTP with only a username and no password

I am trying to connect to GoDaddy auctions FTP server, but I am not able to connect due to it not having a password. It looks like this.options in connect doesn't allow an empty password and changes it to anonymous by default.
this.options.password = options.password || 'anonymous@';
<EMAIL_ADDRESS>is the complete login. How do I connect to GoDaddy's FTP service?

@CodeBradley these options works for me:
const options = {
  host: "ftp.godaddy.com",
  user: "auctions",
  password: new String(""),
};

Hey @greenfrontend those options are not working it is giving following error
Error: Please login with USER and PASS.     at makeError (E:\projects\domain-analysis\node_modules\ftp\lib\connection.js:1068:13)     at Parser.<anonymous> (E:\projects\domain-analysis\node_modules\ftp\lib\connection.js:114:25)     at Parser.emit (node:events:390:28)     at Parser._write (E:\projects\domain-analysis\node_modules\ftp\lib\parser.js:59:10)     at writeOrBuffer (node:internal/streams/writable:389:12)     at _write (node:internal/streams/writable:330:10)     at Parser.Writable.write (node:internal/streams/writable:334:10)     at Socket.ondata (E:\projects\domain-analysis\node_modules\ftp\lib\connection.js:274:20)     at Socket.emit (node:events:390:28)     at addChunk (node:internal/streams/readable:315:12) {   code: 530 }


Wrong dicom error

I try convert this dicom files https://www.dicomlibrary.com/?manage=1b9baeb16d2aeba13bed71045df1bc65 but the software says that file is wrong dicom.

These dicom files are compressed. With the dicom2mesh build from the ppa you cannot read compressed files. There are two ways:

Building Dicom2Mesh yourself by considering following issue https://github.com/AOT-AG/DicomToMesh/issues/14#issuecomment-443224034
Uncompress the files using tools, such as dcmtk

I got error on build vtk-dicom (https://github.com/AOT-AG/DicomToMesh/issues/14#issuecomment-443224034):
By not providing "FindVTK.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "VTK", but
  CMake did not find one.

  Could not find a package configuration file provided by "VTK" with any of
  the following names:

    VTKConfig.cmake
    vtk-config.cmake

  Add the installation prefix of "VTK" to CMAKE_PREFIX_PATH or set "VTK_DIR"
  to a directory containing one of the above files.  If "VTK" provides a
  separate development package or SDK, be sure it has been installed.

and I got install dcmtk but after install the system not recognize it, is like not are installed.

Good morning,
There is no binary named dcmtk. What you are interested in is the command dcmjpeg. This came with dcmtk on your system.
About the cmake error: You need to read the build instructions. In this case the VTK_DIR is not set. Please see this and that

Waht I pretend is generate 3D object to render with javascript

Yes, VTK_DIR was setted, but I thing that VTK is not installed correctly.

https://github.com/AOT-AG/DicomToMesh/issues/18#issuecomment-528225293 should say dcmdjpeg (not dcmjpeg).
Before:
dicom2mesh -i . -v

Read DICOM images located under .
ERROR: In /build/vtk7-d8SNZg/vtk7-7.1.1+dfsg1/IO/Image/vtkDICOMImageReader.cxx, line 105
vtkDICOMImageReader (0x55cb89114650): DICOMParser couldn't parse : ./conv

Progress: 0.0%ERROR: In /build/vtk7-d8SNZg/vtk7-7.1.1+dfsg1/IO/Image/vtkDICOMImageReader.cxx, line 333
vtkDICOMImageReader (0x55cb89114650): There was a problem retrieving data from: ./0196.dcm

ERROR: In /build/vtk7-d8SNZg/vtk7-7.1.1+dfsg1/Common/ExecutionModel/vtkExecutive.cxx, line 784
vtkCompositeDataPipeline (0x55cb891152f0): Algorithm vtkDICOMImageReader(0x55cb89114650) returned failure for request: vtkInformation (0x55cb89120710)

after:
apt install dcmtk
mkdir conv
for x (*.dcm)  { dcmdjpeg -v $x conv/$x }
cd conv
dicom2mesh -i . -v
(success)

I tried, but the generated images are not able to open.

@IgorDePaula try to open the CT-Scan on Slicer, export it like a DICOM and try it again. Usually works.

@IgorDePaula I am closing this issue. Feel free to reopen it. Thx, Adrian


Job file syntax needs to be documented

I just wasted some time trying to figure out job files, specifically the syntax for /XF. It is not mentioned at all  that the file names need to be listed on lines after the /XF line. Trying to put them on the same line as /XF, with a colon or a space separator, results in:
ERROR : Invalid Parameter #1 : "/JOB:j"

Not helpful at all. A simple example of a job file with /XD or /XF would have made it immediately clear.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: bf2281d4-1470-0658-ec5a-687fe2cd451c
Version Independent ID: 2ce2be3e-e3a2-790c-3bff-90efe7318450
Content: robocopy
Content Source: WindowsServerDocs/administration/windows-commands/robocopy.md
Product: windows-server
Technology: windows-commands
GitHub Login: @JasonGerend
Microsoft Alias: jgerend

#assign:xelu86

Hey @jgriffitts,
Thanks again for contacting Microsoft. We value your feedback.
A documentation change has been made and will be live after the next business day. Additional clarity has been added into the job options parameter. We're taking your feedback into consideration regarding command reference examples.
We will be closing this as resolved. Thank you and have a most pleasant day. #please-close


password is required for redis-webui

Description
https://github.com/laradock/laradock/pull/3218
The above pull request requires a password for redis-webui.
Motivation and Context
Allow redis-webui to open.
Types of Changes

[x] Bug fix (non-breaking change which fixes an issue).
[] New feature (non-breaking change which adds functionality).
[] Breaking change (fix or feature that would cause existing functionality to not work as expected).

Definition of Done Checklist:

[x] I've read the Contribution Guide.
[] I've updated the documentation. (refer to this for how to do so).
[x] I enjoyed my time contributing and making developer's life easier :)

thanks for resolve my problem!


Resetting saved Array with WRUserSettings

I am able to set array in userdefaults with this library,but when i call reset() it doesnt clear the array i had saved via WRSuerSttings object

@iOSBonson it should work in latest version of library :)


TODO:Add support anti-meridian

should be able to use anti-meridian for correct calculation of distance

Hey @paulmach . What are your thoughts about it? Are you interested in this modification?


status: failed without any feedback

My challenge http://maproulette.org/ui/admin/list/1986/Challenge/tasks/2747 has status failed.
Geojson is
{"type":"FeatureCollection","features":[{"type":"Feature","geometry":{"type":"Point","coordinates":[8.5306777,11.9908895]},"properties":{"name": "\"Kofar Nassarawa\"", "name_1": "\"Badalar Kano\"", "source": "\"www.steadfast.com.ng\"", "historic": "\"monument\"", "source_1": "\"http://kanostreetsonosm.org.ng/\"", "wikipedia": "\"en:Kano City Walls and Gate\"", "description": "\"One of ancient Kano City Walls and Gate\""}}]}

produced by
[out:json];
(
way["wikipedia"="en:Kano City Walls and Gate"]
node["wikipedia"="en:Kano City Walls and Gate"]
); out body geom qt;

There is no feedback anywhere why status is failed.
From looking at http://maproulette.org/view/2747 there are no signs what went wrong.

That is annoying and something I am looking to improve. Let me see if I can reproduce your case and help out.

Closing old ticket. UI will show errors at this point, and that geojson imported fine


Update the url for the repository

This updated the url for the repository. It's still the old one since before the module was transferred to Mixpanel.

Travis fails, but it doesn't seem to be related

Thanks for the patch, merged in https://github.com/mixpanel/mixpanel-node/pull/55. Don't worry about the build failures, those were ancient node versions and Travis couldn't install dependencies for them.


test: Initial attempt at multidim CI test

WIP until CI passes

Future todos:

[ ] Figure out why firefox webrtc isn't working
[ ] Figure out why webkit tests fail to create a libp2p node

@ckousik I'm seeing a failure with firefox -> webrtc -> rust (Chrome works fine).  https://github.com/libp2p/js-libp2p/actions/runs/4369651820/jobs/7643671824#step:5:5535 " localCert.getFingerprints is not a function". Have you seen that before? https://github.com/libp2p/js-libp2p-webrtc/blob/4c8806c6d2a1a8eff48f0e2248203d48bd84c065/src/transport.ts#L215

I don't know why windows is failing. It seems like it's failing in other PRs as well, so probably not related to these changes

@ckousik I'm seeing a failure with firefox -> webrtc -> rust (Chrome works fine). https://github.com/libp2p/js-libp2p/actions/runs/4369651820/jobs/7643671824#step:5:5535 " localCert.getFingerprints is not a function". Have you seen that before? https://github.com/libp2p/js-libp2p-webrtc/blob/4c8806c6d2a1a8eff48f0e2248203d48bd84c065/src/transport.ts#L215

That's odd, since getFingerprints is required in w3c https://w3c.github.io/webrtc-pc/#rtccertificate-interface . Looking into it now.

This ready for review. This could be faster, our build of head takes 10m. I haven't spent any time investigating this, but we could probably speed this up.

@achingbrain Can we review and get this in this week (meant to follow up on this in triage today)?

Per sync discussion:
@achingbrain is hesistant to merge this while it takes >10min. If I bring it down to <10min, we'll merge.

Alright, I spent a good bit of time optimizing this and the tests run in 15 min. I think this is as good as we can get it without reworking the test runner itself. (unfortunately docker run startup time is the limiting factor, with each test taking about 2s to run).
This could improve a bit if we make the rust images smaller, but I wouldn't expect too much.
@achingbrain can I get another approval?

This could improve a bit if we make the rust images smaller, but I wouldn't expect too much.

Tracking this here: https://github.com/libp2p/rust-libp2p/issues/3881

All tests pass! @achingbrain any objections to merging?

@MarcoPolo Do I understand correctly that with this PR js-libp2p will use the same structure of just pointing to a commit-hash in the test-plans repo?
Much in favor of adopting that!

Do I understand correctly that with this PR js-libp2p will use the same structure of just pointing to a commit-hash in the test-plans repo?

Yes

@achingbrain this is blocking more interop tests in test plans and webrtc interop tests. Since this is only test code, I'll assume your previous approval is still valid unless you say otherwise within the next ~24h.

I've split up the test @achingbrain, but what's annoying now is that one or the other test will always be skipped but it will still spin up and shutdown a libp2p node because of beforeEach and afterEach. Any suggestions? We might be able to replace it(... with it.skip(... dynamically and thus skip the test at the higher level, but that seems a bit hacky.

I agree @MarcoPolo it seems to get a bit messy going that route. What do you think about splitting the tests into two separate directories i.e. one for dialler tests and one for listener tests (albeit they are both just one test each at the moment) and we can use just use an npm script like
"test": if [ \"$is_dialer\" = \"true\" ]; then aegir test -t node -f \".dist/test/dialer/*.{node,spec}.js\"; else aegir test -t node -f \".dist/test/listener/*.{node,spec}.js\"; fi

I know we have been going back and forth on this for a while so if you and @achingbrain agree then I could make these changes so that we can land this.

I've split up the test @achingbrain, but what's annoying now is that one or the other test will always be skipped but it will still spin up and shutdown a libp2p node because of beforeEach and afterEach. Any suggestions? We might be able to replace it(... with it.skip(... dynamically and thus skip the test at the higher level, but that seems a bit hacky.

I agree @MarcoPolo it seems to get a bit messy going that route. What do you think about splitting the tests into two separate directories i.e. one for dialler tests and one for listener tests (albeit they are both just one test each at the moment) and we can use just use an npm script like
"test": if [ \"$is_dialer\" = \"true\" ]; then aegir test -t node -f \".dist/test/dialer/*.{node,spec}.js\"; else aegir test -t node -f \".dist/test/listener/*.{node,spec}.js\"; fi

I'm strongly against separate files for this test. I also don't like using bash scripting as the way to check which test to run. Why are we introducing bash into this when we're already working with TypeScript?

I'm strongly against separate files for this test.

why is that? As it is now they are two separate tests in one file, I think that having them in separate files improves the modularity and reporting (as you wouldn't see the logs of the test being skipped)

I also don't like using bash scripting as the way to check which test to run. Why are we introducing bash into this when we're already working with TypeScript?

I can appreciate that but I think they serve two different purposes in this case. I would say it's more common to see bash scripting accessing environment variables than Typescript. I would also say it's more intuitive than the dynamic way of skipping tests
Regardless, I wouldn't let either of those concerns block this from being landed and I see that tests pass now  🚀. We can always optimize in a follow up PR 😄

why is that? As it is now they are two separate tests in one file, I think that having them in separate files improves the modularity and reporting (as you wouldn't see the logs of the test being skipped)

Having more files here doesn't improve modularity, it adds indirection. You may as well run the test command and grep for only the tests you want to run.
Is it common to see Bash scripting used in npm test? I've done it a couple of times, but only as a last resort.

You may as well run the test command and grep for only the tests you want to run.

Sorry this wasn't clear. I was using this an example of what you wouldn't want to do.

Poking in here: were the discussions mostly resolved i.e. is this guy almost there to getting merged?

I meant something like:
it('should be a listener', function () {
  if (is_dialer) {
    return this.skip()
  }

  // listener test code here
})

The dynamic skip:
(isDialer ? it.skip : it)('should listen for ping', async () => {

is reminiscent of what we did for the js-ipfs interface tests.  It worked, though typing it all was a little hairy.
I think one file for the ping test is ok, definitely don't want to be debugging bash scripts to figure out why tests aren't running.
I think we should get this in and we can finesse it in follow PRs.


core dump on --scheme=SpectreSafeInvisibleSpec run

Runing with --scheme=UnsafeBaseline successfully completes but SpectreSafeInvisibleSpec aborts with core dump. I tried with low number of instructions as well as different cache configurations so find the source of memory leakage. But had no success.
build/X86/gem5.opt -d myout/gcc configs/example/se.py --cmd=../workspace/benchmarks/cpu2006/spec-SE/binaries/x86/linux/gcc -I 100000 --cpu-type=DerivO3CPU --caches --scheme=SpectreSafeInvisibleSpec --needsTSO=1

build/X86/mem/cache/cache.cc:192: void Cache::satisfyRequest(PacketPtr, CacheBlk*, bool, bool): Assertion pkt->hasRespData()' failed.`


[iOS] App crashes/restarts after taking photos

Describe the bug

The iOS app forcibly restarts after having taken a new picture with the iOS Camera app.
To Reproduce
Steps to reproduce the behavior:

Focus the Bluesky app (settings, feeds, new post editor... doesn't matter)
Switch to the Camera app AND take a new picture (note: just adding items to the gallery isn't enough to trigger the crash, screenshots work)
Switch back to Bluesky – app restarts (and kills your new post for which you took the picture)

Expected behavior

Just resume state.
Details

Platform: iOS
Platform version: 17.2.1
App version: 1.63.0 (also earlier versions)

after more testing:

iPhone 13 mini: affected as above
iPhone 11, same iOS and App version: no problems

after more testing:

iPhone 13 mini: affected, as above
iPhone 11, same iOS/app version: no problems

I don't know if this is helpful as it's such an old iOS version, but I'm unable to reproduce this on my iPhone 7 running iOS 15.8 and Bluesky 1.63.

Okay… after app removal, phone reboot, app reinstall, allowing photo gallery access – can't reproduce any longer 🤷♂


FIX: Remove HEAD -> GET request swap workaround

Description
Remove a workaround replacing HEAD requests with GET requests for async requests.
This workaround is no longer needed after the underlying bug was fixed in https://github.com/aio-libs/aiohttp/pull/5012
Issues Resolved
Partially resolves https://github.com/opensearch-project/opensearch-py/issues/698
The rest of the issue is user (my) error.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Is there a way to write a test for this, too? Something that fails without this change?

Do we need to lock aiohttp >= a certain version as part of the PR?

Bug resolved in aiohttp version 3.7.0.
opensearch-py compatibility: aiohttp>=3,<4.
I suggest instead of locking aiohttp to >=3.7.0,<4, we should check the aiohttp version and conditionally remove the workaround for aiohttp >=3.7.0,<4.
Detailed code review required to identify necessary code changes.

I suggest instead of locking aiohttp to >=3.7.0,<4, we should check the aiohttp version and conditionally remove the workaround for aiohttp >=3.7.0.

Disregard the previous comment. To resolve this issue, we're planning to lock aiohttp to >=3.9.2,<4. Thus, simply removing the workaround should suffice."

Hello @vcastane ,
Could you please take a look at finishing this PR when you have a moment? If you need any help, let us know.
Thanks a lot!

Finished this in https://github.com/opensearch-project/opensearch-py/pull/794.


Official config releases on more than just tags

The readme suggests that in order to only build version tags, I should use:
with:
  release: ${{ startsWith(github.ref, 'refs/tags/v') }}

However, I don't think this is correct. When I push a tag, my GitHub Action runs for both the commit to master and the tag that points to it, which expected. However, in both cases, a release occurs:

v0.1.9 tag:
  • publishing      publisher=Github (owner: ***, project: ***, version: 0.1.9)
  • uploading       file=***-0.1.9.dmg.blockmap provider=GitHub
  • uploading       file=***-0.1.9.dmg provider=GitHub
  • creating GitHub release  reason=release doesn't exist tag=v0.1.9 version=0.1.9

commit on master branch (different build with identical code):
  • publishing      publisher=Github (owner: ***, project: ***, version: 0.1.9)
  • uploading       file=***-0.1.9.dmg.blockmap provider=GitHub
  • uploading       file=***-0.1.9.dmg provider=GitHub
  • overwrite published file  file=***-0.1.9.dmg.blockmap reason=already exists on GitHub
  • overwrite published file  file=***-0.1.9.dmg reason=already exists on GitHub

I believe github.ref is set to the tag on which the commit is based, rather than only being equal to refs/tags/v... for the run for the actual tag.
To make matters worse, the ref also appears to be set even if the branch is ahead of the tag. I updated my build config to change the output filename, pushed that change, and it added the resulting files (with the new filename) to the last release!
I hope I'm doing something wrong but since I just copied the release config verbatim, I'm not sure where to look.
Unfortunately, this is for a private repo, but it should be easy to reproduce.

I've done some more investigation myself and I think I've identified the issue.
In my Action's log, I can check whether publishing is supposed to happen based on this log line that I'd missed before:
Building the Electron app…

or
Building and releasing the Electron app…

The logging happens here and tells me that something is going wrong when release is false.
Hence, my analysis above about github.ref was incorrect and the issue is in how the Action processes release = false.
Now, contrary to your Mini Diary project, I have build.publish.github in my package.json:
{
  "publish": [
    {
      "provider": "github",
      "releaseType": "release",
      "publishAutoUpdate": false
    }
  ]
}

My assumption was that this would only get used when actually publishing, but I'm now thinking that it's being interpreted as "always publish to GitHub".
However, the counterpart to the Action's --publish always is documented as --publish never. I think passing this flag will override my package.json config.

So I tried forking and publishing your Action to add --publish never but strangely, it didn't have any effect: it still insisted on publishing in all cases. I guess this is starting to sound more like an issue with electron-builder?

After many, many red herrings, I've identified a non-obvious electron-builder feature as the culprit: on CI, it always publishes.
However, --publish=never does disable that behavior. I'm not sure why my previous test didn't work.
At any rate, here's a workaround that doesn't involve updating the Action:
steps:
  - name: Set electron-builder args
    if: ${{ ! startsWith(github.ref, 'refs/tags/v') }}
    run: echo '::set-env name=EB_ARGS::--publish=never'

  - name: Build/release Electron app
    uses: samuelmeuli/action-electron-builder@v1
    with:
      github_token: ${{ secrets.github_token }}
      release: ${{ startsWith(github.ref, 'refs/tags/v') }}
      args: ${{ env.EB_ARGS }}

Closing this issue since a) it belongs with electron-builder and b) it's been resolved. Sorry for the noise.


One shot learning example

It might be a good thing to give an example of one shot learning :)

This question is better asked on  StackOverflow since it is not a  bug or feature request. There is also a larger community that reads questions there. Thanks!


restore_mode not working

It looks like restore mode, or anything else what needs to be writen to persistent memory is not working. I have LSC bulbs flashed with LibreTiny ESPHome. When I try to set restore_mode it makes only "on" and "off" like stated, but don't remember last state and state of color temperature.

restore_mode works fine on every device I have.  "on" and "off" are not valid settings, see https://esphome.io/components/light/index.html?highlight=restore_mode.  Additionally, settings are not saved instantly, by default, they are only updated every 60 seconds to avoid flash wearing.  Additionally if you use anything like groups, they may be overriding the restored settings when they sync.

Edit: there are also quirks as to when save triggers when the mode changes, but the on/off state does not. You may want to consider calling the preferences save when you set the state you desire.

Like in the original post, restore_mode doesn't work for me (I have CMA19-60W-AL-9ACK  lights). Could you elaborate on how to do preferences save?

There was a bug found and patched in 2024.10.1 that sometimes led to preferences not being saved, so start by updating to that version or newer.
Additionally, you can add a button to manually save preferences on demand;
button:
  - platform: template
    id: save_preferences
    name: "Save Preferences"
    entity_category: diagnostic
    on_press:
      - lambda: global_preferences->sync();

Thanks, I was running the September build. I upgraded to 2024.10.2 and it seems to have fixed the issue. Using restore_mode = RESTORE_DEFAULT_ON now restores the color and brightness of the bulb after cutting power and restoring it.

Is there a way to completely disable saving preferences to flash, like [restore_from_flash](https://esphome.io/components/esp8266#:~:text=package to use.-,restore_from_flash,-(Optional%2C boolean)?

Is there a way to completely disable saving preferences to flash, like restore_from_flash: false?

I'm not certain if there is a way to completely disable it, but there are two things you can do to reduce it in ESPHome.
First is changing the flash_write_interval to never (note, 0s just removes delays, never is a special case).
Secondly, you can set restore_mode to either ALWAYS_ON or ALWAYS_OFF, which should reduce state tracking.

Secondly, you can set restore_mode to either ALWAYS_ON or ALWAYS_OFF, which should reduce state tracking.

This is what I'm doing currently.
Thanks for the hint about flash_write_interval.

No - this option is specific to ESP8266, where there apparently is some other kind of (semi-)non-volatile memory that is used to persist settings across reboots and OTA updates. There is no other kind of memory in LibreTiny, so this option would be equivalent to setting restore_mode and/or flash_write_interval.

Awesome. Thanks a lot.

@Cossid, @kuba2k2, there's no never for flash_write_interval:

And also, I have set all my switches and lights to either ALWAYS_ON and ALWAYS_OFF, but I still see this in my device logs:
00:59:02 | [I] | [safe_mode:041] | Boot seems successful; resetting boot loop counter
00:59:02 | [D] | [lt.preferences:104] | Saving 1 preferences to flash...
00:59:02 | [D] | [lt.preferences:132] | Saving 1 preferences to flash: 0 cached, 1 written, 0 failed

It seems to only happen in initialization/startup though.

Interesting, https://esphome.io/components/esphome.html#adjusting-flash-writes it is a Time object, and never is a valid option according to https://esphome.io/guides/configuration-types#config-time
I believe it just sets it to int.Max or int.Max-1 though.  They must have missed a validation case (or don't have it properly documented when it can an cannot be used)
Preferences will also write when given a reboot command (like before an OTA).  I guess just set it to a high value.


adds phylogenetic diversity metrics

Adds PD, unweighted unifrac, and weighted unifrac (with and without normalization).
To do before merge:

[ ] interface work (how will this be imported/used)

[x] add stability decorators

[ ] should normalized=True be the default for weighted_unifrac?

[x] modify pw_distances to take metric as the first argument and not have a default

[x] update pw_distances to take skbio's distance functions as strings and this should be the preferred/documented way of passing them (similar to SciPy's pdist)

[x] for PD, u becomes counts to match our other alpha diversity metrics

[x] for *unifrac, u and v become u_counts and v_counts, to match the alpha diversity metrics and SciPy

[x] phylogenetic_diversity renamed to faith_pd and move to _base.py; make it accessible from skbio.diversity.alpha

[x] *unifrac should be accessible from skbio.diversity.beta, and skbio/diversity/beta/_stats.py should be renamed skbio/diversity/beta/_unifrac.py

[ ] testing

[ ] expand unit tests to include direct computation of distance metrics with pw_distances, including referencing the metric both by passing the function and the name (as a string)

[ ] expand unit tests to use another tree, including:

[ ] tree with no branch lengths

[ ] tree with duplicated tip names

[ ] tree with one node

[ ] non-trivial case where tips in tree are superset of otu_ids

[x] add test with input vector of length == 0

[ ] add test of vector with negative counts (see skbio.diversity.alpha._validate)

[ ] documentation

[x] add numpy-doc (with cite)

[ ] add formulas for metrics

[x] update skbio.diversity.beta doctests to use a phylogenetic metric

[ ] update skbio.diversity.alpha doctests to use faith_pd

[ ] describe branch length normalization

[ ] misc

[x] inline _to_pdist_metric

[ ] add tree/otu ids _validate function

[ ] add _validate for beta diversity metrics, similar (or the same as skbio.diversity.alpha._validate)

[ ] squash commits

[ ] add ChangeLog note

Addresses #46, #820.

@gregcaporaso, would you like a review now, or the pending items get knocked out?

Probably not worth it yet.
On Wed, Aug 5, 2015 at 11:07 AM, Daniel McDonald<EMAIL_ADDRESS>wrote:

@gregcaporaso https://github.com/gregcaporaso, would you like a review
now, or the pending items get knocked out?
—
Reply to this email directly or view it on GitHub
https://github.com/biocore/scikit-bio/pull/1027#issuecomment-128094416.

kk
On Wed, Aug 5, 2015 at 12:08 PM, Greg Caporaso<EMAIL_ADDRESS>wrote:

Probably not worth it yet.
On Wed, Aug 5, 2015 at 11:07 AM, Daniel McDonald<EMAIL_ADDRESS>
wrote:

@gregcaporaso https://github.com/gregcaporaso, would you like a review
now, or the pending items get knocked out?
—
Reply to this email directly or view it on GitHub
<https://github.com/biocore/scikit-bio/pull/1027#issuecomment-128094416
.

—
Reply to this email directly or view it on GitHub
https://github.com/biocore/scikit-bio/pull/1027#issuecomment-128094764.

This is now ready for a first review. Before reviewing, I recommend reading the list above - there is still a bunch of outstanding items, so it's probably not worth reviewing things related to those items (mostly tests that still need to be added) yet. Also, the purpose of this PR is primarily to define the API and extensive tests of faith_pd, weighted unifrac, and unweighted unifrac so we can then write and test other implementations that we'll benchmark against. So, it's not worth a lot of profiling or optimization of these functions now. We'll get to that after this initial PR is merged.

Done with the review, this is awesome! Few general comments/questions:

should OTU be changed to observation?
should u_counts and v_counts just be u and v?
unifrac -> UniFrac

Also note, build is failing against py27

Thanks for the review @wasade, working through your comments this morning.

should u_counts and v_counts just be u and v?

I started out with u and v, but @jairideout, @ebolyen and I discussed yesterday. The motivation to name them u and v is consistency with the scipy beta diversity metrics, and the motivation to include the term counts is for clarity to users of other ecological software, and consistency with skbio.diversity.alpha. @jairideout came up with these terminology distinctions when reviewing other ecology software, see the note at the top of this page for what we mean by counts specifically. I think we should leave this as is.

should OTU be changed to observation?

I'm torn on this... I think that in our documentation for TreeNode, we should refer to tips as OTUs. This would be consistent with the literature on phylogeny reconstruction. If we go with that, then it makes sense to call these OTUs here since the ids are to map counts to tree tips (OTUs). I of course take your point that some/most/all of these metrics these could be applied to other data types, but I think it might make sense for the phylogenetic metrics to call these OTUs, since e.g. a generalized UniFrac for metagenomic data would probably operate on a DAG, and would ultimately probably be a different function. (For example, we specifically tell users to not use these metrics on shotgun data in the QIIME tutorials.)

Also note, build is failing against py27

So hard to support this antiquated technology... :weary:

Also note, build is failing against py27

So hard to support this antiquated technology... :weary:

:movie_camera:

Okay, sounds good re: u_counts and v_counts. The PR isn't entirely
consistent in the use IIRC -- can flag if you'd like.
On OTUs, I disagree that the TreeNode should describe tips as OTUs given
that phylogenies can be used for orthologous or paralogous genes. Is OTU
really the terminology used in the phylogenetics literature?
On Fri, Aug 7, 2015 at 7:08 AM, Greg Caporaso<EMAIL_ADDRESS>wrote:

Thanks for the review @wasade https://github.com/wasade, working
through your comments this morning.
should u_counts and v_counts just be u and v?
I started out with u and v, but @jairideout
https://github.com/jairideout, @ebolyen https://github.com/ebolyen
and I discussed yesterday. The motivation to name them u and v is
consistency with the scipy beta diversity metrics, and the motivation to
include the term counts is for clarity to users of other ecological
software, and consistency with skbio.diversity.alpha. @jairideout
https://github.com/jairideout came up with these terminology
distinctions when reviewing other ecology software, see the note at the
top of this page
http://scikit-bio.org/docs/latest/generated/skbio.diversity.alpha.html
for what we mean by counts specifically. I think we should leave this as
is.
should OTU be changed to observation?
I'm torn on this... I think that in our documentation for TreeNode, we
should refer to tips as OTUs. This would be consistent with the literature
on phylogeny reconstruction. If we go with that, then it makes sense to
call these OTUs here since the ids are to map counts to tree tips (OTUs). I
of course take your point that some/most/all of these metrics these could
be applied to other data types, but I think it might make sense for the
phylogenetic metrics to call these OTUs, since e.g. a generalized UniFrac
for metagenomic data would probably operate on a DAG, and would ultimately
probably be a different function. (For example, we specifically tell users
to not use these metrics on shotgun data in the QIIME tutorials.)
Also note, build is failing against py27
So hard to support this antiquated technology... [image: :weary:]
—
Reply to this email directly or view it on GitHub
https://github.com/biocore/scikit-bio/pull/1027#issuecomment-128695206.

@wasade, don't worry about it for now, I have a bunch of local changes that may have fixed it.

Automated report from asv run
Benchmark run completed successfully. Results available at
http://powertrip.mggen.nau.edu:5000/runs/gregcaporaso/issue-46/html/index.html

Awesome to see the report @scikitten-bio!
@jairideout, @ebolyen, and @wasade I'm going to wrap up my changes on this tomorrow so should be ready for merge then (and I'll create issues for the "future" items at that point as well).

Automated report from asv run
Benchmark run completed successfully. Results available at
http://powertrip.mggen.nau.edu:5000/runs/gregcaporaso/issue-46/html/index.html

Automated report from asv run
Benchmark run completed successfully. Results available at
http://powertrip.mggen.nau.edu:5000/runs/gregcaporaso/issue-46/html/index.html

Automated report from asv run
Benchmark run completed successfully. Results available at
http://powertrip.mggen.nau.edu:5000/runs/gregcaporaso/issue-46/html/index.html

Automated report from asv run
Benchmark run completed successfully. Results available at
http://powertrip.mggen.nau.edu:5000/runs/gregcaporaso/issue-46/html/index.html

Automated report from asv run
Benchmark run completed successfully. Results available at
http://powertrip.mggen.nau.edu:5000/runs/gregcaporaso/issue-46/html/index.html

Automated report from asv run
Benchmark run completed successfully. Results available at
http://powertrip.mggen.nau.edu:5000/runs/gregcaporaso/issue-46/html/index.html

Automated report from asv run
Benchmark run completed successfully. Results available at
http://powertrip.mggen.nau.edu:5000/runs/gregcaporaso/issue-46/html/index.html

@scikitten-bio is probably a little too active. I'll add some issues to autobencher to try and keep the number of automated comments down, delete outdated ones, etc.

Automated report from asv run
Benchmark run completed successfully. Results available at
http://powertrip.mggen.nau.edu:5000/runs/gregcaporaso/issue-46/html/index.html


FSSpecFileOpenerIterDataPipe raises a NoCredentialsError on large dataloader num_worker

🐛 Describe the bug
There are two issues (both are reproducible using the script below):

FSSpecFileOpenerIterDataPipe gets stuck if one tries to iteratively create DataLoader(num_workers=0, ...) then DataLoader(num_workers=greater_than_zero). Practically speaking this isn't much of an issue since typically a trainer will create the dataloader once but for benchmarking this means that we can't iterate benchmark runs that change dataloader num_workers from the same parent process.
NoCredentialsError when using FSSpecFileOpenerIterDataPipe with large (>64) dataloader num_workers.

Repro Script
from torch.utils.data import DataLoader
from torchdata.datapipes.iter import IterableWrapper

if __name__ == "__main__":
    print(f"=== BEGIN REPRO TEST ===")

    data_s3url = "s3://<REPLACE_WITH_YOUR_S3_URL>"
    # workers = [0, 1, 2] # <-- use this to repro stuckness. You'll observe that the loop below will get stuck when i=1
    workers = [1, 2, 4, 8, 16, 32, 48, 64, 128]
    for i in workers:
        dataset = (
            IterableWrapper([data_s3url])
            .list_files_by_fsspec()
            .open_files_by_fsspec()
            .readlines(return_path=False)
        )
        try:
            for batch in DataLoader(
                dataset,
                batch_size=max(workers) * 2,
                num_workers=i,
            ):
                break
            print(f"Succeeded running with num_workers={i}")
        except Exception as e:
            print(f"Error running with num_workers={i}. Exception: {e}")

    print(f"=== END REPRO TEST ===")

Exception
Traceback (most recent call last):
  File "/home/ubuntu/workspace/mfive/mfive/examples/data/repro.py", line 28, in <module>
    for batch in DataLoader(
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 681, in __next__
    data = self._next_data()
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1376, in _next_data
    return self._process_data(data)
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1402, in _process_data
    data.reraise()
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/torch/_utils.py", line 460, in reraise
    raise RuntimeError(msg) from None
RuntimeError: Caught NoCredentialsError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 32, in fetch
    data.append(next(self.dataset_iter))
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/torch/utils/data/datapipes/_typing.py", line 514, in wrap_generator
    response = gen.send(None)
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/torch/utils/data/datapipes/datapipe.py", line 344, in __iter__
    yield from self._datapipe
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/torch/utils/data/datapipes/_typing.py", line 514, in wrap_generator
    response = gen.send(None)
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/torchdata/datapipes/iter/util/plain_text_reader.py", line 121, in __iter__
    for path, file in self.source_datapipe:
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/torch/utils/data/datapipes/_typing.py", line 514, in wrap_generator
    response = gen.send(None)
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/torchdata/datapipes/iter/load/fsspec.py", line 137, in __iter__
    for file_uri in self.source_datapipe:
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/torch/utils/data/datapipes/_typing.py", line 514, in wrap_generator
    response = gen.send(None)
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/torchdata/datapipes/iter/load/fsspec.py", line 85, in __iter__
    for file_name in fs.ls(path):
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/fsspec/asyn.py", line 91, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/fsspec/asyn.py", line 71, in sync
    raise return_result
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/s3fs/core.py", line 810, in _ls
    files = await self._lsdir(path, refresh)
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/s3fs/core.py", line 593, in _lsdir
    async for i in it:
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/aiobotocore/paginate.py", line 32, in __anext__
    response = await self._make_request(current_kwargs)
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/aiobotocore/client.py", line 173, in _make_api_call
    http, parsed_response = await self._make_request(
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/aiobotocore/client.py", line 193, in _make_request
    return await self._endpoint.make_request(operation_model, request_dict)
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/aiobotocore/endpoint.py", line 77, in _send_request
    request = await self.create_request(request_dict, operation_model)
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/aiobotocore/endpoint.py", line 70, in create_request
    await self._event_emitter.emit(event_name, request=request,
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/aiobotocore/hooks.py", line 27, in _emit
    response = await handler(**kwargs)
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/aiobotocore/signers.py", line 16, in handler
    return await self.sign(operation_name, request)
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/aiobotocore/signers.py", line 63, in sign
    auth.add_auth(request)
  File "/home/ubuntu/.pyenv/versions/venv39/lib/python3.9/site-packages/botocore/auth.py", line 378, in add_auth
    raise NoCredentialsError()
botocore.exceptions.NoCredentialsError: Unable to locate credentials
This exception is thrown by __iter__ of FSSpecFileListerIterDataPipe(kwargs={}, masks='')

Versions

torchdata-0.4.1
torch-1.12.1
fsspec-2022.1.0
s3fs-2022.1.0

Thanks for opening the issue.
I am able to reproduce the first issue that the subsequent DataLoader would be stuck when the prior DataLoader has num_workers=0. I am investigating it now.
However, I am not able to reproduce the second issue. I am not sure why larger number of processes would interfere the credential. I suspect this is tied to boto3 not handling racing on credential read correctly.

For Issue 1, I am currently able to boil down to fork starting method. If I set multiprocessing_context="spawn" for the subsequent DataLoader, there won't be hanging issue.

Hi, regarding issue 2, I am using FSSpecFileOpenerIterDataPipe with 22 workers to load data from s3. I am also getting NoCredentialsError. Any progress on this issue? Thanks!

I ran into the same issue and found that setting AWS_METADATA_SERVICE_NUM_ATTEMPTS helps mitigate the issue as mentioned here.


Fetching from inside element

this is my data.. How to fetch a particular value like name from this..

I'm not seeing any data... could you look at your comment again and see if there a formatting issue or something?

@drmohundro

This is my data

@drmohundro
<PrintLetterBarcodeData state="Delhi" name="Mangal Singh" dob="02/03/1963"></PrintLetterBarcodeData>
This is my data

Are you asking how to pull the name attribute from your XML? That's documented at https://github.com/drmohundro/SWXMLHash#attributes-usage.
So, assuming you've already got that value, it'd be like this:
xml["PrintLetterBarCodeData"].element!.attribute(by: "name")!.text

I'm going to go ahead and close this as there hasn't been any activity - feel free to comment and re-open if you still are having issues. Thanks!


Fix synchronization issue

Fix the following crash experienced twice when extracting features from 250,000 games (so pretty rare):
thread '<unnamed>' panicked at 'called `Option::unwrap()` on a `None` value', libcore/option.rs:335:21
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::print
             at libstd/sys_common/backtrace.rs:68
             at libstd/sys_common/backtrace.rs:57
   2: std::panicking::default_hook::{{closure}}
             at libstd/panicking.rs:380
   3: std::panicking::default_hook
             at libstd/panicking.rs:396
   4: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:576
   5: std::panicking::begin_panic
             at libstd/panicking.rs:537
   6: std::panicking::begin_panic_fmt
             at libstd/panicking.rs:521
   7: rust_begin_unwind
             at libstd/panicking.rs:497
   8: core::panicking::panic_fmt
             at libcore/panicking.rs:71
   9: core::panicking::panic
             at libcore/panicking.rs:51
  10: dream_go::mcts::predict::PredictState::predict
  11: <dream_go::mcts::predict::PredictState as dream_go::parallel::service::ServiceImpl>::process
  12: dream_go::parallel::service::worker_thread
  13: std::panicking::try::do_call
  14: __rust_maybe_catch_panic
             at libpanic_unwind/lib.rs:102
  15: <F as alloc::boxed::FnBox<A>>::call_box
  16: std::sys_common::thread::start_thread
             at /checkout/src/liballoc/boxed.rs:827
             at libstd/sys_common/thread.rs:24
  17: std::sys::unix::thread::Thread::new::thread_start
             at libstd/sys/unix/thread.rs:90
  18: start_thread
  19: clone

Probably this stack trace:
thread '<unnamed>' panicked at 'called `Option::unwrap()` on a `None` value', libcore/option.rs:335:21
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::print
             at libstd/sys_common/backtrace.rs:68
             at libstd/sys_common/backtrace.rs:57
   2: std::panicking::default_hook::{{closure}}
             at libstd/panicking.rs:380
   3: std::panicking::default_hook
             at libstd/panicking.rs:396
   4: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:576
   5: std::panicking::begin_panic
             at libstd/panicking.rs:537
   6: std::panicking::begin_panic_fmt
             at libstd/panicking.rs:521
   7: rust_begin_unwind
             at libstd/panicking.rs:497
   8: core::panicking::panic_fmt
             at libcore/panicking.rs:71
   9: core::panicking::panic
             at libcore/panicking.rs:51
  10: <core::option::Option<T>>::unwrap
             at /checkout/src/libcore/macros.rs:20
  11: <dream_go::nn::network::WorkspaceGuard<'a> as core::ops::deref::DerefMut>::deref_mut
             at src/nn/network.rs:43
  12: dream_go::mcts::predict::PredictState::predict
             at src/mcts/predict.rs:137
  13: dream_go::mcts::predict::PredictState::check
             at src/mcts/predict.rs:186
  14: <dream_go::mcts::predict::PredictState as dream_go::parallel::service::ServiceImpl>::process
             at src/mcts/predict.rs:231
  15: dream_go::parallel::service::worker_thread
             at src/parallel/service.rs:101
  16: __rust_maybe_catch_panic
             at libpanic_unwind/lib.rs:102
  17: std::panicking::try
             at /checkout/src/libstd/panicking.rs:458
  18: std::panic::catch_unwind
             at /checkout/src/libstd/panic.rs:365
  19: std::thread::Builder::spawn::{{closure}}
             at /checkout/src/libstd/thread/mod.rs:405
  20: <F as alloc::boxed::FnBox<A>>::call_box
             at /checkout/src/liballoc/boxed.rs:817
  21: std::sys_common::thread::start_thread
             at /checkout/src/liballoc/boxed.rs:827
             at libstd/sys_common/thread.rs:24
  22: std::sys::unix::thread::Thread::new::thread_start
             at libstd/sys/unix/thread.rs:90
  23: start_thread
  24: clone

This might be because we are using an Rc and relying on it being unique, which is not necessarily true for the following race condition:

Some WorkspaceGuard hands the workspace back but has not released its Rc yet (because we only hold the lock for the duration of the drop function, but the Rc lifes for the duration of the struct).
Another thread requests a workspace and acquires the Rc from step 1.
The thread from step 2 tries to use the Rc before thread 1 can release it.

This can be fixed by not using an Rc to store the workspace. Using, for example, an Option instead which we can call take on to move the ownership from the guard to the pool in the drop function.

I re-wrote the WorkspaceGuard and Network to be less messy. This seems to have fixed this issue.


[EN] Feedback for: /web/tools/chrome-devtools/profile/network-performance/understanding-resource-timing?hl=en

"Hitting throughput capactiy"
Spelling error :)

Thanks @connorshea I've got a fix in one of my branches


chore: refactor messages gas computation for bouncer

This change is

Codecov Report
Attention: Patch coverage is 78.37838% with 8 lines in your changes are missing coverage. Please review.

Project coverage is 73.85%. Comparing base (253af91) to head (a808395).

Files
Patch %
Lines

crates/blockifier/src/bouncer.rs
0.00%
8 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1711      +/-   ##
==========================================
+ Coverage   73.75%   73.85%   +0.10%
==========================================
  Files          59       59
  Lines        8587     8548      -39
  Branches     8587     8548      -39
==========================================
- Hits         6333     6313      -20
+ Misses       1836     1817      -19
  Partials      418      418

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Redundant Immutable.is() for entire props and state?

Hey, I wanted to make sure I understood this bit correctly:
  if (objA === objB || is(objA, objB)) {
    return true;
  }

I don't think you can use Immutable.Map as your entire props/state. is would then check for referential equality, just like ===.

is() is essentially a deep value compare of the Immutable object.

Sorry, here's what I mean. I could be missing something, let me know:
// ImmutableRenderMixin.js
...
...
  shouldComponentUpdate: function(nextProps, nextState) {
    return !shallowEqualImmutable(this.props, nextProps) ||
           !shallowEqualImmutable(this.state, nextState);
  }
...

// shallowEqualImmutable.js
...
// this receives props/state as objects
function shallowEqualImmutable(objA, objB) {
  // consider state with: {myMap: Immutable.Map}
  // you might not always reuse references for props and state even if they're shallow equal

  // Immutable.is tries === first, and only does further work if it's an immutable object
  // Immutable.is(objA, objB) would only be true if objA === objB
  // additionally, Immutable.is('a', 'a') works since 'a' === 'a'

  if (objA === objB || is(objA, objB)) {
    return true;
  }
...

// Immutable.js console
var map = Immutable.Map({a: 1})
Immutable.is({myMap: map}, {myMap: map});
-> false
var a = {myMap: map};
Immutable.is(a, a);
-> true

The problem with your assertions is that your putting an Immutable Object as a value of a Javascript Object so Immutable.is({myMap: map}, {myMap: map}); should be false, but when we actually process the values of the javascript object and compare keys to values we will in fact satisfy our equality.
if (!bHasOwnProperty(keysA[i]) || !is(objA[keysA[i]], objB[keysA[i]]))

First we compare the top level objects (which could be a mixed bag of Immutable or Javascript objects
var map = Immutable.Map({a: 1});
var props = {myMap: map};
Immutable.is(props, props);
-> false

Second since we have not found equality yet we pull the keys from each object and compare the values at the keys:
var map = Immutable.Map({a: 1});
var props = {myMap: map};
Immutable.is(props.myMap, props.myMap);
-> true

Ok I see what u mean now. It looks like something may have changed inside the equality check from Immutable. I'll take a look.

Ok one more thing I think your test is misleading:
var map = Immutable.Map({a: 1})
Immutable.is({myMap: map}, {myMap: map});
-> false

Should be rewritten as for clarity:
var map = Immutable.Map({a: 1})
var objA = {myMap: map};
var objB = {myMap: map};
Immutable.is(objA, objB);
-> false

Ok so I think I understand the original issue; is the === equal to the is() since props/state objects are essentially javascript objects with some Immutable values. Let me think about possible edge cases here. We may be able to remove the is() call; I'll check and see what the source looks like in Immutable to see how expensive this call is in this case.

FWIW the file used to just run is() until it was reformatted to look like the original Facebook code and someone added the extra ===. I think they're completely equivalent, but I guess it's worth checking the source.


Move investment project code to a sub directory

Description of change
This moves all investment project code into a subdirectory of project.
All paths updated.
Checklist

[x] Has a new newsfragment been created? Check changelog/README.rst for instructions
[ ] Have any relevant search models been updated?
[ ] Have any relevant fixtures (fixtures/test_data.yaml) been updated?
[ ] Have any relevant select-/prefetch-related field lists in the views and search apps been updated?
[ ] Has the admin site been updated (for new models, fields etc.)?
[ ] Has the README been updated (if needed)?

https://github.com/uktrade/data-hub-leeloo/pull/1504#pullrequestreview-211670889
Due to time constraints I'm not going to do the subdirectory change. With this not being as complicated a change as I initially thought this could always be done at a later date.


Patching of click in rich-click via litestar cli breaks Typer cli apps

Description
I assume this may have similar impact on attempts to use click, but I am using Typer so that is what I have proven out here.
When attempting to use Typer to build cli tools in the general context of building a Litestar application, Typer loses its ability to properly report help messages on commands. This appears to be due to the patching of click in rich-click via litestar/cli/init.py. Because of imports in both Litestar and Advanced Alchemy that result in this patching, I am unable to integrate Typer code into a larger platform that includes shared code between command line and web application utilities. I am unsure but would expect similar issues with attempting to use click directly or any other library built on click.
For what it's worth, actual execution of the Typer commands still works from what I have seen, so the commands appear to be getting registered, they just don't show up in the help output.
URL to code causing the issue
No response
MCVE
# examples of imports that break typer's help:
import litestar.cli
from litestar.contrib.sqlalchemy.plugins import AsyncSessionConfig

import typer

app = typer.Typer()

@app.command()
def main(arg1: str = "default1", arg2: int = 0):
    print(f"Running main with arg1={arg1} and arg2={arg2}")

@app.command()
def another(arg1: str = "default1", arg2: int = 0):
    print(f"Running pipeline with arg1={arg1} and arg2={arg2}")

if __name__ == "__main__":
    app()

Steps to reproduce
1. To run:

pip install typer
pip install litestar

python -m mcve --help

# alternatively:
python mcve.py --help

The expectation is that typer will provide command help for main and another (shown
as a list of commands in the "Commands section) but the help messages do not show.

The workaround:

pip uninstall rich-click
python -m mcve --help

Help messages now show. My working assumption is that the issue is caused by the
patching of click that occurs in rich-click via litestar/cli/init.py, which is not executed if rich-click is not installed.
Screenshots
No response
Logs
No response
Litestar Version
2.8.3
Platform

[ ] Linux
[X] Mac
[ ] Windows
[ ] Other (Please specify in the description above)

[!NOTE]
While we are open for sponsoring on GitHub Sponsors and
OpenCollective, we also utilize Polar.sh to engage in pledge-based sponsorship.
Check out all issues funded or available for funding on our Polar.sh dashboard

If you would like to see an issue prioritized, make a pledge towards it!
We receive the pledge once the issue is completed & verified
This, along with engagement in the community, helps us know which features are a priority to our users.

I agree that we shouldn't just monkey-patch things out of the box.
I'm curious however what your use case is that involves making use of the Litestar CLI and typer simultaneously?

I am not using LItestar CLI, other than my assumption that it is perhaps used for the litestar command itself. I typically use a mix of Typer, argparse, and sometimes Click for building out command line tools such as administrative utilities, backend processing, etc.
It is not so much that I am using them "simultaneously", i.e. Litestar would typically run in its own runtime environment from whatever utilities and processes I have. The issue is more that they share the same code. The ORM code is a particularly strong case in point, although I don't really expect that to be the only conflict. In my case however, it does seem that orm code is directly implicated: I ran into this issue trying to build a typer-based utility and traced the issue's source through interdependencies between Litestar itself and advanced-alchemy. Even if you are not using Litestar CLI directly, indirect imports will cause the patch to happen. At least I have found this to be the case when advanced-alchemy is being used. I have not yet tried to isolate this issue in a codebase that does not utilize advanced-alchemy.

Ok, looking back at the code I provided, and digging through my history of chasing this down, I see the point you are making. I do not have any examples of imports from Litestar proper that cause this problem other than the one above of directly importing litestar.cli. In this sense, I see your point about them being simultaneous.
However, if you install litestar[sqlalchemy] and make just about any import from advanced-alchemy, the issue will show itself. This happens due to a number of interdependent imports within advanced-alchemy that result in the cli module being loaded and the implicit patch executed.
So, yes, depending on your perspective, you might consider this to be more of an advanced-alchemy issue than a Litestar issue. I suppose it comes down to whether or not any other similar implicit imports might happen within Litestar per se, and/or whether general click interoperability is something you see as a priority.


PR from abrombo/galgebra:master for reference (with merge conflicts solved)

A variant of #52 containing the (trivial) manual merge(s) needed to make the diff readable

Created https://github.com/abrombo/galgebra/pull/2 to get this merge into @abrombo's hands

Rebased with some flake8 fixes and removals of stray files.
I think all that's really in here now is an incomplete odot implementation, and the printer stuff.

Conflicts solved again

Thank you @abrombo @eric-wieser , this should be superseded by #510 which merged gprinter from upstream. Feel free to reopen if there are remaining issues.


Parameter chip

Description
Add dropdown functionality to click on parameter chips and be able to change filter parameters in the filter builder.
Screenshots

License
I confirm that this contribution is made under the terms of the license found in the root directory of this repository's source tree and that I have the authority necessary to make this contribution on behalf of its copyright owner.

Looks like we have a scroll issue:


No support for pytest-cov 4.x.x?

The setup.py is pinned to 3.* for pytest-cov. Is this intentional? pytest-cov 4.0.0 was released in Sep 2022, so I wanted to just check there isn't a way to loosen this restriction.

This should be supported on version v0.3.0


Fix the return value of Cluster.findTopicPartitionMetadata

The return value is not an Object, but an Array (of PartitionMetadata objects). This also means we can
simply check the number of entries and don't need to calculate the keys when sending messages.
Note that this exposed a bunch of problems in the tests as well: the mocked calls not just returned the
wrong type, but in fact returned a completely unrelated value.

Thanks. If you fix the conflict I'll merge this.

Turns out I didn't split another PR properly, and the actual change in the type was already merged with 78fefe2, so this PR is only updating the users to match as well.
(Rebased onto current master)

And now this needs a merge as well.

There you go :)


Add taskids into history

Summary:
Adds taskids to the history of job runs. This is needed to be able to query logs by task id when tasks are finished.
This is from a test cluster I started this morning.
{
   "id":"test",
   "labels":{

   },
   "run":{
      "cpus":1,
      "mem":128,
      "disk":0,
      "gpus":0,
      "cmd":"sleep 1",
      "env":{

      },
      "placement":{
         "constraints":[

         ]
      },
      "artifacts":[

      ],
      "maxLaunchDelay":3600,
      "ucr":{
         "image":{
            "id":"alpine",
            "kind":"docker",
            "forcePull":false
         },
         "privileged":false
      },
      "volumes":[

      ],
      "restart":{
         "policy":"NEVER"
      },
      "secrets":{

      }
   },
   "activeRuns":[

   ],
   "history":{
      "successCount":1,
      "failureCount":2,
      "lastSuccessAt":"2019-06-11T08:28:17.722+0000",
      "lastFailureAt":"2019-06-11T08:26:15.428+0000",
      "successfulFinishedRuns":[
         {
            "id":"20190611082632T0w4y",
            "createdAt":"2019-06-11T08:26:32.342+0000",
            "finishedAt":"2019-06-11T08:28:17.722+0000",
            "tasks":[
               "test_20190611082632T0w4y.dc3c80e6-8c22-11e9-89ca-f6185111971b"
            ]
         }
      ],
      "failedFinishedRuns":[
         {
            "id":"20190611082145nG1ta",
            "createdAt":"2019-06-11T08:21:45.262+0000",
            "finishedAt":"2019-06-11T08:26:15.428+0000",
            "tasks":[
               "test_20190611082145nG1ta.2ea877e5-8c22-11e9-89ca-f6185111971b"
            ]
         },
         {
            "id":"201906110815512U9uW",
            "createdAt":"2019-06-11T08:15:51.321+0000",
            "finishedAt":"2019-06-11T08:21:24.814+0000",
            "tasks":[
               "test_201906110815512U9uW.4b709754-8c21-11e9-89ca-f6185111971b"
            ]
         }
      ]
   }
}

JIRA issues: DCOS-54879

@kensipe the API change just happened "automagically" :) because we serialize whatever is in JobRunInfo
we do need a release notes!! thanks!

@alenkacz  I do have one concern that nagged at me in the review and I should raise the conversation to make sure we are on the same page (and potentially clean up some of the API).
We only allow for 1 task per job run... why would we represent this as a collection?   It seems better to have "a task id" not a collection of "tasks".   perhaps this is to avoid the confusion between id (which is the job run id) and a task_id... but it is implied with this solution.


docs(components): fix jumping checkboxes on tables

fix jumping checkboxes on tables

@shadcn would you mind taking care of this PR?


Added a bunch of exclusions to .gitignore

Migrated PHPUnit config to latest schema to remove a warning when running test
Updates to build-phar.php
Completed renames in composer.json (VCS repo and App name)

This branch also includes a fix to CI Test runs on PHP 7.4 @hboutemy


local_quadratic_regression fails

Hi,
As I was preparing a notebook using WBT and the "whitebox" python package (not whitebox-workflows), I got this error while running local_quadratic_regression:
./whitebox_tools --run="LocalQuadraticRegression" --wd="/mnt/d/Dropbox/USP/extensao/videos_whitebox" --dem='CopDEM_clip_petar.tif' --output='CopDEM_clip_petar_wood_w9' --filter=9 -v --compress_rasters=True

***************************************
* Welcome to LocalQuadraticRegression *
* Powered by WhiteboxTools            *
* [www.whiteboxgeo.com](https://www.whiteboxgeo.com/)                 *
***************************************
Reading data...
thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', whitebox-plugins/src/local_quadratic_regression/main.rs:218:61
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Failure to run plugin subprocess.

0

The python package was installed today with pip and then I downloaded WBT binaries with:
whitebox.download_wbt(linux_musl=True, reset=True)

Just to complement. I tested today with a Windows version (with WhiteboxRunner), and it ran fine. So it is likely related to the MUSL version

Hi Carlos, as you know, I didn't write this tool, but rather Dan Newman did. So I'm not as familiar with this tool as I am many of the others. However, I am confused as to why it would work on Windows and not on Linux for you. The code is exactly the same regardless. Do you have a copy of CopDEM_clip_petar.tif that you can share with me for testing?

Sure, here it is: https://www.dropbox.com/scl/fi/ixyh60i4qemm28k3x6vpm/CopDEM_clip_petar.tif?rlkey=oaxjkq8lzjl3gg84f9fmdch1t&dl=0
Just to provide a bit more info: the linux system is running on WSL2. I created a python virtual environment, installed whitebox with pip, and when I called import whitebox for the first time, it downloaded WBT but it didn't worked (GLIBC error), so I went with whitebox.download_wbt(linux_musl=True, reset=True) and then it worked. Well, except for local_quadratic_regression.

Looking at the line that crashed and the command used, I'm pretty sure the issue is simply that the file extension of the output raster wasn't provided but is required, so --output='CopDEM_clip_petar_wood_w9.tif' instead of --output='CopDEM_clip_petar_wood_w9'.

WHAAAT? That simple?
And it works!
But let's point out that this tool is not behaving like the others. For every other tool I used, I was just creating the name without the extension, like
az = 315
alt = 35
zf = 1

wbt.hillshade(
dem = cop_tif,
output = f'{cop}_shade_{az}_{alt}_z{zf}',
azimuth=az,
altitude=alt,
zfactor=zf)

Hi Carlos,
For all the tools that I've written, I have a bit of code in there that adds a 'tif' extension if the user hasn't provided it. I think perhaps Dan just left this out for this tool. I'll add it in myself when I get a chance.

Yes, likely just a simple check that is missing. The good thing is that it works! Closing the issue.


Fix division by zero error when verifying empty collection

If num_items is 0 then we set step to 0 instead of 1_000_000 / num_items
With this PR sugar verify works correctly for an empty collection:
>>> sugar verify

[1/2] 🍬 Loading candy machine
▪▪▪▪▪ Completed

[2/2] 📝 Verification
Verifying 0 config line(s): (Ctrl+C to abort)
[00:00:00] Config line verification successful █████████████████████████████████████████████████████ 0/0

Verification successful. You're good to go!

See your candy machine at:
  -> https://www.solaneyes.com/address/DqfEXKYn8Nct9xaY8A5xj2DUNkwZw8cJKXWS1Wwqt4oK?cluster=devnet

✅ Command successful.

Fixes #342

Thanks for the PR!


Set up domain & analytics

We have access to set up at map.fpdcc.com and can plug in our Google Analytics info at launch.

[ ] Get map.fpdcc.com subdomain set up with server
[ ] Have fpdcc.com dns set up to direct map.fpdcc.com to server
[ ] Add Google analytics code


update to zig master

Um, why did this not run CI? This should have broken there since the CI didn't update its zig version

Okay perfect, now we can make this PR include a compiler update

Confirmed this zig version works for me locally too.

Thanks!


fix docker image publication

What does this PR do?
Fixes #3604
Will require executing docker image publication outside of a release.

Doesn't this PR have the same goal as #3600? I also think @v1v wanted to manually add the missing 1.49.0 image after that PR is merged


Class Slack already defined

After installing farady manually, running puppet agent -t gives the following error:
Error: Could not send report: Error 400 on SERVER: Could not autoload puppet/reports/slack: Class Slack is already defined in Puppet::Reports

Any where besides slack.rb has this bit:
Puppet::Reports.register_report(:slack)

Could slack.rb be installed in multiple places some how? ... hummm.
I have multiple environments with this class in each environment.  The puppet master is running the production environment though.

FWIW, this issue is currently tracked by Puppetlabs in https://tickets.puppetlabs.com/browse/PUP-1189 and it happens when the report processor has some error.
The quick workaround is to restart the puppetmaster after fixing the report processor error.


⚠ Samsung has degraded performance

In bac4622, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2968 ms

Resolved: Samsung performance has improved in 1452a73.


gateway-api: Name GAMMA controller explicitly

This is to pave the way for controller-runtime upgrade as the newer version will fail if there is naming conflict between existing HTTPRoute and GAMMA.
Relates: https://github.com/cilium/cilium/pull/34463

/test


formatting and doc crs

Addressed some formatting concerns and removed unused methods and interfaces

If you're fine with these changes you can just merge it in and it'll update your main PR


Not showing messages from Named Error Bags

Hello, for some reason I had problems showing messages from Named Error Bags.
Something like:
return redirect('register')
    ->withErrors($validator, 'login');

To access the named ErrorBag it's necessary to do:
{{ $errors->login->first('email') }}

so, I had to add the following code to the ConvertMessagesToSweetAlert.php middleware:
public function handle($request, Closure $next)
{
     ...

     if ($request->session()->has('errors')) {
            $message = $request->session()->get('errors');

             /* Added new condition */
	     if (!$message->hasBag('default')) {
	       $bagName = array_keys($message->getBags())[0];
	       $message = $message->$bagName;
	      }
	     /* End new condition */

            if (!is_string($message)) {
                $message = $this->prepareErrors($message->getMessages());
            }

            alert()->error($message)->html()->persistent();
        }

        return $next($request);
}

It should be improved, but for the moment hope it works for something.
Was there another way to solve it? Thank you.

Thank you @eleazarbr
Right now this package does not support named error bags. If you know how to implement the feature, please send a PR.

Great! Let me see what'd be the best way to do this and make a PR. Regards and nice work.


replacing np.bool with bool

replacing deprecated np.bool with bool

Codecov Report
Patch coverage: 50.00% and no project coverage change.

Comparison is base (5542f96) 71.73% compared to head (0a90ae7) 71.73%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
Additional details and impacted files
@@           Coverage Diff           @@
##           master     #124   +/-   ##
=======================================
  Coverage   71.73%   71.73%
=======================================
  Files          28       28
  Lines        1012     1012
=======================================
  Hits          726      726
  Misses        286      286

Flag
Coverage Δ

unittests
71.73% <50.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pycomlink/processing/baseline.py
31.34% <ø> (ø)

pycomlink/processing/wet_dry/stft.py
0.00% <0.00%> (ø)

pycomlink/processing/blackout_gap_detection.py
100.00% <100.00%> (ø)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


into ImpalaContext from HDFS CSV/Parquet

The impyla.bdf.from_hdfs function creates an Impala table from a CSV file or Parquet file already living on HDFS.  It would be swell to leverage these for into functions.
cc @quasiben

moving to into