JadxArgsValidateException Output directory exists as file

jadx.core.utils.exceptions.JadxArgsValidateException: Output directory exists as file /Users/gucheng/Downloads/qianji.apk
please help me, I have uninstall and clone install many times, but can not fix this error.
I can not find the out dir.
求助，已经不能够正常使用了，用了下 网上推荐的 jadx-gui -d out ***.apk 命令。
删除  重新安装都没啥用。

@skylot

-d out ***.apk

Use different value for -d parameter

-d out ***.apk

Use different value for -d parameter

I try many different apks , but always this error

@otwayz I can't reproduce your issue.
Maybe you need to escape some characters in your file name or put it inside quotes (like jadx-gui "app.apk")
Please add -v option and post full command line and full output, also post jadx version you use?
P.S. I don't see the reason to add -d option to jadx-gui, because output path will be asked on export start.

I believe he tries to use -d app.apk

I use jadx-gui in command line and select app.apk by file system. This error  occured.  Maybe I have used error command and dirty the jadx env.  What can I do to clean the jadx env?

./jadx-gui -v ~/Downloads/release/app-google-release.apk
ERROR - Error load file: /Users/gucheng/Downloads/release/app-google-release.apk
jadx.core.utils.exceptions.JadxArgsValidateException: Output directory exists as file /Users/gucheng/Downloads/qianji.apk
	at jadx.api.JadxArgsValidator.checkDir(JadxArgsValidator.java:102)
	at jadx.api.JadxArgsValidator.validateOutDirs(JadxArgsValidator.java:62)
	at jadx.api.JadxArgsValidator.validate(JadxArgsValidator.java:18)
	at jadx.api.JadxDecompiler.load(JadxDecompiler.java:83)
	at jadx.gui.JadxWrapper.openFile(JadxWrapper.java:36)
	at jadx.gui.ui.MainWindow.openFile(MainWindow.java:187)
	at jadx.gui.ui.MainWindow.open(MainWindow.java:144)
	at jadx.gui.JadxGUI$$Lambda$16/741669172.run(Unknown Source)
	at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311)
	at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:744)
	at java.awt.EventQueue.access$400(EventQueue.java:97)
	at java.awt.EventQueue$3.run(EventQueue.java:697)
	at java.awt.EventQueue$3.run(EventQueue.java:691)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:75)
	at java.awt.EventQueue.dispatchEvent(EventQueue.java:714)
	at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
	at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
	at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)
DEBUG - Loaded settings: {"lastOpenFilePath":"/Users/gucheng/Downloads/release","lastSaveFilePath":"/Users/gucheng","flattenPackage":false,"checkForUpdates":false,"recentFiles":["/Users/gucheng/Downloads/release/classes.dex","/Users/gucheng/Downloads/release/classes2.dex","/Users/gucheng/Downloads/release/app-google-release.apk","/Users/gucheng/Downloads/qianji.apk","/Users/gucheng/Downloads/biquge.apk","/Users/gucheng/tools/jadx-0.8.0/bin/dr"],"fontStr":"","editorThemePath":"/org/fife/ui/rsyntaxtextarea/themes/default.xml","langLocale":{"locale":"zh_CN"},"autoStartJobs":false,"excludedPackages":"","showHeapUsageBar":true,"settingsVersion":7,"windowPos":{"SearchDialog":{"windowId":"SearchDialog","x":440,"y":286,"width":800,"height":500}},"outDir":"out","outDirRes":"/Users/gucheng/Downloads/qianji.apk","skipResources":false,"skipSources":false,"exportAsGradleProject":false,"threadsCount":1,"showInconsistentCode":false,"useImports":true,"replaceConsts":true,"escapeUnicode":false,"respectBytecodeAccessModifiers":false,"deobfuscationOn":false,"deobfuscationMinLength":3,"deobfuscationMaxLength":64,"deobfuscationForceSave":true,"deobfuscationUseSourceNameAsAlias":true,"cfgOutput":false,"rawCfgOutput":false,"fallbackMode":false,"printVersion":false}
DEBUG - Saving settings: {"lastOpenFilePath":"/Users/gucheng/Downloads/release","lastSaveFilePath":"/Users/gucheng","flattenPackage":false,"checkForUpdates":false,"recentFiles":["/Users/gucheng/Downloads/release/app-google-release.apk","/Users/gucheng/Downloads/release/classes.dex","/Users/gucheng/Downloads/release/classes2.dex","/Users/gucheng/Downloads/qianji.apk","/Users/gucheng/Downloads/biquge.apk","/Users/gucheng/tools/jadx-0.8.0/bin/dr"],"fontStr":"","editorThemePath":"/org/fife/ui/rsyntaxtextarea/themes/default.xml","langLocale":{"locale":"zh_CN"},"autoStartJobs":false,"excludedPackages":"","showHeapUsageBar":true,"settingsVersion":7,"windowPos":{"SearchDialog":{"windowId":"SearchDialog","x":440,"y":286,"width":800,"height":500}},"outDir":"out","outDirRes":"/Users/gucheng/Downloads/qianji.apk","skipResources":false,"skipSources":false,"exportAsGradleProject":false,"threadsCount":1,"showInconsistentCode":false,"useImports":true,"replaceConsts":true,"escapeUnicode":false,"respectBytecodeAccessModifiers":false,"deobfuscationOn":false,"deobfuscationMinLength":3,"deobfuscationMaxLength":64,"deobfuscationForceSave":true,"deobfuscationUseSourceNameAsAlias":true,"cfgOutput":false,"rawCfgOutput":false,"fallbackMode":false,"printVersion":false}

This is detail info . in this log, the out dir is my first time to  use jadx to decompile apk, but that maybe 15 days ago.

Oh ,  I only delete the first time decompiled apk file "outDirRes":"/Users/gucheng/Downloads/qianji.apk" , The error fixed !   Thanks for your help,  maybe you need init the env when error occured.

@otwayz thank you! Output directories must not be saved in settings, I have fixed that in latest commit.
And if you want to reset jadx-gui settings you can press Reset button in the settings dialog, this is another solution for your issue.


ESP32 S3 TinyUSB problems with USB unplugging detection (IDFGH-11193)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.
esp-idf-v5.1.1
Espressif SoC revision.
ESP32 S3
Operating System used.
Linux
How did you build your project?
Eclipse IDE
If you are using Windows, please specify command line type.
None
Development Kit.
ESP32-S3-WROOM-1
Power Supply used.
USB
What is the expected behavior?
I'm using an example tusb_msc from esp-idf-v5.1.1 and using spiflash to store files. My device is expected to be Self-Powered Device.
I did everything as described in this documentation in the section Self-Powered Device
const tinyusb_config_t tusb_cfg = {
.device_descriptor = &descriptor_config,
.string_descriptor = string_desc_arr,
.string_descriptor_count = sizeof(string_desc_arr) / sizeof(string_desc_arr[0]),
.external_phy = false,
.configuration_descriptor = desc_configuration,
.self_powered = true,
.vbus_monitor_io = GPIO_NUM_1
};
I expect that when I disconnect the device from USB I will have access to files on the SPI flash. But when I disconnect the device from the PC, the tud_umount_cb() function is not called and tinyusb_msc_storage_in_use_by_usb_host() returns true. Function tud_umount_cb() call only when I  umount device from PC, if remove device without umount from PC there is no event to determine whether the device is connected or not.
What is the actual behavior?
I expected that using the vbus_monitor_io would generate a connection and disconnection event on the USB bus.
Steps to reproduce.
Flash tusb_msc example and remove from PC without manual umount.
Debug Logs.
No response
More Information.
No response

hi @SlySquirrel could you please share your HW connection of VBUS to ESP32-S3?

Hi, I use hw design similar as https://github.com/wuxx/nanoESP32-S3 with battery supply and VBUS connection

VBUS connect only R35(20 kOm)/R36(41.2 KOm). When USB connect on IO1 about 3.3V. C24 not present.

Thanks, I'll try to reproduce the problem with your HW setup and let you know ;)

I am also having this issue.
ESP-IDF v5.1.2
If I read the GPIO with gpio_get_level, it returns the expected value with usb connected and disconnected.

Hi, I'm sorry for the extraordinary slow responses here. I'll get to it ASAP

Hello @silverchris and @SlySquirrel thank you for your patience.
Indeed, there is a regression in our tinyusb version >= 0.15, we will fix soon.
In the meanwhile, you can use tinyusb v0.14. It contains the same features for MSC device as v0.15.
You can do this be requiring specific version in your main/idf_component.yml like this:
## IDF Component Manager Manifest File
dependencies:
  espressif/esp_tinyusb: "^1.4.2"
  espressif/tinyusb: '0.14.2'

Also please make sure that the pin you are using for VBUS monitoring is not used by any other peripheral. Here's a nice summary for ESP32-S3 https://www.luisllamas.es/en/which-pins-can-i-use-on-esp32-s3/

@tore-espressif downgrading to  0.14.2 fixed it for me. Thank you!

Hi,
Question:
when the USB Storage is exposed to the Host, the Storage is no more accessible by the Application ?

so a "Shared" Access is per Standard not possible right ?

This is not per (USB) Standard but per differences how the MSC host and the application access the memory partition.
The MSC host accesses memory sectors while the application accesses files (implemented in FATFS):
in general it is impossible to ensure file consistency when the disk is mounted both to the app and to the host (Unless both the host and the device mount the FS read-only)
The problem is that if the MSC host updates the FAT, then the FAT FS code running on the device won't "know" that something on disk has changed, and won't "reload" the FAT from disk.

Just to add, MTP class tries to address this problem. Here is the issue about supporting it in tinyusb: https://github.com/hathach/tinyusb/issues/1079

@igrr very cool

https://github.com/espressif/esp-iot-solution/tree/master/examples/usb/device/usb_msc_wireless_disk

Hello everyone,
Thank you for your patience.

Indeed, there is a regression in our tinyusb version >= 0.15, we will fix soon.

New version is available: espressif/tinyusb^0.15.0~5
Please update the manifest file accordingly.


refactor: [M3-6571] - RQ-ify Events - Part 2: Event Handlers

Description 📝
See Part 1 here: #9046.
This PR finishes off the RQ Events migration, moving all event handlers over from the old system (Redux middleware + rxjs observable) to a unified events "mega-hook".
The 2 hooks from the last PR are combined into a single hook useEventsInfiniteQuery, which handles both fetching events and allows for registering event handlers for new/updated events. It was necessary to combine the polling hook into this one because we use previously-fetched events to create a list of in-progress eventIds to poll for updates.
The event handlers themselves have been moved to two new hooks, useAppEventHandlers and useToastNotifications, from <App /> where most of them used to live (a few others were scattered around the codebase -- they are all now in one place). I decided to move them out of App to reduce clutter in that file; moving them to hooks instead of components seemed appropriate because they don't render any content.
This PR also introduces a container, withEventsInfiniteQuery that allows our class-based components to use events.
What exactly do Events do ❓
Events have a few different functions in Cloud Manager:

Notify the user of ongoing/completed changes and updates to their account entities (Linodes, NodeBalancers, etc.)

These Event notifications are displayed in the Notification Menu and the Events Landing at /events.
Toast notifications appear when long running tasks complete (i.e., a Linode migration)

Trigger event handlers that update locally cached data

E.g., when a Linode is updated in another tab, the Linodes cache is invalidated and Cloud Manager requests and displays the latest details without having to refresh the page

Display the current status of an entity

E.g., when a Linode resize is in progress, this information, along with the progress will be displayed in the Linode details page

Major Changes 🔄

Creates one RQ hook, useEventsInfiniteQuery, for retrieving events and registering event handlers
Consolidate all event handlers to two new hooks, useAppEventHandlers and useToastNotifications
Delete the Events Redux store and all associated files (reduces, actions, helpers, etc.)
Remove rxjs as a dependency

How to test 🧪

Open two tabs of this version of the app.
Modify some entities in one tab and ensure those modifications are reflected in another tab (e.g., renaming/resizing/rebooting/deleting Linodes/NodeBalancers/Firewalls, etc.)
Perform an event that takes some time to complete and reports progress updates (e.g., resizing or migrating a Linode). Ensure both tabs receive and display updates as the event progresses and eventually completes.
Refresh in the middle of an in-progress event and ensure it continues to update until completion.
Perform some action that creates a toast notification (see useToastNotifications.tsx for examples of this, such as attaching a volume, resizing a disk, or migrating a Linode). Ensure this toast actually appears in both tabs.

Additional things to check:

Ensure all event handlers in useAppEventHandlers are registered and invoked as expected
Ensure all toast notifications in useToastNotifications work as expected
Ensure the events polling interval (via resetPollingInterval) is correctly reset when adding/updating/deleting/performing actions on entities
Ensure the NotificationMenu, EventsLanding and LinodeActivity components continue to display events as expected
Ensure unit and E2E tests pass

1,500+ fewer LOC to maintain between Part 1 and Part 2! 🎉

Noticing a couple of issues while testing:

Issue
Possible Resolution

Events consumed by one tab are missed by another open tab due to the seen-based filtering used in polling
Filter based on created time instead of seen

Multiple calls to /events API every polling cycle due to different polling filters
Use the same polling query and filter client-side

EventsLanding re-downloads all loaded pages when new events are detected
Use 'page-zero' idea to add new events instead of reloading all events; disable EventsLanding when not visible

I'm seeing more GETs than I'd expect when loading Cloud Manager directly to http://localhost:3000/events. How many do we expect to see? Also, is it okay we request pages 0, 2, then 3?

It should be the same as prod (i.e., one initial request followed by polling requests at increasing intervals). In the screenshot, I think the 3 requests starting with '5cd...' are from LD.
Starting from page zero was a testing change I made that I accidentally committed (although it has no effect since the API returns page 1).

Good eye @mjac0bs. Added sorting in-progress events to the top in EventsLanding to match NotificationMenu.

PR is looking solid but I don't quite understand why the events load into the dom in batches less then 25... What am I not seeing in the code that causes this?
https://github.com/linode/manager/assets/115251059/8117ead8-9891-40f2-bca3-e822748c3708

PR is looking solid but I don't quite understand why the events load into the dom in batches less then 25... What am I not seeing in the code that causes this?
Screen.Recording.2023-07-17.at.9.50.45.AM.mov

Some types of events, like profile_update, are not displayed.


Adding config state handling before/after line comments

Added functions to serve as hooks for saving some specific options before commenting and recovering them afterwards. I found out that folding plugins can be really slow (mainly because each line changed triggers evaluating foldexpr), so I set foldmethod to manual, do the commenting and then recover what was configured.
The ignorecase option was also changed, so I moved both to get a consistent way of saving and recovering options.

Fork notice: With the upstream project sitting stale after nearly 8 months since the last activity, 71 outstanding pull requests, no comments from the original developer, and dozens of developers duplicating efforts while scratching their own itches, I've decided to step in and run a fork for a while. Hopefully the original developer will eventually consider picking this plugin up again or adding more maintainers so that the user base doesn't end up fragmented, but for now I would invite anyone interested in the latest updates to consider installing from and contributing pull requests to my fork.
About this PR: I have merged this PR as is. Please consider installing the plugin from alerque/nerdcommenter for this and (so far) over two dozen other updates.

Change of plans. Thanks to scrooloose who has added me as a maintainer to this repo, I'll be going through and taking care of business here instead of in a fork. This PR has been merged directly ustream.
Thanks for your contribution @suoto.


SolarEdge Component Serial Numbers Missing from Device Info and Meter Option/Type missing from Meter Entity Device Info

Describe the feature

The individual SolarEdge system components (e.g. inverters, battery unit(s), backup interface, production meters, consumption meters, etc.) are able to report serial numbers and these should be included in the module's device enumeration.  I have not found the serial numbers within the device entity descriptions or device information.  The serial number is available from the modbus details alongside the component model numbers (as "c_serialnumber" in JSON?), and could be included for visual clarity sake, especially when dealing with multiple inverters and meters.

The meters themselves report the type of meter they are (along with their serial number).  This information should also be included in the module's device details.  Meters show their type within the modbus details (as "c_option" in JSON?) for meter types like:
"Export+Import"
"Production"

Perhaps my understanding is off base, but I think the python solaredge_modbus module may in fact be missing on reporting of some meters (likely because the values are derived) based on what is shown by the SolareEdge Monitoring site, but that is a separate problem not withstanding the feature request herein.

Home Assistant (please complete the following information):

Home Assistant Core Version:  2024.2.0
solaredge-modbus-multi Version:  v2.4.11

Additional context
For reference, I have the following Solar Edge US system components:
SE Energy Hub 7.6kW with LG RESU10H battery and string of 30 panels
SE Energy Hub 3.8kW (follower, no battery) and string of 12 panels
SE Backup Interface (Whole Home)
The solaredge-modbus HA module (and the latest solaredge_modbus v0.8.0 python native version) report my system has 3 meters, however, the SolarEdge Monitoring Site Details page indicates 6 meters, 3 (+1) with unique serial numbers, so it's a bit confusing:

Production Meter Type SE-RGMTR-1D-240C-B - attached to Inverter 1
Production Meter Type SE-RGMTR-1D-240C-B - attached to Inverter 2
Export Meter Type SE-MTR-3Y-240V-A (*) - attached to Inverter 1
Import Meter Type SE-MTR-3Y-240V-A (*) - attached to Inverter 1
Self Consumption Meter  (no serial, likely derived?) - attached to Inverter 1
Consumption Meter (no serial, derived?) - attached to Inverter 1

Export & Import Meters share the same serial number, but are listed separately in the site details

I don't really understand what you're asking for, the integration already reports these things.

No serial numbers are displayed on the device enumeration, only part model numbers?

   "meters": {         "Meter1": {             "c_manufacturer": "SolarEdge",             "c_model": "SE-RGMTR-1D-240C-B",             "c_option": "Production",             "c_version": "78",             "c_serialnumber": "6064xxxxx",             "c_deviceaddress": 1,             "c_sunspec_did": 203,             "c_sunspec_length": 105,

I still don't understand. The integration reports serial number in the device attributes.

I guess I must not be seeing it.   The Meter Serial Numbers and Type/Option isn't showing.  The device serial numbers aren't showing.  I see:
Device
Manufacturer
Model
Area
Integration
Battery
No serial numbers are listed?

None of those are "serial number" columns.
So your suggestion is to change HA front end to show a serial number column?

Am I missing the SolarEdge gear serial numbers somewhere else?
Perhaps serial numbers could be shown on the individual device pages?  And if possible, the reported Meter Type for Meters.
The individual Device Info pages show just about everything else, less Serial Number:
e.g.
Device info
SE7600H-USS3BBC14
by SolarEdge
Firmware: 0004.0017.0221

It already shows the meter type like this:

And like this:

So I still am not sure what you're asking for that isn't already in there.
If you think the serial number or hardware fields should be displayed in the device page columns, that's something you need to suggest to HA frontend. I don't control how that data is displayed by Home Assistant.
https://github.com/home-assistant/frontend
As far as serial numbers I can add that to the device properties, but again, I don't control how HA will display it.

Release v2.4.13-pre.1 adds serial number to device properties.
As far as having serial numbers or other information beyond what currently shows up in the device list, it's a good suggestion to have the option to display that information in the device list, but unfortunately it's nothing I can control from the integration. That would be something HA frontend needs to do.

Yep, it sure is.  My bad on the meter details.  It is there in plain sight.   SolarEdge monitoring must derive the other meter values, or perhaps they are pulling from the Backup Interface somehow since I'm quite sure it has a CT meter as well.
I can't seem to find the Inverter Serial Numbers in the device details.  Though, it looks like there is already a request out there for that.
Thanks for your time and efforts!  This truly is great stuff!   Also, apologies for my confusion.

Serial number added to device info in PR #557.


Tutorial for reusing a SUT class.

This is probably the most common way someone will add a SUT, but maybe not the first SUT they add.

Looks good!


Unable to deploy a 'Compute Instance' User Resource to a Workspace AML Service

Deployment of AML Compute Instance fails
When adding a Compute Instance to a TRE Workspace AML Service, the deployment fails with the following error:
desired number of dedicated nodes could not be allocated. This error has been happening consistently for the past
two days. Have not tried it before then with this version of the TRE.
This error occurs when deploying via:

the TRE UI using the 'aml_compute' user-resource template and
Logging into the AML Workspace from a workspace VM and trying to create new compute instance

Steps to reproduce

Create New Workspace and User
Add User to 'Workspace Owners'
Add User to 'Workspace Researchers'
Login to TRE UI with User account
Add Virtual Desktops (Guacamole) Service to Workspace
Add a User Resource (VM) to Virtual Desktops Service
Add Azure ML Service to Workspace ('expose externally' = False)
Add a Compute instance User Resource to AML Service

Additional Steps taken

Grant User 'Network Contributor' on the TRE Workspace VNet
Grant User 'AzureML Compute Operator' on the Workspace AML Workspace

Additional Info

there is sufficient quota for the selected compute-size in my deployment region
I have tried a number of different compute sizes
I have confirmed that there are free IP addresses in the AML Subnet.
all resources are deployed to the UK South Region.

Azure TRE release version: v0.19.1
tre-workspace-base: 1.5.7
tre-service-azureml: 0.8.11
tre-user-resource-aml-compute-instance: 0.5.7
deployment location: UKSouth

Hi @dram1964, can you create an AML in the portal manually?

Interesting development - Decided to re-deploy the AML Service into a workspace, this time with expose externally set to True. When I tried to add a compute instance from the user resource template it succeeded, and I can connect and run code on it.

Could potentially be something to do with private endpoints within the vnet?

@dram1964 did you get any further with this? If it is compute size, it doesn't really make sense that it works in one network configuration, but not the other. As @tim-allen-ck says if can deploy the instance through the AML studio it would be useful to identify if the issue is the templates in this project, or a subscription/quota issue.

@marrobi , @tim-allen-ck:  I couldn't see any quota issues with private endpoints in the subscription/region (25/65,000). I've destroyed my original TRE deployment, and created a new one (without my custom templates) in the same subscription/region: but I'm still having the same issue.

Have you tried to create the compute instance via the AML studio?

Only on a Workspace that had public access. A private AML workspace looked a bit complicated on the Portal - seems I need to create a VNet beforehand to then create private endpoints.

Just a quick update: thought I'd try a re-deployment of the TRE in the westeurope region: but the same error occurs when trying to deploy the compute instance.
So I'm going to manually deploy a private AML instance and see how that goes. I'm going to use the terraform quickstart template rather than attempt this in the portal.

I've setup a AML workspace with public_network_access_enabled = false, and successfully deployed a compute instance using this code.

Have you tried to create the compute instance via the AML studio?

Hi @marrobi - may have misread this question originally. I've had the same error trying to deploy the compute instance to a private 'azureml' Workspace Service in two different ways:

Using the TRE Portal and deploying a 'aml_compute' user resource
Using the AML studio from a VM within the workspace


[ALLUXIO-3290] Update UFS mode to journal as a part of modify

https://alluxio.atlassian.net/browse/ALLUXIO-3290

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/252/
Test PASSed.


Refitter failed to write generated code: System.IO.IOException: The process cannot access the file 'Generated.cs' because it is being used by another process.

Describe the bug
Refitter failed to write generated code: System.IO.IOException: The process cannot access the file 'Generated.cs' because it is being used by another process.
When I run the following commands in my build pipeline:

dotnet restore
dotnet pack

The dotnet restore works as expected, however the dotnet pack fails with the exception above. My assumption is the dotnet restore regenerates the file, doesn't close the stream and when dotnet pack runs, it complains about the file being in use.
I have seen similar behaviour when trying to move the generated source into a different directory in Rider. Where I have to close Rider then move the file and then open Rider again.
Im not sure what is causing the file to still be in use. As File.WriteAllText should release the file immediately after writing. However this is still an issue.

@fabioloreggian Thanks for taking the time to report this
Have you tried shutting down the C# build-server? I have run into this a couple of times and this helped when the source generator started acting up
dotnet build-server shutdown

I have not used dotnet pack with Refitter yet as dotnet build runs the source generator. I use Refitter from my CI/CD pipeline and in this case I only want to build once, so I use the --no-build argument with dotnet when I do things like dotnet test or dotnet publish

@all-contributors please add @fabioloreggian for bugs

Thanks for the response.
I will try updating the build pipelines config to only trigger the build once. Hopefully that will sort out this issue.

Thanks for the response.
I will try updating the build pipelines config to only trigger the build once. Hopefully that will sort out this issue.

If you use the source generator, you will probably need to build 2x, as the source generator produces actual files that are written to disk post-compilation, so you will need to rerun the build to build the Refit types.
The way I do this is that I use dotnet build to trigger Refitter, then I do the release build once the Refit types are generated using dotnet build --configuration Release -warnaserror /p:SourceRevisionId=x.x.x.x /p:Version=x.x.x.x

There is something interesting happening here.
I have the following:
dotnet restore -> dotnet build.
Inbetween the restor and build I run dotnet build-server shutdown.
I can see the build server shutting down successfully. However the build step is failing with the same error as before.
Also just an FYI the built code is already checked into the repo. (not sure if that makes any difference)

@fabioloreggian, does this only happen in your CI/CD pipeline and not locally? I mostly use Refitter on projects with a CI/CD pipeline on Azure DevOps using Microsoft-hosted agents running Ubuntu and have no problems.

Yeah it is only happening in our Azure devops pipeline. It is super strange. Because it is repeatable every time.
I did play around with the dotnet commands and the ordering. But nothing really seemed to work. I will do some more investigation tomorrow.

@fabioloreggian I would like to investigate this further. May I ask what version of .NET you target and what build agent you're using on Azure DevOps? We build, test, package, and deploy everything on ubuntu-latest. I have over 100 deployment pipelines running Refitter to generate client SDK's and don't really have any problems


A question about perferance.

A great job. Your work solves a wider range of LT problems.
But I m confused with TADE performance on the vanilla LT test set.
Actually, with the same backbone and training strategy, the following methods adopt almost the same loss, but the top-1 ACC varies, for example on CIFAR100-LT-IR-100:

ICLR'21  logit adjustment [43.89% cf. origin paper Tab.3 ]
CVPR'21 LADE without test prior [45.6% cf. this paper Tab.8(a)]
NeurIPS'20 Balanced Softmax which can be rewritten as Eq.3 in this paper [46.1% cf. this paper Tab.8(a)]

In such a situation, TADE should get the best performance when the expert E2 (Eq.3 in this paper) mainly works. If so, it should not outperform the above methods by a large margin, right?
However, the TADE's top-1 ACC is 49.8% (cf. this paper Tab.8(a)) and the weight of experts is [0.40 0.35 0.24] (cf. this paper Tab. 12). The E1 mainly works.
So I just wondering how to explain the improvement of TADE on the vanilla test dataset?

Hi, thanks very much for your attention to our work.

In such a situation, TADE should get the best performance when the expert E2 (Eq.3 in this paper) mainly works.

Yes. As shown in the below table (c.f. Table 11 in the appendix),  the forward expert E2 performs the best.

However, the TADE's top-1 ACC is 49.8% (cf. this paper Tab.8(a)) and the weight of experts is [0.40 0.35 0.24] (cf. this paper Tab. 12). The E1 mainly works.

This is quite an interesting phenomenon. We also considered this question before, when we saw the learned weights are not equal on the uniform test distribution of some datasets. It does not meet our initial expectation that the three weights should be roughly equal and the overall performance is the same as the average ensemble (i.e., without using our test-time self-supervised aggregation strategy). However, as shown in the below table (c.f. Table 13 in appendix), the performance on the uniform test distribution of CIFAR100-LT-100 is improved by 0.4% by our test aggregation strategy.

Therefore, we thought it is not a technical issue, which generally leads to performance degradation. Here, we give one potential speculation for this phenomenon. As shown in the above first table, the average performance of the forward expert E1 on all classes is higher than that of the backward expert E3 on CIFAR100-LT-100. Considering this difference, the optimal weighting scheme on the uniform test distribution may not necessarily be the average ensemble; instead, it may mean a better trade-off among these three experts. Following this perspective, although there is no data number imbalance on the uniform test distribution, the results show that our test-time aggregation strategy can adaptively achieve a better trade-off among experts that leads to better overall performance. Such a phenomenon, demonstrating a potential advantage of our test-time aggregation strategy on the uniform test distribution, also surprises us.
In addition, since the performance improvement on the uniform test distribution by our test-time aggregation strategy is slight, it is okay to directly use the average ensemble (w/o the test-time aggregation) in practice, if you know the actual test distribution is uniform in advance. Happy to discuss further.

Thank you for your patient and detailed reply! This may be worth further study. It seems that just a multi-expert architecture can improve LT without special aggregation strategies. It is amazing. Anyway, many thanks again！

It seems that just a multi-expert architecture can improve LT.

Yes, this has been demonstrated. You may refer to a recent survey of deep long-tailed learning (https://arxiv.org/pdf/2110.04596.pdf) for more related work of ensemble-based LT (c.f. Sec 3.3.4 and Sec 4 in this survey).

without special aggregation strategies.

Aggregation is still important, especially when you face the test-agnostic LT scenarios that are more practical. Also, I agree with your opinion that this is worth further studying.


[GSOC] Added Image Fetch

Added coroutine Image fetch from the device.

Tasks remaining.

[ ] Permission
[ ] Folder count
[ ] Image thumbnail
[ ] Sha1 of original image
[ ] Documentation.

Codecov Report

Merging #4449 (7a9650c) into gsoc21-custom-picture-selector (dc26621) will decrease coverage by 0.04%.
The diff coverage is 0.00%.

@@                         Coverage Diff                          @@
##             gsoc21-custom-picture-selector    #4449      +/-   ##
====================================================================
- Coverage                             10.08%   10.04%   -0.05%
  Complexity                              482      482
====================================================================
  Files                                   357      358       +1
  Lines                                 13539    13595      +56
  Branches                               1114     1123       +9
====================================================================
  Hits                                   1365     1365
- Misses                                12103    12159      +56
  Partials                                 71       71

Impacted Files
Coverage Δ

...e/nrw/commons/customselector/helper/ImageHelper.kt
0.00% <0.00%> (ø)

...commons/customselector/ui/adapter/FolderAdapter.kt
0.00% <ø> (ø)

...tomselector/ui/selector/CustomSelectorViewModel.kt
0.00% <0.00%> (ø)

...mmons/customselector/ui/selector/FolderFragment.kt
0.00% <0.00%> (ø)

...mons/customselector/ui/selector/ImageFileLoader.kt
0.00% <0.00%> (ø)

...ommons/customselector/ui/selector/ImageFragment.kt
0.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dc26621...7a9650c. Read the comment docs.

Up for review @nicolas-raoul, @madhurgupta10.
Folder framgment is almost final.
State of images in image fragment would change depending on the selected and uploaded images.

It works great!
The scrolling is as smooth as before, thumbnails load very quickly.
Images seem to be faded compared to other gallery apps, maybe some setting when loading thumbnails? Or when applying the overlay?
Colors seen on Galaxy's stock gallery, the open source OpenGallery app, Google Photos, as well as when seeing each of the pictures themselves:

Colors seen in the Commons custom selector:

Disabled overlay, now it should work just fine.

Ah, I understand now, the fading was intentional, sorry I forgot that! 😅


add nix flake support

This PR adds:

a homepage definition to [package] in Cargo.toml,
a flake.nix which allows building oranda with nix build and dropping into a dev shell with nix develop,
a line to .gitignore preventing accidental commits of the nix build result directory.

Note: I have only tested this on aarch64-darwin and x86_64-linux.

This actually adds a flake.lock too, for waves hands ~reproducibility~


Introduce Switch.padding

fixes Switch has some padding that leads to uncentered UI
Code sample

expand to view the code sample
import 'package:flutter/material.dart';

void main() => runApp(const MyApp());

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      debugShowCheckedModeBanner: false,
      home: Scaffold(
        body: Center(
          child: Column(
            mainAxisSize: MainAxisSize.min,
            children: <Widget>[
              ColoredBox(
                color: Colors.amber,
                child: Switch(
                  padding: EdgeInsets.zero,
                  value: true,
                  materialTapTargetSize: MaterialTapTargetSize.padded,
                  onChanged: (bool value) {},
                ),
              ),
              const SizedBox(height: 16),
              ColoredBox(
                color: Colors.amber,
                child: Switch(
                  value: true,
                  materialTapTargetSize: MaterialTapTargetSize.shrinkWrap,
                  onChanged: (bool value) {},
                ),
              ),
            ],
          ),
        ),
      ),
    );
  }
}

Default Switch size

Update Switch size using the new Switch.padding to address  Switch has some padding that leads to uncentered UI

Pre-launch Checklist

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] I read the Tree Hygiene wiki page, which explains my responsibilities.
[x] I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
[x] I signed the CLA.
[x] I listed at least one issue that this PR fixes in the description above.
[x] I updated/added relevant documentation (doc comments with ///).
[x] I added new tests to check the change I am making, or this PR is test-exempt.
[ ] I followed the breaking change policy and added Data Driven Fixes where supported.
[x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

I've been thinking about breaking padding too but I still went for it since it has same padding on Android as well as specs. If this breaks too many tests, we could consider something optional. WDYT? :)

I've been thinking about breaking padding too but I still went for it since it has same padding on Android as well as specs. If this breaks too many tests, we could consider something optional. WDYT? :)

Sounds good! Not sure why Google testing is not triggered🧐

@QuncCccccc WYDT about this? https://github.com/flutter/flutter/pull/149884#discussion_r1647183901 :)

@QuncCccccc Please see this https://github.com/flutter/flutter/pull/149884#discussion_r1668595305

@QuncCccccc
Thanks so much for the feedback, really appreciate it.
This wouldn't have been easy to land due to breaking changes so we decided to go for padding parameter.

if this is not default, I guess you/we need to introduce this attribute on ListTileSwitch too?

if this is not default, I guess you/we need to introduce this attribute on ListTileSwitch too?

You can wrap a SwitchTheme with just the list or column containing ListTileSwitch. Best to use composition instead of adding same parameter again in ListTileSwitch and tile theme.

Shouldn't it be the default since this is different than M3 specs?

Shouldn't it be the default since this is different than M3 specs?

Since switch is a common widget. Simply changing the defaults isn't feasible considering the result is minor. This parameter is both optional and provides more flexibility than just updated default padding.


Finding best fitting hit fails for JAMP and BOLDigger method

Hi,
I having issues getting the top hit for the JAMP and BOLDigger methods. I am using the most recent version.
Error message (BOLDigger method, JAMP method fails with the same error):
$ boldigger-cline digger_hit BOLDResults_COI_cluster_reps_curated_no_contam.xlsx
12:08:01: Opening resultfile.
12:08:23: Filtering data for JAMP hits.
Traceback (most recent call last):
File "c:\users\nauras\programs\python\python39\lib\runpy.py", line 197, in _run_module_as_main
return run_code(code, main_globals, None,
File "c:\users\nauras\programs\python\python39\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Users\Nauras\Programs\Python\Python39\Scripts\boldigger-cline.exe_main.py", line 7, in
File "c:\users\nauras\programs\python\python39\lib\site-packages\boldigger_cline_main.py", line 70, in main
digger_sort.main(args.xlsx_path)
File "c:\users\nauras\programs\python\python39\lib\site-packages\boldigger_cline\digger_sort.py", line 15, in main
jamp_hits = [jamp_hit(otu) for otu in otu_dfs]
File "c:\users\nauras\programs\python\python39\lib\site-packages\boldigger_cline\digger_sort.py", line 15, in
jamp_hits = [jamp_hit(otu) for otu in otu_dfs]
File "c:\users\nauras\programs\python\python39\lib\site-packages\boldigger\jamp_hit.py", line 55, in jamp_hit
threshold, level = get_threshold(df)
File "c:\users\nauras\programs\python\python39\lib\site-packages\boldigger\jamp_hit.py", line 13, in get_threshold
elif threshold >= 98:
TypeError: '>=' not supported between instances of 'str' and 'int'
There seems to be an error with how similarity values are being assessed? The first hit method works fine. My results file is attached.
BOLDResults_COI_cluster_reps_curated_no_contam.xlsx
Cheers
Nauras

I ran the exact same command on a HPC now and it worked. Not sure what is going on here.

Also fixed with the newest version.

yes, cool

Hi,
I experienced a very rare error today, which imho doesn't need a separate thread. So I just report here in case it's useful for anyone in the future. Similar to Nauras' case, my process crashed in the step of Filtering data for JAMP hits within digger_hit, but with an "IndexError: list index out of range".
The reason appeared to be empty lines in the middle of the BOLDResults xlsx file - a single batch got saved incorrectly without crashing the identification process. Looks like some glitch, because it happened only in one out of >10 000 batches that I ran over the weekend in multiple terminal windows simultaneously. I could continue with finding best hits after I simply put that batch of sequences into a separate fasta, ran identification for it and replaced the empty lines in BOLDResults by the new result.
Cheers!


onchain-signer example runs thrice, with different Truffle versions

See: https://github.com/oasisprotocol/sapphire-paratime/actions/runs/6184389347/job/16788067786
We had a matrix which was trying multiple versions of Truffle for the examples/ projects.
However, if you add a new item to the matrix it will end up running the tests once for every version of truffle (even though it doesn't use truffle).
TODO: split the multi-version Truffle matrix from the rest of the tests

Fixed by #185


Implement whence=2 for the utils.PackedObjectReader.seek method

This value for whence is used for example by the constructor of the File class of the h5py package. Currently users will have to implement a workaround with first reading the file entirely into memory or write to disk to a temporary file.

The issue with implementing this is that, for compressed objects, seeking backwards cannot be implemented efficiently (at least not in the easy manner): every time you seek backwards, you need to get the total size, convert into a positive offset, and seek back from the beginning (i.e., decompress again the whole file).
I suggest creating a mid-size HDF5 file and checking performance of random access in the file:

time to extract it + time to seek from a disk uncompressed copy
time to seek randomly directly from the packed compressed file

I think there might be slightly more clever ways to do it - like keeping a "cache" of where to restart from in the compressed stream, e.g. every 1MB or so, that needs to include the uncompressed and compressed offsets, and the part of the buffer that the decompresser needs from the previous chunk of file. But this might require some testing and performance tuning.

(i.e.: let's merge it only once we know that the performance hit is reasonable, I think)

(i.e.: let's merge it only once we know that the performance hit is reasonable, I think)

I would say that anything that is more efficient than fully copying the file to a temporary file on the local file system first, would be a win already, wouldn't it?

I think in any case having large HDF5 file stored as compressed may hinder the read performence. Would it even be possible to read a compressed stream from a offset in the middle?

This value for whence is used for example by the constructor of the File class of the h5py package.

Is there any reference to understand what exactly this does? I tried the h5py docs but did not find anything related to the whence keyword.

whence is a keyword of the seek method on a file handle: https://docs.python.org/3/library/io.html#io.IOBase.seek . It determines with respect to what you are seeking. The File constructor must be doing something like:
class File

    def __init__(self, handle):
        handle.seek(10, whence=2) # Seek to 10 bytes from the end of the stream

I would say that anything that is more efficient than fully copying the file to a temporary file on the local file system first, would be a win already, wouldn't it?

I'm not sure. Imagine you read the file backwards, 1 byte at a time (with N bytes).
If you extract the file, you uncompress once and then read backwards (relatively quickly).
If you don't, you will uncompress N bytes to read the last one, then N-1 bytes to read the second byte (from the end), then N-2, .... This means you decompress O(N^2) bytes that I'm sure at some (not so large) file size starts to be worse than extracting it and storing it.

Just out of interest, because I have never really looked into detail how these compression algorithms work, but where do you get this heuristic that to read the last byte of N bytes, you need to decompress just those N bytes. Wouldn't this also be dependent on the algorithm with which it was compressed. Is it even possible for all algorithms to arbitrarily decompress a subchunk of the compressed stream? I seem to vaguely remember that to decompress you would have to start at the beginning, or some point, for it all to make sense.
Anyway, when I opened the issue I went back in the history of the original implementation of whence=1 to see why I also didn't do whence=2 because I was sure we had discussed this and there was a reason, but I couldn't find any (not in commits, nor in issue/PR). But you clearly highlighted the reason why. Maybe it is indeed best to leave this not implemented and just document that the technique of copying the entire file to a temp file on disk is the work around.

Anyway, when I opened the issue I went back in the history of the original implementation of whence=1 to see why I also didn't do whence=2 because I was sure we had discussed this and there was a reason, but I couldn't find any (not in commits, nor in issue/PR). But you clearly highlighted the reason why. Maybe it is indeed best to leave this not implemented and just document that the technique of copying the entire file to a temp file on disk is the work around.

I think the directly implementing whence=2 would be difficult for compressed stream. However, the PackedObjectReader is just for reading the bytes as they are, the and decompression is actually done by ZlibLikeBaseStreamDecompresser that wraps it. So there is no limitation for whence=2 I think?
Copying to in memory buffer also works for smaller files. Perhaps we can implement some kind of logic:

If the object is small, just copy it to an in memory buffer
If the object is large, then we unpack the object into the loose folder, and read from the loose file. Of course, if such file exists then we can open it directly. The loose file will then be deleted in the next mantainance.

But this should be the job of ZlibLikeBaseStreamDecompresser not PackedObjectReader?

As a future reference to myself. I was thinking to keep a track of a copy of some decompressers object in memory, e.g. every 1MB or 10MB, and when seeking only uncompress from the closest previous decompresser object.
I tried this implementation: but for the few tests I did, it looks like this is actually not very fast (or at least, does not really make things very fast for a lot of random access).
For future ref, I started from
8e4db6c306fdd5e9762ed82a1db3fbea3e186f3e
and I did these changes:
diff --git a/disk_objectstore/utils.py b/disk_objectstore/utils.py
index 0dcf5c6..121f219 100644
--- a/disk_objectstore/utils.py
+++ b/disk_objectstore/utils.py
@@ -641,6 +641,7 @@ class ZlibLikeBaseStreamDecompresser(abc.ABC):
     """

     _CHUNKSIZE = 524288
+    _BUFFER_CACHE_SIZE = 1 * 1024 * 1024 ## Should be larger than the _CHUNKSIZE

     def __init__(self, compressed_stream: StreamSeekBytesType) -> None:
         """Create the class from a given compressed bytestream.
@@ -653,6 +654,9 @@ class ZlibLikeBaseStreamDecompresser(abc.ABC):
         self._internal_buffer = b""
         self._pos = 0

+        self._cache_unconsumed = {0: (0, self._decompressor.copy())}
+        self._last_cache_unconsumed = b"" # Only set when manually seeking
+
     @property
     def mode(self) -> str:
         return getattr(self._compressed_stream, "mode", "rb")
@@ -713,6 +717,11 @@ class ZlibLikeBaseStreamDecompresser(abc.ABC):
                     "the compressed stream: there must be a problem in the incoming buffer"
                 )

+            caching_position = ((self._pos + len(self._internal_buffer)) // self._BUFFER_CACHE_SIZE) * self._BUFFER_CACHE_SIZE
+            if caching_position > max(self._cache_unconsumed.keys()):
+                self._cache_unconsumed[self._pos + len(self._internal_buffer)] = (self._compressed_stream.tell(), self._decompressor.copy())
+                data = self._cache_unconsumed[self._pos + len(self._internal_buffer)]
+
         # Note that we could be here also with len(self._internal_buffer) < size,
         # if we used 'break' because the internal buffer reached EOF.
         to_return, self._internal_buffer = (
@@ -783,10 +792,23 @@ class ZlibLikeBaseStreamDecompresser(abc.ABC):
             self._pos = 0
             return 0

-        if target < self.tell():
-            # I cannot go backwards. In this case, I am forced to go back to zero and restart
-            # (I always know how to go back to zero). Otherwise, I just continue from where I am.
-            self.seek(0)
+        # I cannot go backwards, and in general, if I seek forward, I try to
+        # skip anything I don't really need.
+        # In this case, I go back to the latest cached position before
+        # the current one (worst case is zero, setup in the __init__).
+        cached_positions = sorted(self._cache_unconsumed)
+        for pos in cached_positions:
+            if pos <= target:
+                cache_pos = pos
+            else:
+                break
+
+        self._pos = cache_pos
+        self._internal_buffer = b""
+        compressed_stream_pos, decompressor = self._cache_unconsumed[cache_pos]
+        self._decompressor = decompressor.copy() # I need to copy again otherwise it will fail
+        self._compressed_stream.seek(compressed_stream_pos)
+        #print(f'seek, {target=}, {cache_pos=}, need to decompress only {target-cache_pos}')

         # Read target bytes, but at most `read_chunk_size` at a time to avoid memory overflow
         while self.tell() < target:

The main run script is
#!/usr/bin/env python
import os
import random
import time
import string

from disk_objectstore import Container

RANDOM_BYTES = True # If False, only use ASCII characters
TOT_SIZE = 100_000_000
#TOT_SIZE = 1_000
piece_length = 10
num_pieces = 10000
COMPRESS = True

def get_random_string(length):
    letters = (string.ascii_letters + string.digits)
    return ''.join(random.choice(letters) for i in range(length)).encode()

def main():
    print("Creating new empty container")
    container = Container("/tmp/test-container")
    container.init_container(clear=True)

    print(f"Generating {TOT_SIZE / 1024 / 1024:.2f} MB file in memory")
    if RANDOM_BYTES:
        content = os.urandom(TOT_SIZE)
    else:
        content = get_random_string(TOT_SIZE)
        #print(content)

    # Store a few small random sequence of bytes in random positions, to be
    # checked later
    to_check = []

    print(f"Fetching {num_pieces} random pieces for later check")
    for piece_idx in range(num_pieces):
        offset = random.randint(0, TOT_SIZE - 1)
        to_check.append((offset, content[offset:offset+piece_length]))

    print("Adding object directly to pack, compressed")
    hashkey = container.add_objects_to_pack([content], compress=COMPRESS)[0]

    print("Running speed test for all pieces")
    t = time.monotonic()
    with container.get_object_stream(hashkey) as stream:
        stream.read()
        print(f"Time to uncompress all: {time.monotonic() - t} s")
        t = time.monotonic()
        stream.seek(0)
        #print(stream._cache_unconsumed)
        for offset, check_val in to_check:
            stream.seek(offset, 0)
            piece = stream.read(piece_length)
            assert piece == check_val, f"{piece=} {check_val=}"
        #print(stream._cache_unconsumed)
    print(f"Time to read {num_pieces=} random-offset {piece_length}-bytes pieces: {time.monotonic() - t} s")

if __name__ == "__main__":
    main()  # pylint: disable=no-value-for-parameter

The results show that (at least this implementation) is to very convenient:  for a 95.37 MB file, uncompressing (in memory) takes ~0.18s, but read num_pieces=10000 with random access takes ~7.2 s.
(As a comparison, if it's uncompressed it takes 0.08s). So indeed the approach suggested by @zhubonan is better - we uncompress the file and cache it on disk, then random access is fast. The only case where this is not convenient is if when one just wants to read the first few bytes and ignore the rest.
For reference, the time to decompress seems to be heavily dependent on the data. E.g. if I use random data (incompressible, actually it might take more space), fetching 1000 random pieces from a 9.54 MB file takes ~0.44s, while if I only use ASCII letters (uppercase+lowercase) + digits, it takes ~2.7s.


Changes to boto3 and botocore to implement AWS rate limiting client side

Hello
This PR is for changes to both boto3 and botocore. The changes are explained in detail on the README of master project I've created to test them together here - https://github.com/museadmin/botorate/tree/master
Your contributing guidelines suggest that I should be supporting some deprecated versions of Python. I'm not clear if this is a real requirement or if your docs are just a little out of date. I raised an issue asking the question with boto3, but have decided to submit the PR to see what other specific issues you might want resolving.
If accepted, I think you would need to coordinate the merge with boto3, whom I'm also raising a PR with.
Bradley
boto3 PR - https://github.com/boto/boto3/pull/2086

This is much needed as AWS's implementation of rate limiting is very hard to handle.

Would it be possible to get a little help with this PR from someone with some experience of managing the travis ci build process?
I'm still struggling with the build process for python2.x versions but will have a crack at the setup.py syntax tomorrow and see if I can work out why it doesn't install two packages I require.
The issue I need help with is for the 3.x builds which I think would now pass if you were pulling in my fork of boto3 instead of the current version. Looking at the failure it's complaining that I have an unexpected parameter being passed, (api_rate=0) which is the parameter I would be passing in from my fork of boto3:
AssertionError: Expected call: create_client(api_version=<ANY>, client_config=<botocore.config.Config object at 0x7f367a082438>, credentials=<ANY>, endpoint_url=<ANY>, is_secure=<ANY>, region_name=<ANY>, scoped_config=<ANY>, service_name=<ANY>, verify=<ANY>)
Actual call: create_client(api_rate=0, api_version=None, client_config=<botocore.config.Config object at 0x7f367a082438>, credentials=<botocore.credentials.Credentials object at 0x7f367a1a6e80>, endpoint_url=None, is_secure=True, region_name='us-west-11', scoped_config={'aws_access_key_id': 'fie', 'aws_secret_access_key': 'baz'}, service_name='sts', verify=None)

I don't know how to get your ci pipeline to use two forked PRs for boto3 and botocore at the same time....

Ok boto
Finally have a green build.
My PR for boto3 won't pass until this is merged.
Do I have to do anything else to get this in?

Thanks for putting this together, it looks like you've put quite a bit of work into it. Unfortunately, we feel that this is a little out of scope for what botocore should handling. I would suggest you pull this into it's own package and hook into one of the events the clients emit:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/events.html
https://botocore.amazonaws.com/v1/documentation/api/latest/topics/events.html
For what it's worth, we do agree that the current retry behavior isn't perfect and could handle being throttled much more gracefully. This is something we're actively researching to improve, though I cannot share much information on this now.

Thanks for the feedback @joguSD
With regard to the scope of botocore, I'm not really asking it to handle anything beyond making the queue available as an option. It is entirely up to the consumer to instantiate, start and use it if they need it.
It would be a shame if you refuse it, because I was going to go on to submit PRs to all of the SDKs in each language to do the same. Then approach AWS to persuade them that leveraging client side limiting by exposing a limits API would put an end to this problem.
You comment that botocore, "could handle being throttled much more gracefully" suggests that you are unaware of the serious outages being caused by this issue. Presumably because those issues are being raised on other projects that are consuming yours, not your own.
May I suggest that you contact the ScoutSuite developers at NCCGroup to try and get the measure of this issue? Because it is certainly the case at present that we have conscientious AWS customers attempting to scan their production environments for vulnerabilities, and bringing down those environments as a result.
Something of an irony I am sure you will agree...

Closing for lack of interest from boto

It's really a shame this has been closed. AWS API rate limiting is poorly implemented and very hard to handle. Having the official libraries help avoid hitting the limits would be tremendously helpful to the community and organizations that build upon them.

@museadmin  @x4v13r64  I was looking for a similar solution for handling route53's throttling in the client side with time based rate limiting since it has only 5 calls allowed per second and I hit upon this issue.
By any chance , would you mind releasing a separate package that can patch the relevent modules / class of botocore or boto3 during runtime to achieve this. I know it's little hacky but it will definitely help others who are in same boat as you to achieve this without much modification to the underlying boto* package.  For reference, I had made something for caching over boto which uses unittest's patch method to achieve call based caching here.


Make PollingFileSystemWatcher serializable

Part of #2222

@ahsonkhan @JeremyKuhne Can we get this reviewed?

@KrzysztofCwalina, why did you want to make the watcher serializable?

@JeremyKuhne @KrzysztofCwalina
Part of the purpose of PollingFileSystemWatcher is that file events are too important to miss. If that is the case, then we need to account for file changes during downtime. I initially proposed import/export functions, but I was guided to aim for the more consistent ISerializable. Whatever direction we go, I believe we need a story for "file changes during downtime".

Part of the purpose of PollingFileSystemWatcher is that file events are too important to miss. If that is the case, then we need to account for file changes during downtime.

@MisinformedDNA Ah, I forgot that you were trying to do that. Sorry.


Work around upstream memory leak with APIServerIPOverride

When ClusterDeployment.Spec.ControlPlaneConfig.APIServerIPOverride is in use, we experience the memory leak described by
https://github.com/kubernetes/kubernetes/issues/118703. That issue has not been resolved at the time of this writing, but describes a couple of workarounds. Try the one described in this comment:

For now, I would probably just set the Proxy func to
utilnet.NewProxierWithNoProxyCIDR(http.ProxyFromEnvironment) so that
the TLS cache is skipped for your code. This isn't guaranteed behavior
and may stop working at any time, but it will at least solve the issue
for you for now (by restoring the old behavior).

HIVE-2272

This worked.
/assign @vincepri
/cc @dlom

/lgtm


Fix include for introduction in userguide_single.adoc

Introduction.adoc has been renamed.
Fixes https://github.com/gradle/gradle/issues/27847

@bot-gradle merge

The merge queue build has started. Click here to see all failures if any.


Integrate TFLite Delegate into TfLite in C++

Hi,
The document Integrate delegate using Python shows how to use TFLite delegate with ARM NN using Python.
Is it possible to integrate TFLite delegate using C++? If so, is there any tutorial or sample to do this?
Thank you

Hi @Rahn80643,
Yes it certainly is. At the bottom of the Arm NN Delegate build guide is a small snippet that shows how it can be achieved. This can be found here: https://arm-software.github.io/armnn/latest/md_delegate__build_guide_native.xhtml
In the codebase there is a number of examples of it being used which may be helpful too. E.g. In the ExecuteNetwork TfLiteDelegateMainImpl function there is an example which can be found in armnn/tests/ExecuteNetwork/ExecuteNetwork.cpp. Also, there are multiple nice examples within the TestHelper.hpp files for each operator. These can be found in the armnn/delegate/src/test directory.
I hope this helps, but if you have any further questions feel free to reach out. Thank you!
Kind regards,
Matthew

Hi @MatthewARM,
Thank you for providing the information of sample. Since the hardware we use contain both GPU and NPU, we want to ask:

Is it possible to assign a certain operation to be executed in GPU or NPU by using TFLite delegate?
How to copy and transfer data between GPU and NPU?
The tutorial of TFLite delegate model mainly focus on mobile devices, but our system will be an embedded system. Is there other tutorial of TFLite delegate for creating TFLite delegate model for embedded systems?
Do ARM NN/ TFLite delegate support both TensorFlow 1 and TensorFlow 2 model ?

Thank you,
Rahn

Hi @Rahn80643,
You're very welcome, I will reply to your questions in the same manner as above.

Yes, you can call this function below with your with your backend's ID, this will provide a hint for the optimizer as to which backend to prefer for this layer. This is declared in the armnn/include/INetwork.hpp interface.

void BackendSelectionHint(Optional<BackendId> backend) final
{
      m_BackendHint = backend;
}

This should be handled by the TFLite Delegate and ArmNN.
The tutorial I provided above is the only ArmNN tutorial at the moment. However, embedded systems are supported by TensorFlow Lite as seen here, to use the ArmNN TFLite Delegate you simply have to plug the ArmNN delegate into the TFLite Interpreter. The ArmNN TFLite Delegate can use any TFLite Lite model (.tflite). These models can be converted using TensorFlow Lite Converter. I hope this answers your question.
The Arm TFLite Delegate uses TensorFlow version 2.3.1 so any model supported by this version should work with the delegate. It will need TFLite models. If you have TF models you can convert them with the TensorFlow Lite Converter as mentioned above.

Thank you!
Kind regards,
Matthew

Hi @MatthewARM,
Thank so much for providing the information, we have successfully loading a MobileNetv1 classifier model with TFLite delegate parser based on armnn/tests/ExecuteNetwork/ExecuteNetwork.cpp
The following are questions we want to figure out:

From the tutorial, does libarmnnDelegate.so in PyARMNN provide the API of ArmNN delegate for Python to call?
Based on the list of supported operations of ArmNN TFLite parser and TFLite delegate, gpu advanced. If a model contains ARGMAX operation which doesn't appear in supported list of TFLite delegate, will the model be interpreted successfully with ArmNN TFLite delegate parser?
Is it possible to mixedly use ARMNN TF lite parser and TFLite delegate to parser a single model at the same time, and assign specific operation to a certain parser? For example, loading a MobileNetv1 classifier, and assign ArmNN TFLite to execute the first convolution operation, and TFLite delegate to execute the rest of convolution operations.
When a model is parsed through TFLite delegate parser, it is possible to split the model into two parts, the first part is executed on NPU, and the second part is executed on GPU?
If a model whose operations are partially supported in NPU can be fully interpreted by TFLite delegate parser, will these unsupported operations be executed in GPU without the implentation of backend?
In our understanding, the requred reserved memory for NPU is 256MB. Is the memory stored for temporary processed data? How to make sure the processed data tranfered between GPU and NPU?
The role of TFLite custom delegate  looks similar to ArmNN backend, both of these features allows users to implement a custom operation. What's the difference between these features?
Is it possible to provide a list of supported operations of TFLite parser for NPU?

Hi @Rahn80643,
You're very welcome and that's great news! Of course, I will do my best to answer them below and I will update 7 & 8 once I have more information.

Yes the libarmnnDelegate.so provides the API for the ArmNN TFLiteDelegate.
ARG_MAX is supported by the ArmnNN TFLiteParser and ArmNN TFLiteDelegate but I am currently working on a fix for the TFLiteParser implementation when the axis is negative. ARG_MIN support will also be added in this patch for the TFLiteParser. However, with the ArmNN TFLiteDelegate if there is an operator that isn't supported it will fallback to the TFLite default implementation, whereas with the ArmNN TFLiteParser it simply won't run.
I think there maybe some confusion here between the ArmNN TFLiteParser and the ArmNN TFLiteDelegate. Both provide the same accelerated operator support and performance. The difference resides the benefits, how it is created and how it is used. The ArmNN TFLiteDelegate plugs into TFLite's own interpreter as seen in the ArmNN TFLiteDelegate build guide. The TFLiteParser is created using the ArmNN TFLiteParser API as seen in this guide. So both wouldn't be used together if that makes sense. Also, you have the fallback benefit mentioned in point 2 with the ArmNN TFLite Delegate so it can be used to execute any TFLite model. A full list of accelerated operators can be found here.
Yes you should be able to split it on a layer by layer basis, you can specify which backend a layer will run on using the BackendSelectionHint method mentioned in my previous comment.
If unsupported in the preferred backend it will fallback to the next available backend.
I am not too sure about this question, however, the data transfer should be handled by the ArmNN TFLiteDelegate and ArmNN for you.

Thank you!
Kind regards,
Matthew

Hi @matthewsloyanARM
Thank you for your reply, the information is very helpful.
I will try to use the function BackendSelectionHint into our code.
Thank you!
Best regards,
Rahn


Possible to split optional and required dependencies on conda

See how hist splits its installation as hist_base (with required dependencies) and hist (with all the optional dependencies) -
https://github.com/conda-forge/hist-feedstock/blob/33fb4d946e7f9672c6b448b65ff199adcabdb8d1/recipe/meta.yaml#L15-L59

Another possible medium task for someone who does this is to compile the IDAKLU solver for PyBaMM in conda-forge since we currently ship without it. I shall open a new issue about this, though.

Being worked in https://github.com/conda-forge/pybamm-feedstock/pull/70

Another possible medium task for someone who does this is to compile the IDAKLU solver for PyBaMM in conda-forge since we currently ship without it. I shall open a new issue about this, though.

Should I open an issue (if not yet) and start working on this?

Another possible medium task for someone who does this is to compile the IDAKLU solver for PyBaMM in conda-forge since we currently ship without it. I shall open a new issue about this, though.

Should I open an issue (if not yet) and start working on this?

I've already opened it here: https://github.com/conda-forge/pybamm-feedstock/issues/65. Please feel free to get started and fork away!

This is up now - https://anaconda.org/conda-forge/pybamm-base!


Segmentation fault when class assigned to instance variable raises in its initializer

When a class's initializer invokes the initialize of a different class to assign it to an instance variable, and this is done inside a begin, if the different class raises during initialization, the instance variable will not get nil, leading to a seg fault.
class Foo
	def initialize
		begin
			@bar = Bar.new   # `Bar.new` will raise
		rescue
		end
	end

	def to_s : String
		(b = @bar) ? b.to_s : "DEFAULT"   # `@bar` has invalid memory
	end
end

class Bar
	def initialize
		raise "error"
	end
end

f = Foo.new
puts f.to_s

If we merely acknowledge the existence of @bar in the rescue, it works.
class Foo
	def initialize
		begin
			@bar = Bar.new
		rescue
			@bar   # adding this line makes it work
		end
	end

	def to_s : String
		(b = @bar) ? b.to_s : "DEFAULT"
	end
end

Naturally, adding a default initialization of @bar also fixes it.
My expectation was that @bar would either get nil or more generally would be in such a state as though it was never created.

Crystal 0.9.1 (Fri Oct 30 16:14:17 UTC 2015)

The compiler is making @bar of type NoReturn, but it's true, it should be Nil as it can never be assigned a value.


Nestable Nav Section slots

Create a way to have parent/child relationships between a parent nav-section slot and zero-to-many child nav-section slots.  This should allow grouping together of documentation and allow for collapsing and expanding of the children so that the entire list of nav-section slots need to be shown and take up screen real estate all the time.  There should be a default collapse/expand setting like there is with components and operations.
The use case for this is that there is a need for grouping and displaying related documentation if you have a considerable number of custom documentation pages for different areas of an API.  For instance if you have a pet store and there is a single API that has endpoints for retail, grooming and training and you have documentation for each of those areas of the API.  The documentation needs to be grouped by the particular area it belongs and then only show when you click on the nav-section associated with that area.
It would be great to have this functionality sometime before the end of the year, if possible.

Every nav tag component is already slottable. So you can already override what it is in there put custom other logic in. If there is something specific that is stopping you from doing that we should definitely provide improvements, but at this moment isn't exactly clear what that would be (if anything).
As far as additional functionality, it isn't clear from what you have provided what this would even look like. If you have an idea for an improved user experience, please share concrete UX flow diagrams in figma or another tool that can convey how exactly this would look.
By "endpoints" I'm going to assume you meant tags retail, grooming, etc...
Given that there is already full support for
 slotting. An example of this could be:

  Retail
  Other Link
  Other Link

Paths

    GET /
    POST /
    ...

Alternatively, I've enabled the functionality to support the tag description, so if you update the description for the tag, you'll see it as an expanding view in the navbar, but also visible in the right navigation panel.
I'me assuming all of this is sufficient, so I'm going to close this issue.

@wparad below is a screenshot of what I am trying to accomplish.  As you can see there would be a main "nav-section" group, and within it are multiple links that would belong to that group.  In my original example, there could be a "nav-section" for "grooming" and then below that there could be one or more sub documents that discuss different aspects of "grooming".  There could be other main groups with sub documents in each.  This would be expandable and collapsible to make the UI cleaner when they don't need to be displayed.  Is this something that can be done with existing functionality?  If so, could you provide some guidance on how this would be achieved?  As you can see, this isn't specific to an endpoint as it may discuss many endpoint details, but I need to be able to organize it logically so that the user can find it without extensive scrolling.  Thoughts?

Does the nav-section slot not solve for this?

I can see how the nav-section would solve creating the My Custom Doc Group, but how would it deal with the expand/collapse and the nested links that display when expanded?

@silverbullettruck2001, I'm not sure I'm following the issue. Are you asking how you would write html and JavaScript to have a list of buttons that change your html elements when clicked? You can put html in the custom nav slot, not just text.
You can write that functionality, right?
Or are you asking a more nuanced question?


Proposal: Incremental build context sending

There have been lots of discussions to optimize the rebuild flow of building a project with Docker. We have seen workarounds where volumes are used instead to get access to the source data to optimize the process. But that approach is really dangerous as the input data is mutable and can't be properly cached.
I'm proposing adding an endpoint to the remote API that could be used to transfer data between client and the builder using a negotiated protocol. For incremental build use case one of these protocols could be one that can skip the transfer of files that have not been changed.
@dmcgowan and I have been prototyping a solution for this using the changes stream from containerd https://godoc.org/github.com/docker/containerd/fs#Changes . Usually, this is used for finding the difference of changes between a read-write and image layer when committing a container. But the same concept also works for detecting the difference between current and a previous context transfer from a client. This is also the same algorithm used by rsync recursive directory sync (rsync delta update algorithm could be added later if it can help performance).
Having a special endpoint for context transfer also means we don't need to do any hacks with Dockerfile/.dockerignore where we currently try to combine build description and build inputs to the same archive and then extract and separate on daemon side.
Other benefits of this include daemon possibly detecting what files are actually needed for the build and skipping the transfer of the rest. Later this could be used for providing multiple sources for a build operation, skipping context transfer completely when parallel builds require same context, syncing files back to the client etc.
This would not remove any existing features from the builder. Sending context with a tar archive in a POST request would still be supported.
I hope to share code soon, was hoping to get some initial thoughts/feedback.
The transfer would go over a hijacked connection similar to the one used by the attach endpoint. For the framing, the prototype currently uses grpc. Could probably switch to websockets as grpc doesn't really provide much extra value in this case. All transferred data is encapsulated by protobuf. Initial benchmarks have been encouraging.

grpc doesn't really provide much extra value in this case.

I'd have to look at the implementation, but it seems like grpc here would be good for formalizing the protocol, handling errors, etc.. where as with websockets we'd be on our own to handle this.
What's the motivation for wanting to use websockets at all vs http2?

but it seems like grpc here would be good for formalizing the protocol

To me that is the main selling point. The only downside right now with the implementation is that the upgrade to HTTP2 and ultimately to GRPC feels pretty hacky and it only upgrades from a specific endpoint. Nothing is violating any of the RFCs but I think if we were to use GRPC we should try and centralize to single endpoint and possibly allow for more services to attach to it in the future. While I think it would be great that does open up a opportunity (or slippery slope depending on your view) for more to be added to the API with GRPC.
Setting aside the specifics of the protocol for the doing the file sync, I think this is the right approach for optimizing context. It will require slightly more state on the daemon side (and maybe client side in the future) but we could easily integrate it with docker system prune to allow flushing out this cached build data.

@cpuguy83
The endpoint for this is more like an interactive session between 2 clients. While grpc is designed for anonymous response handler for client side request. Also, there needs to be a way to negotiate protocol because they will be updated in the future.
This is the base outline for moving context form client to daemon:
client   -> i_am_ready_for_fs_jobs(session)
                 request_fs(paths, supported_protocols)   <- daemon (when it receives a build job)
            -> response_fs(ok, chosen protocol)
             ... actual transfer ...

Atm, this is solved by single bidirectional grpc handler where the request_fs, response_fs are the grpc stream types. After successful negotiation, the grpc stream(https://github.com/grpc/grpc-go/blob/master/stream.go#L66-L81) is passed to the transfer implementation. As you see the grpc is only minimally used and there isn't a way to share a session for multiple builds or transfer data from daemon to client.
The other way would be to for every response_fs to define a separate method but there isn't a reliable way to connect multiple requests(would need to fake a callback ID). Also, we can't predict what kind of data stream a transfer protocol would need, not sure if a single bidirectional grpc method with only 2 types always works.
I did make a TCP only version for test that just adds grpc wire protocol 5 byte header to every message and it is actually much cleaner.

Could we use this to also stream registry credentials ? that would be really nice to avoid sending unnecessary credentials to the daemon.

@simonferquel In theory yes, but that is not part of the current implementation. A related problem to this is DCT that atm has a parse step in the client that limits lots of use cases. One way could be that this stream could be used to get the DCT info while the builder is running.

@tonistiigi About grpc, could we simply upgrade the build api http connection to raw tcp (like what is done with docker run) and use this tcp connection to expose grpc methods like get_file_hash / get_file_content / get_authentication_creds, as soon as the client uses the next api version ? That would have the advantage of working with whatever underlying communication protocol is in use, and avoiding protocol negociation.


Should expose getCurrentItem()

It is possible to setCurrentItem(), but not to get it. This should be implemented to find out which item is currently selected.

@Nutomic # getCurrentItem() exposed in 0.0.10.

Thanks :)


Acceleration and climb are way off

I just made a comparison between a real flight, A320 leaving EDDM via SID MAMOR and the ATC-Simulation. I did that, because i always found the climb-rate to be way of reality. I also found the acceleration was way to fast. I don´t know, if acceleration and climb-rate are calculated together, but they should be ... after all, climb rate affects speed-gain and vis versa.
Long story short, here are two screenshots.

I should mention, that i set the max speed of the sim-A320 to 400 and gradually set higher speeds as he goes, because otherwise, the sim-A320 will be at 400 knots with 1900 feet height. I also should mention, that of course this is the same SID, same aircraft, same runway, same departure heading and of course same location of to planes where the screenshot is taken.
The real data are taken from a Lufthansa A320 from Munich to Berlin using flightradar24.com:
http://www.flightradar24.com/data/airplanes/d-aipa/#8844290

I should also clarify, that of course you can adjust ascend-rate and acceleration-rate to match the outcome in the real world, but ascend would be 5000 ft/min or more and acceleration so low, that the runway is barely long enough.

Oh, this is so much better now, thx!


chore: cache node_modules

Fixed in https://github.com/maxogden/menubar/commit/8a49b9f848080fdfe8f580c56fbbfdb0bbe03e78


Fix nyc dataset documentation

Variable names don't match up to documentation, discovered in devtools::check(). Mentioned by Jakub in #8:

One other thing that need to be solved is the documentation of nyc:

 checking for code/documentation mismatches ... WARNING
  Data codoc mismatches from documentation object 'nyc':
  Variables in data frame 'nyc'
    Code: CODE FORHIS06 FORHIS07 FORHIS08 FORHIS09 FORWH06 FORWH07
          FORWH08 FORWH09 HHSIZ00 HHSIZ02 HHSIZ05 HHSIZ08 HHSIZ1990
          KIDS2000 KIDS2005 KIDS2006 KIDS2007 KIDS2008 KIDS2009 NAME
          PUBAST00 PUBAST90 RENT2002 RENT2005 RENT2008 RENTPCT02
          RENTPCT05 RENTPCT08 SUBBOROUGH YRHOM02 YRHOM05 YRHOM08 bor_subb
    Docs: FORHIS06 FORWH06 HHSIZ00 HHSIZ02 HHSIZ1990 KIDS2000 PUBAST90
          RENT2002 RENTPCT02 YRHOMO02

I do not know what is the best solution here: (a) list all of the variables in the documentation, (b) remove this list entirely, (c) something else?...

Ended up adding documentation for all variables in commit 9d4b65ff35816209e13ee2ccd901fae311867d81


samples/net/dhcpv4_client does not work with sam_e70_xplained

Building this sample for sam_e70_explained reveals two problems:

By default the MAC address is 00:00:00:00:00:00.  This can be eliminated by adding CONFIG_ETH_SAM_GMAC_MAC_I2C_EEPROM=y somewhere.
Once that's fixed wireshark shows that the device is not transmitting any DHCP requests.

The same sample with frdm_k64f works as expected.

Works for me (after enabling the MAC address)
[00:00:00.000,000] <inf> i2c_sam_twihs: Device I2C_2 initialized
[00:00:00.000,000] <inf> i2c_sam_twihs: Device I2C_0 initialized
uart:~$ *** Booting Zephyr OS build zephyr-v2.4.0-2666-gfe8311843285  ***
[00:00:00.001,000] <inf> eth_sam: MAC: fc:c2:3d:11:c1:47
[00:00:00.001,000] <inf> eth_sam: Queue 0 activated
[00:00:00.001,000] <inf> eth_sam: Queue 1 set to idle
[00:00:00.001,000] <inf> eth_sam: Queue 2 set to idle
[00:00:00.001,000] <inf> eth_sam_phy: Soft Reset of ETH PHY
[00:00:00.091,000] <inf> eth_sam_phy: PHYID: 0x221561 at addr: 0
[00:00:00.097,000] <inf> net_dhcpv4_client_sample: Run dhcpv4 client
[00:00:03.122,000] <inf> eth_sam: Link up
[00:00:06.027,000] <inf> eth_sam_phy: common abilities: speed 100 Mb, full duplex
[00:00:07.030,000] <inf> net_dhcpv4: Received: <IP_ADDRESS>
[00:00:07.030,000] <inf> net_dhcpv4_client_sample: Your address: <IP_ADDRESS>
[00:00:07.030,000] <inf> net_dhcpv4_client_sample: Lease time: 864000 seconds
[00:00:07.030,000] <inf> net_dhcpv4_client_sample: Subnet: <IP_ADDRESS>
[00:00:07.030,000] <inf> net_dhcpv4_client_sample: Router: <IP_ADDRESS>

Huh.  Wonder if it's a board problem.  Mine has on the back A09-2637/02 sn<PHONE_NUMBER>0. On the front there's a JX02 94V09 26 18 where the last two suggest it may have been built in July 2018.

Mine has on the back A09-2637/02

I have earlier version A09-2637/01 from 2017. I remember that there was two versions (A vs B) of this board, perhaps this is related to board version. I though the difference between the boards was related to number of TSN queues, and not about IP connectivity.

Aha.  Yes, it turns out I do have the B variant, as indicated by this cryptic output:
[00:00:00.000,000] <wrn> soc: CIDR mismatch: chip = 0xa1020e01 vs HAL = 0xa1020e00

When I switch to sam_e70b_xplained it works.


DOCKER_CERT_PATH should not imply TLS is enabled

With docker CLI, DOCKER_CERT_PATH specifies the cert path, but the user still has to specify --tls or --tlsverify or set DOCKER_TLS_VERIFY to turn on TLS.
Probably we should actually make sure we are implementing the verification side (I think we are not) and support DOCKER_TLS_VERIFY.

I'm not sure I agree.  Should we also add a -UseTLS parameter to the cmdlets?  Would it every be correct to supply one without the other?  If they always have to both be specified, but one is just a flag and the other is a value, then I think DOCKER_CERT_PATH does indeed imply TLS and the additional flag has no value.

Talked offline, I see the reasoning.  I'll get this fixed soon.

Finally fixed by #155.


Wayland: Switch pointer position handling to doubles

This reduces even further the amount of work we have to do when scaling and potentially improves input accuracy as now the input code is free from any form of rounding.

Thanks!


Adds buid flags as in maliput_malidrive to pair builds.

🎉 New feature
Pairs with https://github.com/maliput/maliput_malidrive/pull/255
Summary
Uses build flags variables and applies it to all cc_* targets. Note that we are currently using a subset of all the flags that should  be used (see cmake/DefaultCFlags.cmake).
Test it
It should simply improve builds and parity against cmake builds.
Checklist

[x] Signed all commits for DCO
[ ] Added tests
[ ] Added example and/or tutorial
[ ] Updated documentation (as needed)
[ ] Updated migration guide (as needed)
[ ] Consider updating Python bindings (if it affects the public API)

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by messages.

Related to #569


Supported QUIC version?

Hi all
I was just wondering if there's an easy way to know which QUIC version(s) a release of quic_go supports - i didn't find anything obvious from a look around the repo. I use Caddy so this would feed in to when i should update that and what to expect.
Cheers
Neil

Hi Neil,
you can find the currently supported versions here: https://github.com/lucas-clemente/quic-go/blob/master/internal/protocol/version.go#L21-L27.
We're adding support for new QUIC versions as soon as Chrome Canary or Chrome Beta adds them, and we're dropping old versions a couple of weeks after Chrome Stable drops them. That way, quic-go is always compatible with basically all Chrome version that are currently in use.
If you're running Caddy, there's even an easier way to see which QUIC versions are supported: Just look at the Alt-Svc HTTP Header, which will contain a list of the supported versions.

Thanks for replying so quickly @marten-seemann - much appreciated. That's exactly the info i was looking for.
The reason for asking is that i noticed when Chrome 61 came out this week that my website's QUIC support stopped working as it's doing QUIC/38- and Chrome stable now uses QUIC/39. I think i am on Caddy 0.10.8 so upgrading to 0.10.9 might fix that, i'll give it a look/try.
Thanks again


Remove code smells reported by InCode <layout,naming,logic

The InCode code smell reporting tool found some problems in our code. The found code smells were 'Data Clumb', 'Data Class' and 'Feature Envy'. This branch fixes those problems by refactoring code.

maybe data is a little too generic name

Yea I could change it to movedata, but at the places where it is referenced the name will be longer. So I deliberately made my choice to keep it simple and thus use the name data.

It seems like something goes wrong with BigOpponentTest and thus the build fails, maybe try to merge development back into this branch and then see if the build still fails

Great! The build does not fail anymore! Can be merged


Only use lt in list.sort

Prep work to eliminate __cmp__.

Can you highlight what tests we have that cover this scenario already?

You mean tests to make sure sorting still works? We're running CPythonTests.test_sort.


Client documentation is too limited

A few aspects of this were discussed in a users forum thread, but based on initial feedback in PRs #1565 and #1568, it seems a more clear problem statement is needed.
Problem Statement
The current examples/client.rs, its guide explanation, and the client module doctest use run and no other alternatives are shown. In order to use run, these utilize futures::future::lazy to move the client into a single request future. Outwardly this makes things look minimal, clean and friendly, but as a pattern this is a dead end for building an application beyond the most trivial, single request case:

The Client in this formulation can't be reused for more than the single request, in order to take advantage of important features like connection keep-alive.

Methods run as used, or block_on, are inappropriate in the runtime context. Only spawn is non-blocking, but no current client example shows that usage.

Its not obvious why lazy is needed or what the shutdown requirements of the runtime and Client are.  Omitting lazy or moving the Client outside of the future can result in these examples halting (never shutting down.)

Code examples could be improved to some extent with comments.  Going further, in #1565 I propose using block_on in the doctest, and in #1568: spawn and explicit shutdown in the examples/client.rs.  Alternatively, instead of replacing the run + lazy examples, these could be adapted to include the alternatives along side.

I agree with the sentiment of better explaining how to execute asynchronous client futures. As stated in the linked PRs, I'm not sure using block_on is the best thing to teach users, but it definitely is too tricky that the examples use futures::lazy without explaining why.
When looking at the doc example, what would users likely be trying to do? Just getting a simple fetch in fn main working? Or perhaps trying to tie in some asynchronous requests with the rest of an app?
Perhaps the docs the can be updated to suggest looking at the examples/client.rs for a "here's a simple fn main example", and that example could be updated with a comment about why rt::lazy is used. Then, the doc example could instead show that the futures should be spawned, removing rt::run and rt::lazy, and just using rt::spawn.

"What would [new] users [of the Client] likely be trying to do?"

I think we've covered all of these potential starting points/use cases:

Unit tests of client usage (ala TDD, even if its for testing a hyper Server application)
Initially simple main for experimentation (scaling to a real CLI)
client requests in an existing asynchronous server or other tokio/futures application

For (1) I'm surprised and would like to better understand your reluctance to include a  block_on example. For my body-image testing, I've found that pattern quite useful, even for non-trivial tests:
https://github.com/dekellum/body-image/commit/2c77175edcff382441aef90d7c0c2c86d3e43cbd#diff-31bbf71c54bca98a0ae3d40a327af940R642
As I'm sure you are aware, you can't just replace printing in the current example or doctest with asserts for unit tests. Asserts need to be performed in the test thread to properly panic the test runner.  I would also imagine you've built up a much more elaborate testing harness in hyper itself, but as far a I know, you haven't packaged that for end user reuse? Thus end users need to start from scratch.
In all of these use cases I think its important that the first bullet of this problem statement is addressed so that these aren't dead ends to achieve a real application.  My proposals do just that for use cases (1) and (2), as I've learned ways of doing it the hard way, without a helpful guide. It would also be nice to see an example of that in the full setting of use cases (3).

I'm sorry if my reluctance seems rude, I'm not trying to be! Throughout the lifetime of hyper 0.11, and future.wait() and core.run(fut), the issue tracker has received bug reports of when users inappropriately used those synchronous tools. I've gotten plenty of personal email asking about the same, and people would show in the IRC channel likewise having the same confusion.
 ;)
I do agree that block_on is very useful when writing tests, but I personally feel that the docs and examples should be guiding people towards using the Future combinators whenever they don't have the value right away. It seems to come up super often in server, when people want to return a Response, but only after getting the body, so they reach for a blocking utility, and harm the server (if it still works at all).

Can you update documentation and show how to extract body from response? I need to parse response and I dont have any idea how to do it. Never was a problem to using Python or Go for this simple action.  I need to use https (thanks to Google its a default method for all popular websites) and authorization headers, extract body and parse string, using some regex.

Just want to note that this modified example in #1568 was important for helping me figure out 3) from above, sending requests from a client with an existing hyper server. I had thought about going down that path, and that example pushed me towards a solution.

@Crandel sorry, I was responding to the thread in general, and not specifically to your concerns. I think that if you want to have a user-friendly client, have you looked at https://github.com/seanmonstar/reqwest, which is an ergonomic layer over hyper? The experience is a lot closer to Python's request or Go.
Hyper might be a little lower level than you need.

As far as I can see when looking at the source code, Client appears to be fully internally reference counted, thus meaning that a Client could be cheaply cloned when needed several places.
If this is a suggested pattern for using the client, it should probably be mentioned in the documentation.

Regardin block_on, I'd like to mention an issue I've run into while testing my library which uses hyper under the hood:
I created a rather complicated future using a bunch of joins and and_thens, which resulted in the type length in the compiler exponentially exploding and requiring annotations like #![type_length_limit="4194304"] to compile.
I imagine this is related to this issue on the rust compiler.
 I realize this might not be the best way to handle it, and maybe one should be doing something else, but I'm not sure what that way is.

I'm writing an asynchronous client application based on hyper, and wow - has this been painful.  I'm writing an app that takes a stack of queries to a single server, and issues them up to x num of concurrent requests at a time.
I've achieved getting x num of concurrent requests to be sent at a server, but I don't yet have all the request results.  Based on what I've learned from the examples, its like I either can use the request status & headers, or I can use the request body, but not both.
It's been... tough.  I want the results all wrapped in a  response type struct (which includes the body) and then sent to another thread via mpsc channel.
What I'd really like to see is some asynchronous client example that gives me some kind of environment (I'm guessing via a .map call?  ) which has the full request available.
So far, all the examples just immediately print return status, headers and then enter a completely different 'task' to chunk the response body and handle that separately.
I just want access to the full response, at one point in time.  By "full response" I mean return status, the headers, and the body... the full response.  Instead, I've spent over ten hours just reading up on tokio trying to figure out what futures are, or what the hell concat2 is even returning, all while pondering if this is lazily evaluated. Or even map, for that matter - is it lazy?  How can I ensure a web request is finished?  I feel like "let body = res.into_body().concat2();" should allow me to at least do a "tx.try_send( body.to_vec() )" but that doesn't work.
I'm a novice in rust, and way out of my league apparently, but even from where I'm standing I'm pretty sure an example of an asynchronous app that creates a "full" response could wipe away a 1000 tears.

You're incredible! I love you.  A lot.  Your code is so clean!?!  How long did that take you?  I think.... I think I can accomplish my goal now.  Please include Darsonn's example in the hyper client source files.

@frenchtoastbeer Well I'm just glad I could help. It took me around an hour, although I've already done a bunch of stuff with futures and hyper in the past, as I've been writing a library using hyper (although it's currently temporarily dropped until async fn is stable).
As for including it as an example, notice that it makes use of spawn, which was mentioned in the top post of this issue, although there should probably also be an example using the other spawn intended for usage inside a future.