Split Makefile to be self-contained per directory

Split the src/Makefile out to each directory where installation actions are needed:

config_repo
notification_images
scripts
src
Top-level

The Makefile in each sub-directory can be called individually and it'll perform the proper actions, independent of other directories.
The top-level Makefile calls all sub-dir's functions as needed.
Tweaked install.sh to use the top-level Makefile.

Michael, Makefiles have gained a lot of capabilities since I used them 20 years ago in UNIX.  We used to only use them to compile source code.  What's the advantage of using a Makefile to do other things (copy files, etc.) rather than an install.sh script?
As we move to supporting upgrades which may include copying the user's configuration settings to new files, what will be better to handle that - a Makefile or install.sh?

For packaging, Makefile is the default workhorse of building and putting files in the proper location.
The overall idea for future maintenance will be to use the package manager ( apt update && apt install allsky ) and it'll bring in the new binaries.  Any config file changes will be asked to the User to resolve conflicts.
In general, package maintainers warn if there are breaking changes and let the end-user migrate their config to the new format.
If a programmatic migration can be done, we can also put a settings migration script in place which checks during systemd startup ( as an ExecStartPre directive ) and migrates to the new format, then moves the old config out of the way, etc.

I prefer makefiles to installer scripts because it's easier to re-run or modify a stage if necessary. Install.sh can call make.

Ok, this is ready to merge.
I also moved the config files into $ALLSKY_HOME/config  -- this should help keep the $ALLSKY_HOME directory tidy.
Note -- aside from Makefile(s), install.sh(s), and moving $ALLSKY_TMP to /tmp by default -- all other code remains the same.

don't commit just yet.. fixing up a Makefile issue...

In general, package maintainers warn if there are breaking changes and let the end-user migrate their config to the new format.

And any time apt upgrade wants to  change a file I've modified it always makes me choose to (at least) keep my file, install the new file, or run a diff to see what changed. Programs that update with shell scripts (eg. pihole) have caused me varying amounts of grief when the upgrader runs without showing me what would change.

Please source in variables.sh on line 2, then delete setting RED, GREEN, and NC since they are set in variables.sh. Then use $ALLSKY_HOME and $ALLSKY_CONFIG as necessary.

I assume this and the rest all refer to gui/install.sh ?

Yes, they refer to gui/install.sh.   I did the review of that file but apparently it doesn't indicate that anywhere.

@EricClaeys I figured someone may decide to tweak allsky.sh in their top level directory.  While unlikely, this will prevent it from getting clobbered.

Re: moving config files to allsky/config vs allsky-portal -- As it was, we still leave dangling settings_$CAMERA.json behind in the allsky folder... this would be no worse in the interim.

I'm with Eric on this one; allsky.sh is as much of an executable as capture/keogram/startrails and if someone doesn't like what it does they can file a bug, we can discuss it, and possibly add a configuration knob or some other hook.

Ok, I agree then... however.. I still think it's best to leave it as a .repo file, and move it to the proper location...
-- When I was going to revert that change, I remembered the driving reason was for future packaging.. the allsky.sh will be loaded into /usr/libexec/allsky  -- this is where the systemd unit will execute it from.  While this may seem odd, it's common practice for packaged assisting scripts to live in libexec, while compiled binaries live in /usr/bin
Everyone OK with leaving it in scripts/allsky.sh.repo and installing it to the proper location?

I talked to Thomas awhile ago about discouraging users from editing any scripts, other than endOfNight_additionalSteps.sh which is designed to be edited.  We agreed the GUI's Editor link should only show that file and ftp-settings.sh in the drop-down list (and ftp-settings.sh will go away once we combine all the config files).  Users that are Linux knowledgeable will know how to edit a script; others shouldn't edit scripts (it makes support hard) but instead should enter issues for new features as Chris said.  With the three of us working on Allsky we'll likely be able to add user-requested features quickly once we go through our current "to do" list.
I'd like to keep allsky.sh in ~/allsky so users can continue to do a simple ./allsky.sh.
If you want to move it to /usr/libexec/allsky then I suggest making ~/allsky/allsky.sh a symbolic link to /usr/libexec/allsky/allsky.sh
My vote would be to keep all of the Allsky files in ~/allsky as is currently done.  It's really simple for users to understand and is easy to explain when troubleshooting.  I understand that binaries typically go in /usr/bin and other files typically elsewhere, but from a novice user's perspective that makes it complex.  $ALLSKY_TMP would point to $ALLSKY_HOME/tmp and all temporary files (including image.jpg) would go there.  Users can easily make it a memory filesystem to decrease writes to the SD card because the directory will never get very big.  I'm not sure having ALLSKY_TMP=/tmp would work for low-memory Pi's since /tmp can grow quite big.
With variables.sh, users that want to separate pieces of allsky can easily do so by editing the variables.
I also talked to Thomas about NOT leaving leave dangling settings_$CAMERA.json files behind in the allsky folder when installing the GUI, but instead MOVE them to /etc/raspap.  Having them in two places is very confusing - it took me a while to figure it out and at least one user was confused.
Putting all config files in $ALLSKY_CONFIG (regardless of where it points to) solves that problem since we'd never move/copy the files.
Comments?

Sorry, this was accidentally closed ( by my rename of the branch ... doh ).  Submitting new PR.


Brian/minimal plutarch test

AFAIK There's no one I can really ask to review this so I'll just merge it


make the levels table appear without horizontal scrollbar

The right side of the table was hidden for me and I did not notice the scrollbar, since it's only on the bottom of the rather large table
Before

After

Great idea, I took it even one level further. 😄

Thank you, @courier-new & @kopancek! Good stuff


Example command incorrect

Example text has the following line:
Get-WinLanguageBarOptions
There should be no 's' on the end of the command.
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c0d25509-d895-a936-8472-5396205269e2
Version Independent ID: 07c2f956-7f02-ea5d-57ec-9a5cd0670431
Content: Get-WinLanguageBarOption (International)
Content Source: docset/winserver2019-ps/international/Get-WinLanguageBarOption.md
Product: w10
Technology: windows
GitHub Login: @JasonGerend
Microsoft Alias: jgerend

here is the proof

@e0i Please assign to verify and merge this PR #2526


Latest VHD VM's admin username is "localadmin" and not "administrator"

The latest VHD on LCS (10.0.29) has an admin user localadmin instead of the built in administrator account.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: fe3913a6-4daa-9973-6f0b-d7ae4c1007c2
Version Independent ID: f2cd311b-fcb8-cff4-57a3-47c3c52b19d6
Content: Deploy and access development environments - Finance & Operations | Dynamics 365
Content Source: articles/fin-ops-core/dev-itpro/dev-tools/access-instances.md
Service: dynamics-365-cross-app
GitHub Login: @laneswenka
Microsoft Alias: laswenka

Bug 793948 has been logged to address this issue.

Reached out to PM for guidance.


csv file upload failed lynxkite

details: {}
message: File upload failed.
url: >-
https://demo.lynxkite.com/user/a<EMAIL_ADDRESS>version: |
LynxKite Enterprise 4.2.2

Hi Myra,
The demo instances are fairly small and the upload size is limited. You can upload a Gzipped file to possibly get under the size limit. But if you want to use LynxKite with really big data, you will have to run your own instance. It's fairly easy to run even on a laptop. See https://lynxkite.com/getting-started#docker-linux. Let me know if you need help!

@darabos  I was using the cloud version and figured out it was because of the file size, since it was accepting smaller file sizes but uploading the same file to the pc version it works without zipping.

While it makes sense to limit file upload sizes, I think that the error message can be more helpful 😅  I will make a PR for this some day


Doesn't build on arm64 linux

Linux debian-4gb-fsn1-1 6.1.0-11-arm64 #1 SMP Debian 6.1.38-4 (2023-08-08) aarch64 GNU/Linux
error[E0308]: mismatched types
   --> /home/mozgiii/.cargo/registry/src/index.crates.io-6f17d22bba15001f/ls-qpack-0.1.2/src/decoder.rs:374:31
    |
374 |             this.header.buf = this.decoding_buffer.as_mut_ptr() as *mut i8;
    |             ---------------   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `*mut u8`, found `*mut i8`
    |             |
    |             expected due to the type of this binding
    |
    = note: expected raw pointer `*mut u8`
               found raw pointer `*mut i8`

error[E0308]: mismatched types
   --> /home/mozgiii/.cargo/registry/src/index.crates.io-6f17d22bba15001f/ls-qpack-0.1.2/src/header.rs:199:18
    |
199 |             buf: header.buffer.as_mut_ptr() as *mut i8,
    |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `*mut u8`, found `*mut i8`
    |
    = note: expected raw pointer `*mut u8`
               found raw pointer `*mut i8`

I'm not sure why is this.

I'm not sure why is this.

This comes from the FFI (i.e., the C bindings).
The variable this.header.buf has type c_char, that is, char type of C-language.
Now, the C standard, does not guarantee the sign for the char type.

char - type for character representation. Equivalent to either signed char or unsigned char (which one is implementation-defined and may be controlled by a compiler commandline switch), [...]

It means char is both valid to be signed char or unsigned char. It depends on the implementation.
On x86_64 char seems to be defined as signed (thus, i8 in Rust), while it is unsigned on arm64 (this, u8 in Rust)

@MOZGIII I don't have an easy way to test the patch on arm64, if you have some spare time it would be great if you can see if you are able to compile it applying the patch: https://github.com/BiagioFesta/ls-qpack-rs/pull/6
Thank you :)


New Feature: Auto realtime chat backup

This is inbuilt in whatsapp. And won't implement it because security issues and trust issues.


recipes: Update SRC_URI protocols for github

Use protocols=https for GitHub SRC_URIs using the conversion script in
openembedded-core.

Can one of the admins verify this patch?

@agherzan-ci ok to test


sys vs _sys

Don't we want to import sys as _sys?

Why?

I'm just thinking consistency. Since all functions are prefixed with _, why not also make sure everything else is?
Or perhaps that's not standard procedure?
To be honest, I'm not sure why you wanted the prefix for the functions, as the module is being replaced at sys.modules["Qt"] = binding() anyways.

Underscore prefix is convention for "private" members; that is, members that isn't part of an interface, in this case from the importer of the module.
You're right, it technically won't matter because it's being replaced, but it's more to communicate to whomever reads the code that these are implementation detail, and not interface.
When we do have functions that are meant to be used from the importer, then those would not have an underscore and the reader could gather than "aha, I can rely on these members to remain consistent, versioned and backwards compatible when this interface changes". The same is not true for the ones with an underscore.
Importing modules into a module with an underscore would make sense from a technical point of view, because they would otherwise be part of the outside interface. But on the other hand it makes the code itself less readable so there's a tradeoff to be made in readability.

Ok, that's fair. Figured I'd just bring it up. :)

It's good that you do. :) I take a lot of stuff for granted, so it can be healthy to sometimes shine a light on things in case they are based on some shenanigans. Hopefully this makes sense though, it's worked well long-term and I read a lot of code in other projects that follow the same convention.


Combine REST with Socket protocol in microservice architecture

hi, firstly, i must say this is super cool project, and i very love it. But i have some confuse, if you have time, please explain clearly for me.
i want to build a microservice, Client will call to one main service by HTTP, and then main service call to another service by message (websocket). But how i can skip the HTTP request and wait the message go to all other services, get the response and send back to to client :(
i want to use websocket to communication between internal microserivce because it's faster than HTTP protocol
many thank guys

@cjus do you have any idea about it? :D many thank

@tybq93 since this isn't but rather a question about architecture I invite you to join our slack channel. While you're considering that I can say that HTTP and messages (socket) can be combined at will. You can have an HTTP endpoint which in turns sends another service a message via hydra.sendMessage.
Hydra doesn't offer a messaging synchronization feature to keep track of messages sent and acknowledged. Rather it simply offers a message transmission and routing. To use socket only messaging you can create hydra message listeners in your services:
hydra.on('message', (message) => {
  // handle message / dispatch message
  this.myMessageHandlerAndDispacher(message);
});

When a message is received and you'd like to reply to it you can use hydra.sendMessageReply()
Also, you mentioned that you'd like to use WebSockets. Is that via HydraRouter or some other means?

yeah, i mean websocket is via HydraRouter.
for example:
step 1: client ----call http api to get list post---> API Service
step 2: API Service ----sendMessage get list post to PostService ----> PostService
step 3: PostService  --- sendMessageReply back to APIService -----> API Service
step 4: API Service ---send result back----> Client
but the problem here is we can not delay the step 1 request and wait util we have the response with result back on step 3 and then send back to user in step 4, so i think this is impossible, right?
I already sent you email to join slack channel, i'm very happy to discuss more about your team. :+1:

Correct that doesn't sound possible. It's an issue of whether you close the HTTP connection in step one or leave it open pending a response.  The way we do this is that we create websocket connections to HydraRouter (from mobile devices) and then our microservices able to route message responses back out through hydraRouter to specific devices.  This isn't fully documented and I need to do a write-up for anyone else wanting to do the same.
I sent you an invite to our Slack channel. See you there.


IPAM managed AR need stub IP on create

Description
When new IPAM managed AR range (e.g., by Packet) is created, it needs to be provided with fake initial IP, although this IP doesn't have to be respected in IPAM range registration.
(This issue is created based on notes from Packet IPAM, more description needs to be provided).
Details

Affected Component: Core
Version: 5.8.1

Progress Status

[ ] Branch created
[ ] Code committed to development branch
[ ] Testing - QA
[ ] Documentation
[ ] Release notes - resolved issues, compatibility, known issues
[ ] Code committed to upstream release/hotfix branches
[ ] Documentation committed to upstream release/hotfix branches

Problem only in Sunstone.

This shouldn't be mandatory if IPAM driver is selected.

PR to be approve:

ONE: https://github.com/OpenNebula/one/pull/4435
Docs: https://github.com/OpenNebula/docs/pull/1015

good work! Let's go for testing now

@tinova, PR to be aprove:

Development: https://github.com/OpenNebula/development/pull/906

good work!


feature: Pushdown LimitDistinct Through Join

Proposed changes
Push down limit-distinct through left/right outer join or cross join.
such as select * from t1 left join t2 on t1.c1 = t2.c1 order by t1.c1 limit 1;

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall

run buildall


Title column for special filenames should be prefixed with the movie name

Currently we are getting bad Title for trailers, etc:
trailer

These should be:
<dir_name> - trailer

Merged to develop


NuGet  v1.1.4 fails to construct TransferManager from TransferManagerConfiguration

I am noticing the following compile time error on upgrading to v1.1.4 via NuGet package manager.
error C2248: 'Aws::Transfer::TransferManager::TransferManager': cannot access private member declared in class 'Aws::Transfer::TransferManager'

yes for safety reasons we had to move everything to shared_ptr. You need to call the Create() function


Update LICENSE

replace name Johann Cohen-Tanugi by LePHARE collaboration

Change Description

[ ] My PR includes a link to the issue that I am addressing

Solution Description

Code Quality

[ ] I have read the Contribution Guide
[ ] My code follows the code style of this project
[ ] My code builds (or compiles) cleanly without any errors or warnings
[ ] My code contains relevant comments and necessary documentation

Project-Specific Pull Request Checklists

Bug Fix Checklist

[ ] My fix includes a new test that breaks as a result of the bug (if possible)
[ ] My change includes a breaking change

[ ] My change includes backwards compatibility and deprecation warnings (if possible)

New Feature Checklist

[ ] I have added or updated the docstrings associated with my feature using the NumPy docstring format
[ ] I have updated the tutorial to highlight my new feature (if appropriate)
[ ] I have added unit/End-to-End (E2E) test cases to cover my new feature
[ ] My change includes a breaking change

[ ] My change includes backwards compatibility and deprecation warnings (if possible)

Documentation Change Checklist

[ ] Any updated docstrings use the NumPy docstring format

Build/CI Change Checklist

[ ] If required or optional dependencies have changed (including version numbers), I have updated the README to reflect this
[ ] If this is a new CI setup, I have added the associated badge to the README

Other Change Checklist

[ ] Any new or updated docstrings use the NumPy docstring format.
[ ] I have updated the tutorial to highlight my new feature (if appropriate)
[ ] I have added unit/End-to-End (E2E) test cases to cover any changes
[ ] My change includes a breaking change

[ ] My change includes backwards compatibility and deprecation warnings (if possible)

This change looks sensible. I think we still need to add some details in the new documentation showing all the efforts over the full timeline of LePHARE development. I wonder if it might be good to have a full page on the development history with all the contributions explicitly detailed.


Introduce a no-wasm compatible ExtrinsicSigner

The current ExtrinsicSigner depends on a Runtime. But the node Runtime is not available in an environment where wasm is not supported.
Therefore, a No wasm compatible ExtrinsicSigner (implementing the same trait SignExtrinsic) where no Runtime type input is needed could be introduced.
If not possible in any other way, a hardcoded extrinsic_address type conversion may be used.
@clangenb Would that help your usecase?

Answering myself: yes, definitely. Necessary for signing extrinsic in the TEE.

We can use the runtime inside the TEE. We do this already. :)

Ah wait, this is only the sgx runtime. :P

Hah. I just panicked for a short moment here. Was already double checking 😨

But anyway: For simply signing extrinsics, importing the full Runtime is not worth it. So it's useful anyway


Dart extension methods are not autocompleted

I have to import the file manualy every time I want to use one of my extensions from my extensionsfile
I use android studio.

[✓] Flutter (Channel unknown, v1.12.13+hotfix.5, on Mac OS X 10.15.1 19B88, locale en-BE)
    • Flutter version 1.12.13+hotfix.5 at /Users/vanlooverenkoen/flutter
    • Framework revision 27321ebbad (7 weeks ago), 2019-12-10 18:15:01 -0800
    • Engine revision 2994f7e1e6
    • Dart version 2.7.0

[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.2)
    • Android SDK at /Users/vanlooverenkoen/Library/Android/sdk
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-29, build-tools 29.0.2
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_202-release-1483-b49-5587405)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 11.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 11.3, Build version 11C29
    • CocoaPods version 1.7.5

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 3.5)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 42.1.1
    • Dart plugin version 191.8593
    • Java version OpenJDK Runtime Environment (build 1.8.0_202-release-1483-b49-5587405)

[!] VS Code (version 1.41.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    ✗ Flutter extension not installed; install from
      https://marketplace.visualstudio.com/items?itemName=Dart-Code.flutter

[✓] Connected device (5 available)
    • AOSP on IA Emulator • emulator-5554                        • android-x86    • Android 9 (API 28) (emulator)
    • iPhone 11 Pro Max   • F2704E51-185C-4E5F-BC54-8FB1BA3C43BE • ios            • com.apple.CoreSimulator.SimRuntime.iOS-13-3 (simulator)
    • macOS               • macOS                                • darwin-x64     • Mac OS X 10.15.1 19B88
    • Chrome              • chrome                               • web-javascript • Google Chrome 78.0.3904.108
    • Web Server          • web-server                           • web-javascript • Flutter Tools

! Doctor found issues in 1 category.

https://github.com/dart-lang/sdk/issues/40365

Hi @vanlooverenkoen
From what I can see, the issue is related to the Dart language rather than to Flutter itself. Please open the issue in the dedicated repository.
This may be a duplicate of one of the issues in that repository, please see https://github.com/dart-lang/sdk/issues/38894.
Closing, as this isn't an issue with Flutter itself.
Thank you


part of PR 340 adding test_make_parent_mount_private, test_setup_namespaced_hierarchy, test_setup_comount_symlinks

as requested by separating #340 into small PRs
this part is adding:

test_make_parent_mount_private
test_setup_namespaced_hierarchy
test_setup_comount_symlinks

please help review 🙇🏻

Codecov Report

Merging #374 (efc634d) into main (c86c6bb) will increase coverage by 0.70%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #374      +/-   ##
==========================================
+ Coverage   74.73%   75.44%   +0.70%
==========================================
  Files          51       51
  Lines        7763     7836      +73
==========================================
+ Hits         5802     5912     +110
+ Misses       1961     1924      -37

I noticed there has a PR
https://github.com/containers/youki/pull/373
did the same things for this PR's

test_setup_namespaced_hierarchy ---> this one's name has changed
test_setup_comount_symlinks ---> this one tested in that PR

so I'll remove those two tests in this PR


Doc - ArangoDB Starter doc integration

Internal Ref: https://github.com/arangodb/planning/issues/1479

Waiting on https://github.com/arangodb-helper/arangodb/pull/126


Tree-sitter: experiment with dropping the truncation strategy dependency on block start chars

Slack thread
Follow up for https://github.com/sourcegraph/cody/issues/1619

Still in-progress / todo?


Another exception was thrown: 'package:flutter/src/gestures/monodrag.dart': Failed assertion: line 387 pos 12: '_initialButtons == kPrimaryButton': is not true.

Steps to Reproduce

1. build a page
  @override
  Widget build(BuildContext context) {
    return DefaultTabController(
      length: 2,
      child: Scaffold(
          appBar: AppBar(
            title: Text('tabbar'),
            bottom: TabBar(
                tabs: Iterable.generate(2, (index) => Tab(text: 'tab$index'))
                    .toList()),
          ),
          body: TabBarView(children: [
            ListView.builder(
                itemCount: 20,
                itemBuilder: (context, index) => ListTile(
                      title: Text('tab 0 list: Item $index'),
                    )),
            ListView.builder(
                itemCount: 30,
                itemBuilder: (context, index) => ListTile(
                      title: Text('tab 1 list: Item $index'),
                    ))
          ])),
    );
  }

2.  two fingers simultaneously click on ListView
Logs

I/HiTouch_PressGestureDetector: ACTION_POINTER_DOWN.
I/flutter: ══╡ EXCEPTION CAUGHT BY GESTURE LIBRARY ╞═══════════════════════════════════════════════════════════
I/flutter: The following assertion was thrown while dispatching a pointer event:
I/flutter: 'package:flutter/src/gestures/monodrag.dart': Failed assertion: line 357 pos 12: '_initialButtons ==
I/flutter: kPrimaryButton': is not true.
I/flutter: Either the assertion indicates an error in the framework itself, or we should provide substantially
I/flutter: more information in this error message to help you determine and fix the underlying cause.
I/flutter: In either case, please report this assertion by filing a bug on GitHub:
I/flutter:   https://github.com/flutter/flutter/issues/new?template=BUG.md
I/flutter: When the exception was thrown, this was the stack:
I/flutter: #2      DragGestureRecognizer._checkStart (package:flutter/src/gestures/monodrag.dart:357:12)
I/flutter: #3      DragGestureRecognizer.acceptGesture (package:flutter/src/gestures/monodrag.dart:298:7)
I/flutter: #4      GestureArenaManager.sweep (package:flutter/src/gestures/arena.dart:156:27)
I/flutter: #5      _WidgetsFlutterBinding&BindingBase&GestureBinding.handleEvent (package:flutter/src/gestures/binding.dart:222:20)
I/flutter: #6      _WidgetsFlutterBinding&BindingBase&GestureBinding.dispatchEvent (package:flutter/src/gestures/binding.dart:198:22)
I/flutter: #7      _WidgetsFlutterBinding&BindingBase&GestureBinding._handlePointerEvent (package:flutter/src/gestures/binding.dart:156:7)
I/flutter: #8      _WidgetsFlutterBinding&BindingBase&GestureBinding._flushPointerEventQueue (package:flutter/src/gestures/binding.dart:102:7)
I/flutter: #9      _WidgetsFlutterBinding&BindingBase&GestureBinding._handlePointerDataPacket (package:flutter/src/gestures/binding.dart:86:7)
I/flutter: #13     _invoke1 (dart:ui/hooks.dart:250:10)
I/flutter: #14     _dispatchPointerDataPacket (dart:ui/hooks.dart:159:5)
I/flutter: (elided 5 frames from class _AssertionError and package dart:async)
I/flutter: Event:
I/flutter:   PointerUpEvent#a8059(position: Offset(101.6, 324.5), timeStamp: 429:52:23.054000, pointer: 101,
I/flutter:   kind: touch, down: false, pressure: 0.1, pressureMin: 0.0)
I/flutter: Target:
I/flutter:   <WidgetsFlutterBinding>
I/flutter: ════════════════════════════════════════════════════════════════════════════════════════════════════
I/flutter: Another exception was thrown: 'package:flutter/src/gestures/monodrag.dart': Failed assertion: line 387 pos 12: '_initialButtons == kPrimaryButton': is not true.

CC @dkwingsmt

I'm able to reproduce and I'm investigating.

not yet.
1.17

not yet.
1.17

The same error can be caused by different reasons. Can you describe how to reproduce yours?

i have same issue. Are there any bug fix or workaround solution ? When i use tabbar like dummy user with two or three finger (gesture) it freeze all tab bar views.
flutter: ══╡ EXCEPTION CAUGHT BY GESTURE LIBRARY ╞═══════════════════════════════════════════════════════════
flutter: The following assertion was thrown while routing a pointer event:
flutter: 'package:flutter/src/gestures/monodrag.dart': Failed assertion: line 416 pos 12: '_initialButtons ==
flutter: kPrimaryButton': is not true.
flutter:
flutter: Either the assertion indicates an error in the framework itself, or we should provide substantially
flutter: more information in this error message to help you determine and fix the underlying cause.
flutter: In either case, please report this assertion by filing a bug on GitHub:
flutter: https://github.com/flutter/flutter/issues/new?template=BUG.md
flutter:
flutter: When the exception was thrown, this was the stack:
flutter: #2 DragGestureRecognizer._checkEnd
package:flutter/…/gestures/monodrag.dart:416
flutter: #3 DragGestureRecognizer.didStopTrackingLastPointer
package:flutter/…/gestures/monodrag.dart:358
flutter: #4 OneSequenceGestureRecognizer.stopTrackingPointer
package:flutter/…/gestures/recognizer.dart:343
flutter: #5 DragGestureRecognizer._giveUpPointer
package:flutter/…/gestures/monodrag.dart:367
flutter: #6 DragGestureRecognizer.handleEvent
package:flutter/…/gestures/monodrag.dart:256
flutter: #7 PointerRouter._dispatch
package:flutter/…/gestures/pointer_router.dart:78
igunes@igunes ios % flutter doctor -v
[✓] Flutter (Channel stable, 1.20.3, on Mac OS X 10.15.6 19G2021, locale en-TR)
• Flutter version 1.20.3 at /users/igunes/flutter
• Framework revision 216dee6 (13 days ago), 2020-09-01 12:24:47 -0700
• Engine revision d1bc06f032
• Dart version 2.9.2
[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.0)
• Android SDK at /Users/igunes/Library/Android/sdk
• Platform android-30, build-tools 30.0.0
• Java binary at: /Applications/Android
Studio.app/Contents/jre/jdk/Contents/Home/bin/java
• Java version OpenJDK Runtime Environment (build
1.8.0_242-release-1644-b3-6222593)
• All Android licenses accepted.
[✓] Xcode - develop for iOS and macOS (Xcode 11.7)
• Xcode at /Applications/Xcode.app/Contents/Developer
• Xcode 11.7, Build version 11E801a
• CocoaPods version 1.9.3
[✓] Android Studio (version 4.0)
• Android Studio at /Applications/Android Studio.app/Contents
• Flutter plugin version 48.1.2
• Dart plugin version 193.7361
• Java version OpenJDK Runtime Environment (build
1.8.0_242-release-1644-b3-6222593)
[✓] VS Code (version 1.49.0)
• VS Code at /Applications/Visual Studio Code.app/Contents
• Flutter extension version 3.14.1
[✓] Connected device (1 available)
• Mustafa’s iPad (mobile) • ec9cc0171e4772f3dbca05f0c732828429310826 • ios •
iOS 13.5.1


A09: Remaining platform gaps / enhancements (TBC)

Description:
TBC once we learn more about the gaps in the platform.

May I take care of this?

Can I be assigned to this?

May I handle this issue?

Can I try solving this issue?

I’d like to help with this.

Can I be assigned this issue when it's ready for contribution please?


Update jwt_verify_lib to latest commit

Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

cc @qiwzhang

Updates look OK but DCO needs to be fixed first

Updates look OK but DCO needs to be fixed first

Fixed.

/retest


ShareRoutes tests are done

Test for ShareRoute component are written, I would like that you take a look at the tests, because despite 160 lines of code in the test, I can not reach more than a 47% of coverage on it. I leave the PR open to take into account that this tests are done just need to be improved.

Also, a little bit of CSS is left to be changed, but I will do that when this PR is merged onto dev, and in another branch.

Ready to merge @alvarezGarciaMarcos


class.bind currently overrides any existing class definitions on the element

If i have

I expect that the div either becomes

or

Currently it becomes

or

I'm not sure this would be the expected behavior. I would not want to do

in a lot of cases

@shyamseshadri something seems to have go wrong with your formatting as nothing is showing after your statements for code (e.g. "if I have" is followed by blank).

class.bind is expected to overwrite the existing classes. If you do not want to do this, you will want to use interpolation instead. Something like this class="some-class another-class ${boundClass}"

I'd like to reopen this, there should be a way to do this class interpolation thing without overwriting current classes but without using the example of @EisenbergEffect.
I have an use case where I want to add a dynamic class to a custom element. <my-element class="${myClass}"></my-element> but the element in its inner logic toggles a class from Js which I don't(and shouldn't) control in this scope.
I don't know there should be a special syntax for this or a different behavior but I think it should be possible.

Not sure if it's important but also note that class.bind="..." and class="${...}" overwriting deletes the au-target class also.

With respect to au-target it's no longer important to the framework by the time that the binding expression is applied.

@jdanyow Can you an @olanod continue discussion here on his use case?

So as I previously mentioned. I want to set a class conditionally
<my-element class="${condition?'active':''}"></my-element>

but this will replace the classes the element already has. In my case the classes I want to keep are added dynamically from inside the custom element(or from anywhere anytime?).
E.g.
In the custom element logic in the bind method I add a class with element.classList.add('fuu'), it gets added to the host element but then sometime in the future some event changes the condition and the element gets the 'active' class but it also deletes any previous classes it could have.

How was this issue resolved?

I haven't seen that issue, but you could use string interpolation instead of .bind


Fix for the inconsistent internal parser state bug

This code change fixes the following issues:

fix #2
fix #20
fix #46
fix #53

and makes the following pull requests superfluous:

close #21
close #37

As described in issue #53, there's a bug in the Elm.Kernel.Parser.findSubString function,
that leads to inconsistent internal parser positions where:

the offset into the source string is positioned before a token
the row and column are positioned after the token

This code change fixes that bug, so that both offset and row/column are consistently positioned after the token.

There are two reasons why I chose the after token position and not the before token position:

The users seem to expect it:

#2
#20

Even @evancz has implemented it this way in the multiComment parser when using the Nestable mode

in this mode the parser is implemented differently and therefore isn't affected by the bug
in this mode the parser moves to a position after the last closing token:

import Parser exposing ((|.), (|=), Parser)

testParser : Parser { row : Int, col : Int, offset : Int }
testParser =
    Parser.succeed (\row col offset -> { row = row, col = col, offset = offset })
        |. Parser.multiComment "{-" "-}" Parser.Nestable
        |= Parser.getRow
        |= Parser.getCol
        |= Parser.getOffset

Parser.run testParser "{- -}"
--> Ok { row = 1, col = 6, offset = 5 }

The real bug fix is on this line:
src/Elm/Kernel/Parser.js
@@ -133 +133 @@ var _Parser_findSubString = F5(function(smallString, offset, row, col, bigString
-       return __Utils_Tuple3(newOffset, row, col);
+       return __Utils_Tuple3(index < 0 ? -1 : target, row, col);

where we either return -1 to signal the "subString not found" case, or the offset after the subString, which is stored in variable target.
So what about all the other changes?

First, I decided to rename the newOffset variable:
src/Elm/Kernel/Parser.js
@@ -122,2 +122,2 @@ var _Parser_findSubString = F5(function(smallString, offset, row, col, bigString
-       var newOffset = bigString.indexOf(smallString, offset);
-       var target = newOffset < 0 ? bigString.length : newOffset + smallString.length;
+       var index = bigString.indexOf(smallString, offset);
+       var target = index < 0 ? bigString.length : index + smallString.length;

It doesn't contain the new offset anymore (since we return either -1 or target), and I thought the name would be misleading now. Therefore I changed the name to "index", because it contains the result of the indexOf function.

In the Parser.Advanced module, there's a wrapper function for every Kernel function. The comment of the findSubString wrapper function has been wrong before this code change (see #37 "Fix comment in findSubString"), and it was still wrong after the code change, so I changed it to document the fact, that we return the position after the subString:
src/Parser/Advanced.elm
@@ -1125,7 +1125,7 @@ isAsciiCode =
     findSubString "42" offset row col "Is 42 the answer?"
         --==> (newOffset, newRow, newCol)

-If `offset = 0` we would get `(3, 1, 4)`
+If `offset = 0` we would get `(5, 1, 6)`
 If `offset = 7` we would get `(-1, 1, 18)`
 -}
 findSubString : String -> Int -> Int -> Int -> String -> (Int, Int, Int)

The wrapper functions hide the fact, that they are implemented in JavaScript rather than in Elm, from the rest of the module code. The rest of the code only uses the wrapper functions, just as if they had been implemented in Elm.
The only exception from this rule was a line where the findSubString Kernel function has been called directly, and I thought it was appropriate to change this line, too, to be consistent with the rest of the code, when we are modifying the findSubString function itself:
src/Parser/Advanced.elm
@@ -913 +913 @@ chompUntilEndOr str =
-        Elm.Kernel.Parser.findSubString str s.offset s.row s.col s.src
+        findSubString str s.offset s.row s.col s.src

If this patch fixes:
https://github.com/elm/parser/issues/20
Does that mean it should be preferred to the PR that fixes just that issue:
https://github.com/elm/parser/pull/21
This PR seems more general than that one and fixes a number of issues together.

Hi Rupert, I think you addressed the question to me, but as the author I'm biased, of course. If I wouldn't prefer this PR to #21, then I wouldn't have added it.


Support subfolder */ glob pattern?

I've organized a bunch of images into subfolders and am trying to do a batch resize of them. I've tried using the subfolder **/* glob pattern in the source, but it doesn't seem to work. Below is my config:
gulp.src(['_source/assets/images/**/*.jpg'])
 	.pipe(responsive(
 	{
 		'*.jpg':
 		[{
 			width: 160,
 			rename:
 			{
 				suffix: '--mini'
 			}
 		}]
 	},
 	{
 		strictMatchImages: false,
 		strictMatchConfig: false
 	}))
.pipe(gulp.dest('images'));

When I run the task with strictMatchConfig set to true I get the error:
stream.js:94
      throw er; // Unhandled stream error in pipe.
            ^
Error: Available images do not match following config:
  - `*.jpg`

When set to false the error doesn't show, as expected, but either way all images in the subfolders are ignored.
Changing the src to one of the subfolders like _source/assets/images/portraits/*.jpg runs fine.
Am I doing something wrong or is there a workaround? I appreciate any help.

Hi @patrickcate
gulp-responsive uses the same filename matching library as gulp (https://github.com/isaacs/node-glob).
So in case when you specify sources as  _source/assets/images/portraits/*.jpg then all files are matched by pattern *.jpg.
When you specify sources as _source/assets/images/**/*.jpg only first level files are matched by pattern *.jpg.
To math all files in sources and in gulp-responsive config you should specify _source/assets/images/**/*.jpg and for example **/*.jpg.

works perfect thanx ! :beers:

@agborkowski thank you for using gulp-responsive :wink:
Maybe you will be interested in the following module: gulp-responsive-config
It is just a proof of concept generation responsive configuration from CSS or HTML, for example following CSS:
.test {
  background-image: url(image-100x200.png);
}
.retina-test {
  background-image: url(image-100x200@2x.png);
}

will be parsed to:
[{
  name: "image.png",
  width: 100,
  height: 200,
  rename: "image-100x200.png"
},{
  name: "image.png",
  width: 200,
  height: 400,
  rename<EMAIL_ADDRESS>}]

and it can be passed directly to gulp-responsive. Profit!
Short example:
gulp.src('/images/**/*.png')
  .pipe(gulpResponsive(gulpResponsiveConfig('/path/to/styles.css')))
  .pipe(gulp.dest('build'));

Currently gulp-responsive-config has no documentation and tests, but I plan to do it.

:+1:

Confirmed, this works.

:beers: !


Recaptcha v3 is a bad choice for this type of app

Is your feature request related to a problem? Please describe.
Recaptcha v3 is more reliable for large, complex websites/apps. It uses analytics on as many pages as possible to track a user's interactions with the site over a long period of time. A small app that consists of a single page and that is rarely visited by a user and only for a few seconds is not enough data for Recaptcha v3 to be reliable.
Without a fallback to select images, etc. Recaptcha v3 fails for me frequently in PassCore v4
Describe the solution you'd like
Re-add support for Recaptcha v2

@Serk352 there is a way to support both versions in the React component that you used?

@Serk352 can you give it a chance to this issue this week?

@geoperez  sure, I will.

@xxbiohazrdxx can you verify the new implementation?

@geoperez getting Could not verify you are not a robot after update from 3 to 4 version all time.

Hi @dragoangel, there's a new release that fixed this issue, please try it.

I had same issue as dragoangel with the captcha v2 verification. Just updated to 4.2.2 and issue remains.

@Serk352 ok will try 4.2.2

@Serk352 build from 4.2.2 GET https://change-pwd.ua.pnnsoft.com/api/password net::ERR_ABORTED 500 and page loading endless

Hi All,
Everything is working except the Recaptcha part. I keep getting could not verify you are not a robot error. Anything specific I need to check? I used V2 & V3 with the correct site and secret keys.

@Serk352 can you take a look in the Recaptcha issue?

@geoperez Hi, after update to 4.2.3 I can successfully build app for linux. I had issues with nuget packages via npm install cli, but via Visual Studio 2019 Community it works. Doesn't know why exactly.


Android RAM req?

Hi! running the quantized models on android sounds pretty exciting. were you able to run inference with a 4gb device? It is also interesting to know what kind of processor you had used! thanks!

Hi! running the quantized models on android sounds pretty exciting. were you able to run inference with a 4gb device? It is also interesting to know what kind of processor you had used! thanks!

never tried on devices with 4 GiB or less. I work for a SoC company, we have all kinds of fancy Android devices with large amount of DRAM. Mainly on devices with MediaTek 9000/9000+/9200 series

thank you for your reply, I hope to learn more by experiment!


Permutation matrices

The following is an (almost complete) implementation of #131. Compared with the list in #131, what remains is:

Products between permutation matrices
Determinant/parity computation for permutation matrices

In addition, I have a few questions:

So far I have not implemented multiplication for MatrixSlice(Mut), only &MatrixSlice(Mut). I see that i.e. Mul<T> is implemented for MatrixSlice. Consuming a matrix slice does not really make much sense in its own, but I suppose this is implemented so that the user does not have to deal with compilation errors when trying to multiply matrix slices and not their references? In that case I will also provide implementations also for &MatrixSlice(Mut).
For some of the Px mul implementations (P permutation matrix, x a Vector), I added some documentation detailing the complexity of each operation. I have not so far done it for the matrix-matrix products. Do you think this is worthwhile, or is the risk of the documentation getting out of sync too large for now, and it is something we could instead revisit once the library is more stable?

Also, note that I've separated all the logic of dealing with permutations into a very general Permutation data structure (which PermutationMatrix internally wraps). This way it's more focused and easier to test specifically only the permutation functionality. This leads to some duplication of APIs (and increased line count), but I think it makes the overall code cleaner. Currently this Permutation struct is exposed through the utils module, but in fact I want to propose making the whole utils module private (will make a separate issue for this eventually).
Finally, note that currently PermutationMatrix has a phantom marker that restricts it to a given type. In fact, one could implement and provide PermutationMatrix completely without any associations to a given scalar type (since it only deals with mapping indices). Put differently, it's possible to make PermutationMatrix not a generic type, and rather a single implementation which could interact with Matrix<T> for any T. However, since BaseMatrix is generic over the scalar field rather than having the scalar field as an associated type, one cannot write functions that operate on PermutationMatrix and BaseMatrix<T> without also associating PermutationMatrix with T, hence the phantom marker.
@AtheMathmo: This is one of the reasons I mentioned the possibility of rewriting BaseMatrix to use an associated type for the scalar field rather than generic type parameter in some other issue that I can not currently find.
In any case, this is not a big issue as usually you'd just work with a single type in the first place.

Also note that I have not been very concerned with performance in this initial PR. The implementation is in general rather simple, using only simple data structures and iteration. I'd expect it to be reasonably fast, but as no area of rulinalg currently uses PermutationMatrix, I do not have any reference points to work with. I'd say we rather get the APIs and functionality right, and then we can possibly improve performance in later PRs (possibly as part of a rewritten #94).

Just a heads-up: I think I will try to refactor the code so that I can avoid some of the duplication with Permutation. That is, I'll outright remove Permutation and leave PermutationMatrix behind, but I'll try to make sure that the permutation logic can still easily be tested on its own.
Just letting you know so that perhaps any sort of detailed review should wait a little. Also, the times I have to work on this are rather unpredictable, so progress may move at an erratic pace.

All right, I consider this ready for review now.
I added itertools as a dev-dependency. We've briefly discussed it before even for a general dependency. Adding it as a dev dependency means we can leverage it for tests, at least. If you don't want it, I can rewrite the test in which I use it, but I have to say that the test is a lot more understandable with it.
I realize it might be hard to review this code if you're not familiar with working with permutations. However, I've added ample tests and I've tried to use QuickCheck to cover the majority of the input space. So hopefully reviewing the tests should convince you that the implementations are most likely correct (unless I've forgotten to include relevant tests).

Thank you for your work on this!
I'll take some time to look through it tomorrow. I've started getting a little more settled so should be able to put some more time into this soon.

Before I review, I'll address some of the questions in the initial post. Sorry that I missed these (I thought we had discussed some before...)

In that case I will also provide implementations also for &MatrixSlice(Mut).

As you pointed out this is messy and for MatrixSlice not really necessary at all (it derives Copy). However I think that for now we should implement for references (as you have) just to save the compilation errors.

[On documenting complexity] Do you think this is worthwhile, or is the risk of the documentation getting out of sync too large for now [...]?

I think that this could be worthwhile even now, but honestly may not be the best use of your time. It will be a little tricky to maintain and the existing documentation you have is plenty. It's up to you but I'm more than happy for you to stick with what you have.

This is one of the reasons I mentioned the possibility of rewriting BaseMatrix to use an associated type

I think it is a pretty compelling reason. Thanks for pointing it out.
Ok now I'll try to read the actual code :)

As you pointed out this is messy and for MatrixSlice not really necessary at all (it derives Copy). However I think that for now we should implement for references (as you have) just to save the compilation errors.

Heh, I made a typo there. At the moment I'm actually missing the implementations for the value types. I'll add them when I get the chance.

I think that this could be worthwhile even now, but honestly may not be the best use of your time. It will be a little tricky to maintain and the existing documentation you have is plenty. It's up to you but I'm more than happy for you to stick with what you have.

I think that while we should start documenting complexity in general, I think it's not necessarily the best usage of time, because rulinalg as a whole is still very volatile. However, in the future, once things are more stable and the algorithms are more battle-tested, we should perhaps start documenting these things. For now I decided to leave it out.

Would you be able to annotate areas of code that you know can be improved (performance-wise)? I just mean if there is anything you came across in your research but that was too hefty for a first implementation it would be good to know.

Well, apart from more efficient indexing, there are no low-hanging fruits that I'm aware of. Unlike e.g. matrix-matrix products, the performance of permutation matrices depend highly on the data itself. For example, applying the identity matrix in place is basically equivalent to iterating over n items, while a worst-case shuffle might involve moving every element in the cache-wise worse possible way.
In fact, I believe that it is proven that there exists no cache-optimal algorithm for applying a permutation. One can however improve cache performance in most cases by representing a permutation by two arrays instead of one, and then sorting them in some specific way, although I never saw an explanation of exactly how they should be sorted. Still, this seems more like a theoretical thing, because the costs of sorting them and having two arrays add significant overhead in practice, which means that it's not universally faster, which means you'd have to apply it conditionally depending on some heuristics. Ugh, it wouldn't be pretty, at least.
To summarize, I'm not sure exactly where to put this in the code, but this is at least my impression of performance characteristics. Bear in mind, however, that I'm certainly no expert on this, I've only skimmed a few papers.

Very minor: I wonder if to stick with the theme permutation_matrix_mul.rs should be impl_permutation_mul.rs?

Yes, I can certainly make that change :)

I think a condition for the merge of this PR should be to make an issue taking note of the areas in code where we would like to introduce this new type (lup being a big one).

Agree. Though, to be fair, I can't think of any other applications than LU right now, off the top of my head. They come up quite often when working with sparse direct solvers, however, so we might see more use of them in the future.
Actually, there's one more application in the current code base. I've known for some time that I actually incorrectly implemented the sorting of the SVD values. I'll have to get around to making an issue for that (or add it to the existing SVD issue). Turns out, this should be possible to trivially fix with the new PermutationMatrix type.
Thanks a lot for your insightful comments and suggestions! I'll try to get around to fixing these things during the week :)

Heh, I made a typo there. At the moment I'm actually missing the implementations for the value types. I'll add them when I get the chance.

No worries! This is certainly minor, I thought it was useful mostly for consistency.
Thank you for explaining the performance a little more. The reason I asked is that I couldn't identify anything myself and didn't want to miss anything in the future. Things look great to me so don't worry about any minor theoretical performance gains :)
In addition to LUP and SVD there is a parity function in matrix/mod.rs that we can remove once this is merged. I'm happy to write up this issue myself once we merge.

All right, unless I missed anything, I believe I've addressed the points you brought up!


Wrong result when a shift involves a DST change.

In paris in 2013, we changed to DST in 2013-03-31 (at 2 AM it was 3 AM), from UTC+1 to UTC+2 (DST)
Here's an example of the issue I have with arrow :
>>> import arrow, datetime

>>> just_before = arrow.get(datetime.datetime(2013, 03, 31, 1, 50, 45), "Europe/Paris").ceil("hour")
>>> just_before
<Arrow [2013-03-31T01:59:59.999999+01:00]>

>>> just_after = just_before.replace(microseconds=+1)
>>> just_after
<Arrow [2013-03-31T02:00:00+02:00]>
>>> # That is not right... It should be either 2 AM UTC+1 or preferably  3AM UTC +2
>>> # One could argue that depending on what you ask for, it may be the correct answer, but then the following is :

>>> (just_after - just_before).total_seconds()
1e-06
>>> # That is right but not consistent with the dates

>>> (just_after.to("utc") - just_before.to("utc").total_seconds()
-3599.999999
# That should be the same value as the previous computation. Plus, the value is negative because :

>>> just_before.to('utc'), just_after.to('utc')
(<Arrow [2013-03-31T00:59:59.999999+00:00]>,
 <Arrow [2013-03-31T00:00:00+00:00]>)

I think the problem is pretty clear...

Isn't the "proper" solution to only add and subtract datetimes in UTC and then convert back to localtime?

@chebee7i I don't think so. When you're on 2013-03-30 12:00:00+01:00, in Paris and you you want "1 day later", if you shift to UTC, you'll get 2013-03-30 11:00:00Z, add 1 day that's 2013-03-30 11:00:00Z, shift back to Paris, it's 2013-03-30 13:00:00+02:00.
Going to UTC is "useless" in this case, it's just like considering "1 day equals 24 hours, period." and that's the heart of the problem. Everything under the day (second, minute, hour) works like that (well except with leap seconds but that's something else) and everything from the day and up, what you need (most  of the time, or at least what I do need) is not a time shift but a calendar digits replace.

I'm still a bit confused. Here is a script:
from __future__ import print_function

import datetime
import pytz

paris = pytz.timezone('Europe/Paris')
utc = pytz.utc

before_naive = datetime.datetime(2013, 3, 31, 1, 59)
before_paris = paris.localize(before_naive)
before_utc = before_paris.astimezone(utc)

delta = datetime.timedelta(minutes=1)
after_utc = (before_utc + delta)
after_paris = after_utc.astimezone(paris)

# Compute aware deltas
minutes_paris = (after_paris - before_paris).total_seconds() / 60
minutes_utc = (after_utc - before_utc).total_seconds() / 60

# Compute naive deltas
naive_minutes_paris = (after_paris.replace(tzinfo=None) - before_paris.replace(tzinfo=None)).total_seconds() / 60
naive_minutes_utc = (after_utc.replace(tzinfo=None) - before_utc.replace(tzinfo=None)).total_seconds() / 60

print("Paris")
print(before_paris)
print(after_paris)
print("Minutes:", minutes_paris) # 1 minute, as expected
print("Naive Minutes:", naive_minutes_paris) # 61 minutes, as visually seen.
print()
print("UTC")
print(before_utc)
print(after_utc)
print("Minutes:", minutes_utc) # 1 minute, as expected.
print("Naive Minutes:", naive_minutes_utc) # 1 minute, as visually seen.

Can you tell me what is wrong here? And what you want?

Anyone want to submit a PR for this?

See #227 for another example
`
import arrow
import pytz
from datetime import date, datetime, timedelta
print arrow.Arrow.fromtimestamp(0, pytz.timezone('Europe/Paris'))
just_before = arrow.Arrow(2013, 3, 31, 1, 59, 59, 999999, 'Europe/Paris')
#just_after = just_before.replace(microseconds=1, hours=1)
just_after = just_before.shift(microseconds=1)
print just_before
print just_after
print just_before.to('utc')
print just_after.to('utc')
print (just_after.to('utc') - just_before.to('utc')).total_seconds()`
Produces
1970-01-01T01:00:00+01:00 None 2013-03-31T01:59:59.999999+01:00 2013-03-31T02:00:00+02:00 2013-03-31T00:59:59.999999+00:00 2013-03-31T00:00:00+00:00 -3599.999999
The problem is the hour should be 3 which is then throwing off the other calculations.

I think @chebee7i nails it. It's all in the localize with pytz. Without doing that (which is recommended by pytz) the calculation is off.

@andrewelkins It seems that all the examples you point out as to be fixed comes from here: https://github.com/sdispater/pendulum#why-not-arrow (I am the author of Pendulum by the way)
(And if you took the time to look at it you would have noticed that this is the same example as the first post of this issue)
I should point out that this approach is wrong since those are just some non-exhaustive examples and should not be treated as specific cases but rather understanding the underlying problem, in this case understanding how DST transitions work.
This is an issue that has been here for 3 years and neither Chris nor you have been able to fix this while being critical for a datetime library, so the promise made by Arrow seems dubious, at best. I started Pendulum because Arrow is so broken (you can just look at your issues piling up) I wondered if its maintainers had any knowledge of what a datetime library should be or if they had any desire to go forward with the library.
Sorry if I sound too harsh but I have been bitten so many times by Arrow that I could no longer rely on it. To be honest, the first stable version of Pendulum is around the corner and does everything Arrow does and more, without the bugs.
I don't know what you plan to do with Arrow (it seems to me that you just expect people to fix the issues for you at this point) but you owe it to your users to warn them about the critical shortcomings of the library.

I get it that you're upset at Arrow not working how you would expect, but there's no need to be aggressive and condescending here. We're in @crsmithdev 's space, AFAIK we both have no idea of what might have happened in their live that kept them from fixing the bug (or I should say, to spend countless hours and/or days maintaining a library they may not even use anymore themselves, working completely for nothing, on their free time in addition to whatever else they might have to deal with) and it's dishonest to assume it has anything to do with their competence.
Using someone else's software won't make them owe you anything. It's a pure question of gratefulness, of trust, of sharing.

A library does not grow with time but with contributions. Contributions are more frequent when a project is useful and welcoming. Having people taking over issues to say their own project is better and this project is shit does not really help making the space welcoming. Just saying.

I've been using Arrow a little bit (not quite much, I admit), but I've always looked at that project with interest. When I discovered pendulum lately (and we had an exchange on twitter), I was equally interested. Sadly, I'm not sure I really want to go further now. I'm aware that you have no reason to care, but I sincerly hope that by the time pendulum becomes a must-have in many projects, which might happen, I can find in your community a more welcoming atmosphere :/

I agree with what you say. But if you don't have time or the interest anymore to maintain it (and I completely understand that, believe me) you can add new contributors to the project so that you can pass it on and it keeps on living. But Arrow seems to be stagnating bugfixes wise, that's what I think is a shame. When Andrew took over for Chris a lot of issues were fixed and improvements made which I found really good because Arrow could catch up on what was broken. But critical issues were not fixed, like this one.
And don't get me wrong, I am not advocating for Arrow to be completely replaced by Pendulum, Arrow has its user base because they find it useful, which I understand. I merely pointed some flaws and bugs to raise awareness not to force anyone to use something else (How could I?). And, to be honest, I would be really glad if both libraries were to remain and be used along because, in my opinion, while they are close they don't have the exact same set of features.
The thing that bothers me is that Arrow is seen by a lot of people as the definitive library when it comes to datetime handling in Python while it is not necessarily, mostly due to critical bugs, and alternatives exist (and I am not talking only about pendulum which is one of many). If these bugs were to be fixed I would have nothing to say about it.
And I agree that open source maintainers and creators don't owe you anything as a user. However, I think that as a library maintainer you owe it to people using your library to, at least, be true to the promise you made initially since this is the reason why people chose to use it, and if not just point the shortcomings of your library. That's all. And if you can't do it anymore just bring someone along to help you with it, everyone would benefit from it.
And sorry if you felt animosity in my post, that was not the intention. Sometimes I get a little carried away when it comes to subject I really care about and I sound like rude. I am sorry about that.
But I want to tell @andrewelkins and @crsmithdev that it is not a personal attack. I mean Chris made a really popular library, useful to a lot of people and that is a great achievement in itself. And as a developer myself I know what it can cost and the time it can take, so I will always recognize the work that has been done. And Andrew took over a project that was already largely popular and that is not small work.
What I would like to see now, however, is what they want to do with the library, what they plan to do with it. Nothing more.

FYI, this is relatively simple to fix, because you just need to detect whether it's an imaginary datetime. There's a convenience function in python-dateutil>=2.6.0 called tz.datetime_exists. The way it works is that it puts a datetime through a round trip from UTC and back, and if it doesn't get the same answer, you know it's imaginary, e.g.:
from dateutil import tz
from datetime import datetime

PAR = tz.gettz('Europe/Paris')

im_dt = datetime(2013, 3, 31, 2, 0, tzinfo=PAR)   # This time does not exist

dt_u = im_dt.astimezone(tz.tzutc())
im_dt_back = dt_u.astimezone(PAR)

print(im_dt)
# 2013-03-31 02:00:00+02:00
print(im_dt_back)
# 2013-03-31 01:00:00+01:00

r_dt = datetime(2013, 3, 31, 3, 0, tzinfo=PAR)
r_dt_back = r_dt.astimezone(tz.tzutc()).astimezone(PAR)

print(r_dt)
# 2013-03-31 03:00:00+02:00
print(r_dt_back)
# 2013-03-31 03:00:00+02:00

So you could do something like this in shift:
from dateutil.tz import datetime_exists

def resolve_imaginary(dt):
    if dt.tzinfo is not None and not datetime_exists(dt):
        # You need to shift this by the difference between the current offset
        # and the previous one. Historically, there has never been a time zone
        # shift of more than 24 hours, nor has there been two changes to a time
        # zone within 24 hours, so if we take the "wall time" - 24 hours, we
        # should be OK, barring any insane time zone behavior in the future.
        curr_offset = dt.utcoffset()
        old_offset = (dt - timedelta(hours=24)).utcoffset()

        dt += curr_offset - old_offset

    return dt

Then you can just call resolve_imaginary() on current before returning it in shift().
If you don't want to peg your dateutil dependency at 2.6.0, then you can stub it out when missing:
try:
    from dateutil.tz import datetime_exists
except ImportError:
    def datetime_exists(dt, tz=None):
        # Implement handling the tz-attachment part if necessary here

       return (dt.astimezone(dateutil_tz.tzutc()).astimezone(dt.tzinfo) == dt)

@sdispater Thanks for chiming in. Do you have a more exhaustive list known bugs? I would love to have that contribution.
I'm aware the DST translations is an area that needs work. As you can see this ticket is pretty old and was never resolved. It is the number one thing to fix and why I reupped this issue.
I wish you success with your library.
@ewjoachim Thanks for chiming in on my behalf :+1:

@pganssle Thanks for the example. Seems straight forward and I'll test it out. Do you have time to make it a PR w/ tests?

@andrewelkins Pretty busy schedule, so probably not, feel free to take what I've got and run with it.
FYI, relevant edge cases:

The entire day of datetime(2011, 12, 30) is imaginary in the 'Pacific/Apia' zone.
You should probably test against Europe/London, because that's often a source of DST-related issues, since it shifts from 0 to +1.
Possibly relevant: on datetime(1969, 10, 1, 0) in 'Pacific/Kwajalein', the time zone offset went back 23 hours, so the interval of [datetime(1969, 9, 30, 1), datetime(1969, 10, 1, 0))is ambiguous. (I don't have an example immediately at hand where there was a full 24 hour adjustment).

@andrewelkins Thanks for the kind words :-) Like I said, it is not directed towards you personally but rather to raise awareness about this issue. I know quite well DST is a tricky problem.
I can't really provide any solution for Arrow since I came up with my own implementation in the end rather than relying on pytz but it should be doable.
And @pganssle's solution seems like a good starting point.

FWIW, I did find a -24 hour edge case, but it's a bit dubious because they're all LMT -> LMT transitions that happened in 1901 for some reason:
 ((datetime.timedelta(-1), datetime.datetime(1901, 12, 13, 9, 22, 52)),
  'US/Samoa'),
 ((datetime.timedelta(-1), datetime.datetime(1901, 12, 13, 9, 22, 52)),
  'Pacific/Samoa'),
 ((datetime.timedelta(-1), datetime.datetime(1901, 12, 13, 9, 22, 52)),
  'Pacific/Pago_Pago'),
 ((datetime.timedelta(-1), datetime.datetime(1901, 12, 13, 9, 22, 52)),
  'Pacific/Midway'),
 ((datetime.timedelta(-1), datetime.datetime(1901, 12, 13, 9, 18, 52)),
  'Pacific/Apia')

@sdispater Not looking for solutions, just issues to fix other than DST. Seemed like you had a few issues, and I'd like to get them addressed for the user base. We won't be relying on pytz either, it has some odd edge cases itself.
@pganssle Thanks for bringing that up. I'll start the PR and writing the tests over the next couple days.

@andrewelkins Most of the issues I have encountered were in timezone handling and DST due to the fact that I implemented my own timezone library since I was not satisfied by the existing solutions (being pytz or dateutil) and I wanted a performant and simple implementation (https://pendulum.eustace.io/blog/a-faster-alternative-to-pyz.html).
Other issues were more related to the API design rather than bugs

@sdispater Great, Thanks for the info!

I'd just like to be clear about this - doing a time shift should automatically resolve DST shifts, both .dst() timedelta-wise and time-wise. That's the goal, right?

Here's another example around the EST DST change:
import arrow
import pytz
tz = pytz.timezone('America/New_York')
end_date = tz.localize(datetime(2018, 3, 13, 4, 58, 28))
 # end_date could be in UTC but showing it in the same timezone here for simplicity (hence we need the `.to(tz)` below.
# arrow datetime at the end of the day
end = arrow.get(end_date).to(tz).ceil('day')
# -6 days as we go from the start of the first day to the end of the last day
start = end.shift(days=-6).floor('day')

# Note: If I convert to the timezone again after shifting I get the result I want
correct_start = end.shift(days=-6).to(tz).floor('day')

Results
end # This is correct
Out[23]: <Arrow [2018-03-13T23:59:59.999999-04:00]>
start # This is wrong - should be -5 hours as the date at the start is before DST
Out[22]: <Arrow [2018-03-07T00:00:00-04:00]>
correct_start # This works but I shouldn't have to convert to the timezone again after shifting!
Out[27]: <Arrow [2018-03-07T00:00:00-05:00]>

@intiocean I would think that's because you're using pytz, which needs to be normalized if it crosses a DST boundary.

@pganssle Interesting. I would have thought that passing in a tzinfo to the to() method would be the same as passing the timezone name. However after your suggestion it was pytz I've experimented and it's not the same.
tz = pytz.timezone('America/New_York')
end_date = tz.localize(datetime(2018, 3, 13, 4, 58, 28))
end = arrow.get(end_date).to('America/New_York').ceil('day')
start = end.shift(days=-6).floor('day')

Results in
In [6]: end_date
Out[6]: datetime.datetime(2018, 3, 13, 4, 58, 28, tzinfo=<DstTzInfo 'America/New_York' EDT-1 day, 20:00:00 DST>)

In [7]: end
Out[7]: <Arrow [2018-03-13T23:59:59.999999-04:00]>

In [8]: start
Out[8]: <Arrow [2018-03-07T00:00:00-05:00]>

I'll change my code to use the timezone name but the different behaviour seems very counter intuitive - maybe we should add a warning to the to() docs?
PS. I'm not clear if this is the same issue others in this thread are having or if their issue is more around times that repeat over the actual hour just before/after DST

@intiocean I imagine there are a lot of subtle ways that using a pytz zone could go wrong because pytz doesn't use the standard datetime time zone interface. You may end up having to sprinkle warnings all over the place in arrow. If it were up to me I'd say "pytz is not supported`", and be done with it, but it's not clear how and where to best communicate this.

This issue isn't limited to .shift(), but also functions like .range().
That part is clear, but here's a quick test case across the spring DST boundary to check:
before = arrow.get('2018-03-10 23:00:00', 'YYYY-MM-DD HH:mm:ss', tzinfo='US/Pacific')
after = arrow.get('2018-03-11 04:00:00', 'YYYY-MM-DD HH:mm:ss', tzinfo='US/Pacific')
[ (t, t.to('utc')) for t in arrow.Arrow.range('hour', before, after) ]

In the normalized-to-UTC column, note the two 2018-03-11T09:00:00+00:00 entries:
[(<Arrow [2018-03-10T23:00:00-08:00]>, <Arrow [2018-03-11T07:00:00+00:00]>),
 (<Arrow [2018-03-11T00:00:00-08:00]>, <Arrow [2018-03-11T08:00:00+00:00]>),
 (<Arrow [2018-03-11T01:00:00-08:00]>, <Arrow [2018-03-11T09:00:00+00:00]>),
 (<Arrow [2018-03-11T02:00:00-07:00]>, <Arrow [2018-03-11T09:00:00+00:00]>),  <-- duplicated!
 (<Arrow [2018-03-11T03:00:00-07:00]>, <Arrow [2018-03-11T10:00:00+00:00]>),
 (<Arrow [2018-03-11T04:00:00-07:00]>, <Arrow [2018-03-11T11:00:00+00:00]>)]

FWIW, the way pandas handles DST boundaries on a range/span seems to me to be the right way (ie treat it as absolute hours localized into the timezone, not as a range across local hours). Useful to consider it as a model?
For example, the equivalent of Arrow.range('hour', ...) over a DST boundary in pandas would be something like these (in both cases, it's always an hour shift if you consider the UTC equivalents):
Spring boundary:
pd.date_range('2018-03-11', tz='US/Pacific', periods=4, freq='H')

[
  '2018-03-11 00:00:00-08:00',
  '2018-03-11 01:00:00-08:00',
  '2018-03-11 03:00:00-07:00',
  '2018-03-11 04:00:00-07:00'
]

Fall boundary:
pd.date_range('2018-11-04', tz='US/Pacific', periods=4, freq='H')

[
  '2018-11-04 00:00:00-07:00',
  '2018-11-04 01:00:00-07:00',
  '2018-11-04 01:00:00-08:00',
  '2018-11-04 02:00:00-08:00'
]

I am not sure if this is the same problem, but I found that I could not subtract times around daylight savings, either.
For example (I am using pandas, and hoping everyone knows it well enough).  I have a DataFrame, and I have adjusted the start time so that it "works", but I have not adjusted the end time.  Here is a sample around daylight savings (apologies for all the red font, I don't know why GitHub is rendering that):
>>> orig_df.loc[127857:127861,"StartTimeArrow"]

127857    2019-03-31T01:57:00+01:00
127858    2019-03-31T01:58:00+01:00
127859    2019-03-31T01:59:00+01:00
127860    2019-03-31T02:00:00+02:00
127861    2019-03-31T02:01:00+02:00
Name: StartTimeArrow, dtype: object

>>> orig_df.loc[127857:127861,"EndTimeArrow"]

127857    2019-03-31T01:58:00+01:00
127858    2019-03-31T01:59:00+01:00
127859    2019-03-31T03:00:00+02:00
127860    2019-03-31T03:01:00+02:00
127861    2019-03-31T03:02:00+02:00
Name: EndTimeArrow, dtype: object

If we include the time zone, which is capturing daylight savings properly, then I have actually corrupted the data by shifting it.  But if I want the proper datetime.timedelta calculations returned, I need to do the shift:
>>> orig_df.loc[127861,"StartTimeArrow"] - orig_df.loc[127857,"StartTimeArrow"]

datetime.timedelta(seconds=240)

Great!  I have captured 4 minutes, the true difference in time, and also 2:01 - 1:57.
But... this delta is not considering the time zone shift, which you can see is still there (note that it shifts from +1 to +2).
Considering the time zone shift, the original objects that arrow created which I didn't modify - the "EndTimeArrow" values - should be correct.  But when I try to subtract them, it's a mess:
>>> orig_df.loc[127861,"EndTimeArrow"] - orig_df.loc[127857,"EndTimeArrow"]

datetime.timedelta(seconds=3840)

Obviously this is wrong.  I am going to use the adjustment I'd made in the "StartTimeArrow" column, but I'm nervous since this is wrong and will break as soon as the time zone is properly captured.

Is there a better way to subtract times using arrow?  I tried to play around with Arrow.shift, but that didn't seem like the right solution?

I haven't dug deeply enough, but is there a pull request I can help with?  Is this really a dateutil issue, as @ManThursday suggests?  In that case, I doubt I can do anything to help.

@lazarillo honestly, if you're already using pandas and you're doing timezone-aware stuff, just use pd.Timestamp everywhere rather than trying to mix arrow into pandas. pd.Timestamp handles DST boundaries correctly, and all the pandas stuff like DatetimeIndex, indexing, vector operations, etc. are built using it. The pd.Timestamp + pd.Timedelta interface for time shifts isn't as nice as arrow's APIs, but since arrow has lots of unpatched DST bugs (eg this one first reported in 2013), I've found arrow just doesn't work if you're doing anything timezone-aware. fwiw, I'm migrating everything over to pd.Timestamp, even when not using in a dataframe.

Hello @dlopuch, I have a branch locally that is nearly ready to PR which should fix most of the DST issues currently present in arrow (including this issue).
pandas uses pytz internally which may cause some problems (article) when dealing with DST. pandas does not particularly support arrow objects at the moment (see #755 ) but they seem willing to help out.

OK, thanks @dlopuch  and @systemcatch .  I'd read somewhere that pytz had more issues with it than arrow when dealing with these corner cases, but I did not realize / research that pandas used pytz under the hood anyway.
So I'll just adjust my code to work with pandas "purely", and avoid arrow in this case.  That's a shame, since I really like how arrow works; it's a great API.
I guess I didn't realize how much the Python ecosystem has left to deal with in this datetime world.  I'm not even asking about leap seconds or anything bizarre (IMO).  I know it's a tricky, pain-in-the-ass problem to solve, but it's also a hugely important one.

@lazarillo that's a pity but I hope you'll revisit arrow in the future.
python's datetime module is rather lacking in basic features, but from 3.6+ there have been some improvements.
For reference this is the branch where I'm working on handling DST properly https://github.com/systemcatch/arrow/tree/DST-handling

@systemcatch Yes, I'll definitely revisit arrow (and in fact, I have not pulled it from my flow yet).  I really like it, but I cannot give up the practicality of using pandas for this project.
If there is something you want me to help with on your fork, let me know.  I don't have a ton of free time, but I would like to help give back.  :)

Hopefully most of the bugs mentioned in this issue have been solved now. We will release 0.17.0 with these fixes asap.


Upgrade Gardener to 1.10.1

Description
Changes proposed in this pull request:

Upgrade Gardener to 1.10.1 (version from master)

Related issue(s)

/test pre-master-control-plane-gke-provisioner-integration

/test pre-master-control-plane-gke-provisioner-integration

/test pre-master-control-plane-gke-integration

/test pre-master-control-plane-gke-provisioner-integration

/test pre-master-control-plane-gke-integration

The Upgrade Shoot test failed with the timeout:
time="2020-08-31T12:24:05Z" level=info msg="Asserting operation 527b6010-ab41-44cf-a84b-424b03c5965e is in Succeeded state."
time="2020-08-31T12:24:05Z" level=info msg="Operation message: error: timeout while processing operation"
    operation.go:26:
                Error Trace:    operation.go:26
                                                        operation.go:13
                                                        upgrade_shoot_test.go:76
                Error:          Not equal:
                                expected: "Succeeded"
                                actual  : "Failed"
                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1,2 +1,2 @@
                                -(gqlschema.OperationState) (len=9) Succeeded
                                +(gqlschema.OperationState) (len=6) Failed
                Test:           TestShootUpgrade/Azure

I'm retriggering the job
/test pre-master-control-plane-gke-provisioner-integration

There is still problem with this KEB thing

And it's gone :)

/test pre-master-control-plane-gke-provisioner-integration

The provisioning test failed:
=== CONT  Test_E2E_Gardener/Azure
    helpers.go:12:
                Error Trace:    helpers.go:12
                                                        e2e_test.go:51
                Error:          Received unexpected error:
                                Failed to receive proper Operation Status response
                                github.com/kyma-project/control-plane/tests/provisioner-tests/test/testkit/compass/provisioner.client.ProvisionRuntime
                                        /go/src/github.com/kyma-project/control-plane/tests/provisioner-tests/test/testkit/compass/provisioner/client.go:56
                                github.com/kyma-project/control-plane/tests/provisioner-tests/test/provisioner.Test_E2E_Gardener.func1.1
                                        /go/src/github.com/kyma-project/control-plane/tests/provisioner-tests/test/provisioner/e2e_test.go:50
                                testing.tRunner
                                        /usr/local/go/src/testing/testing.go:1039
                                runtime.goexit
                                        /usr/local/go/src/runtime/asm_amd64.s:1373
                Test:           Test_E2E_Gardener/Azure
    helpers.go:16: Received unexpected error: Failed to receive proper Operation Status response [Error while starting Runtime provisioning]

this error seems to be flaky. I'll retest the job
/test pre-master-control-plane-gke-provisioner-integration

It happened again. @rafalpotempa , could you take a look at it? 😢

/test pre-master-control-plane-gke-provisioner-integration

/test pre-master-control-plane-gke-integration

/test pre-master-control-plane-gke-provisioner-integration

/test pre-master-control-plane-gke-provisioner-integration

/test pre-master-control-plane-gke-provisioner-integration

/test pre-master-control-plane-gke-provisioner-integration

/meow

/test pre-master-control-plane-gke-provisioner-integration

/test pre-master-control-plane-gke-provisioner-integration

/test pre-master-control-plane-gke-provisioner-integration

/test pre-master-control-plane-gke-provisioner-integration

/test pre-master-control-plane-gke-provisioner-integration

/test pre-master-control-plane-gke-provisioner-integration

/test pre-master-control-plane-gke-provisioner-integration

/test
pre-master-control-plane-gke-provisioner-integration

/test pre-master-control-plane-gke-integration

/test pre-master-control-plane-integration

/test pre-master-control-plane-gke-provisioner-integration

/test pre-master-control-plane-gke-provisioner-integration

/test pre-master-control-plane-gke-provisioner-integration

/test pre-master-control-plane-gke-integration

/test pre-master-control-plane-gke-provisioner-integration


Update repository URL for Flake8

Committed via https://github.com/asottile/all-repos

Pull Request Test Coverage Report for Build 760925448

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+100.0%) to 100.0%

Totals

Change from base Build 505910609:
100.0%

Covered Lines:
122

Relevant Lines:
122

💛  - Coveralls


Patch 3

Implementation of Boolean class.

@freakboy3742  I have removed the extra lines and merged them into a single else statement.

Fantastic! Thanks for the Bool and Float patches!


Trial

Could you delete the  "2023 and 2024 ALL" file? Although we are nowhere near close to the disk quota for now, I think we better be conservative about it.

Sorry, I meant deleting all of the datasets in the "Merging the Data" folder. I prefer to have only the notebooks in that folder.
If it causes any issues please let me know.


Remove unused imports

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.04%) to 94.188%

Totals

Change from base Build<PHONE_NUMBER>:
-0.04%

Covered Lines:
551

Relevant Lines:
585

💛  - Coveralls


Add structured binary logger to MSBuild

To improve diagnostics for builds, MSBuild should include a binary logger that can produce a structured and compressed diagnostic log. The logger should be included by default and easily turned on, e.g. by adding a /bl switch similar to /fl which produces a text file log. The log file should be as as small as possible in size (compressed) and minimal impact to build performance/memory to encourage users to turn it on all the time. This should be especially useful in CI scenarios where a full diagnostic log is too large and intermittent issues are therefor hard to diagnose.
My original idea and proof of concept used the pre-existing binary serialization on log event objects to produce a binary serialized log. I was able to get this working and the compressed size was <10% the size of a full diagnostic log file. I liked this approach as it would allow to potentially de-serialize log event objects and "replay" them to an existing ILogger. For example, one could produce a text log of a specified verbosity or an XML log to use in a viewer (like https://github.com/KirillOsenkov/MSBuildStructuredLog).

If you wrap the binary serialized stream in a GZipStream, it should compress substantially because of all those very similar strings you get in logs.

Binary Serialization is coming back in .NET Standard 2.0, but it's not a format that's going to be compatible across different runtimes, operating systems, or processor architectures, so it doesn't seem appropriate to use as a logging format.

@dsplaisted agree, that's why I haven't pursued that any further. I just liked the idea that we could run the binary format through an existing logger. The MSBuildStructuredLog could use that without having to understand the binary format because it can already attach as a logger.

I also like the idea of replayability.
As for saving space, I've had huge savings of space (500 MB XML -> 3 MB binary) by employing two tricks:

Deduplicate all strings through a string table
Compress everything at the end using GZipString like @danmosemsft suggests

The details about my format are here:
https://github.com/KirillOsenkov/MSBuildStructuredLog/wiki/Log-Format
The downsides I ran into is that it's not a streaming format, i.e. it keeps accumulating everything in memory before writing it out in one fell swoop at the end. This way you achieve string deduplication, however for large builds 32-bit MSBuild.exe runs out of memory. Workarounds are to use 64-bit MSBuild.exe or one should investigate distributed loggers (or another way to accumulate the logger data in a separate 64-bit process).

This would also be helpful in CLI to generate useful failure messages. For example, if we had programatic access to property & item values after a failure occurs then we could use those to give feedback. For example, we could inspect the value of TargetFrameworks to tell the user --framework netcoreapp10 does not exist in the project.

Hey, @KirillOsenkov did this for 15.3! See #1747