Fix pipeline module evaluation when contiguous activation checkpoin…

When activation checkpointing with contiguous checkpoints are enabled calling consecutive eval_batch() will result in
File "/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 421, in partition_activations
contiguous_data_buffers[i][data_offsets[i]].data[range(
IndexError: list index out of range
Must call deepspeed.runtime.activation_checkpointing.reset() after forward to reset internal buffers.
In training these buffers are reset in CheckpointFunction.backward()

@microsoft-github-policy-service agree company="Habana"
From: microsoft-github-policy-service[bot] @.>
Sent: Monday, 13 March 2023 18:31
To: microsoft/DeepSpeed @.>
Cc: Liran Bachar @.>; Mention @.>
Subject: Re: [microsoft/DeepSpeed] Fix pipeline module evaluatiion when contiguoush activation checkpoin… (PR #3005)
@hablbhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fhablb&data=05|01|lbachar%40habana.ai|4b14c900aa034de09ef108db23e0525d|0d4d4539213c4ed8a251dc9766ba127a|0|0|638143218589293535|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=pAOolri%2F7DrGotkA0tLeSuy9%2FJezY8cojp67MECw97g%3D&reserved=0 please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
@microsoft-github-policy-service agree [company="{your company}"]
Options:

(default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

(when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"
Contributor License Agreement
Contribution License Agreement
This Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”),
and conveys certain license rights to Microsoft Corporation and its affiliates (“Microsoft”) for Your
contributions to Microsoft open source projects. This Agreement is effective as of the latest signature
date below.

Definitions.
“Code” means the computer software code, whether in human-readable or machine-executable form,
that is delivered by You to Microsoft under this Agreement.
“Project” means any of the projects owned or managed by Microsoft and offered under a license
approved by the Open Source Initiative (www.opensource.orghttps://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.opensource.org%2F&data=05|01|lbachar%40habana.ai|4b14c900aa034de09ef108db23e0525d|0d4d4539213c4ed8a251dc9766ba127a|0|0|638143218589293535|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=WTG6uoQHP38k6C3XbKj7OIh%2BFXWswBhnn7Yn24asw1U%3D&reserved=0).
“Submit” is the act of uploading, submitting, transmitting, or distributing code or other content to any
Project, including but not limited to communication on electronic mailing lists, source code control
systems, and issue tracking systems that are managed by, or on behalf of, the Project for the purpose of
discussing and improving that Project, but excluding communication that is conspicuously marked or
otherwise designated in writing by You as “Not a Submission.”
“Submission” means the Code and any other copyrightable material Submitted by You, including any
associated comments and documentation.
Your Submission. You must agree to the terms of this Agreement before making a Submission to any
Project. This Agreement covers any and all Submissions that You, now or in the future (except as
described in Section 4 below), Submit to any Project.
Originality of Work. You represent that each of Your Submissions is entirely Your original work.
Should You wish to Submit materials that are not Your original work, You may Submit them separately
to the Project if You (a) retain all copyright and license information that was in the materials as You
received them, (b) in the description accompanying Your Submission, include the phrase “Submission
containing materials of a third party:” followed by the names of the third party and any licenses or other
restrictions of which You are aware, and (c) follow any other instructions in the Project’s written
guidelines concerning Submissions.
Your Employer. References to “employer” in this Agreement include Your employer or anyone else
for whom You are acting in making Your Submission, e.g. as a contractor, vendor, or agent. If Your
Submission is made in the course of Your work for an employer or Your employer has intellectual
property rights in Your Submission by contract or applicable law, You must secure permission from Your
employer to make the Submission before signing this Agreement. In that case, the term “You” in this
Agreement will refer to You and the employer collectively. If You change employers in the future and
desire to Submit additional Submissions for the new employer, then You agree to sign a new Agreement
and secure permission from the new employer before Submitting those Submissions.
Licenses.

Copyright License. You grant Microsoft, and those who receive the Submission directly or
indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license in the
Submission to reproduce, prepare derivative works of, publicly display, publicly perform, and distribute
the Submission and such derivative works, and to sublicense any or all of the foregoing rights to third
parties.
Patent License. You grant Microsoft, and those who receive the Submission directly or
indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license under
Your patent claims that are necessarily infringed by the Submission or the combination of the
Submission with the Project to which it was Submitted to make, have made, use, offer to sell, sell and
import or otherwise dispose of the Submission alone or with the Project.
Other Rights Reserved. Each party reserves all rights not expressly granted in this Agreement.
No additional licenses or rights whatsoever (including, without limitation, any implied licenses) are
granted by implication, exhaustion, estoppel or otherwise.

Representations and Warranties. You represent that You are legally entitled to grant the above
licenses. You represent that each of Your Submissions is entirely Your original work (except as You may
have disclosed under Section 3). You represent that You have secured permission from Your employer to
make the Submission in cases where Your Submission is made in the course of Your work for Your
employer or Your employer has intellectual property rights in Your Submission by contract or applicable
law. If You are signing this Agreement on behalf of Your employer, You represent and warrant that You
have the necessary authority to bind the listed employer to the obligations contained in this Agreement.
You are not expected to provide support for Your Submission, unless You choose to do so. UNLESS
REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING, AND EXCEPT FOR THE WARRANTIES
EXPRESSLY STATED IN SECTIONS 3, 4, AND 6, THE SUBMISSION PROVIDED UNDER THIS AGREEMENT IS
PROVIDED WITHOUT WARRANTY OF ANY KIND, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY OF
NONINFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
Notice to Microsoft. You agree to notify Microsoft in writing of any facts or circumstances of which
You later become aware that would make Your representations in this Agreement inaccurate in any
respect.
Information about Submissions. You agree that contributions to Projects and information about
contributions may be maintained indefinitely and disclosed publicly, including Your name and other
information that You submit with Your Submission.
Governing Law/Jurisdiction. This Agreement is governed by the laws of the State of Washington, and
the parties consent to exclusive jurisdiction and venue in the federal courts sitting in King County,
Washington, unless no federal subject matter jurisdiction exists, in which case the parties consent to
exclusive jurisdiction and venue in the Superior Court of King County, Washington. The parties waive all
defenses of lack of personal jurisdiction and forum non-conveniens.
Entire Agreement/Assignment. This Agreement is the entire agreement between the parties, and
supersedes any and all prior agreements, understandings or communications, written or oral, between
the parties relating to the subject matter hereof. This Agreement may be assigned by Microsoft.

—
Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2FDeepSpeed%2Fpull%2F3005%23issuecomment-1466480519&data=05|01|lbachar%40habana.ai|4b14c900aa034de09ef108db23e0525d|0d4d4539213c4ed8a251dc9766ba127a|0|0|638143218589293535|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=b%2FjYErWpkScJeYNazvCi8OyQRwimAlnPEd4W9Yk%2Fjwo%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAWMN6H5TM6J4OSTOYTBZOBTW35DT7ANCNFSM6AAAAAAVZJ6444&data=05|01|lbachar%40habana.ai|4b14c900aa034de09ef108db23e0525d|0d4d4539213c4ed8a251dc9766ba127a|0|0|638143218589293535|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=hQ%2FFKUJSYJaJ%2BqHPCmK%2BxLiPrn8BvRTj%2FfLwsZ6k04s%3D&reserved=0.
You are receiving this because you were mentioned.Message ID<EMAIL_ADDRESS>

@tjruwase Can you please review?

@tjruwase Can you please approve running the workflows?


Update to Rails 7

Trello
Update from Rails 6.1 to Rails 7 following the Rails guide

Changes made from updating rails in the gemfile and running bin/rails app:update command

Add net-http package
This removes deprecation warnings caused by double loading of the net-protocol library

Run in zeitwerk mode
Zeitwerks auto-loading of paths expects explicit namespaces, move these two files into an Exceptions folder to comply with zeitwerk's conventions

Enable ActiveSupport::Cache serialization format

Acceptance tests: CircleCI

@njseeto good stuff. a few comments.
do we test for the two token errors anywhere?


Configure/disable all keyboard shortcut

I want to use the tree using custom keyboard shortcut as it collide with other parts of my app,
Is there a way to do it?
Disabling all keyboard shortcut is also acceptable as I control the data and can drive the needed event using tree api.
Thanks

There is no way right now other than intercepting the keyboard events on the nodes and calling e.stopPropagation. There are plans to allow custom keyboard navigation settings.

I though about supplying my own RenderContainer, but the current implementation uses part of the package that  are not exported.
Is there a sample on how to supply my own container?

No, that DefaultContainer is in the docs but it's not really possible to provide your own because it uses all those external and internal libs.
But if you provide a custom RowRenderer, you can intercept the onMouseDown event and call stopPropagation on it. Then the event will not bubble up to the default container handler.

I will close this, but configurable keyboard shortcuts are being tracked here #57


Changed issue template structure

This PR fixes #146
Description
Made changes to template structure
Screenshot
N/A
Checks

[x]  I ensured that my changes are working fine locally
[x] I signed my commits (Optional) (Read about Signing commits)

Good job. Contribute more 🎉

Good job. Contribute more 🎉

review more 🤣


Add presto jdbc dependency for reportal module

Presto jdbc is required by presto reportal jobtype.

Pull Request Test Coverage Report for Build 4064

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.09%) to 36.083%

Totals

Change from base Build 4061:
0.09%

Covered Lines:
10507

Relevant Lines:
29119

💛  - Coveralls


Failing to Serialize Custom Map

Here's a minimum reproduction of a failing test I have. Other similar tests without maps pass, and I can't figure out what's wrong.
failing_test.dart:
import 'package:flutter_test/flutter_test.dart';

void main() {
  group("Failing Group", () {
    test("Serialization Test Reproduction", () {
      Failing failing = Failing.random;
      expect(failing, FailingMapper.fromJson(failing.toJson()));
    });
  });
}

failing.dart:
import 'package:chness/math/map_case/failing_key.dart';
import 'package:chness/math/map_case/failing_value.dart';
import 'package:dart_mappable/dart_mappable.dart';

part 'failing.mapper.dart';

@MappableClass()
class Failing with FailingMappable {
  final Map<FailingKey, FailingValue> map;

  Failing({required this.map});

  static Failing get random => Failing(
        map: {
          for (int i = 0; i < 10; i++) FailingKey.random: FailingValue.random,
        },
      );
}

failing_key.dart:
import 'dart:math';

import 'package:dart_mappable/dart_mappable.dart';

part 'failing_key.mapper.dart';

@MappableClass()
class FailingKey with FailingKeyMappable {
  final int a, b, c;
  FailingKey({required this.a, required this.b, required this.c});

  static int get randint => Random().nextInt(99) + 1;
  static FailingKey get random =>
      FailingKey(a: randint, b: randint, c: randint);
}

failing_value.dart:
import 'dart:math';

import 'package:dart_mappable/dart_mappable.dart';

part 'failing_value.mapper.dart';

@MappableClass()
class FailingValue with FailingValueMappable {
  final int a, b, c;
  FailingValue({required this.a, required this.b, required this.c});

  static int get randint => Random().nextInt(99) + 1;
  static FailingValue get random =>
      FailingValue(a: randint, b: randint, c: randint);
}

The test gives the following output:
Converting object to an encodable object failed: _Map len:10
dart:convert                                                   jsonEncode
package:dart_mappable/src/mappers/interface_mapper.dart 220:7  InterfaceMapperBase.encodeJson
package:chness/math/map_case/failing.mapper.dart 53:10         FailingMappable.toJson
test/failing_test.dart 8:54                                    main.<fn>.<fn>

Output of flutter doctor -v:
[✓] Flutter (Channel stable, 3.24.0, on macOS 14.5 23F79 darwin-arm64, locale en-US)
    • Flutter version 3.24.0 on channel stable at /opt/homebrew/Caskroom/flutter/3.22.1/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 80c2e84975 (13 days ago), 2024-07-30 23:06:49 +0700
    • Engine revision b8800d88be
    • Dart version 3.5.0
    • DevTools version 2.37.2

[!] Android toolchain - develop for Android devices (Android SDK version 33.0.2)
    • Android SDK at /Users/aweaver/Library/Android/sdk
    ✗ cmdline-tools component is missing
      Run `path/to/sdkmanager --install "cmdline-tools;latest"`
      See https://developer.android.com/studio/command-line for more details.
    ✗ Android license status unknown.
      Run `flutter doctor --android-licenses` to accept the SDK licenses.
      See https://flutter.dev/to/macos-android-setup for more details.

[✓] Xcode - develop for iOS and macOS (Xcode 15.4)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 15F31d
    • CocoaPods version 1.15.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2022.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)

[✓] VS Code (version 1.91.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.94.0

[✓] Connected device (5 available)
    [ redacted ]

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 1 category.

Also. Sorry for naming everything failing, it's unnecessarily harsh and I didn't mean it to be oops. This is probably some silly thing I'm missing anyway.

Update: I can work around this by using from/toMap instead of from/toJson, but it would definitely still be preferable to use the json methods if possible, and I still have no idea what's wrong with those.

 This isn't really anything to do with dart_mappable, just that the json format does not allow anything else than Strings as keys of a map.
As you discovered, to/fromMap work fine since this is just dealing with Dart Maps. But serializing this to json won't work.


Support for VSCodium

Any chance to add a configuration option to support VSCodium: 100% Open Source Version of Microsoft VS Code?
The cmd line shortcut for for VSCode is code.   It's codium for VSCodium

0.0.7 has the fix released, thanks!

Thanks @rebornix !   Just tested and it works.
FWIW, the "Always in new window" setting is not obeyed when the URL Scheme is set to VS Codium.
Its not a problem for me, but if you need a tester I'd be happy to try again with a new version.


Fixing purview test issues and improve performance

Currently there are two issues:

https://github.com/linkedin/feathr/issues/349 where connecting to Purview always yield this error:

E           requests.exceptions.ConnectionError: HTTPSConnectionPool(host='some-purview-name.catalog.purview.azure.com', port=443): Max retries exceeded with url: /api/atlas/v2/types/typedefs (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f90c3e7ea60>: Failed to establish a new connection: [Errno 111] Connection refused'))

Listing entities in purview is extremely slow.

This is caused by the fact that we are using the search_entities API in pyapacheatlas package. This API will try to get ALL the entities and return it back in a huge array, with paging internally implemented. This is:

not useful in most of the Feathr use cases
might be throttled backend,
will be slow (see implementation details here: https://github.com/wjohnson/pyapacheatlas/blob/master/pyapacheatlas/core/discovery/purview.py#L226)

In this case, instead of using search_entities API, we should use the query API which is just a single call to the backend. Note that this API might need the caller to page correctly, but usually we don't need that.

@YihuiGuo this should also fix your earlier issue

Could you please add more tech details and investigations about the bug? I'm not sure what happened actually.
To confirm that the server side throttling is the root cause, you may need try to change server side parameters to relax the restriction and check if it fixes the problem, otherwise the root cause could be somewhere else.

Could you please add more tech details and investigations about the bug? I'm not sure what happened actually. To confirm that the server side throttling is the root cause, you may need try to change server side parameters to relax the restriction and check if it fixes the problem, otherwise the root cause could be somewhere else.

As described in the PR, the implementation of the search_entities API in the underlying pyapacheatlas package is here (https://github.com/wjohnson/pyapacheatlas/blob/master/pyapacheatlas/core/discovery/purview.py#L226) where it's using a while True statement to keep querying the query API, and might got throttled.
More details are available in this issue: https://github.com/wjohnson/pyapacheatlas/issues/206 where I talked with Will offline, and he agrees to add some backoff in the search_entities API.

Also update the PR description to make it a bit more descriptive.

The canonical way to retrieve entity/entities from PurView is to fetch by GUID, also we already stored related GUIDs in our data model.
Querying/Searching is not the correct approach to load a project and may cause unwanted result if this PurView contains data other than what we've created by Feathr, it only fits for-eyes-only scenarios but not cases require accuracy and consistency such as Feathr client.
Please re-think and re-design the whole solution.

The canonical way to retrieve entity/entities from PurView is to fetch by GUID, also we already stored related GUIDs in our data model.

I agree, when CLI save data to purview, it does not specify any hits to partition data by project, that means search via startsWith might still experience perf issues when data volume grows. Since CLI already writes lineage relationship, for example, project contains feature/anchor/derived, using guid list to fetch registered features sounds more efficient and scalable.

The canonical way to retrieve entity/entities from PurView is to fetch by GUID, also we already stored related GUIDs in our data model. Querying/Searching is not the correct approach to load a project and may cause unwanted result if this PurView contains data other than what we've created by Feathr, it only fits for-eyes-only scenarios but not cases require accuracy and consistency such as Feathr client. Please re-think and re-design the whole solution.

Agree, but the goal of this PR is not to solve all those issues. I have a separate PR solving those issues and please take a look: https://github.com/linkedin/feathr/pull/368


online code editor is now functioning properly

the online code editor only works few times. after code is once executed i cannot write or delete or modify the editor. pressing backspace key actually works as "Go back one page"

This sounds like it may have been due to #17, which is fixed. Were you using FireFox?
If you're still seeing this issue, please reopen and let us know what browser you're using.


Implement smoothing for fonts

Description
Following on from forum discussion here this implements smoothing for fonts, via setSmooth();
Tasks

[x] Tested on Linux
[x] Tested on Windows
[x] Tested on macOS
[ ] Tested on iOS
[ ] Tested on Android

How to test this PR?
Can test using the following code. Space toggles the font's smoothing

#include <SFML/Graphics.hpp>
int main()
{

    sf::RenderWindow window( sf::VideoMode( 800, 600, 32 ), "SFML text smoothing",
                             sf::Style::Titlebar | sf::Style::Close );

    sf::Font font;
    if (!font.loadFromFile("resources/font.ttf"))
        return EXIT_FAILURE;

    sf::Text message;
    message.setFont(font);
    int textSize = 40;
    message.setScale( { 1.5f,1.5f } );
    message.setCharacterSize(textSize);
    message.setFillColor(sf::Color::White);

    message.setString( "Some text about something or other" );

    while (window.isOpen())
    {
        // Handle events
        sf::Event event;
        while ( window.pollEvent( event ) )
        {
            // Window closed or escape key pressed: exit
            if ( (event.type == sf::Event::Closed) ||
                 ((event.type == sf::Event::KeyPressed) && (event.key.code == sf::Keyboard::Escape)) )
            {
                window.close();
                break;
            }

            if ( event.type == sf::Event::KeyPressed )
            {
                switch ( event.key.code )
                {
                case sf::Keyboard::Space:
                    font.setSmooth( !font.isSmooth() );
                    break;

                case sf::Keyboard::Up:
                    message.setCharacterSize( ++textSize );
                    break;

                case sf::Keyboard::Down:
                    message.setCharacterSize( --textSize );
                    break;
                }
            }
        }

        window.clear(sf::Color(50, 200, 50));
        window.draw(message);
        window.display();
    }

    return EXIT_SUCCESS;
}

Yeah he also clarified afterwards that he misunderstood and to ignore that comment, unless I misunderstand? the smoothing is not an issue when rendering glyphs, but when rendering texts

Feedback addressed and rebased onto master

SFMLBuildMaster: Build this please.


[Backport #277620] nixos/pam: Add option for ssh-agent auth's trusted authorized_keys files

Backport of #277620
Description of changes

Rename option pam.enableSSHAgentAuth to pam.sshAgentAuth.enable
Add option pam.sshAgentAuth.authorizedKeysFiles, so system administrators can specify a set of trusted authorized_keys files.
The default remains as-is (services.openssh.authorizedKeysFiles) which is insecure; see #31611
Add a warning for insecure pam.sshAgentAuth settings;
in particular, the warning will trigger when sshAgentAuth.enable is set but authorizedKeysFiles isn't.

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
no relevant NixOS test available in 23.11

23.11 release notes

[x] (Module updates) Added a release notes entry if the change is significant

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

As NixOS 23.11 reaches end-of-life today, this PR is closed as part of the EOL cleanup.


Fix statistics of DEM test after update of default

Codecov Report
Base: 84.83% // Head: 84.79% // Decreases project coverage by -0.04% :warning:

Coverage data is based on head (bd266a2) compared to base (c5837c0).
Patch has no changes to coverable lines.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #166      +/-   ##
===========================================
- Coverage    84.83%   84.79%   -0.05%
===========================================
  Files           50       50
  Lines         4175     4175
===========================================
- Hits          3542     3540       -2
- Misses         633      635       +2

Impacted Files
Coverage Δ

eogrow/core/logging.py
97.31% <0.00%> (-1.35%)
:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Add file/tree message for language server

Pull Request Description

closes #538
PR adds file/tree command that returns a tree representation of a directory. The algorithm follows symlinks and is able to detect symlink loops.

add: file/tree message that returns a DirectoryTree for the specified file system object
add: DirectoryTruncated to FileSystemObject indicating that directory contents was truncated
aad: SymlinkLoop to FilesSystemObject indicating that symbolic link creates a loop
~add: SystemPath = RelativePath(rootId, segments) | AbsolutePath(segments)~
~refactor: rename Path to RelativePath~

Important Notes
Interface changed. FileSystemObject ADT extended with DirectoryTruncated and SymlinkLoop:
type FileSystemObject
  = Directory
  | DirectoryTruncated
  | SymlinkLoop
  | File
  | Other;

/**
 * Represents a directory which contents have been truncated.
 *
 * @param name a name of the directory
 * @param path a path to the directory
 */
interface DirectoryTruncated {
  name: String;
  path: Path;
}

/**
 * Represents a symbolic link that creates a loop.
 *
 * @param name a name of the symlink
 * @param path a path to the symlink
 */
interface SymlinkLoop {
  name: String;
  path: Path;
}

Checklist
Please include the following checklist in your PR:

[x] The documentation has been updated if necessary.
[x] All code conforms to the Scala, Java, Rust or Haskell style guides as appropriate.
[x] All code has been tested where possible.

I have finished implementing the requested changes.
ChangeLog:

update: tree algorithm follows simliks and detects loops
update: FileSystemObject is extended with DirectoryTruncated and SymlinkLoop
refactor: remove Symlink
refactor: remove AbsolutePath
refactor: rename RelativePath back to Path

@lolczak PR is ready for review :ok_hand:

ChangeLog:

add: NotDirectory error indicating that requested path is not a directory
add: target to SymlinkLoop as discussed with Marcin  /**
  * Represents a symbolic link that creates a loop.
  *
  * @param name a name of the symlink
  * @param path a path to the symlink
  * @param target a target of the symlink. Since it is a loop,
  * target is a subpath of the symlink
  */
case class SymlinkLoop(name: String, path: Path, target: Path)

refactor: tree implementation using the list function
refactor: DirectoryTree conversion helper functions into an appropriate Path and FileSystemObject modules
refactor: replace TreeSet with Vector to remove the Ordering abomination

@lolczak ready for review :ok_hand:


[SMALLFIX] supply variable name for WorkerInfo.java (alluxio.wire)

supply variable name for WorkerInfo.java (alluxio.wire)

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/16272/Failed Tests: 1org.alluxio:alluxio-tests: 1alluxio.master.file.ConcurrentRenameIntegrationTest.sharedPrefixDirConcurrentRenameNonPersisted
Test FAILed.

jenkins, test this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/16278/
Test PASSed.


ESPN: The Last Dance not working

[x] I'm reporting a broken site support
[x] I've verified that I'm running youtube-dl version 2020.05.08
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar issues including closed ones

Verbose log
$ youtube-dl --verbose 'https://www.espn.com/watch/player/_/id/6fbad816-31dd-47a6-ad2e-776e96eaa6c3/bucketId/33086'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--verbose', u'https://www.espn.com/watch/player/_/id/6fbad816-31dd-47a6-ad2e-776e96eaa6c3/bucketId/33086']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2020.05.08
[debug] Python version 2.7.17 (CPython) - Darwin-18.7.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.2.2, ffprobe 4.2.2, rtmpdump 2.4
[debug] Proxy map: {}
[ESPN] 6: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 400: Bad Request (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 627, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2238, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File<EMAIL_ADDRESS>line 435, in open
    response = meth(req, response)
  File<EMAIL_ADDRESS>line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File<EMAIL_ADDRESS>line 473, in error
    return self._call_chain(*args)
  File<EMAIL_ADDRESS>line 407, in _call_chain
    result = func(*args)
  File<EMAIL_ADDRESS>
Description
Downloading The Last Dance from ESPN fails. Per another request, copying out the content URL and using that also fails (see below).
$ youtube-dl 'https://content.uplynk.com/ext/d09b16c953aa40c98dd8c513526aca5a/a113018464.m3u8?oid=d09b16c953aa40c98dd8c513526aca5a&amp'
[uplynk] a113018464: Downloading m3u8 information
[uplynk] a113018464: Downloading JSON metadata
[hlsnative] Downloading m3u8 manifest
WARNING: hlsnative has detected features it does not support, extraction will be delegated to ffmpeg
[download] Destination: The Last Dance - Episode One-84a8e5ca2b774ebb854c88ce98e1317e.mp4
ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with Apple clang version 11.0.0 (clang-1<IP_ADDRESS>)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.2.2_3 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags=-fno-stack-check --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libsoxr --enable-videotoolbox --disable-libjack --disable-indev=jack
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[hls @ 0x7ff932000000] Skip ('#EXT-X-VERSION:5')
[hls @ 0x7ff932000000] Skip ('#UPLYNK-SEGMENT: 84a8e5ca2b774ebb854c88ce98e1317e,00000000,segment')
[hls @ 0x7ff932000000] Skip ('#UPLYNK-KEY:https://content-ausc4.uplynk.com/check?b=84a8e5ca2b774ebb854c88ce98e1317e&v=84a8e5ca2b774ebb854c88ce98e1317e&pbs=')
[hls @ 0x7ff932000000] Skip ('#EXT-X-PROGRAM-DATE-TIME:2020-05-17T14:54:35.718000+00:00')
[hls @ 0x7ff932000000] Skip ('#EXT-X-DISCONTINUITY')
[hls @ 0x7ff932000000] Skip ('#EXT-X-PROGRAM-DATE-TIME:2020-05-17T15:12:06.067312+00:00')
[hls @ 0x7ff932000000] Skip ('#EXT-X-DISCONTINUITY')
[hls @ 0x7ff932000000] Skip ('#EXT-X-PROGRAM-DATE-TIME:2020-05-17T15:24:03.717583+00:00')
[hls @ 0x7ff932000000] Skip ('#EXT-X-DISCONTINUITY')
[hls @ 0x7ff932000000] Skip ('#EXT-X-PROGRAM-DATE-TIME:2020-05-17T15:35:59.432583+00:00')
[hls @ 0x7ff932000000] Opening 'https://content-ausc4.uplynk.com/check2?b=84a8e5ca2b774ebb854c88ce98e1317e&v=84a8e5ca2b774ebb854c88ce98e1317e&r=f&pbs=' for reading
[https @ 0x7ff931e28840] HTTP error 403 Forbidden
[hls @ 0x7ff932000000] Unable to open key file https://content-ausc4.uplynk.com/check2?b=84a8e5ca2b774ebb854c88ce98e1317e&v=84a8e5ca2b774ebb854c88ce98e1317e&r=f&pbs=
[hls @ 0x7ff932000000] Opening 'crypto+http://x-default-stgec.uplynk.com/ause/slices/84a/d09b16c953aa40c98dd8c513526aca5a/84a8e5ca2b774ebb854c88ce98e1317e/F00000000.ts?x=0&si=0' for reading
[hls @ 0x7ff932000000] Error when loading first segment 'http://x-default-stgec.uplynk.com/ause/slices/84a/d09b16c953aa40c98dd8c513526aca5a/84a8e5ca2b774ebb854c88ce98e1317e/F00000000.ts?x=0&si=0'
http://content-ausc4.uplynk.com/84a8e5ca2b774ebb854c88ce98e1317e/f.m3u8: Invalid data found when processing input

ERROR: ffmpeg exited with code 1

Did something change with ESPN in mid May? Seems like something changed as ESPN download attempts have since failed with:
ERROR: Unable to download JSON metadata: HTTP Error 400: Bad Request
In case it helps, I use Disney+ to log into ESPN, so I don't authenticate separately with a TV service. I still get that same error.

Did something change with ESPN in mid May? Seems like something changed as ESPN download attempts have since failed with:
ERROR: Unable to download JSON metadata: HTTP Error 400: Bad Request
In case it helps, I use Disney+ to log into ESPN, so I don't authenticate separately with a TV service. I still get that same error.


docs(admin) add docs for vault_beta entity

Summary
Adds "autogen" docs to vault_beta entity.

see suggestions

Thanks, committed them!


ci: fix potential setuptools_scm issues in gh actions

In reference to https://github.com/PyPSA/PyPSA/pull/907 #279 and #292 I just add a fetch for the tags here so that setuptools_scm can detect the version correctly. Right now that doesn't lead to errors, so only to avoid the potential bug search in the future (which I just had).
Fore more details see here.

flake 8 does not like some syntax here https://github.com/PyPSA/linopy/actions/runs/9273219169/job/25512792210?pr=298 (?)

don't know what is going on with codecov, I am closing and opening the pr in the hope that codecov with work then


Question:  Delete vertex syntax in SQL strings

OrientDB Version, operating system, or hardware.

v2.0 SNAPSHOT[ ] - .18[ ] .17[ ] .16[ ] .15[ ] .14[ ] .13[ ] .12[ ] .11[ ] .10[ ] .9[ ] .8[ ] .7[ ] .6[ ] .5[ ] .4[ ] .3[ ] .2[ ] .1[ ] .0[ ]
v2.1 SNAPSHOT[ ] - .16[ ] .15[ ] .14[ ] .13[ ] .12[ ] .11[ ] .10[ ] .9[ ] .8[ ] .7[ ] .6[ ] .5[ ] .4[ ] .3[ ] .2[ ] .1[ ] .0[ ]
v2.2 SNAPSHOT[x] - .rc1[ ] .beta2[ ] .beta1[ ]

Operating System

[x] Linux
[ ] MacOSX
[ ] Windows
[ ] Other Unix
[ ] Other, name?

Expected behavior and actual behavior
Use delete sql syntax for classes that don't extend V or do extend V
Steps to reproduce the problem
Create a class that doesn't extend V and one that does.  Delete vertex syntax is required for classes that extend V.  What is the benefit of this requirement?  In Orient Spring Data, the deleteByXYZ methods will fail unless annotated when deleting vertices due to the syntax differences. Not sure why it shouldn't be legal to use regular delete on a class that happens to extend V.

Hi @RAndrews137
This is mainly for historical reasons. The DELETE VERTEX is implemented in the graph module, that has all the components to manage (and cascade delete) connected edges, while simple DELETE is implemented in the core module, that does not know anything about edges, so it would leave dangling pointers (ie. inconsistent graph).
For sure, DELETE VERTEX could be safely used to delete a non-vertex element without any problems.
In V. 3.0 we are trying to unify the core APIs to also include graph components, so probably we will also review the DELETE commands.
Thanks
Luigi

Thanks for the quick response @luigidellaquila.   That makes sense.  Looking forward to 3.x  :)


"Reload to deactivate" after uninstall message is confusing

Issue Type: Bug
When i click "uninstall" of an extension in the extension's editor, the button changes to "Reload to Deactivate".

I understand that it is the inverse of "Reload to Activate" after clicking on "Install", which makes sense.  But, in the case of uninstall, we are not "deactivating" the extension on reload, we are removing it.  "Deactivate" sounds like the extension will not be removed, but instead just disabled.
I suggest something like this (there is plenty of room):

or this is even more accurate I believe.

I am leaning towards Reload to finish uninstall instead of Reload to remove the extension.
User has already clicked on Uninstall to "remove the extension"...

I think the extension is already is installed/uninstalled and Reload to finish gives an expression that it is still not installed / uninstalled. Actually we need to Reload to load or unload the extension in the window. It would be nice if the wordings are along these lines.

Ok.

@ramya-rao-a Good point. You can always check if an extension is disabled or not right? A newly installed extension is enabled but not loaded.
I was also thinking about enable/disable scenario and what would be the message in this case?

Good point. You can always check if an extension is disabled or not right? A newly installed extension is enabled but not loaded.

@sandy081 Turns out, by the time extensionWorkbenchService fires the onChange event when a disabled extension is enabled, the enablementService marks it as "enabled". So the below if condition (which checks for !isDisabled) evaluates to true for both "enabling" and "installing" an extension.
https://github.com/Microsoft/vscode/blob/125260e63f179a87cfa1f8228cff9d6afc435c99/src/vs/workbench/parts/extensions/electron-browser/extensionsActions.ts#L1080-L1084

I was also thinking about enable/disable scenario and what would be the message in this case?

I was thinking of the below:

Reload to complete installing the extension.
Reload to complete uninstalling the extension.
Reload to complete updating the extension.
Reload to complete enabling the extension.
Reload to complete disabling the extension.

May be you need to compute both disabled and enabled and an installed extension has false for both

installed extension has false for both

Wow, I didnt know that.

@NavyAdmiral The PR for the changes is already linked to this issue.

@NavyAdmiral sqlite migrations will not help us completely in avoiding reload. It just removes one obstacle away and there is more needed for that.

@ramya-rao-a Changes are good, but not sure if long labels look good in the UI.

Sorry if I forgot, what is the reason to have verbose labels instead of tool tips?

@sandy081 it originally came from #56028 that I filed about users not understanding that reloading is required to load an extension
While I like how verbose the text is, the text is quite long. Maybe @gregvanl has any ideas on how to make it shorter but yet succinct?

We can skip the this extension. in the end?

Also removing complete? Also since enabled extensions are those which are currently running... How about following?
Reload to Enable
Reload to Disable
Reload to Uninstall

That would result in Enable button turning to Reload to Enable. Wouldnt that be confusing/annoying? The user just clicked a button to enable the extension, and now we ask them to reload to do the same.
How about going back to simple "Reload"? A simple text below the buttons can have the full text of Please reload Visual Studio Code to complete the installing of this extension

I like it.

I'll move this to November then

I'm sorry if I missed some part of the conversation, but why exactly is the "to activate" being removed from the reload button? I think that "reload to activate" is more informative than just "reload". I remember the first time I installed an extension and I didn't know what "reload" meant nor that I needed to do it in order to activate the extension.

@luabud Its not just about activate, there are other scenarios like uninstall where we cannot have small and proper label. To be consistent we have Reload label and description showing why it has to be reloaded.


[Feature] .SayIn Command

Add a .sayin CHANNEL MESSAGE command to tell the bot to say a message in a specific channel.

Added in v1.4.0


Deploy to Tutum broken

Clicking deploy to Tutum and trying to deploy it leads to Tutum giving an error saying "Service 'db': 'environment' must be a 'list' or a 'dict' object. See 'https://support.tutum.co/support/solutions/articles/5000583471' for more details"

Nevermind, this was an error on my part.


ARROW-2784: [C++] MemoryMappedFile::WriteAt allow writing past the end

I also added a pure WriteAt test, since there wasn't one.

Codecov Report

Merging #2206 into master will decrease coverage by 0.09%.
The diff coverage is 100%.

@@           Coverage Diff            @@
##           master   #2206     +/-   ##
========================================
- Coverage    84.2%   84.1%   -0.1%
========================================
  Files         288     288
  Lines       43749   43784     +35
========================================
- Hits        36837   36826     -11
- Misses       6883    6927     +44
- Partials       29      31      +2

Impacted Files
Coverage Δ

cpp/src/arrow/io/file.cc
96.07% <100%> (+2.4%)
:arrow_up:

cpp/src/arrow/io/io-file-test.cc
93.19% <100%> (+0.51%)
:arrow_up:

go/arrow/math/float64_avx2_amd64.go
0% <0%> (-100%)
:arrow_down:

go/arrow/memory/memory_avx2_amd64.go
0% <0%> (-100%)
:arrow_down:

go/arrow/math/int64_avx2_amd64.go
0% <0%> (-100%)
:arrow_down:

go/arrow/math/uint64_avx2_amd64.go
0% <0%> (-100%)
:arrow_down:

go/arrow/math/int64_amd64.go
33.33% <0%> (-33.34%)
:arrow_down:

go/arrow/math/uint64_amd64.go
33.33% <0%> (-33.34%)
:arrow_down:

go/arrow/math/float64_amd64.go
33.33% <0%> (-33.34%)
:arrow_down:

go/arrow/math/math_amd64.go
31.57% <0%> (-31.58%)
:arrow_down:

... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 782c5a5...a8987bb. Read the comment docs.


New version: meshmakers.octo-cli version <IP_ADDRESS>

[ ] Have you signed the Contributor License Agreement?
[ ] Have you checked that there aren't other open pull requests for the same manifest update/change?
[ ] This PR only modifies one (1) manifest
[ ] Have you validated your manifest locally with winget validate --manifest <path>?
[ ] Have you tested your manifest locally with winget install --manifest <path>?
[ ] Does your manifest conform to the 1.6 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

Automatic Validation ended with:

No errors to post.

(Automated response - build 729.)

Automatic Validation ended with:

No errors to post.

(Automated response - build 729.)


perlPackages.{CryptCurve25519,MathGMP}: fixed build failures

ZHF: #80379
https://hydra.nixos.org/build/112817446
https://hydra.nixos.org/build/112813918
https://hydra.nixos.org/build/112814931
https://hydra.nixos.org/build/112833536
https://hydra.nixos.org/build/112804942
https://hydra.nixos.org/build/112809869
perlPackages.CryptCurve25519: apply patch from gentoo to fix fmul conflicting
types build breakage.
https://gitweb.gentoo.org/repo/gentoo.git/commit/dev-perl/Crypt-Curve25519?id=e07299f804a8376bb5bf85d28916e0a360199f3e
perlPackages.MathGMP: 2.19 -> 2.20
Updated to latest upstream, which passes tests. This is a dependency of
perlPackages.NetSSH
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[X] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

@GrahamcOfBorg build perl528Packages.CryptCurve25519 perl528Packages.MathGMP perl528Packages.NetSSHPerl
@GrahamcOfBorg build perl530Packages.CryptCurve25519 perl530Packages.MathGMP perl530Packages.NetSSHPerl


Support basic ECS via docker

After adding lambda support via docker, I think it makes sense to start mocking out some simple use cases of ECS via docker.  Right now in my test-case I want to be able to start/stop some simple containers.
If I have some time I'll start working on this.

talking with a co-worker it would then not really be a mock.  What's interesting is that the lambda treads water of being a "mock" or not...hrm, going to close for now but please pipe in your thoughts


Warn before overwriting existing genesis block

Add warning to approve overwriting of the genesis block when it is
already existing.
Signed-off-by: ZeekHuge<EMAIL_ADDRESS>

Description of the Change
Ask for conformation to overwrite the genesis block if already existing, instead of silently overwrting it.
Changes in irohad/main/application.hpp adds string constants YES and SHORT_YES to be used to provide choice in the above mentioned conformation, which is added in the irohad/main/irohad.cpp

Benefits
Prevents un-intentional removal of existing transaction data.
This pull request fixes #1231

Possible Drawbacks
None

@lebdron : I have made the required changes (addition of --overwrite_genesis flag) in my local repo. Should I also add this flag in the docker entrypoint scripts like here ?

@ZeekHuge I don't think that default command should overwrite existing ledger, so no need to change the scripts. You can also modify this branch, so that changes will appear in this pull request.

The docker image will always run in non-interactive mode. right ? In that case, shouldnt the --overwrite_genesis flag added in the docker image ? It probably wont let the daemon start otherwise (as it will be stuck at the confirmation). Am I missing something ?
Also, I have added more commits to the pull-request. Please review them.

@ZeekHuge any updates on the pull request?

Hi ! Got busy at work. Will update it tonight.

Sorry for the delay. Had updated the pull request last night. Please review it.

I am not sure if this is the right way to do it ..
But as discussed here, I rebased this fix branch over develop and then force pushed it.
Please review.


Mirrored request with host rewrite has wrong Host header

Description:
I'm currently trying to use request_mirror_policy in a route that is also using auto_host_rewrite.
The problem are the host/:authority headers in the mirrored request is giving me weird results...
Repro steps:
I created a simplified version of my setup in this gist including a full log of execution
docker-compose up --build

curl localhost:5000/any/host_rewrite

Mock server logs 2 requests
# The actual request
mockserver_1    | 	{
mockserver_1    | 	  "method" : "GET",
mockserver_1    | 	  "path" : "/any/host_rewrite",
mockserver_1    | 	  "headers" : [ {
mockserver_1    | 	    "name" : "host",
mockserver_1    | 	    "values" : [ "api" ] # successfully rewritten
mockserver_1    | 	  }, {
....

# The mirror request
mockserver_1    | 	{
mockserver_1    | 	  "method" : "GET",
mockserver_1    | 	  "path" : "/any/host_rewrite",
mockserver_1    | 	  "headers" : [ {
mockserver_1    | 	    "name" : "host",
mockserver_1    | 	    "values" : [ "localhost-shadow:5000" ] # Why localhost?
mockserver_1    | 	  }, {
....

Calling again...
curl localhost:5000/any/host_rewrite

Mock server logs 2 requests
# The actual request
mockserver_1    | 	{
mockserver_1    | 	  "method" : "GET",
mockserver_1    | 	  "path" : "/any/host_rewrite",
mockserver_1    | 	  "headers" : [ {
mockserver_1    | 	    "name" : "host",
mockserver_1    | 	    "values" : [ "api" ] # successfully rewritten
mockserver_1    | 	  }, {

# The mirrored request
mockserver_1    | 	{
mockserver_1    | 	  "method" : "GET",
mockserver_1    | 	  "path" : "/any/host_rewrite",
mockserver_1    | 	  "headers" : [ {
mockserver_1    | 	    "name" : "host",
mockserver_1    | 	    "values" : [ "api-shadow" ] # successfully rewritten

From now on it's always successfully rewritten, until I restart the containers and the same thing happen. Maybe this is some caching problem in the host resolution?
Tested versions
envoyproxy/envoy:v1.10.0
envoyproxy/envoy-dev:daeb9850dbcd78da21a2e8ff9854564830cefb19 # 1.13.0-dev
Additional question
As you can see in my envoy.yaml I want to mirror the request to another cluster with another domain (I'm migrating one feature from a monolith to a microservice).
So in my case the real request is going to api, and the mirror request is going to another cluster called api-mirror. In this case, shouldn't the request mirror get the host=api-mirror-shadow?
The documentation is not really clear (at least to me) which cluster is going to be used when rewriting, the mirror cluster or the original one?
Thanks for the help!

cc. @derekargueta who recently touched codes related to this.

👀
my pr hasn't landed yet but I can help investigate

So there's 2 main observations I have so far.

shouldn't the request mirror get the host=api-mirror-shadow?

Not quite, because the way that shadowing works is that it takes an exact clone of the original request, and just adds -shadow as a suffix to the host, and doesn't do its own host resolution for the api-mirror cluster (code here) regardless of auto_host_rewrite. We can definitely update the docs to clarify that. So what's happening here is the auto-rewrite kicks in to rewrite the host to api on the main request, then the ShadowWriter takes that host string as-is and just adds -shadow. The hostname of the mirroring cluster is not considered or looked up.

Maybe this is some caching problem in the host resolution?

My hypothesis so far:
From observing -l trace level logs and looking at /stats on the admin server, it seems that this only happens if you hit an Envoy worker thread doesn't already have a connection to upstream, and has to establish a new connection (each worker has its own connection pool). After hitting an established connection you don't have this problem.
A few important facts:

The mirrored request is queued immediately once the entire request is parsed. (link)
host resolution is an asynchronous event, requiring a callback to be triggered (link).
Envoy needs to resolve the host for every new connection is establishes upstream.

So I think what we are seeing here is:

Envoy receives a new request on a worker thread that doesn't have an upstream connection
Envoy initiates an upstream connection, and before the connection is established (which is prerequisite to getting the hostname in a callback), it also queues the mirrored request in the cluster manager with a copy of whatever it has at that time - in this case localhost:5000 since the connection callback hasn't been triggered. (attempts to call doRequestComplete() after parsing headers/body/trailers, which in turn is what executes the mirrored request).
Connection callback is triggered, which sets the host rewrite (code link) so the main request has the right host, but there's no bridge to the mirrored request - it's already been queued elsewhere in the system, so when it gets sent it only has the original localhost:5000 to work with.

Then on subsequent calls to the same upstream connection the host is immediately resolved because the connection is already open, so the host rewrite is set before the mirrored request is queued in the cluster manager

Still gathering supporting evidence for the above, but if true means that either 1) we document this as expected behavior or 2) fix it so that the mirrored request also waits for onPoolReady (and not just "once the request is fully read) so that they have the expected host when combined with auto_host_rewrite on the first request of a new connection.

Thank you for the quick and detailed answer!
Regarding the cluster used in the host
I'll try to wrap my head around it and maybe open a SO/issue with some more questions if needed.
Regarding the mirror host resolution
Thanks for clarifying it, I think it's not going to be a huge problem for me, at least for now.
Let me know if there is something else I can do to help on this
Thanks!

Yeah @derekargueta your assessment is correct. We rewrite the host after we get an upstream host. Note though that we are not waiting for DNS resolution, just connection creation. DNS resolution is async. I think we could likely fix this by doing shadowing after pool complete like you said.

@derekargueta @mattklein123 - making the shadow request wait till the primary cluster's upstream host connection is ready might introduce unnecessary dependency and potentially unwanted latency on the shadow request. In https://github.com/envoyproxy/envoy/issues/9094 it was suggested to add disable_shadow_host_suffix_append under request_mirror_policies entries. How about we add auto_host_rewrite along with that? After all, it will be helpful to use the mirror cluster's hostname for the Host header in the shadow request instead of the primary cluster's hostname. What do you think?
If my understanding is correct, could you point me to the code that does "Connection callback is triggered, which sets the host rewrite". I think the code link in @derekargueta message above is now obsolete.  I can look into making these changes. Thanks.

If my understanding is correct, could you point me to the code that does "Connection callback is triggered, which sets the host rewrite". I think the code link in @derekargueta message above is now obsolete. I can look into making these changes. Thanks.

Sure the above makes sense to me. Here is the shadow writer code: https://github.com/envoyproxy/envoy/blob/main/source/common/router/shadow_writer_impl.h

Are there any updates?
Ignoring auto_host_rewrite parameter by mirror module seems to be a critical problem for cases when envoy sends duplicate request to somewhere with TLS, or for example to k8s. When host field in header becomes, in fact, invalid (add -shadow).
Adding auto_host_rewrite parameter for mirror sounds like a real solution to the problem, if the host field is overwritten correctly.

Pump! Are these any updates on this issue?

adding auto_host_rewrite sounds like a reasonable approach, if sounds good to @envoyproxy/api-shepherds I'll take a stab at this next week.


Reload in Androids Dev app not working for TypeScript project

Problem description
I'm serving a TypeScript app created by tabris init. Changes in the app.ts code are not recognized on reloads in the Android dev app. Same procedure for a JavaScript app works fine.
Expected behavior
Code changes take effect on reload in Android dev app for TypeScript projects
Environment

Tabris.js version: tabris-cli 2.7.0 in Termux
Device: Gemini PDA
OS: Android 7.1.1

Are you running tabris serve -w?
PS, 3.0.0 (in beta) has automatic reloading

The -w switch does help indeed! Why is this just for TypeScript apps?

It would work equally well if you're using vanilla JS, e.g. Babel or some other transpiler.

OK, thank you.


Transitmatters Data Dashboard

https://github.com/transitmatters/t-performance-dash
https://github.com/coding-to-music/t-performance-dash
https://dashboard.transitmatters.org/
https://api-v3.mbta.com/portal
https://www.mbta.com/developers/v3-api
https://api-v3.mbta.com/
https://api-v3.mbta.com/docs/swagger/index.html
https://github.com/mbta/api
https://mbtastringlines.com/
https://www.mbta.com/developers
http://mbtaviz.github.io/
https://mbtastringlines.com/

Need to set python version to 3.7 using pipenv
root@docker-ubuntu-s-1vcpu-2gb-nyc1-01:~/ap/t-performance-dash# npm start

><EMAIL_ADDRESS>start /root/ap/t-performance-dash
> concurrently npm:start-python npm:start-react

[start-python]
[start-python] ><EMAIL_ADDRESS>start-python /root/ap/t-performance-dash
[start-python] > cd server && pipenv run chalice local --port=5000
[start-python]
[start-react]
[start-react] ><EMAIL_ADDRESS>start-react /root/ap/t-performance-dash
[start-react] > react-scripts start
[start-react]
[start-python] Warning: Python 3.7 was not found on your system...
[start-python] You can specify specific versions of Python with:
[start-python]   $ pipenv --python path/to/python
[start-python] npm ERR! code ELIFECYCLE
[start-python] npm ERR! errno 1
[start-python] npm ERR<EMAIL_ADDRESS>start-python: `cd server && pipenv run chalice local --port=5000`
[start-python] npm ERR! Exit status 1
[start-python] npm ERR!
[start-python] npm ERR! Failed at the<EMAIL_ADDRESS>start-python script.
[start-python] npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
[start-python]
[start-python] npm ERR! A complete log of this run can be found in:
[start-python] npm ERR!     /root/.npm/_logs/2021-07-07T05_20_17_813Z-debug.log
[start-python] npm run start-python exited with code 1
[start-react] ℹ ｢wds｣: Project is running at http://<IP_ADDRESS>:3000/
[start-react] ℹ ｢wds｣: webpack output is served from
[start-react] ℹ ｢wds｣: Content not from webpack is served from /root/ap/t-performance-dash/public
[start-react] ℹ ｢wds｣: 404s will fallback to /
[start-react] Starting the development server...
[start-react]
[start-react] Compiled successfully!
[start-react]
[start-react] You can now view t-dash in the browser.
[start-react]
[start-react]   http://localhost:3000
[start-react]
[start-react] Note that the development build is not optimized.
[start-react] To create a production build, use npm run build.
[start-react]
[start-react] Proxy error: Could not proxy request /git_id from localhost:3000 to http://localhost:5000.
[start-react] See https://nodejs.org/api/errors.html#errors_common_system_errors for more information (ECONNREFUSED).

How to Install Python 3.7 on Ubuntu
Start by updating the packages list and installing the prerequisites:
sudo apt update
sudo apt install software-properties-common

Next, add the deadsnakes PPA to your sources list:
sudo add-apt-repository ppa:deadsnakes/ppa

When prompted press Enter to continue:
Press [ENTER] to continue or Ctrl-c to cancel adding it.

Once the repository is enabled, install Python 3.7 with:
sudo apt install python3.7

At this point, Python 3.7 is installed on your Ubuntu system and ready to be used. You can verify it by typing:
python3.7 --version
Python 3.7.3

root@docker-ubuntu-s-1vcpu-2gb-nyc1-01:~/ap/t-performance-dash# npm start

><EMAIL_ADDRESS>start /root/ap/t-performance-dash
> concurrently npm:start-python npm:start-react

[start-python]
[start-python] ><EMAIL_ADDRESS>start-python /root/ap/t-performance-dash
[start-python] > cd server && pipenv run chalice local --port=5000
[start-python]
[start-react]
[start-react] ><EMAIL_ADDRESS>start-react /root/ap/t-performance-dash
[start-react] > react-scripts start
[start-react]
[start-python] Creating a virtualenv for this project...
[start-python] Using /usr/bin/python3.7m (3.7.11) to create virtualenv...
[start-python] created virtual environment CPython3.7.11.final.0-64 in 2184ms
[start-python]   creator CPython3Posix(dest=/root/.local/share/virtualenvs/server-ZG6U0lHf, clear=False, global=False)
[start-python]   seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, pkg_resources=latest, via=copy, app_data_dir=/root/.local/share/virtualenv/seed-app-data/v1.0.1.debian.1)
[start-python]   activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
[start-python]
[start-python] Virtualenv location: /root/.local/share/virtualenvs/server-ZG6U0lHf
[start-python] Error: the command chalice could not be found within PATH or Pipfile's [scripts].
[start-react] ℹ ｢wds｣: Project is running at http://<IP_ADDRESS>:3000/
[start-react] ℹ ｢wds｣: webpack output is served from
[start-react] ℹ ｢wds｣: Content not from webpack is served from /root/ap/t-performance-dash/public
[start-react] ℹ ｢wds｣: 404s will fallback to /
[start-react] Starting the development server...
[start-react]
[start-python] npm ERR! code ELIFECYCLE
[start-python] npm ERR! errno 1
[start-python] npm ERR<EMAIL_ADDRESS>start-python: `cd server && pipenv run chalice local --port=5000`
[start-python] npm ERR! Exit status 1
[start-python] npm ERR!
[start-python] npm ERR! Failed at the<EMAIL_ADDRESS>start-python script.
[start-python] npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
[start-python]
[start-python] npm ERR! A complete log of this run can be found in:
[start-python] npm ERR!     /root/.npm/_logs/2021-07-07T05_49_45_160Z-debug.log
[start-python] npm run start-python exited with code 1
[start-react] Compiled successfully!
[start-react]
[start-react] You can now view t-dash in the browser.
[start-react]
[start-react]   http://localhost:3000
[start-react]
[start-react] Note that the development build is not optimized.
[start-react] To create a production build, use npm run build.
[start-react]
[start-react] Proxy error: Could not proxy request /git_id from localhost:3000 to http://localhost:5000.
[start-react] See https://nodejs.org/api/errors.html#errors_common_system_errors for more information (ECONNREFUSED).

Install Chalice
https://pypi.org/project/chalice/
Chalice is a framework for writing serverless apps in python. It allows you to quickly create and deploy applications that use AWS Lambda. It provides:

A command line tool for creating, deploying, and managing your app
A decorator based API for integrating with Amazon API Gateway, Amazon S3, Amazon SNS, Amazon SQS, and other AWS services.
Automatic IAM policy generation
You can create Rest APIs:

pip install chalice

Getting more messages
root@docker-ubuntu-s-1vcpu-2gb-nyc1-01:~/ap/t-performance-dash# npm start

><EMAIL_ADDRESS>start /root/ap/t-performance-dash
> concurrently npm:start-python npm:start-react

[start-react]
[start-react] ><EMAIL_ADDRESS>start-react /root/ap/t-performance-dash
[start-react] > react-scripts start
[start-react]
[start-python]
[start-python] ><EMAIL_ADDRESS>start-python /root/ap/t-performance-dash
[start-python] > cd server && pipenv run chalice local --port=5000
[start-python]
[start-python] Exception in thread Thread-1:
[start-python] Traceback (most recent call last):
[start-python]   File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
[start-python]     self.run()
[start-python]   File "/usr/local/lib/python3.8/dist-packages/chalice/local.py", line 727, in run
[start-python]     self._server = self._server_factory()
[start-python]   File "/usr/local/lib/python3.8/dist-packages/chalice/cli/__init__.py", line 143, in create_local_server
[start-python]     app_obj = config.chalice_app
[start-python]   File "/usr/local/lib/python3.8/dist-packages/chalice/config.py", line 141, in chalice_app
[start-python]     app = v()
[start-python]   File "/usr/local/lib/python3.8/dist-packages/chalice/cli/factory.py", line 277, in load_chalice_app
[start-python]     app = importlib.import_module('app')
[start-python]   File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
[start-python]     return _bootstrap._gcd_import(name[level:], package, level)
[start-python]   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
[start-python]   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
[start-python]   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
[start-python]   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
[start-python]   File "<frozen importlib._bootstrap_external>", line 848, in exec_module
[start-python]   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
[start-python]   File "/root/ap/t-performance-dash/server/app.py", line 6, in <module>
[start-python]     from chalicelib import data_funcs, aggregation, s3_alerts, secrets
[start-python]   File "/root/ap/t-performance-dash/server/chalicelib/data_funcs.py", line 2, in <module>
[start-python]     import pytz
[start-python] ModuleNotFoundError: No module named 'pytz'
[start-react] ℹ ｢wds｣: Project is running at http://<IP_ADDRESS>:3000/
[start-react] ℹ ｢wds｣: webpack output is served from
[start-react] ℹ ｢wds｣: Content not from webpack is served from /root/ap/t-performance-dash/public
[start-react] ℹ ｢wds｣: 404s will fallback to /
[start-react] Starting the development server...
[start-react]
[start-react] Compiled successfully!
[start-react]
[start-react] You can now view t-dash in the browser.
[start-react]
[start-react]   http://localhost:3000
[start-react]
[start-react] Note that the development build is not optimized.
[start-react] To create a production build, use npm run build.
[start-react]
[start-react] Proxy error: Could not proxy request /git_id from localhost:3000 to http://localhost:5000.
[start-react] See https://nodejs.org/api/errors.html#errors_common_system_errors for more information (ECONNREFUSED).

Verify Chalice is set up correctly
root@docker-ubuntu-s-1vcpu-2gb-nyc1-01:~/ap/t-performance-dash# chalice --help
Usage: chalice [OPTIONS] COMMAND [ARGS]...

Options:
  --version             Show the version and exit.
  --project-dir TEXT    The project directory path (absolute or
                        relative).Defaults to CWD
  --debug / --no-debug  Print debug logs to stderr.
  --help                Show this message and exit.

Commands:
  delete
  deploy
  dev                Development and debugging commands for chalice.
  gen-policy
  generate-models    Generate a model from Chalice routes.
  generate-pipeline  Generate a cloudformation template for a starter CD...
  generate-sdk
  invoke             Invoke the deployed lambda function NAME.
  local
  logs
  new-project
  package
  url
root@docker-ubuntu-s-1vcpu-2gb-nyc1-01:~/ap/t-performance-dash# chalice --version
chalice 1.23.0, python 3.8.10, linux 5.4.0-52-generic

root@docker-ubuntu-s-1vcpu-2gb-nyc1-01:~/ap/t-performance-dash# python --version
Python 3.8.10
root@docker-ubuntu-s-1vcpu-2gb-nyc1-01:~/ap/t-performance-dash# pipenv shell
Spawning environment shell (/bin/bash). Use 'exit' to leave.
root@docker-ubuntu-s-1vcpu-2gb-nyc1-01:~/ap/t-performance-dash# . /root/.local/share/virtualenvs/t-performance-dash-meyo44Iz/bin/activate
(t-performance-dash-meyo44Iz) root@docker-ubuntu-s-1vcpu-2gb-nyc1-01:~/ap/t-performance-dash# python --version          Python 3.7.11
(t-performance-dash-meyo44Iz) root@docker-ubuntu-s-1vcpu-2gb-nyc1-01:~/ap/t-performance-dash# npm install
npm WARN lifecycle<EMAIL_ADDRESS>cannot run in wd<EMAIL_ADDRESS>cd server && pipenv install --python=3.7 (wd=/root/ap/t-performance-dash)
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>|| >= 3.2.0-dev || >= 3.3.0-dev || >= 3.4.0-dev || >= 3.5.0-dev || >= 3.6.0-dev || >= 3.6.0-beta || >= 3.7.0-dev || >= 3.7.0-beta but none is installed. You must install peer dependencies yourself.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY<EMAIL_ADDRESS>(node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for<EMAIL_ADDRESS>wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY<EMAIL_ADDRESS>(node_modules/webpack-dev-server/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for<EMAIL_ADDRESS>wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY<EMAIL_ADDRESS>(node_modules/watchpack/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for<EMAIL_ADDRESS>wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY<EMAIL_ADDRESS>(node_modules/jest-haste-map/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for<EMAIL_ADDRESS>wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

audited 1826 packages in 31.824s

61 packages are looking for funding
  run `npm fund` for details

found 10237 vulnerabilities (5051 low, 204 moderate, 4982 high)
  run `npm audit fix` to fix them, or `npm audit` for details
(t-performance-dash-meyo44Iz) root@docker-ubuntu-s-1vcpu-2gb-nyc1-01:~/ap/t-performance-dash# npm audit fix
npm WARN deprecated<EMAIL_ADDRESS>Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm WARN deprecated<EMAIL_ADDRESS>fsevents 1 will break on node v14+ and could be using insecure binaries. Upgrade to fsevents 2.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY<EMAIL_ADDRESS>(node_modules/watchpack/node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for<EMAIL_ADDRESS>wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY<EMAIL_ADDRESS>(node_modules/watchpack-chokidar2/node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for<EMAIL_ADDRESS>wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>|| >= 3.2.0-dev || >= 3.3.0-dev || >= 3.4.0-dev || >= 3.5.0-dev || >= 3.6.0-dev || >= 3.6.0-beta || >= 3.7.0-dev || >= 3.7.0-beta but none is installed. You must install peer dependencies yourself.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY<EMAIL_ADDRESS>(node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for<EMAIL_ADDRESS>wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY<EMAIL_ADDRESS>(node_modules/webpack-dev-server/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for<EMAIL_ADDRESS>wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY<EMAIL_ADDRESS>(node_modules/jest-haste-map/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for<EMAIL_ADDRESS>wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

added 21 packages from 19 contributors, removed 8 packages, updated 31 packages and moved 6 packages in 47.881s

70 packages are looking for funding
  run `npm fund` for details

fixed 10117 of 10237 vulnerabilities in 1826 scanned packages
  7 vulnerabilities required manual review and could not be updated
  1 package update for 113 vulnerabilities involved breaking changes
  (use `npm audit fix --force` to install breaking changes; or refer to `npm audit` for steps to fix these manually)
(t-performance-dash-meyo44Iz) root@docker-ubuntu-s-1vcpu-2gb-nyc1-01:~/ap/t-performance-dash# npm start

><EMAIL_ADDRESS>start /root/ap/t-performance-dash
> concurrently npm:start-python npm:start-react

[start-python]
[start-python] ><EMAIL_ADDRESS>start-python /root/ap/t-performance-dash
[start-python] > cd server && pipenv run chalice local --port=5000
[start-python]
[start-react]
[start-react] ><EMAIL_ADDRESS>start-react /root/ap/t-performance-dash
[start-react] > react-scripts start
[start-react]
[start-python] Exception in thread Thread-1:
[start-python] Traceback (most recent call last):
[start-python]   File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
[start-python]     self.run()
[start-python]   File "/usr/local/lib/python3.8/dist-packages/chalice/local.py", line 727, in run
[start-python]     self._server = self._server_factory()
[start-python]   File "/usr/local/lib/python3.8/dist-packages/chalice/cli/__init__.py", line 143, in create_local_server
[start-python]     app_obj = config.chalice_app
[start-python]   File "/usr/local/lib/python3.8/dist-packages/chalice/config.py", line 141, in chalice_app
[start-python]     app = v()
[start-python]   File "/usr/local/lib/python3.8/dist-packages/chalice/cli/factory.py", line 277, in load_chalice_app
[start-python]     app = importlib.import_module('app')
[start-python]   File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
[start-python]     return _bootstrap._gcd_import(name[level:], package, level)
[start-python]   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
[start-python]   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
[start-python]   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
[start-python]   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
[start-python]   File "<frozen importlib._bootstrap_external>", line 848, in exec_module
[start-python]   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
[start-python]   File "/root/ap/t-performance-dash/server/app.py", line 6, in <module>
[start-python]     from chalicelib import data_funcs, aggregation, s3_alerts, secrets
[start-python]   File "/root/ap/t-performance-dash/server/chalicelib/data_funcs.py", line 2, in <module>
[start-python]     import pytz
[start-python] ModuleNotFoundError: No module named 'pytz'
[start-react] ℹ ｢wds｣: Project is running at http://<IP_ADDRESS>:3000/
[start-react] ℹ ｢wds｣: webpack output is served from
[start-react] ℹ ｢wds｣: Content not from webpack is served from /root/ap/t-performance-dash/public
[start-react] ℹ ｢wds｣: 404s will fallback to /
[start-react] Starting the development server...
[start-react]
[start-react] Compiled successfully!
[start-react]
[start-react] You can now view t-dash in the browser.
[start-react]
[start-react]   http://localhost:3000
[start-react]
[start-react] Note that the development build is not optimized.
[start-react] To create a production build, use npm run build.
[start-react]
[start-react] Proxy error: Could not proxy request /git_id from localhost:3000 to http://localhost:5000.
[start-react] See https://nodejs.org/api/errors.html#errors_common_system_errors for more information (ECONNREFUSED).
[start-react]
^C[start-react] npm run start-react exited with code SIGINT
[start-python] npm run start-python exited with code SIGINT
(t-performance-dash-meyo44Iz) root@docker-ubuntu-s-1vcpu-2gb-nyc1-01:~/ap/t-performance-dash# python --version
Python 3.7.11
(t-performance-dash-meyo44Iz) root@docker-ubuntu-s-1vcpu-2gb-nyc1-01:~/ap/t-performance-dash# npm start

><EMAIL_ADDRESS>start /root/ap/t-performance-dash
> concurrently npm:start-python npm:start-react

[start-python]
[start-python] ><EMAIL_ADDRESS>start-python /root/ap/t-performance-dash
[start-python] > cd server && pipenv run chalice local --port=5000
[start-python]
[start-react]
[start-react] ><EMAIL_ADDRESS>start-react /root/ap/t-performance-dash
[start-react] > react-scripts start
[start-react]
[start-python] Exception in thread Thread-1:
[start-python] Traceback (most recent call last):
[start-python]   File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
[start-python]     self.run()
[start-python]   File "/usr/local/lib/python3.8/dist-packages/chalice/local.py", line 727, in run
[start-python]     self._server = self._server_factory()
[start-python]   File "/usr/local/lib/python3.8/dist-packages/chalice/cli/__init__.py", line 143, in create_local_server
[start-python]     app_obj = config.chalice_app
[start-python]   File "/usr/local/lib/python3.8/dist-packages/chalice/config.py", line 141, in chalice_app
[start-python]     app = v()
[start-python]   File "/usr/local/lib/python3.8/dist-packages/chalice/cli/factory.py", line 277, in load_chalice_app
[start-python]     app = importlib.import_module('app')
[start-python]   File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
[start-python]     return _bootstrap._gcd_import(name[level:], package, level)
[start-python]   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
[start-python]   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
[start-python]   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
[start-python]   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
[start-python]   File "<frozen importlib._bootstrap_external>", line 848, in exec_module
[start-python]   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
[start-python]   File "/root/ap/t-performance-dash/server/app.py", line 6, in <module>
[start-python]     from chalicelib import data_funcs, aggregation, s3_alerts, secrets
[start-python]   File "/root/ap/t-performance-dash/server/chalicelib/data_funcs.py", line 2, in <module>
[start-python]     import pytz
[start-python] ModuleNotFoundError: No module named 'pytz'
[start-react] ℹ ｢wds｣: Project is running at http://<IP_ADDRESS>:3000/
[start-react] ℹ ｢wds｣: webpack output is served from
[start-react] ℹ ｢wds｣: Content not from webpack is served from /root/ap/t-performance-dash/public
[start-react] ℹ ｢wds｣: 404s will fallback to /
[start-react] Starting the development server...
[start-react]
[start-react] Compiled successfully!
[start-react]
[start-react] You can now view t-dash in the browser.
[start-react]
[start-react]   http://localhost:3000
[start-react]
[start-react] Note that the development build is not optimized.
[start-react] To create a production build, use npm run build.
[start-react]
[start-react] Proxy error: Could not proxy request /git_id from localhost:3000 to http://localhost:5000.
[start-react] See https://nodejs.org/api/errors.html#errors_common_system_errors for more information (ECONNREFUSED).

Install Chalice inside this python3.7 shell
(t-performance-dash-meyo44Iz) root@docker-ubuntu-s-1vcpu-2gb-nyc1-01:~/ap/t-performance-dash# python3 --version
Python 3.7.11
(t-performance-dash-meyo44Iz) root@docker-ubuntu-s-1vcpu-2gb-nyc1-01:~/ap/t-performance-dash# python3 -m venv venv37
Error: Command '['/root/ap/t-performance-dash/venv37/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.
(t-performance-dash-meyo44Iz) root@docker-ubuntu-s-1vcpu-2gb-nyc1-01:~/ap/t-performance-dash# $ python3 -m pip install chalice
$: command not found
(t-performance-dash-meyo44Iz) root@docker-ubuntu-s-1vcpu-2gb-nyc1-01:~/ap/t-performance-dash# python3 -m pip install chalice
Collecting chalice
  Using cached chalice-1.23.0-py2.py3-none-any.whl (387 kB)
Collecting six<2.0.0,>=1.10.0
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting jmespath<1.0.0,>=0.9.3
  Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting inquirer<3.0.0,>=2.7.0
  Using cached inquirer-2.7.0-py2.py3-none-any.whl (31 kB)
Requirement already satisfied: wheel in /root/.local/share/virtualenvs/t-performance-dash-meyo44Iz/lib/python3.7/site-packages (from chalice) (0.34.2)
Requirement already satisfied: pip<21.2,>=9 in /root/.local/share/virtualenvs/t-performance-dash-meyo44Iz/lib/python3.7/site-packages (from chalice) (20.0.2)
Collecting click<8.0,>=7
  Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
     |████████████████████████████████| 82 kB 1.0 MB/s
Requirement already satisfied: setuptools in /root/.local/share/virtualenvs/t-performance-dash-meyo44Iz/lib/python3.7/site-packages (from chalice) (44.0.0)
Collecting pyyaml<6.0.0,>=5.3.1
  Downloading PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)
     |████████████████████████████████| 636 kB 4.9 MB/s
Collecting mypy-extensions==0.4.3
  Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Collecting attrs<20.4.0,>=19.3.0
  Downloading attrs-20.3.0-py2.py3-none-any.whl (49 kB)
     |████████████████████████████████| 49 kB 2.8 MB/s
Collecting botocore<2.0.0,>=1.14.0
  Downloading botocore-1.20.107-py2.py3-none-any.whl (7.7 MB)
     |████████████████████████████████| 7.7 MB 4.2 MB/s
Collecting python-editor==1.0.4
  Using cached python_editor-1.0.4-py3-none-any.whl (4.9 kB)
Collecting readchar==2.0.1
  Using cached readchar-2.0.1-py3-none-any.whl (6.9 kB)
Collecting blessed==1.17.6
  Using cached blessed-1.17.6-py2.py3-none-any.whl (76 kB)
Collecting python-dateutil<3.0.0,>=2.1
  Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting urllib3<1.27,>=1.25.4
  Downloading urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
     |████████████████████████████████| 138 kB 4.0 MB/s
Collecting wcwidth>=0.1.4
  Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Installing collected packages: six, jmespath, python-editor, readchar, wcwidth, blessed, inquirer, click, pyyaml, mypy-extensions, attrs, python-dateutil, urllib3, botocore, chalice
Successfully installed attrs-20.3.0 blessed-1.17.6 botocore-1.20.107 chalice-1.23.0 click-7.1.2 inquirer-2.7.0 jmespath-0.10.0 mypy-extensions-0.4.3 python-dateutil-2.8.1 python-editor-1.0.4 pyyaml-5.4.1 readchar-2.0.1 six-1.16.0 urllib3-1.26.6 wcwidth-0.2.5
(t-performance-dash-meyo44Iz) root@docker-ubuntu-s-1vcpu-2gb-nyc1-01:~/ap/t-performance-dash# chalice --version
chalice 1.23.0, python 3.7.11, linux 5.4.0-52-generic

npm start
now finding correct version of python3.7
(t-performance-dash-meyo44Iz) root@docker-ubuntu-s-1vcpu-2gb-nyc1-01:~/ap/t-performance-dash# npm start

><EMAIL_ADDRESS>start /root/ap/t-performance-dash
> concurrently npm:start-python npm:start-react

[start-python]
[start-python] ><EMAIL_ADDRESS>start-python /root/ap/t-performance-dash
[start-python] > cd server && pipenv run chalice local --port=5000
[start-python]
[start-react]
[start-react] ><EMAIL_ADDRESS>start-react /root/ap/t-performance-dash
[start-react] > react-scripts start
[start-react]
[start-python] Exception in thread Thread-1:
[start-python] Traceback (most recent call last):
[start-python]   File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
[start-python]     self.run()
[start-python]   File "/root/.local/share/virtualenvs/t-performance-dash-meyo44Iz/lib/python3.7/site-packages/chalice/local.py", line 727, in run
[start-python]     self._server = self._server_factory()
[start-python]   File "/root/.local/share/virtualenvs/t-performance-dash-meyo44Iz/lib/python3.7/site-packages/chalice/cli/__init__.py", line 143, in create_local_server
[start-python]     app_obj = config.chalice_app
[start-python]   File "/root/.local/share/virtualenvs/t-performance-dash-meyo44Iz/lib/python3.7/site-packages/chalice/config.py", line 141, in chalice_app
[start-python]     app = v()
[start-python]   File "/root/.local/share/virtualenvs/t-performance-dash-meyo44Iz/lib/python3.7/site-packages/chalice/cli/factory.py", line 277, in load_chalice_app
[start-python]     app = importlib.import_module('app')
[start-python]   File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
[start-python]     return _bootstrap._gcd_import(name[level:], package, level)
[start-python]   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
[start-python]   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
[start-python]   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
[start-python]   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
[start-python]   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
[start-python]   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
[start-python]   File "/root/ap/t-performance-dash/server/app.py", line 6, in <module>
[start-python]     from chalicelib import data_funcs, aggregation, s3_alerts, secrets
[start-python]   File "/root/ap/t-performance-dash/server/chalicelib/data_funcs.py", line 2, in <module>
[start-python]     import pytz
[start-python] ModuleNotFoundError: No module named 'pytz'
[start-python]
[start-react] ℹ ｢wds｣: Project is running at http://<IP_ADDRESS>:3000/
[start-react] ℹ ｢wds｣: webpack output is served from
[start-react] ℹ ｢wds｣: Content not from webpack is served from /root/ap/t-performance-dash/public
[start-react] ℹ ｢wds｣: 404s will fallback to /
[start-react] Starting the development server...
[start-react]
[start-react] Compiled successfully!
[start-react]
[start-react] You can now view t-dash in the browser.
[start-react]
[start-react]   http://localhost:3000
[start-react]
[start-react] Note that the development build is not optimized.
[start-react] To create a production build, use npm run build.
[start-react]
[start-react] Proxy error: Could not proxy request /git_id from localhost:3000 to http://localhost:5000.
[start-react] See https://nodejs.org/api/errors.html#errors_common_system_errors for more information (ECONNREFUSED).
[start-react]

ModuleNotFoundError: No module named 'pytz'
and
Proxy error: Could not proxy request /git_id from localhost:3000 to http://localhost:5000.
[start-react] See https://nodejs.org/api/errors.html#errors_common_system_errors for more information (ECONNREFUSED).


DRAFT, DO NOT MERGE: proof-of-concept JobRunr for async APIs

this shows how we'd implement async APIs using JobRunr.
My initial assessment of JobRunr …
Pros

actively-maintained library (https://github.com/jobrunr/jobrunr)
easy to integrate with Spring Boot
nice property-file based config options for compatibility with our schemas
natively understands MDC, so we can correlate async jobs to the request that spawned them
support for retries

we need to make sure our jobs are idempotent if we want to use retries!

built-in UI dashboard for local debugging
uses our existing db for persistence
nice features but I didn't test them:

distributed optimistic locking to support multiple replicas/horizontal scaling
jobs automatically restart if WDS restarts
multiple options for persistence if we don't want to use the db

Cons

requires JobRunr pro ($$$) for programmatic access to job status and job results

workaround: query the JobRunr database tables directly. Fragile?

using our existing db for job persistence complicates cloning a little bit. We don't want to clone job history (or do we?)

workaround: ensure we drop/recreate the JobRunr tables after cloning

more reading: https://www.baeldung.com/java-jobrunr-spring

closing; we will not be using JobRunr


Teamcity

Quite a lot of commits in this, bit of messing around without being able to test locally.

Can now build this in TC -> http://teamcity.nice.org.uk/project.html?projectId=KbQs_Git2Prov&tab=projectOverview


Updated license the source files headers

Signed-off-by: Nikolay Vlasov<EMAIL_ADDRESS>

Testing approval


Update block proposer API

Can we add proposer and signers props to block details endpoint https://nexus.oasis.io/v1/consensus/blocks/{height}? Props were added to blocks list endpoint in #646 and we need to show the same data in block details page.

Can we define type for  entity_metadata  so it is not unknown ?

expected type is
interface EntityMetadata {
  v: number
  serial: number
  name?: string
  url?: string
  email?: string
  keybase?: string
  twitter?: string
}

based on comment:
When available, it is an object with some subset of the following fields:

v: The version of the metadata structure (always present).
serial: The serial number of the metadata statement (always present).
name: The name of the entity.
url: The URL associated with the entity.
email: The email address associated with the entity.
keybase: Tne entity's keybase.io handle.
twitter: The twitter handle associated with the entity.

Can we add proposer and signers props to block details endpoint

This was done.

Can we define type for entity_metadata so it is not unknown ?

We could, but we don't enforce the type in the DB level so I'm not sure if we should do it in the API.