Failing to simulate authentication with mock_iam and mock_s3 due to generated keys not validated successfully

I'm writing unit tests for a function that should authenticate to AWS S3 and then perform some operations on S3. I have a bunch of functions that do various things (like downloading/uploading files, checking for existence, etc).
Since in production these will need authentication, I want to simulate it in the tests, so that I can check that this part of the code is ok too.
For mocking the AWS environment I'm using moto. As of now I have the following code and traceback - all below. A quick intro to it: my_head_bucket is an example function that needs to have unit tests. create_user_with_access_key_and_policy should mock the IAM user and policy needed for authenticated S3 access. It is almost the same as in the examples in the documentation.
Then there are two tests. The first should pass without errors (having the correct authentication). The second should fail with ClientError, because "invalid" keys are being passed.
For some reason I am not able to pass through the creation of the mock user and policy, getting botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the AttachUserPolicy operation: The security token included in the request is invalid. It seems that no authentication should be needed for create_access_key, so what am I doing wrong?
import os
import unittest
import boto3
import botocore
from moto import mock_s3, mock_iam
from botocore.client import ClientError
from moto.core import set_initial_no_auth_action_count

import json

def my_head_bucket(bucket, aws_access_key_id, aws_secret_access_key):
    """
    This is a sample function. In the real case, this function will do more than
    just heading a bucket (like uploading/downloading files or other actions).

    It would be imported from another module and should not have any decorators (like moto's)
    """
    s3_client = boto3.client("s3", aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key)
    s3_client.head_bucket(Bucket=bucket)

@mock_iam
def create_user_with_access_key_and_policy(user_name="test-user"):
    """
    Should create a user with attached policy allowing read/write operations on S3.
    """
    policy_document = {
        "Version": "2012-10-17",
        "Statement": [
            {"Effect": "Allow", "Action": "s3:*", "Resource": "*"}
        ],
    }

    # Create client and user
    client = boto3.client("iam", region_name="us-east-1")
    client.create_user(UserName=user_name)

    # Create and attach the policy
    policy_arn = client.create_policy(
        PolicyName="policy1", PolicyDocument=json.dumps(policy_document)
    )["Policy"]["Arn"]
    client.attach_user_policy(UserName=user_name, PolicyArn=policy_arn)

    # Return the access keys
    return client.create_access_key(UserName=user_name)["AccessKey"]

class TestMyTest(unittest.TestCase):
    @set_initial_no_auth_action_count(0)
    @mock_s3
    def test_correct_credentials(self):
        """
        Sets the environment (creates user with keys and policy, creates the bucket), then calls
        the function-to-be-tested and expects it to run without exceptions.
        """
        ### Arrange
        iam_keys = create_user_with_access_key_and_policy()
        print(iam_keys)

        s3 = boto3.client('s3', aws_access_key_id=iam_keys["AccessKeyId"],aws_secret_access_key=iam_keys["SecretAccessKey"])
        s3.create_bucket(Bucket='mock_bucket')

        my_head_bucket('mock_bucket', aws_access_key_id=iam_keys["AccessKeyId"],aws_secret_access_key=iam_keys["SecretAccessKey"])

    @set_initial_no_auth_action_count(0)
    @mock_s3
    def test_incorrect_credentials(self):
        """
        Sets the environment (creates user with keys and policy, creates the bucket), then calls
        the function-to-be-tested and expects it to run without exceptions.
        """
        ### Arrange
        iam_keys = create_user_with_access_key_and_policy()
        print(iam_keys)

        s3 = boto3.client('s3', aws_access_key_id=iam_keys["AccessKeyId"],aws_secret_access_key=iam_keys["SecretAccessKey"])
        s3.create_bucket(Bucket='mock_bucket')

        with self.assertRaises(ClientError):
            my_head_bucket('mock_bucket', aws_access_key_id=iam_keys["AccessKeyId"],aws_secret_access_key="invalid")

==================================================================================================== test session starts =====================================================================================================
platform linux -- Python 3.6.8, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
rootdir: /home/trusso.storage.maintenance
collected 2 items

some_test.py FF                                                                                [100%]

========================================================================================================== FAILURES ==========================================================================================================
____________________________________________________________________________________________ TestMyTest.test_correct_credentials _____________________________________________________________________________________________

self = <some_test.TestMyTest testMethod=test_correct_credentials>

    @set_initial_no_auth_action_count(0)
    @mock_s3
    def test_correct_credentials(self):
        """
        Sets the environment (creates user with keys and policy, creates the bucket), then calls
        the function-to-be-tested and expects it to run without exceptions.

The @set_initial_no_auth_action_count(0) can be read as:
@authenticate_all(except_first_0_requests)
When starting the server, there are no users/roles at all, so almost all requests will fail. To get around this, all requests that deal with creating users/roles should be exempt from authentication.
In your case, there are 4: create_user, create_policy, attach_user_policy and create_access_key, so the decorator-parameter should be set to 4:
@set_initial_no_auth_action_count(4)
The initial 4 IAM requests will go through without any authentication like this, but the 5th request to S3 will be validated as expected.
Changing this will pass the first test.
The second test will still fail though, with this change, and I'm not sure why. Will mark it as a bug until we can figure out what's going wrong there.

Potentially related: #4240

Thank you for the clarification about set_initial_no_auth_action_count !
Confirmed that the first test works and the second still doesn't (which is, passes when it must fail). I'll wait for the resolution.

Hmmm... played around a little bit. Actually, inserting a call to s3_client.list_objects does make the second test pass, like this:
Interestingly enough, AWS's official documentation says that head_bucket should return 403 if the caller has no access. Thus, the call to head_bucket should raise ClientError anyways.
Pasting my code below for full reference and with linting:
import json
import os
import unittest

import boto3
import botocore
from botocore.client import ClientError
from moto import mock_iam, mock_s3
from moto.core import set_initial_no_auth_action_count

def my_head_bucket(bucket, aws_access_key_id, aws_secret_access_key):
    """
    This is a sample function. In the real case, this function will do more than
    just heading a bucket (like uploading/downloading files or other actions).

    It would be imported from another module and should not have any decorators (like moto's)
    """
    s3_client = boto3.client("s3",
                             aws_access_key_id=aws_access_key_id,
                             aws_secret_access_key=aws_secret_access_key)
    s3_client.head_bucket(Bucket=bucket)
    s3_result = s3_client.list_objects(Bucket=bucket, Prefix='')

@mock_iam
def create_user_with_access_key_and_policy(user_name="test-user"):
    """
    Should create a user with attached policy allowing read/write operations on S3.
    """
    policy_document = {
        "Version": "2012-10-17",
        "Statement": [{
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "*"
        }],
    }

    # Create client and user
    client = boto3.client("iam", region_name="us-east-1")
    client.create_user(UserName=user_name)

    # Create and attach the policy
    policy_arn = client.create_policy(PolicyName="policy1",
                                      PolicyDocument=json.dumps(policy_document))["Policy"]["Arn"]
    client.attach_user_policy(UserName=user_name, PolicyArn=policy_arn)

    # Return the access keys
    return client.create_access_key(UserName=user_name)["AccessKey"]

class TestMyTest(unittest.TestCase):
    @set_initial_no_auth_action_count(4)
    @mock_s3
    def test_correct_credentials(self):
        """
        Sets the environment (creates user with keys and policy, creates the bucket), then calls
        the function-to-be-tested and expects it to run without exceptions.
        """
        ### Arrange
        iam_keys = create_user_with_access_key_and_policy()
        print(iam_keys)

        s3 = boto3.client('s3',
                          aws_access_key_id=iam_keys["AccessKeyId"],
                          aws_secret_access_key=iam_keys["SecretAccessKey"])
        s3.create_bucket(Bucket='mock_bucket')

        my_head_bucket('mock_bucket',
                       aws_access_key_id=iam_keys["AccessKeyId"],
                       aws_secret_access_key=iam_keys["SecretAccessKey"])

    @set_initial_no_auth_action_count(4)
    @mock_s3
    def test_incorrect_credentials(self):
        """
        Sets the environment (creates user with keys and policy, creates the bucket), then calls
        the function-to-be-tested and expects it to run without exceptions.
        """
        ### Arrange
        iam_keys = create_user_with_access_key_and_policy()
        print(iam_keys)

        s3 = boto3.client('s3',
                          aws_access_key_id=iam_keys["AccessKeyId"],
                          aws_secret_access_key=iam_keys["SecretAccessKey"])
        s3.create_bucket(Bucket='mock_bucket')

        with self.assertRaises(ClientError):
            my_head_bucket('mock_bucket',
                           aws_access_key_id=iam_keys["AccessKeyId"],
                           aws_secret_access_key="invalid")

Hmmm... played around a little bit. Actually, inserting a call to s3_client.list_objects call does make the second test pass.

Yeah, I noticed that as well. We do check authentication for the list_objects requests, so that will fail with invalid credentials, but we do not validate the authentication on head_bucket requests.
There's a PR coming soon with a fix for this.


Importing long list causes "Maximum call stack size exceeded"

I got this error trying to run tests, as they rely on the repl it seems. Anyway I tracked it down to the import of a file that contains only a 640 item list (of countries and states). I reduced the list to 290 and it worked, but I don't know what is the actual cutoff point.
This has made me think I should move this into asynchronously downloaded json, but the bug still stands I think.

I got this error trying to run tests, as they rely on the repl it seems

elm-test doesn't use elm-repl at all, so I suspect it's more likely to be https://github.com/elm-lang/elm-compiler/issues/1521


fix broken build

Combination of recently separately merged PRs #865 and #859 broke it.

I have exceptionally merge this PR without even waiting for Travis - because it was already broken.
@awasum @xurror @percyashu @thesmallstar @ptuomola just FYI

Thanks Mike.
On Fri, May 15, 2020, 21:56 Michael Vorburger ⛑<EMAIL_ADDRESS>wrote:

I have exceptionally merged this PR without even waiting for Travis -
because it was already broken.
@awasum https://github.com/awasum @xurror https://github.com/xurror
@percyashu https://github.com/percyashu @thesmallstar
https://github.com/thesmallstar @ptuomola https://github.com/ptuomola
just FYI
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/fineract/pull/893#issuecomment-629479020, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ADWPAYGFIM7X4F46I3XXRQDRRWUABANCNFSM4NCRWYOQ
.


two icons were disappear

Do you have the latest versions of Spicetify, Spotify and Lucid?

[X] I'm using the latest versions of Spicetify, Spotify and Lucid

Is there already an issue for your problem?

[X] I have checked older issues, open and closed

Is your problem definitely caused by the theme?

[X] I have checked the app behavior without the theme and it is different

Environment & Computer Info
- Spotify version:
- Installed from: (Windows Store, Spotify site, Flatpak, etc)
- Spicetify version:
- Color scheme:
- Extensions:
- Custom apps:
- Lucid installation method: (Spicetify Marketplace/PowerShell script/Bash script/Manual)

Description
The shuffle button and loop play button are gone,
Screenshots

This was a known issue for some of our users. I'm unable to reproduce it and am unsure of the root cause. We are working on a fix, so please be patient

can you provide your spotify spicetify version @mellasitus

Is this?

Yes, I will check on that. What language are you using in Spotify?

usually Chinese

My guess was correct. It was an issue specific to some languages. I'll fix it as soon as possible.

usually Simplified Chinese.But when i used English,it's right

yes i will update it soon

think

thank


Dart test returned: Error: Not found: 'dart:ui'

Context
Working on a project that views data in the CSV and stores it all in a list. Before, when testing out the file: dart data_file.dart it was able to return the content that I requested to be printed showed in the terminal.

Enviroment

VScode
macOSX
flutter version:

Flutter 1.25.0-9.0.pre.88 • channel master • https://github.com/flutter/flutter.git
Framework • revision e24e763872 (6 hours ago) • 2020-12-16 07:33:03 -0500
Engine • revision e9f2f0eaea
Tools • Dart 2.12.0 (build 2.12.0-157.0.dev)

Any solutions, tips or help will be greatly appreciated!

If you test depends on flutter or dart:ui, you need to use flutter test, not dart or pub commands.

Thanks @jonahwilliams, and by doing so i have the following error:
══╡ EXCEPTION CAUGHT BY WIDGETS LIBRARY ╞═══════════════════════════════════════════════════════════
The following HiveError was thrown attaching to the render tree:
Box not found. Did you forget to call Hive.openBox()?

When the exception was thrown, this was the stack:
#0      HiveImpl._getBoxInternal (package:hive/src/hive_impl.dart:170:7)
#1      HiveImpl.box (package:hive/src/hive_impl.dart:181:33)
#2      _MyAppState.initState (package:app/main.dart:50:25)
#3      StatefulElement._firstBuild (package:flutter/src/widgets/framework.dart:4821:57)
#4      ComponentElement.mount (package:flutter/src/widgets/framework.dart:4658:5)
#5      Element.inflateWidget (package:flutter/src/widgets/framework.dart:3624:14)
#6      Element.updateChild (package:flutter/src/widgets/framework.dart:3386:20)
#7      RenderObjectToWidgetElement._rebuild (package:flutter/src/widgets/binding.dart:1211:16)
#8      RenderObjectToWidgetElement.update (package:flutter/src/widgets/binding.dart:1189:5)
#9      RenderObjectToWidgetElement.performRebuild (package:flutter/src/widgets/binding.dart:1203:7)
#10     Element.rebuild (package:flutter/src/widgets/framework.dart:4378:5)
#11     BuildOwner.buildScope (package:flutter/src/widgets/framework.dart:2777:33)
#12     AutomatedTestWidgetsFlutterBinding.drawFrame (package:flutter_test/src/binding.dart:1110:19)
#13     RendererBinding._handlePersistentFrameCallback (package:flutter/src/rendering/binding.dart:321:5)
#14     SchedulerBinding._invokeFrameCallback (package:flutter/src/scheduler/binding.dart:1144:15)
#15     SchedulerBinding.handleDrawFrame (package:flutter/src/scheduler/binding.dart:1082:9)
#16     AutomatedTestWidgetsFlutterBinding.pump.<anonymous closure> (package:flutter_test/src/binding.dart:977:9)
#19     TestAsyncUtils.guard (package:flutter_test/src/test_async_utils.dart:72:41)
#20     AutomatedTestWidgetsFlutterBinding.pump (package:flutter_test/src/binding.dart:964:27)
#21     WidgetTester.pumpWidget.<anonymous closure> (package:flutter_test/src/widget_tester.dart:522:22)
#24     TestAsyncUtils.guard (package:flutter_test/src/test_async_utils.dart:72:41)
#25     WidgetTester.pumpWidget (package:flutter_test/src/widget_tester.dart:519:27)
#26     main.<anonymous closure> (file://.../app/test/widget_test.dart:16:18)
#27     main.<anonymous closure> (file://.../app/test/widget_test.dart:14:48)
#28     testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:144:29)
<asynchronous suspension>
<asynchronous suspension>
(elided 5 frames from dart:async and package:stack_trace)

I do have the box opened in the initilaizer. I personally think it's a possible issue with the dart SDK inside flutter because when i run dart file.dart with out even importing 'dart:io'; it get the same return as before Error: Not found: 'dart:ui'`

@jeremybeal11
Can you provide a complete reproducible code sample along with steps you take which shows this issue ?

file name: drinklist.dart
import 'dart:io';
import 'package:csv/csv.dart' as csv;
//import 'package:hive/hive.dart';
import 'package:path_provider/path_provider.dart';

List drinkNames = [];
List drinkDesc = [];
File myData = new File('./myCSV.csv');

//Box _drinkList;

void main() {
  //Hive.registerAdapter(MenuAdapter());
  getDrinkNames();
  getDrinkDesc();
}

getDrinkNames() {
  List<List<dynamic>> csv = csvToList(myData);
  for (int i = 0; i < csv.length; i++) {
    drinkDesc.add(csv[i][1]);
    drinkNames.add(csv[i][0]);

  }
  print(drinkNames);
}

getDrinkDesc() {
  List<List<dynamic>> csv = csvToList(myData);
  for (int i = 0; i < csv.length; i++) {
    drinkDesc.add(csv[i][1]);
  }
  //print(drinkDesc);
}

List<List> csvToList(File myData) {
  csv.CsvToListConverter c =
      new csv.CsvToListConverter(eol: "\r\n", fieldDelimiter: ",");
  List<List> listCreated = c.convert(myData.readAsStringSync());
  return listCreated;
}

All i did was run dart drinklist.dart and recieved the error.

@jeremybeal11
Thanks for the code sample. By running dart drinklist.dart I see below in console log:
dhs@Dhss-MacBook-Pro testapp % dart test/drinklist.dart
../../Fluttersdk/flutter/packages/flutter/lib/src/foundation/basic_types.dart:9:1: Error: Not found: 'dart:ui'
export 'dart:ui' show VoidCallback;
^
../../Fluttersdk/flutter/packages/flutter/lib/src/foundation/binding.dart:8:8: Error: Not found: 'dart:ui'
import 'dart:ui' as ui show SingletonFlutterWindow, Brightness, PlatformDispatcher, window;
       ^
../../Fluttersdk/flutter/packages/flutter/lib/src/foundation/debug.dart:5:8: Error: Not found: 'dart:ui'
import 'dart:ui' as ui show Brightness;
       ^
../../Fluttersdk/flutter/packages/flutter/lib/src/foundation/key.dart:5:8: Error: Not found: 'dart:ui'
import 'dart:ui' show hashValues;
       ^

By running flutter test test/drinklist.dart, I get below error:
dhs@Dhss-MacBook-Pro testapp % flutter test test/drinklist.dart
00:02 +0 -1: loading /Users/dhs/Documents/NCFlutter/testapp/test/drinklist.dart [E]
  Failed to load "/Users/dhs/Documents/NCFlutter/testapp/test/drinklist.dart": Cannot open file, path = './myCSV.csv' (OS Error: No such file or directory, errno = 2)
  dart:io                    _File.readAsStringSync
  test/drinklist.dart 39:45  csvToList
  test/drinklist.dart 19:29  getDrinkNames
  test/drinklist.dart 14:3   main

00:02 +0 -1: Some tests failed.

What's the expected result by running both type of commands ?

flutter doctor -v
[✓] Flutter (Channel master, 1.26.0-2.0.pre.86, on Mac OS X 10.15.4 19E2269
    darwin-x64, locale en-IN)
    • Flutter version 1.26.0-2.0.pre.86 at
      /Users/dhs/documents/Fluttersdk/flutter
    • Framework revision b793285329 (3 days ago), 2020-12-18 10:14:36 -0800
    • Engine revision 7983c6fda0
    • Dart version 2.12.0 (build 2.12.0-167.0.dev)

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.0)
    • Android SDK at /Users/dhs/Library/Android/sdk
    • Platform android-30, build-tools 30.0.0
    • Java binary at: /Applications/Android
      Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build
      1.8.0_242-release-1644-b3-6222593)
    • All Android licenses accepted.

[!] Xcode - develop for iOS and macOS (Xcode 12.0.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 12.0.1, Build version 12A7300
    ! CocoaPods 1.9.3 out of date (1.10.0 is recommended).
        CocoaPods is used to retrieve the iOS and macOS platform side's plugin
        code that responds to your plugin usage on the Dart side.
        Without CocoaPods, plugins will not work on iOS or macOS.
        For more info, see https://flutter.dev/platform-plugins
      To upgrade see
      https://guides.cocoapods.org/using/getting-started.html#installation for
      instructions.

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.0)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 46.0.2
    • Dart plugin version 193.7361
    • Java version OpenJDK Runtime Environment (build
      1.8.0_242-release-1644-b3-6222593)

[✓] VS Code (version 1.51.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.16.0

[✓] Connected device (3 available)
    • sdk gphone x86 arm (mobile) • emulator-5554 • android-x86    • Android 11
      (API 30) (emulator)
    • macOS (desktop)             • macos         • darwin-x64     • Mac OS X
      10.15.4 19E2269 darwin-x64
    • Chrome (web)                • chrome        • web-javascript • Google
      Chrome 87.0.4280.88

! Doctor found issues in 1 category.

Hey @darshankawar thanks for looking into it. The expected behavior is that it's supposed to return the list created from the CSV. I surprisingly enough worked when I kept running the file(almost got excited) but resume to display the above errors.
When I ran the flutter test command I have received the same errors as you have.
when running flutter doctor -v i got this:
flutter doctor -v
[✓] Flutter (Channel master, 1.26.0-2.0.pre.45, on Mac OS X 10.15.7 19H2 darwin-x64, locale en-US)
    • Flutter version 1.26.0-2.0.pre.45 at /Users/.../flutter
    • Framework revision e24e763872 (5 days ago), 2020-12-16 07:33:03 -0500
    • Engine revision e9f2f0eaea
    • Dart version 2.12.0 (build 2.12.0-157.0.dev)

[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.2)
    • Android SDK at /Users/.../Android/sdk
    • Platform android-29, build-tools 29.0.2
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 12.0.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 12.0.1, Build version 12A7300
    • CocoaPods version 1.9.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)

[✓] VS Code (version 1.52.0)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.17.0

[✓] Connected device (1 available)
    • Chrome (web) • chrome • web-javascript • Google Chrome 87.0.4280.88

• No issues found!

I'm just not sure why it randomly displayed the list(as it should) and all of a sudden returned the error.
my speculation is that it's something to do with the Dart SDK thats inside the flutter package, but not entirely sure just yet.

Closing because this is not a bug. flutter test is for flutter programs, dart test is for dart programs.

Closing because this is not a bug. flutter test is for flutter programs, dart test is for dart programs.

Is this issue still open because facing the same issue but still haven't found any solution @jonahwilliams

The same issue here.

I got the same issue with a brand new flutter app, running the test/widget_test.dart in android studio.

Android Studio 4.1.2
Build #AI-201.87<IP_ADDRESS>42882, built on December 20, 2020
Runtime version: 1.8.0_242-release-1644-b01 amd64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
Windows 10 10.0

C:\src\flutter\bin\cache\dart-sdk\bin\dart.exe --enable-asserts C:\projects\flutter-research\unit_testing\test\widget_test.dart
/C:/src/flutter/packages/flutter/lib/src/material/animated_icons.dart:9:8: Error: Not found: 'dart:ui'
import 'dart:ui' as ui show Paint, Path, Canvas;
       ^
/C:/src/flutter/packages/flutter/lib/src/material/animated_icons.dart:10:8: Error: Not found: 'dart:ui'
import 'dart:ui' show lerpDouble;
       ^
/C:/src/flutter/packages/flutter/lib/src/material/app.dart:5:8: Error: Not found: 'dart:ui'
import 'dart:ui' as ui;
       ^
/C:/src/flutter/packages/flutter/lib/src/material/app_bar_theme.dart:5:8: Error: Not found: 'dart:ui'
import 'dart:ui' show lerpDouble;
       ^
/C:/src/flutter/packages/flutter/lib/src/material/arc.dart:6:8: Error: Not found: 'dart:ui'
import 'dart:ui' show lerpDouble;
       ^
/C:/src/flutter/packages/flutter/lib/src/material/bottom_app_bar_theme.dart:5:8: Error: Not found: 'dart:ui'
import 'dart:ui' show lerpDouble;
       ^
/C:/src/flutter/packages/flutter/lib/src/material/bottom_navigation_bar_theme.dart:5:8: Error: Not found: 'dart:ui'
import 'dart:ui' show lerpDouble;
       ^
/C:/src/flutter/packages/flutter/lib/src/material/bottom_sheet.dart:5:8: Error: Not found: 'dart:ui'
import 'dart:ui' show lerpDouble;
       ^
/C:/src/flutter/packages/flutter/lib/src/material/bottom_sheet_theme.dart:5:8: Error: Not found: 'dart:ui'
import 'dart:ui' show lerpDouble;
       ^
/C:/src/flutter/packages/flutter/lib/src/material/button_bar_theme.dart:5:8: Error: Not found: 'dart:ui'
import 'dart:ui' show lerpDouble;
       ^

Process finished with exit code 254

flutter doctor:

Doctor summary (to see all details, run flutter doctor -v):
[√] Flutter (Channel stable, 2.0.2, on Microsoft Windows [Version 10.0.19042.867], locale en-US)
[√] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
[√] Chrome - develop for the web
[√] Android Studio (version 4.1.0)
[√] IntelliJ IDEA Community Edition (version 2020.3)
[√] VS Code (version 1.54.3)
[√] Connected device (3 available)
• No issues found!

The same issue here.

I had this issue, and was the wrong import path
for example:
instead of importing
import 'package:buzzao_ble_protocol/src/models/session.dart';
it imported the package file where export this file.

same here

Same problem here,
I was launching a flutter test command, but at some point, for no reason, those errors start to pop.
I am definitely using a "flutter" command,
flutter drive -d emulator-5554 --target=test_integration/driver/app_target.dart --driver=/Users/my/app/test_integration/tests/intialize/splashscreen_test.dart --dart-define=MOCK_SESSION_ID=6160edc03437c894.07960931
Running "flutter pub get" in application_fr_kador_02...          1 312ms
Running Gradle task 'assembleDebug'...
Running Gradle task 'assembleDebug'... Done                        10,9s
✓  Built build/app/outputs/flutter-apk/app-debug.apk.
Installing build/app/outputs/flutter-apk/app.apk...              1 488ms
I/flutter (10759): Observatory listening on http://<IP_ADDRESS>:40189/Kdwqk8Y8CIE=/
Installing build/app/outputs/flutter-apk/app.apk...              1 488ms
I/flutter (10759): Observatory listening on http://<IP_ADDRESS>:40189/Kdwqk8Y8CIE=/
/usr/local/opt/flutter/packages/flutter/lib/src/cupertino/action_sheet.dart:5:8: Error: Not found: 'dart:ui'
import 'dart:ui' show ImageFilter;
       ^
/usr/local/opt/flutter/packages/flutter/lib/src/cupertino/bottom_tab_bar.dart:5:8: Error: Not found: 'dart:ui'
import 'dart:ui' show ImageFilter;
       ^
/usr/local/opt/flutter/packages/flutter/lib/src/cupertino/colors.dart:5:8: Error: Not found: 'dart:ui'

Flutter doctor below
$ flutter doctor -v

[✓] Flutter (Channel stable, 2.2.3, on macOS 11.4 20F71 darwin-x64, locale fr-FR)
    • Flutter version 2.2.3 at /usr/local/opt/flutter
    • Framework revision f4abaa0735 (12 days ago), 2021-07-01 12:46:11 -0700
    • Engine revision 241c87ad80
    • Dart version 2.13.4

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at /Users/evaisse/Library/Android/sdk
    • Platform android-30, build-tools 30.0.3
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 12.5.1, Build version 12E507
    • CocoaPods version 1.10.1

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)

[✓] VS Code (version 1.56.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension can be installed from:
      🔨 https://marketplace.visualstudio.com/items?itemName=Dart-Code.flutter

[✓] Connected device (3 available)
    • sdk gphone x86 arm (mobile) • emulator-5554                            • android-x86    • Android 11 (API 30) (emulator)
    • iPhone SE Manu (mobile)     • 3106633b7f0a4a8cb24c445801b913b39a9d1b33 • ios            • iOS 14.6
    • Chrome (web)                • chrome                                   • web-javascript • Google Chrome 91.0.4472.114

• No issues found!


🧪 cassette reformatting

Description
Removes binary YAML entries on cassettes
import pathlib

import yaml
from vcr.filters import decode_response
from vcr.serialize import deserialize, serialize
from vcr.serializers import yamlserializer

repo = pathlib.Path.home() / "git" / "camply"
cassettes = list(repo.rglob("**/cassettes/*.yaml"))
for cassette in cassettes:
    cassette_value = cassette.read_text()
    cassette_yaml = yaml.safe_load(cassette_value)
    requests, responses = deserialize(cassette_value, yamlserializer)
    for index, (request, response) in enumerate(zip(requests, responses)):
        decoded_response = decode_response(response)
        if isinstance(decoded_response["body"]["string"], bytes):
            string_value = decoded_response["body"]["string"].decode("utf-8")
            decoded_response["body"]["string"] = string_value
            responses[index] = decoded_response
    cassette_dict = dict(
        requests=requests,
        responses=responses,
    )
    yaml_dump = serialize(cassette_dict, yamlserializer)
    cassette.write_text(yaml_dump)

:tada: This PR is included in version 0.32.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Display authentification URL when running "m365 setup"

Priority
(Low) Something is a little off
Description
When setting up m365 you are required to do an interactive login to create an AppId in Office 365 Tenant. On a headless server this is not easy (you have to install a browser an tunnel a X.Windows session within ssh for example). I suggest the command could print the required URL in the terminal, so that you can copy/paste it in, a local browser
Steps to reproduce
m365 setup, create a new appId in the Tenant
Expected results
Display the URL as well as trying to start the browser
Actual results
Just starts the browser
Diagnostics
No response
CLI for Microsoft 365 version
v9.0.0
nodejs version
v18.19.0
Operating system (environment)
Windows
Shell
bash
cli doctor
No response
Additional Info
No response

@apc005 thanks for this suggestion 👍. We will look into it ASAP

Is there a way for me to formulate the required URL from the error message below? I am attempting to login from a headless installation and getting the following (ID's replaced by xxx):
$ m365 login
To sign in, use the web browser that just has been opened. Please sign-in there.
node:events:491
throw er; // Unhandled 'error' event
^
Error: spawn xdg-open ENOENT
at ChildProcess._handle.onexit (node:internal/child_process:283:19)
at onErrorNT (node:internal/child_process:476:16)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
Emitted 'error' event on ChildProcess instance at:
at ChildProcess._handle.onexit (node:internal/child_process:289:12)
at onErrorNT (node:internal/child_process:476:16)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
errno: -2,
code: 'ENOENT',
syscall: 'spawn xdg-open',
path: 'xdg-open',
spawnargs: [
'https://login.microsoftonline.com/xxxxx-xxxx-xxxx-xxxx-xxxxxxx/oauth2/authorize?response_type=code&client_id=xxxxxxx-e9d3-xxxx-xxxx-xxxxxxxx&redirect_uri=http://localhost:34891&state=xxxxxxxxxx&resource=https://graph.microsoft.com&prompt=select_account'
]
}

Is there a way for me to formulate the required URL from the error message below? I am attempting to login from a headless installation and getting the following (ID's replaced by xxx):
$ m365 login To sign in, use the web browser that just has been opened. Please sign-in there. node:events:491 throw er; // Unhandled 'error' event ^
Error: spawn xdg-open ENOENT at ChildProcess._handle.onexit (node:internal/child_process:283:19) at onErrorNT (node:internal/child_process:476:16) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) Emitted 'error' event on ChildProcess instance at: at ChildProcess._handle.onexit (node:internal/child_process:289:12) at onErrorNT (node:internal/child_process:476:16) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { errno: -2, code: 'ENOENT', syscall: 'spawn xdg-open', path: 'xdg-open', spawnargs: [ 'https://login.microsoftonline.com/xxxxx-xxxx-xxxx-xxxx-xxxxxxx/oauth2/authorize?response_type=code&client_id=xxxxxxx-e9d3-xxxx-xxxx-xxxxxxxx&redirect_uri=http://localhost:34891&state=xxxxxxxxxx&resource=https://graph.microsoft.com&prompt=select_account' ] }

I don't really understand what you're asking. Is it related to this issue?

Sorry for the confusion. Our installation is on a "headless" linux server with no web browser installed. So, the operation to popup an interactive web browser during the m365 setup and/or login was failing and produced the above error message. I was wondering if there was some manual way to put together a URL that could be copy/pasted into a browser to do the authentication. The error message does show a URL but it did not seem to work when I pasted it into a web browser.
Anyway, as I indicated in the other issue (6333) our installation is working again after manually adding the app registration permissions and doing a m365 login with the appID and tenant ID. So, my comment can be disregarded. Thank you for looking at this.
The feature requested in this ticket, to display the authentication URL when running m365 setup, would be very useful and I hope it can be implemented. Thank you!

Thanks for the clarification. Glad to hear your setup works again. The site that we try to open is to authenticate the current user and be able to create a new application registration in Azure that CLI can consume. Logging the URL, just like we do for m365 login is indeed a good suggestion.

@dshirk-uci you may already get the URL just by adding the --debug to the setup or login command.
You will find the redirect URL in the printed logs

For headless servers you should use the device code login method which is made specifically for this case: it prints a code in the terminal which you copy and paste to the login page. Have you considered using it @apc005?

Thx but I was talking about the "setup" process, not the login one

To clarify @waldekmastykarz, when running m365 setup and creating a new app registration, the URL to authenticate to Azure CLI is not printed in the terminal as is the case for m365 login.

Ah sorry, I misunderstood. Thank you for clarifying. In that case, we should consider allowing you to choose whether you want to login to Azure using the browser or device code auth.
Is there btw a reason you need to create the app from the headless server rather than say locally and on the headless server use the option to use an existing app?

If we assume that the most common use case for the m365 setup is right after installing CLI, then we should assume unconfigured, default state. I wonder if we shouldn't consider device code as automatic fallback for browser when we detect that we can't open the browser.

To be honest, I've used m365 setup on different tenants to set up the app registration with all the needed permissions. It's a convenient way of using CLI without the need to add all permissions yourself. I think we shouldn't assume that people just run it after installing CLI and take the config into account as we should do in every command.

That's a good point. I think we talked about it already, that we should consider exposing the app creation and configuration part outside of the setup command so that it's not necessary to reset/change your settings if all you need is to create an app.

And also an additional command that adds all missing scopes from the allScopes file.

You mean to an existing app reg and/or service principal?

You mean to an existing app reg and/or service principal?

Yes

I suggest those would be two different commands.

I had the same idea.

@waldekmastykarz, @milanholemans do we want to conclude this issue with some action and open it up?
Will logging the auth URL in case of m365 setup be good enough for start?

Logging the URL definitely got my vote. Just like the login command we should print it and auto open it in a browser if the right config is set.

Logging the URL definitely got my vote. Just like the login command we should print it and auto open it in a browser if the right config is set.

+1 to that as well
@waldekmastykarz if you agree we could start with this improvement and open this up to be taken

I don't think we fully answered the question. And even if we print the URL it doesn't address the issue with trying to create an app on a remote machine that doesn't have a browser and where we should be using device code auth. So to recap: let's understand first the need behind creating the app on a remote machine that doesn't have a browser. We can then decide the best way forward.

Nevertheless, I use browser containers for almost everything, especially M365 tenants. For me, it's no help that CLI automatically opens my browser window because every time this happens, the URL is opened in a containerless tab.
So for me printing the consent URL would be a nice enhancement already.
I also maintain my position that we should implement autoOpenLinksInBrowser everywhere to maintain consistency in the product.

That's a good reason @milanholemans albeit a different one than the problem outlined in the original issue. If I'm not mistaken, it wouldn't solve the issue with trying to use browser auth on a system that doesn't support it because it's not capable of opening a browser.


Prep for v0.0.7

Proposed changes
Prep for v0.0.7
Note, v0.0.6 was not skipped -- its release was reverted as #91 produced a bad build.
Related issues (optional)
Closes #93

Thanks for the great suggestions!
We'll incorporate these steps going forward, and we'll delete the v0.0.6 tag and Docker container.


There is no assertModelExists method when testing

Laravel Version: 8.59.0
PHP Version: 7.4.3
Database Driver & Version: mysql version 15.1

Description:
When testing with PHPUnit, and using the  RefreshDatabase trait, assertModelExists and assertModelMissing methods should exist, but they don't.
Steps To Reproduce:
Start with a new Laravel project.
Use the RefreshDatabase trait in Tests/TestCase.php.
In tinker run the following:
>>> $test = new \Tests\Feature\ExampleTest();
>>> $test->assertModelExists()
PHP Error:  Call to undefined method Tests\Feature\ExampleTest::assertModelExists() in Psy Shell code on line 1
>>> $test->assertModelMissing()
PHP Error:  Call to undefined method Tests\Feature\ExampleTest::assertModelMissing() in Psy Shell code on line 1

You can see from the output, those methods were undefined
For testing the code above, it throws other errors than undefined method when calling an existing method, as it should:
>>> $test->assertDeleted()
PHP Error:  Call to protected method Illuminate\Foundation\Testing\TestCase::assertDeleted() from context '' in Psy Shell code on line 1

Update your Laravel installation.


style: Revamp Dark Mode, part 1

Begin revamping dark mode.
Discord user tartarts has offered up some mockups, advice, and feedback for improving the default dark mode. Apply these changes with as few side effects to light mode as possible.
With this first pass, we're focused on reducing unnecessary noise with bright borders, bright separators, oversaturated or confusingly colored things.
This will not be a comprehensive overhaul in one go. It will be a gradual refinement, starting with this first pass.

Released in 0.1.18


[Ionic 4] error TS2339: Property 'width' does not exist on type 'HTMLIonMenuElement'.

Ionic Info
v4.0.0-alpha.9
Support Question
In v3 Menu use to expose a width() method. Didn't found it in v4. Did I miss something or are they still on the roadmap?
Code
 const menu: HTMLIonMenuElement = await this.menuController.get('myMenu');
 menu.width();

In v4.0.0-alpha.12 getWidth is there, could confirm it compiles
I'm not there to test it but looks good
Side note:
In Ionic v3 method's name is width() and in v4 getWidth() just in case someone would land on that in the future

@manucorporat Can we document the breaking change for the method name?

Ok so I just tested, I probably did something wrong, but I was unable to get the width of the menu thru this exposed function, I always get undefined
But I found I guess a workaround
const menu: HTMLIonMenuElement = await this.menuController.get('myMenu');
console.log(menu.getWidth()); // --> undefined
console.log(menu.clientWidth); // --> the width of the menu


PowerSpectrum TensorMap can not be fully merged in a single block

When computing a descriptor with PowerSpectrum from two frames that have different atomtypes i.e one with only an H and the other one only with an O the result can not be densified.
Consider the following example
import ase
import numpy as np
from equistore import EquistoreError
from rascaline import SoapPowerSpectrum, SphericalExpansion
from rascaline.utils import PowerSpectrum

frames = [
    ase.Atoms("H", positions=np.zeros([1, 3])),
    ase.Atoms("O", positions=np.zeros([1, 3])),
]

hypers = dict(
    cutoff=3.0,
    max_radial=6,
    max_angular=1,
    atomic_gaussian_width=0.3,
    center_atom_weight=1.0,
    radial_basis={"Gto": {}},
    cutoff_function={"ShiftedCosine": {"width": 0.5}},
)

calculator_pyps = PowerSpectrum(SphericalExpansion(**hypers))
descriptor_pyps = calculator_pyps.compute(frames)
print(descriptor_pyps)

try:
    descriptor_pyps.keys_to_samples("species_center")
except EquistoreError as err:
    print(err)

which prints
TensorMap with 2 blocks
keys: species_center
            1
            8
invalid parameter: can not move keys to samples if the blocks have different property labels

The same happens when calling the SOAPPowerSpectrum
calculator_ps = SoapPowerSpectrum(**hypers)
descriptor_ps = calculator_ps.compute(frames)
print(descriptor_ps)

descriptor_moved = descriptor_ps.keys_to_properties(
    ["species_neighbor_1", "species_neighbor_2"]
)
print(descriptor_moved)

try:
    descriptor_moved.keys_to_samples("species_center")
except EquistoreError as err:
    print(err)

leading to
TensorMap with 2 blocks
keys: species_center  species_neighbor_1  species_neighbor_2
            1                 1                   1
            8                 8                   8
TensorMap with 2 blocks
keys: species_center
            1
            8
invalid parameter: can not move keys to samples if the blocks have different property labels

For SOAPPowerSpectrum the error can be solved by changing the order of keys_to_properties and keys_to_samples. For PowerSpectrum we can not do this because we already move the keys to properties within the computation to avoid nasty bookkeeping. However, it might be that we have to do this because I see no current way to solve this.
Somebody has ideas?

I could be wrong, but I believe the solution is to provide labels here:
descriptor_moved = descriptor_ps.keys_to_properties(
    ["species_neighbor_1", "species_neighbor_2"]
)

instead of a list of strings. If you provide labels with all the species you want to be represented in the properties, then the extra ones will be filled with zeros and then you'll be able to do keys_to_samples. See the documentation of keys_to_properties


Stylization and support local html

I can remove the dist files or anything else, let me know

Can you please leave the dist folder and any unrelated changes out of this?

ok @dominykas , I think this is ready for you to review again


[BUG] wrangler publish generates empty Worker

Is there an existing issue for this?

[X] I have searched the existing issues

What version of workers-rs are you using?
0.0.11
Describe the bug
When publishing a Worker the WASM binary is not included in the output, resulting in Cannot read properties of undefined (reading 'fetch').
This occurs on both MacOS and WSL2 as well as with later versions/commits of workers-rs.
The WASM binary does get created:
$ ls -lah build/worker
total 960
drwxr-xr-x  4 sebastiaan  staff   128B Feb  6 16:59 .
drwxr-xr-x  7 sebastiaan  staff   224B Feb  6 16:59 ..
-rw-r--r--  1 sebastiaan  staff   453K Feb  6 16:59 index.wasm
-rw-r--r--  1 sebastiaan  staff    21K Feb  6 16:59 shim.mjs

but not included in the output:
$ npx wrangler publish --dry-run --name foo | grep "Total Upload"
Total Upload: 21.23 KiB / gzip: 4.58 KiB

Versions:
$ cargo tree | grep worker
todo-worker v0.0.0
└── worker v0.0.11
    ├── worker-kv v0.5.1
    ├── worker-macros v0.0.6 (proc-macro)
    │   └── worker-sys v0.0.6
    └── worker-sys v0.0.6 (*)

$ rustup --version
rustup 1.25.1 (bb60b1e89 2022-07-12)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.66.0 (69f9c33d7 2022-12-12)`

$ node --version
v16.16.0

Steps To Reproduce

npm init cloudflare project_name worker-rust
npx wrangler publish --name foo (--dry-run optionally)
Check upload size: Total Upload: 21.23 KiB / gzip: 4.58 KiB
Accessing the Worker results in Cannot read properties of undefined (reading 'fetch')

I am having the same issue with my project on Windows 10.
I did some research and found out this issue doesn't exist when using the worker version 0.0.9.
Since I already wrote something about this on the Discord, I just copied and pasted what I wrote there:

Found the issue, it wasn't the command nor the template. The issue is that worker = "0.0.11" doesn't work in the templates. I just cloned the old rust template (https://github.com/cloudflare/rustwasm-worker-template) and it worked. So I tested out what changes there are to the new template (https://github.com/cloudflare/templates).
Here is the important line: https://github.com/cloudflare/templates/blob/main/worker-rust/Cargo.toml#L14 In the old templates its 0.0.9 which works.
Now its just the question why did it break with the 0.0.10? I am using Windows 10 but I could also check it out on WSL to see how it works under linux.

Instead of using worker-build directly, I rewrote it as a bash script, and ran each command individually to try and figure out the problem.
In worker-build, there is a function called use_glue_import which alters the JS generated by wasm-pack.
// Replaces the wasm import with an import that instantiates the WASM modules itself.
fn use_glue_import() -> Result<()> {
    let bindgen_glue_path = worker_path(format!("{OUT_NAME}_bg.js"));
    let old_bindgen_glue = read_file_to_string(&bindgen_glue_path)?;
    let old_import = format!("import * as wasm from './{OUT_NAME}_bg.wasm'");
    let fixed_bindgen_glue = old_bindgen_glue.replace(&old_import, "import wasm from './glue.js'");
    write_string_to_file(bindgen_glue_path, fixed_bindgen_glue)?;
    Ok(())
}

It expects an import statement like import * as wasm from './{OUT_NAME}_bg.wasm' in {OUT_NAME}_bg.js, and replaces it with import wasm from './glue.js'.
With worker = "0.0.9", I can see the expected import statement in the wasm-pack generated JS file.
import * as wasm from "./index_bg.wasm";

But with worker = "0.0.10", that import statement is gone, and seemingly replaced with this:
let wasm;
export function __wbg_set_wasm(val) {
  wasm = val;
}

As for why the import statement is no longer there - no clue! Probably a dependency update somewhere?

I did some research on GitHub, and it seems like this: __wbg_set_wasm function was added to wasm-bindgen on the 17th November 2022: https://github.com/rustwasm/wasm-bindgen/commit/721c86c3467750bf2a46bf85941e6c211ae9e87a and this repo just happens to be one of the dependencies for workers-rs and more: https://github.com/rustwasm/wasm-bindgen/network/dependents?owner=cloudflare&dependent_type=REPOSITORY&owner=cloudflare
So, I guess it's probably from somewhere around there. My best guess would be that this method should be called by *something* on cloudflares end / from the workers, but I might just be entirely wrong, since I don't work with JS at all.

Opened a PR with the simple/naive solution, changing the replacement format to match the newer wasm-bindgen import.

Until that PR gets merged, a short-term solution for the TypeError: Cannot read properties of undefined (reading 'fetch') issue is to change the build command in wrangler.toml to use my forked version of worker-build
[build]
command = "cargo install --git https://github.com/CathalMullan/workers-rs worker-build && worker-build --release"

Any chance worker-build version 0.0.9 gets released soon, so we no longer have to use https://github.com/CathalMullan/workers-rs or workers version 0.0.9?

Any chance worker-build version 0.0.9 gets released soon, so we no longer have to use https://github.com/CathalMullan/workers-rs or worker version 0.0.9?

I plan on cutting a new workers-rs release tomorrow or Monday, depending on how some testing goes.

Awesome, looking forward to it! 😄

Still happening for me using the doc https://developers.cloudflare.com/browser-rendering/get-started/screenshots/
but the fetch error is still present


Add ability for runner to handle missing objects or lists

The runner should have the ability to add patches for when a plugin has a patch for a object or list that the original object does not contain.
This should remove the ability for the applier to have to handle new annotations

Duplicate of #4


Update Slack invitation link with new link to form

Describe the bug
Update Slack invitation link with new link to form as Slack links can become deprecated.
Replace all instances of old Slack invite link with new invite link to form.
https://universelabs-slackin.herokuapp.com/

[x] universe.engineering.
[x] GitHub repository README.md files.
[x] Once URL resolves, create a new issue to update again in Sprint 10. Done https://github.com/universelabs/universe/issues/49.

References

Map Slack invite URL slack.universe.engineering to Slackin URL
Build Slack invite form
Fix Slack invitation link as it's no longer working

Done


Move Trees question to Trees Folder

This issue aims at moving trees questions to trees folder

@amritansh22 I would like to take this up.

Yes sure. Have assigned it to you


MX: Duplicate buyers

The website indicates 473 buyers: https://open-contracting.health/country/mexico/buyers
The spreadsheet has less than 300: https://docs.google.com/spreadsheets/d/1IlrAa5BGyKMNa_kh1oeSiBKi5InEKWrTT37YkcVZUdQ/edit#gid=1323249922
There are at least some duplicates like:
https://open-contracting.health/buyers/404
https://open-contracting.health/buyers/815
They have the same buyer ID (009J2Y002), though the names are slightly different. The code seems to perform lookups on both name and ID ("OR" logic), so I don't know how the duplicates could have been introduced.
cc @VTarnay1

Aha, the old method of loading Excel directly (save_tender_excel_to_db) used OR logic:
supplier_obj = Supplier.objects.filter(Q(supplier_name=supplier_name) | Q(supplier_id=supplier_id)).first()

The new method (import_tender_from_batch_id) uses AND logic:
supplier_obj = Supplier.objects.filter(supplier_id=supplier_id, supplier_name=supplier_name).first()

So, if two suppliers have the same ID but different names, then they are duplicated.

Another issue:
PODER's mapping uses "Institución" from CompraNet as the buyer name, which identifies the overall organization, not the specific buyer, which is in "Nombre de la UC". However, the organization has no ID, so PODER uses "Clave de la UC" as the buyer ID. In other words, the name is the overall organization but the ID is the specific buyer. Some examples:

Institución
Clave de la UC
Nombre de la UC

Instituto Mexicano del Seguro Social
050GYR024
IMSS-ESTADO DE MEXICO PONIENTE #050GYR024

Instituto Mexicano del Seguro Social
050GYR006
IMSS-Coordinación de Abastecimiento y Equipamiento #050GYR006

We should either:

Empty the Buyer ID column. The current ID doesn't actually identify the organization in the Buyer column, so it will just lead to duplicate buyers. This is the simplest option.
Use "Nombre de la UC" for the Buyer column. In some cases, this includes the name of a hospital "IMSS-UMAE Hospital de Especilidades No.71 Depto de Abastecimiento #050GYR045", but most of the time it's an acronym (that users will not recognize) followed by the particular name of the procurement department, e.g. "INCMNSZ-Subdirección de Recursos Materiales y Servicios Generales #012NCG001". Ideally, we would replace the acronym and remove the "#ID" at the end. Then, it would be specific and readable.

@VTarnay1 What do you prefer?

I think the way to address this is in the source data (if needed), so closing.


Add programming port

Should breakout

3v3
GND
DIO0
TX
RX

Add debugging with a programming port, maybe?

Debugging like JTAG or SWD?
The port I've added is the serial port with reset and DIO0 (clock pin?) which allows it to be programmed and read serial output.

I was thinking JTAG if possible. The newer ones I'm ordering have jtag and programming over USB built in, idk about the current one.


Add module support

Modules are a nice C++20 feature but the support of the compilers and tooling can still be buggy.
It would be nice, if there is a module support who those who wants to try, but also, the old header includes should not break.
This PR adds module support while retaining the headers.

could maybe be fixed with std module support, but this is not supported yet
https://cmake.org/cmake/help/latest/manual/cmake-cxxmodules.7.html

Support for import std is limited to the following toolchain and standard library combinations:

Clang 18.1.2 and newer with -stdlib=libc++

MSVC toolset 14.36 and newer (provided with Visual Studio 17.6 Preview 2 and newer)


Research: Reward users for inviting others

Expected Behavior
Research how to determine that a user invited another user onto the platform. (valora in this case?)
Eligibility

Sender address must be verified
Sender address must invite someone else via phone number by sending them at least XX cUSD (tbd if minimum amount)
Receiver phone number must not be already on the Celo platofrm
Receiver must confirm phone number

https://www.notion.so/clabsco/CELO-rewards-9d31cf64f82f4f1e9a3894fdc6672924?p=e85e2a0afe8847a198ee4fcadc8ff762
Current Behavior
What is the current behavior?

Design Doc:
https://docs.google.com/document/d/1FgglpekEgRiEC1rOdJWMg88uRDV9N41pwwCezCbJyyY/edit?usp=sharing

@carterqw2 posted suggested plan for feedback here: https://celo-org.slack.com/archives/CP5V8KF51/p1614118046078800?thread_ts=1613510571.197600&cid=CP5V8KF51


Workspace not recognized after upgrading to 7.1.2

After upgrading to 7.1.2, DBeaver no longer recognizes my workspace. My preferences are gone, my db connections are gone, my open scripts are gone. I checked the folder where these should be saved (C:\Users\xxxxxx\AppData\Roaming\DBeaverData\workspace6\General) and everything seems to be there. Scripts are in the \Scripts subfolder and connections are in .dbeaver. However, DBeaver doesn't seem to "see" them. Except it does. I tried File > Switch Workspace > Other and the workspace there is the exact folder I mentioned above. In fact, if I check any of the 3 options in the "Copy Settings" section below, DBeaver warns me that the target workspace already exists and files will be overwritten. So if it knows workspace6 is there, why won't it show me the connections, scripts, etc.?
I tried upgrading to the early access 7.1.3 version with no success. No change.
Is this a known bug?

Hi @razor1973 , what version do you use before upgrading?
Is there something in the log?

I was on either 7.1.0 or 7.1.1, don't recall now. This is the end of my log file, which I assume is C:\Users\rgutier1\AppData\Roaming\DBeaverData\workspace6.metadata.log (please let me know otherswise):. I see a mention to the workspace folder:
`!ENTRY org.eclipse.e4.ui.workbench 4 0 2020-07-14 08:54:39.629
!MESSAGE Unable to load resource file:/C:/Users/rgutier1/AppData/Roaming/DBeaverData/workspace6/.metadata/.plugins/org.eclipse.e4.workbench/workbench.xmi
!STACK 0
org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$1DiagnosticWrappedException: org.xml.sax.SAXParseExceptionpublicId: file:/C:/Users/rgutier1/AppData/Roaming/DBeaverData/workspace6/.metadata/.plugins/org.eclipse.e4.workbench/workbench.xmi; systemId: file:/C:/Users/rgutier1/AppData/Roaming/DBeaverData/workspace6/.metadata/.plugins/org.eclipse.e4.workbench/workbench.xmi; lineNumber: 1; columnNumber: 1; Premature end of file.
at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.handleDemandLoadException(ResourceSetImpl.java:319)
at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLoadHelper(ResourceSetImpl.java:278)
at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getResource(ResourceSetImpl.java:406)
at org.eclipse.e4.ui.internal.workbench.ResourceHandler.getResource(ResourceHandler.java:287)
at org.eclipse.e4.ui.internal.workbench.ResourceHandler.loadResource(ResourceHandler.java:263)
at org.eclipse.e4.ui.internal.workbench.ResourceHandler.loadMostRecentModel(ResourceHandler.java:169)
at org.eclipse.e4.ui.internal.workbench.swt.E4Application.loadApplicationModel(E4Application.java:384)
at org.eclipse.e4.ui.internal.workbench.swt.E4Application.createE4Workbench(E4Application.java:260)
at org.eclipse.ui.internal.Workbench.lambda$3(Workbench.java:579)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:338)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:557)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:154)
at org.jkiss.dbeaver.ui.app.standalone.DBeaverApplication.start(DBeaverApplication.java:233)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:137)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:107)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:401)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:657)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:594)
at org.eclipse.equinox.launcher.Main.run(Main.java:1447)
Caused by: org.xml.sax.SAXParseExceptionpublicId: file:/C:/Users/rgutier1/AppData/Roaming/DBeaverData/workspace6/.metadata/.plugins/org.eclipse.e4.workbench/workbench.xmi; systemId: file:/C:/Users/rgutier1/AppData/Roaming/DBeaverData/workspace6/.metadata/.plugins/org.eclipse.e4.workbench/workbench.xmi; lineNumber: 1; columnNumber: 1; Premature end of file.
at java.xml/com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
at java.xml/com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(Unknown Source)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(Unknown Source)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(Unknown Source)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(Unknown Source)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(Unknown Source)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source)
at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
at java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)
at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)
at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(Unknown Source)
at org.eclipse.emf.ecore.xmi.impl.XMLLoadImpl.load(XMLLoadImpl.java:175)
at org.eclipse.emf.ecore.xmi.impl.XMLResourceImpl.doLoad(XMLResourceImpl.java:261)
at org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(ResourceImpl.java:1563)
at org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(ResourceImpl.java:1342)
at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLoad(ResourceSetImpl.java:259)
at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLoadHelper(ResourceSetImpl.java:274)
... 23 more
!ENTRY org.eclipse.e4.ui.workbench 4 0 2020-07-14 08:54:39.636
!MESSAGE The persisted application model has no top-level window. Reinitializing with the default application model.
!STACK 0
java.lang.Exception
at org.eclipse.e4.ui.internal.workbench.ResourceHandler.loadMostRecentModel(ResourceHandler.java:174)
at org.eclipse.e4.ui.internal.workbench.swt.E4Application.loadApplicationModel(E4Application.java:384)
at org.eclipse.e4.ui.internal.workbench.swt.E4Application.createE4Workbench(E4Application.java:260)
at org.eclipse.ui.internal.Workbench.lambda$3(Workbench.java:579)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:338)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:557)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:154)
at org.jkiss.dbeaver.ui.app.standalone.DBeaverApplication.start(DBeaverApplication.java:233)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:137)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:107)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:401)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:657)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:594)
at org.eclipse.equinox.launcher.Main.run(Main.java:1447)
`

Could you try to reset workspace? https://github.com/dbeaver/dbeaver/wiki/Reset-workspace

Could you try to reset workspace? https://github.com/dbeaver/dbeaver/wiki/Reset-workspace

I closed DBeaver, erased the workbench.xmi file, reopened DBeaver... Nothing. Connections and scripts aren't there.
I then closed DBeaver again, dropped the entire .metadata folder, reopened DBeaver... Nothing either. Closed DBeaver again.
I want to clarify files .data-sources.json, data-sources.json, credentials-config.json and project-metadata.json are still under C:\Users\rgutier1\AppData\Roaming\DBeaverData\workspace6\General\.dbeaver. It's just DBeaver doesn't seem to be using them.
I then went to get the log file (.log) in the same location where I found it before (the .metadata folder) and it isn't there. The only file there is dbeaver-debug.log. These are its contents in case it helps (I don't believe it does).
> Start Eclipse IDE UI Application [org.eclipse.ui.ide.application 1.3.600.v20200116-2127]
> Start Eclipse e4 Workbench SWT [org.eclipse.e4.ui.workbench.swt 0.14.900.v20200213-1442]
> Start Eclipse Application Services [org.eclipse.e4.core.services 2.2.200.v20200127-0814]
> Start Eclipse Contexts [org.eclipse.e4.core.contexts 1.8.400.v20191217-1710]
> Start Eclipse UI Application Services [org.eclipse.e4.ui.services 1.3.700.v20190930-1643]
> Start EMF Common [org.eclipse.emf.common 2.18.0.v20191225-1014]
> Start EMF Ecore [org.eclipse.emf.ecore 2.21.0.v20200127-1342]
> Start EMF XML/XMI Persistence [org.eclipse.emf.ecore.xmi 2.16.0.v20190528-0725]
> Start Eclipse UI Dependency Injection [org.eclipse.e4.ui.di 1.2.800.v20200128-0855]
> Start Eclipse e4 Workbench Add-ons [org.eclipse.e4.ui.workbench.addons.swt 1.3.800.v20191212-1231]
> Start Expression Language [org.eclipse.core.expressions 3.6.700.v20200212-1751]
> Start Eclipse e4 Workbench SWT Renderer [org.eclipse.e4.ui.workbench.renderers.swt 0.14.1100.v20200217-1217]
> Start Eclipse Bindings Support [org.eclipse.e4.ui.bindings 0.12.800.v20191216-0805]
> Start Eclipse Dependency Injection Extensions [org.eclipse.e4.core.di.extensions 0.15.500.v20200106-1259]
> Start Eclipse CSS SWT Support [org.eclipse.e4.ui.css.swt 0.13.900.v20200203-0840]
> Start Eclipse CSS SWT Theme Support [org.eclipse.e4.ui.css.swt.theme 0.12.600.v20200124-0005]
> Start DBeaver UI [org.jkiss.dbeaver.ui <IP_ADDRESS>007102026]
> Start Help System Core [org.eclipse.help 3.8.700.v20191212-1123]
> Start Equinox Provisioning Core [org.eclipse.equinox.p2.core 2.6.300.v20200211-1504]
> Start Equinox Provisioning Operations API [org.eclipse.equinox.p2.operations 2.5.700.v20191211-1713]
> Start Equinox Provisioning Platform Automatic Update Support [org.eclipse.equinox.p2.ui.sdk.scheduler 1.4.600.v20200221-0754]
> Start Equinox Provisioning Metadata [org.eclipse.equinox.p2.metadata 2.4.700.v20191213-1911]
> Start Equinox Provisioning Update Checker [org.eclipse.equinox.p2.updatechecker 1.2.300.v20200222-1600]
> Start Equinox Provisioning Engine [org.eclipse.equinox.p2.engine 2.6.600.v20200110-2121]
> Start Equinox Provisioning Metadata Repository [org.eclipse.equinox.p2.metadata.repository 1.3.400.v20191211-1528]
> Start Equinox Provisioning Repository [org.eclipse.equinox.p2.repository 2.4.700.v20200110-2121]
> Start DBeaver SQL Model [org.jkiss.dbeaver.model.sql <IP_ADDRESS>007102026]
> Start Default Text Editor [org.eclipse.ui.editors 3.13.100.v20200130-1507]
> Start Text Editor Framework [org.eclipse.ui.workbench.texteditor 3.14.100.v20200212-1049]
> Start DBeaver UI Editors - Base [org.jkiss.dbeaver.ui.editors.base <IP_ADDRESS>007102026]
> Start DBeaver UI Editors - SQL [org.jkiss.dbeaver.ui.editors.sql <IP_ADDRESS>007102026]
> Start DBeaver UI Editors - Data [org.jkiss.dbeaver.ui.editors.data <IP_ADDRESS>007102026]
> Start Equinox Provisioning UI Support [org.eclipse.equinox.p2.ui 2.5.800.v20200213-1530]
> Start DBeaver Sample Database [org.jkiss.dbeaver.ext.sample.database <IP_ADDRESS>007102026]
> Start DBeaver Tip of the day [org.jkiss.dbeaver.ext.ui.tipoftheday <IP_ADDRESS>007102026]
> Start DBeaver ER Diagram [org.jkiss.dbeaver.ext.erd <IP_ADDRESS>007102026]
> Start Team Support Core [org.eclipse.team.core 3.8.900.v20200211-0946]
> Start File Buffers [org.eclipse.core.filebuffers 3.6.900.v20200117-0616]

Any news? Thanks.

I had the same issue on both Windows 10 and Ubuntu 18.04. The only solution for me was to downgrade to 7.0.5, which was the version I was on prior to upgrading to 7.1.2.

could you check if issue is reproducible against 7.1.3?
thanks in advance for response

I had the same issue on both Windows 10 and Ubuntu 18.04. The only solution for me was to downgrade to 7.0.5, which was the version I was on prior to upgrading to 7.1.2.
I will be honest and admit I may have upgraded to 7.1.2 directly from 7.0.5 like you. If I had to bet, I'd say I was on 7.1.1, but now, I'm not 100% sure.

I had the same issue on both Windows 10 and Ubuntu 18.04. The only solution for me was to downgrade to 7.0.5, which was the version I was on prior to upgrading to 7.1.2.

I will be honest and admit I may have upgraded to 7.1.2 directly from 7.0.5 like you. If I had to bet, I'd say I was on 7.1.1, but now, I'm not 100% sure.

could you check if issue is reproducible against 7.1.3?
thanks in advance for response

Yes, I mentioned that in my original post. It's an issue in 7.1.3 too. But keep in mind I upgraded to 7.1.3 after having 7.1.2 and this issue. I upgraded to 7.1.3, made sure the files are still there and DBeaver still doesn't see/use them.

Any feedback? This is still tagged as "waiting for response" days later, so not sure this was reviewed. Thanks in advance.

if you install it from windows store. location of workspace is "C:\Users{{user}}\AppData\Local\Packages\DBeaverCorp.DBeaverCE_{{random}}\LocalCache\Roaming\DBeaverData"

cannot reproduce an issue for now.
If it is still actual for you - feel free to reopen it with additional info.


100.000 X covered region included for Freebayes variant calling?

Hi,
Because of a not finishing Freebayes analysis I noticed that a 200 bp region with 100.000 X coverage was included variant calling.
The total callable region BED record was 3.000 bp with also more coverage peaks going up to 5.000 X but the majority of the region was the expected 30 X.
Could it be that there is a bug in the BCBio part that calculates based on coverage in the BED files which regions are suitable for variant calling?
I work with a lot of different reference genomes, some custom and or in-house and with varying degrees of assembly quality. Manually creating blacklist for each reference to exclude collapsed repeat regions and centromeres is too much work.
Rather I would have this handled (more) robust by BCBio it self, based on no or excessive coverage found in the BAM / BED files.
Maybe it's also an idea to use some default or YAML supplied max coverage somewhere between 100 and 1000 X coverage?
I would also already appreciate more information about where and how the callable regions are currently calculated so I could investigate / attempt to  fix this myself.
Best Regards,
Neill

Neill;
Sorry about the issues. The logic in bcbio that tries to find these high depth regions is here:
https://github.com/chapmanb/bcbio-nextgen/blob/master/bcbio/bam/highdepth.py#L26
It should screen out any regions with more than 20x the median coverage in 250bp windows, so it seems like it should capture this.
Is this a whole genome, or capture target region? If the later, we don't try to exclude high depth regions for fear of removing regions due to capture (rather than repetitive regions).
If that's the problem and you still want to screen you could manually set coverage_interval: genome:
https://bcbio-nextgen.readthedocs.org/en/latest/contents/configuration.html#coverage-information
If it's not the problem you might want to look at the align/*-highdepth.bed and align/*-stats.yaml to see if these help explain why it misses identifying the region, and we can try to debug more. Hope this helps.

Thank you Brad for the quick reply and the information. I will investigate further.
Another option that I am considering is trying to fix this (temporary) by setting higher input thresholds for Freebayes (--min-mapping-quality 10 or 20 , --min-supporting-mapping-quality 50)  and if these are not enough maybe  --use-best-n-alleles 10.

Hi Brad,
I think I have narrowed the issue considerably down.
None of my bcbio bwa + freebayes analysis runs have a align/*-highdepth.bed or align/*-stats.yaml files.
According to the commands log sambamba depth is not executed.
grep "sambamba depth" log/bcbio-nextgen-commands.log
There is also no mention of identifying high coverage regions in the bed file.
grep  "Identify high coverage regions"  log/bcbio-nextgen-commands.log
grep  "high"  log/bcbio-nextgen-commands.log

Some runs have different types of coverage but also run were all samples have 'genome' coverage assigned don't have the expected bed and or yaml file. Or any mention of identifying high depth regions in the log file.
I am trying to add some Python print statements to the functions that should handle the calculation of the high depth regions and the subsequent removal to see what is going on.  Should these all appear in the debug.log? Or is there a better way to debug?
An example yaml file that I am using is this one. Here I am starting from aligned bam files but I have the same issue when starting from fastq files.
# Template for whole genome Illumina variant calling with FreeBayes
# This is a GATK-free pipeline without post-alignment BAM pre-processing
# (recalibration and realignment)
---
details:
  - analysis: variant2
    genome_build: MY_REFERENCE
    description:
    # to do multi-sample variant calling, assign samples the same metadata / batch
    metadata:
      batch: Project-123
    algorithm:
      aligner: false
      mark_duplicates: false
      recalibrate: false
      realign: false
      variantcaller: freebayes
      nomap_split_targets: 300
      effects: false
      tools_off:
      - gemini
      # for targetted projects, set the region
      # variant_regions: /path/to/your.bed
resources:
    freebayes:
        options: [--genotype-qualities, --min-mapping-quality 20   ]

Neill;
Sorry about the issues with this. I'm not exactly sure what is going on here as it seems like it should calculate them as long as you have genome coverage. I updated the development version with a bit more debugging which might help. Also I made a mistake in mentioning where the files should be, they'll be in:
align/sample_name/*-highdepth.bed
align/sample_name/*-highdepth-stats.yaml

(or prealign/sample_name if you're not doing alignment).
The best place to look where things are running is in log/bcbio-nextgen-debug.log (or on the console) where you'll see output like:
Processing reference #1 (chrM)
Processing reference #2 (chr22)
[2016-04-20T17:23Z] Identify high coverage regions

You can add print statements (to print to the console) or logger.info statements (to print to log/bcbio-nextgen-debug.log) when it determines whether to call high depth checks or in the run itself:
https://github.com/chapmanb/bcbio-nextgen/blob/master/bcbio/pipeline/sample.py#L185
https://github.com/chapmanb/bcbio-nextgen/blob/master/bcbio/bam/highdepth.py#L36
Thanks for helping dig into this problem.

Hi Brad,
Thank you for the additional information, also on how to debug with the logger. Very helpful.
Somehow dd.get_coverage_interval(data) retuns None in the highdepth.py script.
Therefore this statement equals to false because of the second condition (== "genome")   not being met.
https://github.com/chapmanb/bcbio-nextgen/blob/master/bcbio/bam/highdepth.py#L36
Strange thing is that I have a log statement that mentions that the coverage is genome. But this statement is after my debug statements.
So based on the order of the log statements the coverage_interval variable seems to be set after being required by the high depth script.  Maybe though this is just some async logging issue but I see it for both my two testing samples and I guess this happened for all samples in my previous larger sample sets.
The call to assigning the coverage in sample.py: postprocess_alignment also seems to be after the call for calculating the high depth regions (line 191 for setting the coverage interval comes after line 185 for calculating the high depth)  :
https://github.com/chapmanb/bcbio-nextgen/blob/master/bcbio/pipeline/sample.py#L191
But then I don't know why this would work for anybody else.
[2016-04-20T20:06Z] noden17: assigned coverage is
[2016-04-20T20:06Z] noden17: None
[2016-04-20T20:06Z] noden17: Output bed file does not exist is
[2016-04-20T20:06Z] noden17: True
[2016-04-20T20:06Z] noden17: genome coverage ==  genome is
[2016-04-20T20:06Z] noden17: False
[2016-04-20T20:06Z] noden17: file does not exist and coverage is genome statement is
[2016-04-20T20:06Z] noden17: False
[2016-04-20T20:06Z] noden17: Resource requests: ; memory: 1.00; cores: 1
[2016-04-20T20:06Z] noden17: Configuring 10 jobs to run, using 1 cores each with 1.00g of memory reserved for each job
[2016-04-20T20:06Z] noden17: Assigned coverage as 'genome' with 88.2% genome coverage and 0.0% offtarget coverage

Neill;
Thanks so much. Apologies, this is my mistake. After refactoring the coverage inference got moved after high depth calculation and this is being skipped if not specified. Sorry for the issue and for completely missing this while I was starting at the code trying to debug. I pushed a fix that will resolve this if you get the latest development and will hopefully correctly identify and skip processing for the high depth regions. Sorry again about all the work to track this down and thanks for the help in identifying it.

Hi Brad. No worries. Thank you for fixing it.


How to address non-attribute content in quickscrape

If the landing page contains:
<title>Foo bar</title>

how do I extract the element value using:
"title": {
"selector": "//title",
},

(i.e. I want an analogue to attribute)

Attribute takes two special values,  text (the default) and html. See the scraperJSON README https://github.com/ContentMine/scraperJSON/blob/master/README.md


Generic file processor - attempt 3

In progress

Hello Robin, I was a bit nervous of changing the filenames for images, for fear of broken links. I "did" only aim to fix the path for DITA files that we generate, but I realise they still receive incoming links.
So yes, let's fix filenames for files we generate as well as files we copy over, and rely on the DITA publish process to spot any holes in our logic.

Done in fe0fd97


Default created subsystem configuration is invalid

This command:
box fw1 create app MyApp subsystem new/test

produces a skeleton app with this in the FW/1 configuration:
subsystems = {
    module = ''
}

It's illegal for the configuration to be a string -- it needs to be a struct:
subsystems = {
    module = { }
}

I may have caught this in a local fix I haven't pushed yet as it looks familiar now that you raise it. I'll make time to tackle this as I wanted to make an effort to push some updates this weekend.


TOB-ARGO-TM10 Insufficient default network access controls between pods

Severity: High Difficulty: High
Type: Networking Finding ID: TOB-ARGO-TM10
Description
The namespace a rgo where pods and services reside does not enforce any ingress or egress restrictions. As a result, all pods can communicate with each other, which is unnecessary in many cases.
For example, a new component added to Argo Workflows would be able to communicate with all other components within the same cluster/namespace even if communication were unnecessary. This could facilitate lateral movement should an attacker gain access to a pod within this namespace.
Justification
The difficulty is high for the following reason:
● An attacker must first transit trust boundaries into Argo infrastructure to be able to
speak to other non–external-facing components.
The severity is high for the following reason:
● This allows lateral movement, which may enable an attacker to further compromise
the system and underlying repository credentials.
Recommendation
Short term, consider enforcing ingress and egress restrictions to allow only resources that need to speak to each other to do so. Leverage allow-lists instead of deny-lists to ensure that only expected components can establish these connections. See the Kubernetes documentation for guidance on network policies.
Long term, review exposure of sensitive secrets and services within the network. Ensure the use of appropriate methods of isolation to prevent lateral movement.

@jessesuen any input on how this might be set-up for

Not required due to lack of micro-service architecture.


how many LUT will be needed for a nonlinear test?

take an example, if 48MHz, if 10 different positions that we like to measure the phase offset, (if testing chart vs camera postion), then, should we need to have 10 x (240x320) LUT table? where is your Voxel sdk call those tables?---tnks for your answers!

Nonlinearity compensation is done by OPT9221 online.  You will need to program the 16-entry lookup table value. The LUT is phase-to-phase (using nonlinear phase to lookup ideal phase).

Hello, Larry, could you help let us know whether to get related calibration
kit???
tnks a lot!!!
Kevin Liu，Ph.D
GM, eZimage Technology Co. ltd
易積生活科技
0919377901
2016-09-05 1:36 GMT+08:00 Larry Li<EMAIL_ADDRESS>
Nonlinearity compensation is done by OPT9221 online. You will need to
program the 16-entry lookup table value. The LUT is phase-to-phase (using
nonlinear phase to lookup ideal phase).
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/3dtof/voxelsdk/issues/61#issuecomment-244616729, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AMCb1f9J-cRLaaMJTC9xObp1mO8VXNB9ks5qmwGrgaJpZM4HQOb-
.

Nonlinear calibration is part of VoxelViewer.  We don't currently have a kit to offer. Typically one can use a linear positioning stage to perform such calibration.
-Larry
Sent from my iPhone

On Sep 19, 2016, at 10:15 PM, kevinliu-ez<EMAIL_ADDRESS>wrote:
Hello, Larry, could you help let us know whether to get related calibration
kit???
tnks a lot!!!
Kevin Liu，Ph.D
GM, eZimage Technology Co. ltd
易積生活科技
0919377901
2016-09-05 1:36 GMT+08:00 Larry Li<EMAIL_ADDRESS>
Nonlinearity compensation is done by OPT9221 online. You will need to
program the 16-entry lookup table value. The LUT is phase-to-phase (using
nonlinear phase to lookup ideal phase).
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/3dtof/voxelsdk/issues/61#issuecomment-244616729, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AMCb1f9J-cRLaaMJTC9xObp1mO8VXNB9ks5qmwGrgaJpZM4HQOb-
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

Nothing to do here.


Code refactoring in WindowsSpecification

Linked issues:

https://github.com/pywinauto/pywinauto/issues/710
https://github.com/pywinauto/pywinauto/issues/807

Changes:

rename .wrapper_object() to .find(); wrapper_object marked as deprecated; WrapperObject was removed.
rename .child_window() to .by(); child_window, window marked as deprecated ChildWindow, Window, Window_, window_ were removed
Exists, Wait, WaitNot, PrintControlIdentifiers were removed

import warnings is not used any more in base_application.py. So it can be removed.

import warnings is not used any more in base_application.py. So it can be removed.

import warnings is not used any more in base_application.py. So it can be removed.

removed


Change keyboard primary button color

How to change this button color?

is there any update on this? I've been going through all of the source code and can't figure out a way to change the colors on this keyboard. Any advice?

i have a similar question- how do we change this color? I see no way to do it and yet the color in your readme pics is different than this green color

@sergiyvergun i figured it out in a hacky work-around sort of way. It defaults to the secondary theme's secondary color on the colorswatch. For me, i set mine in my main.dart file. Note that overriding the colorswatch will mess up some other color schemes (like a cupertino dialog background color) so you may have to go in and wrap those other things in themes. You may ALSO be able to wrap the MathKeyboard in a theme but i didnt try that yet


opt: use lookup for small integer in jit

PASS
ok      github.com/bytedance/sonic/generic_test 16.380s
go get golang.org/x/perf/cmd/benchstat && go install golang.org/x/perf/cmd/benchstat && benchstat /tmp/tmpw36l5eys.source.txt /tmp/tmpu980__ul.target.txt
go: added golang.org/x/perf v0.0.0-20230227161431-f7320a6d63e8
goos: linux
goarch: amd64
pkg: github.com/bytedance/sonic/generic_test
cpu: Intel(R) Xeon(R) Platinum 8260 CPU @ 2.40GHz
                                     │ /tmp/tmpw36l5eys.source.txt │      /tmp/tmpu980__ul.target.txt      │
                                     │           sec/op            │    sec/op     vs base                 │
MarshalConcrete/GolangSource_Sonic-8                   2.886m ± 3%   11.746m ± 4%  +307.02% (p=0.000 n=10)

                                     │ /tmp/tmpw36l5eys.source.txt │     /tmp/tmpu980__ul.target.txt      │
                                     │             B/s             │     B/s       vs base                │
MarshalConcrete/GolangSource_Sonic-8                  641.3Mi ± 3%   157.5Mi ± 4%  -75.43% (p=0.000 n=10)

                                     │ /tmp/tmpw36l5eys.source.txt │       /tmp/tmpu980__ul.target.txt       │
                                     │            B/op             │     B/op       vs base                  │
MarshalConcrete/GolangSource_Sonic-8                  1.859Mi ± 1%   24.200Mi ± 0%  +1202.06% (p=0.000 n=10)

                                     │ /tmp/tmpw36l5eys.source.txt │     /tmp/tmpu980__ul.target.txt      │
                                     │          allocs/op          │  allocs/op   vs base                 │
MarshalConcrete/GolangSource_Sonic-8                    4.000 ± 0%   34.000 ± 0%  +750.00% (p=0.000 n=10)
git checkout -- .
git checkout opt/encode

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.0 out of 2 committers have signed the CLA.:x: liuqiang:x: liuq19liuqiang seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Clicking on a folder should have a less strong animation in list view

What's the Problem?
Clicking on a folder in list view should have a less strong animation
Solution/Idea

The Problems

The main problem i have right now is the blur. It makes everything blurry. Blurry text looks not good.
The feel of it, it skews everything in a little bit fiddly unpredictable way
The intensity. It is just selecting a folder. As an opening animation it would probably work. But not when just selecting something.

Solution/Ideas

Make it not blur regardless of animation or not
Make it feel more productive, easier to understand
In my opinion it doesn't really need an animation at all from a ux standpoint, the selection colour is completely enough, everything else will be a little bit over the top for just a selection in my opinion. There could maybe be a cooler smooth animation when opening a folder or something but just with the selection it doesn't sit right for me

Alternatives

Priorities

Capability
Priority

Make it not blur regardless of animation or not
Must

Make selecting feel more productive, easier to understand
Should

Get rid of the zooming animation when single-clicking
Could

Files Version
Version: <IP_ADDRESS> OS Version: 10.0.19043.1110
Windows Version
Win10 Home 21H1
Additional comment
Could also have a slightly longer delay until the renaming function get's called its pretty easy to accidentally go into renaming mode (this also annoys me about the original windows explorer, i think it has a slightly longer delay even)

I also feel like the current animation is a bit confusing. Double clicking produces this zoom / blur effect twice.
Same for the renaming delay, it's too short, making it difficult to actually open the file / folder.

Agreed. Current click/double click animation is weird and confusing. For me, its feels like I'm about to drag the file.
The text shouldn't move to the right and doesn't need to be 'italic' on click. Logically it should have a color on click and doesn't need extra effect on opening.

Marging with #6259


Support URI Template RFC 6570

URI templating is very helpful and a level 1 implementation would be nice. Below is an overview of how the spec would work.

A URI Template provides both a structural description of a URI space
and, when variable values are provided, machine-readable instructions
on how to construct a URI corresponding to those values.  A URI
Template is transformed into a URI reference by replacing each
delimited expression with its value as defined by the expression type
and the values of variables named within the expression.  The
expression types range from simple string expansion to multiple
name=value lists.  The expansions are based on the URI generic
syntax, allowing an implementation to process any URI Template
without knowing the scheme-specific requirements of every possible
resulting URI.
For example, the following URI Template includes a form-style
parameter expression, as indicated by the "?" operator appearing
before the variable names.
 http://www.example.com/foo{?query,number}

The expansion process for expressions beginning with the question-
mark ("?") operator follows the same pattern as form-style interfaces
on the World Wide Web:
 http://www.example.com/foo{?query,number}
                           \_____________/
                              |
                              |
         For each defined variable in [ 'query', 'number' ],
         substitute "?" if it is the first substitution or "&"
         thereafter, followed by the variable name, '=', and the
         variable's value.

If the variables have the values
 query  := "mycelium"
 number := 100

then the expansion of the above URI Template is
 http://www.example.com/foo?query=mycelium&number=100

Alternatively, if 'query' is undefined, then the expansion would be
 http://www.example.com/foo?number=100

https://tools.ietf.org/html/rfc6570

This is rather complex and most users won't benefit from it, so it won't make it to axios. You can implement it in an interceptor. We would be glad to add it to ECOSYSTEM.md if you create it and open source it.

ok, thank you for the consideration.


RFC: Patch datasets

In dataset_description.json, we currently have two DatasetTypes: "raw" and "derivative". Here I propose an additional type "patch" (or similar). This is a particular kind of derivative that would have the interpretation that the data/metadata elements contained in the patch dataset should be added to or supersede those of the source dataset.
Consider the following use cases:

Quality control metrics can be used to exclude data from processing or select among control flow paths. These are additional metadata, but not standalone datasets.
You acquire a fieldmap for every BOLD run. Some are unusable, but all are from the same session, so you find the best fieldmap for each, and provide a new IntendedFor.

I'm sure there are others. Right now, this can be done in an ad hoc way, but a clear directive that this is how derived data is declared to augment/override raw data would help avoid inconsistencies among tools.
xref https://github.com/poldracklab/mriqc/pull/885#issuecomment-762333959

Would it be an alternative to simply update the "raw" or "derivative" dataset, and bump the version of that dataset? If yes, then I can still see how your proposal would save time and perhaps even be more transparent (and less error prone?).
However, without dedicated tooling, this would be way too challenging to handle for a majority of end users IMHO. Do you or (others you know) want to develop tools for this and need support for this in the spec first?

The idea is transparency as it is a kind of derivative dataset, and you could choose another.
It would not be difficult to write a tool to make a view of a dataset with a patch applied.

Briefly mentioned with @christinerogers in context of third-party annotations, where permissions to modify the annotated dataset are limited.
If there's broader interest, this could be a discussion in Copenhagen. Not sure how much flexibility there is for adding topics. cc @melanieganz @CPernet

from a software point of view, I think this could be best represented as another derivative layer, but the concept/ label for it is valuable.

Could you clarify what you mean by "another derivative layer"? I feel like you understood me, but I don't understand the distinction between this and my post.

we can discuss what we want, but annotation is not high on the list of priorities

Note: For a complete "patch" semantic it should also reserve ways to "delete" any file/entry (in .tsv, .json, etc), which IMHO wouldn't be quite trivial if reasonably possible at all.
Overall, probably the task could be solved in a data management solution specific way, e.g. via git/git-annex on top of the forks of e.g. https://github.com/OpenNeuroDatasets, so there could be forks with desired fixes of any kind and there would be no need for any dedicated support of the "patch" dataset type.


useHover sets focus on reference, causing focus indicator to appear

Reproduction: https://codesandbox.io/s/cool-framework-yzkhwh?file=/src/DropdownMenu.tsx
Video just in case:
https://user-images.githubusercontent.com/1892091/205591138-78b7eaa4-1cfa-4c7d-8722-1c08acfda334.mp4
In the video, in the beginning hovering the reference button works as expected, e.g. there is no visible focus indicator. Once I tab to the button at least once and close the menu, after that when hovering the button the focus indicator is shown.
Note that sometimes it does not even need the tabbing, it just shows the focus indicator when hovering.
This also does not seem to be related to the use of useFocus hook, i.e. if I only use useHover, it still behaves like this.

Try with these props instead:
<FloatingFocusManager
  context={context}
  modal={false}
  initialFocus={-1}

I also updated the dropdown menu example in general (on the docs) which works much better

What type of component is this exactly? It doesn't follow WAI-ARIA authoring if it's trying to be a dropdown menu

Pehaps a more realistic example based on my actual needs, I made the buttons links and added multiple of them, so now it should represents a navigation menu more: https://codesandbox.io/s/loving-aj-12v6cc?file=/src/DropdownMenu.tsx

Got it, I'll have to investigate it

@atomiks one scenario when I do want to return focus is when the user has tabbed into the floating element and then closes it via ESC key. If I set returnFocus={false}, it loses the focus completely.

Hmm true, I guess you'd need to conditionally set returnFocus based on some state then


Update Issue - 33711500

default description
URL tested: http://ci.bsstag.com/welcome
Open URL on Browserstack

Property
Value

Browser
Chrome 75.0

Operating System
Windows 8.1

Resolution
1024x612

Screenshot Attached
Screenshot URL

Click here to reproduce the issue on Browserstack

Updated Screenshot Attached