Pylance doesn't like the import from the root module

Python: 3.12
Editor: VSCode 1.88.0 (latest as of 2024-04-17)
pydantic-apply: 0.5.7
OS: macOS 14.4.1
Usually I would import the ApplyModelMixin with from pydantic_apply import ApplyModelMixin.
However, Pylance doesn't like that and shows an error:
"ApplyModelMixin" is not exported from module "pydantic_apply"
  Import from "pydantic_apply.apply" instead

with a link tp reportPrivateImportUsage
A similar issue was reported for pytorch with an explanation of this behavior in this comment.
And in fact, if I add from .apply import ApplyModelMixin as ApplyModelMixin to the __init__ of the module it works.
From PEP 484 about stub files:

Modules and variables imported into the stub are not considered exported from the stub unless the import uses the import ... as ... form or the equivalent from ... import ... as ... form. (UPDATE: To clarify, the intention here is that only names imported using the form X as X will be exported, i.e. the name before and after as must be the same.)

Would it make sense to add this line?
I can open a pull request, if that helps.

I opened an according PR: https://github.com/team23/pydantic-apply/pull/18

Merged your PR just now. Will create a new patch level release containing this change.


MD023 (Headers must start at the beginning of the line) should be ignored within code blocks

Consider the following:
- Something something:

  ```rb
  # A great comment
  foo = bar
  ```

This will trigger the following:
[...]: MD023 Headers must start at the beginning of the line

In Ruby, Python, YAML, Git config, etc., comments start with a #, so rule MD023 should be disabled in code blocks.

I'm not able to reproduce this in the current master of markdownlint. However, I did find a related issue that is fixed in a0cb406 (and should also fix the issue you're seeing if it turns out I was reproducing it wrong somehow). I have also added your issue as a test case so it should be caught in tests if it does show up again.

I can reproduce this exact issue on 0.4.0 and on master. This test suite passes with no failures, but if I run:
bin/mdl test/rule_tests/headers_with_spaces_at_the_beginning.md

I get:
test/rule_tests/headers_with_spaces_at_the_beginning.md:3: MD023 Headers must start at the beginning of the line
test/rule_tests/headers_with_spaces_at_the_beginning.md:5: MD023 Headers must start at the beginning of the line
test/rule_tests/headers_with_spaces_at_the_beginning.md:8: MD023 Headers must start at the beginning of the line


Just to remind us to update the lib side to remove this .first, .second, etc

          Just to remind us to update the lib side to remove this `.first`, `.second`, etc

Originally posted by @joaoantoniocardoso in https://github.com/bluerobotics/navigator-lib/pull/3#discussion_r1189138687

Closed by #4


HttpMethod becomes empty when request hit the server

Updated .net core service from 2.1.7 to 2.1.11, after upgrade we are seeing random tests failure. Reason for failures is HttpMethod becomes empty string after request hits the server.
Verified with proxy server, HttpMethod is correctly sent from the tests. However, when the request hits the server we receive HttpMethod as empty string, RANDOMLY. Sometime all the tests pass, sometime we see failures.
This issue only occurs when we deploy service on AWS, when service and tests are run locally all the test case passes.
On AWS we are running service over .net core 2.1.11 docker image.
what could be the cause of this issue? what can we do to fix it?

Can you reproduce this in a repository? The issue seems very vague.

Are you accessing HttpContext from outside of the middleware pipeline (or an MVC controller action, etc.)? That's generally what causes this kind of issue.
Please provide a runnable sample application so we can get the full context on the issue.

Closing this as we haven't heard from you and generally close issues with no response after some time. Please feel free to comment if you're able to get the information we're looking for and we can reopen the issue to investigate further!

@prabalk this sounds like an old issue where a user was accessing the HttpContext after the request was complete via IHttpContextAccessor.


Update onboarding guide after the switch to Alderaan

Description
We have switched from the Red Eyes to the Alderaan contracts.
Therefore the current onboarding guide (https://github.com/raiden-network/light-client#try-out-the-raiden-demo-dapp) is not working anymore.
CC @taleldayekh
Acceptance criteria

A Goerli test node is provided
The guide is updated to work with Goerli

Tasks

[ ]

This has been fixed so I will close the issue.


tls: improve wildcard matching (#11921)

Patching in 11885 with runtime guards and release notes
Risk Level: Medium (changes to cert matching)
Testing: new unit test
Docs Changes: n/a
Release Notes: inline
Runtime guard: envoy.reloadable_features.fix_wildcard_matching
Signed-off-by: Yann Soubeyrand<EMAIL_ADDRESS>Signed-off-by: Alyssa Wilk<EMAIL_ADDRESS>Signed-off-by: Piotr Sikora<EMAIL_ADDRESS>

/azp run


React Native - can't interact with component

Hi, Ive created a demo project with React Native, I got the component working but if I put it on a view or a button I am unable to interact with said component

@WickednessPro Hello, you can send example of code in react-native?

Sure @Stampoo please find the example below! It just won't allow the button to be pressed however it is protected from screenshots. the same occurs when it's on a text input field. I am using the master branch of SnapshotSafeView
import React from 'react';
import {
  SafeAreaView,
  ScrollView,
  StatusBar,
  StyleSheet,
  Text,
  useColorScheme,
  View,
  requireNativeComponent,
  TextInput,
  Button,
  Alert,
} from 'react-native';

const SnapshotSafeView = requireNativeComponent(
  'RCTReactNativeBridgeSnapshotSafeView',
);

function App(): JSX.Element {
    const isDarkMode = useColorScheme() === 'dark';
    const [text, onChangeText] = React.useState('');

    const backgroundStyle = {
      backgroundColor: isDarkMode ? '#111827' : '#111827',
    };
    return (
      <SafeAreaView style={{backgroundColor: '#111827', height: '100%'}}>
        <StatusBar
          barStyle={isDarkMode ? 'light-content' : 'dark-content'}
          backgroundColor={backgroundStyle.backgroundColor}
        />
        <View
          style={styles.centered}>
          <SnapshotSafeView>
            <Button
              title="Test"
              onPress={() => Alert.alert('Simple Button pressed')}
              color="#fff"
            />
          </SnapshotSafeView>
        </View>
      </SafeAreaView>
    );
  }

Any updates? @Stampoo

@WickednessPro
Yes, sorry. Now I am in the process of moving, which will end by the end of the week. After there will be a result :)

@WickednessPro Hello, you can check from https://github.com/Stampoo/SnapshotSafeView/tree/master

Hi @Stampoo thank you so much it works perfectly!


findOneAndUpdate with return new document is not atomic

Using findOneAndUpdate with returnNewDocument may return the current version of a document as updated by another call - it should return the document as modified by the current request.
We noticed this using $inc of 1 for a sequence. Sometimes it would return the same value, as the returnNewDocument is likely waiting till the next event loop.
So this presently breaks the ability to mock auto increment.

Sorry - wrong project! Closing.


Workflows fail when a default executor is not available, even if it has been overwritten in the workflow task

I'm trying to run the cellpose_segmentation task on the FMI setup which has no gpu (and therefore, we don't have the gpu specified as a possible option of executors).
I used the new ability to edit the executor of a workflow task to cpu-low, which is available. But the workflow still fails early on with this error message:
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 404, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/fastapi/applications.py", line 269, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/starlette/exceptions.py", line 93, in __call__
    raise exc
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
    await self.app(scope, receive, sender)
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/starlette/routing.py", line 670, in __call__
    await route.handle(scope, receive, send)
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/starlette/routing.py", line 266, in handle
    await self.app(scope, receive, send)
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/starlette/routing.py", line 68, in app
    await response(scope, receive, send)
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/starlette/responses.py", line 165, in __call__
    await self.background()
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/starlette/background.py", line 43, in __call__
    await task()
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/starlette/background.py", line 26, in __call__
    await self.func(*self.args, **self.kwargs)
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/fractal_server/app/runner/__init__.py", line 113, in submit_workflow
    output_dataset.meta = await process_workflow(
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/fractal_server/app/runner/_parsl/__init__.py", line 211, in process_workflow
    final_task_pars_future = recursive_task_assembly(
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/fractal_server/app/runner/_parsl/__init__.py", line 150, in recursive_task_assembly
    task_pars_depend_future = recursive_task_assembly(
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/fractal_server/app/runner/_parsl/__init__.py", line 150, in recursive_task_assembly
    task_pars_depend_future = recursive_task_assembly(
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/fractal_server/app/runner/_parsl/__init__.py", line 150, in recursive_task_assembly
    task_pars_depend_future = recursive_task_assembly(
  [Previous line repeated 1 more time]
  File "/opt/fmi/gliberal/fractal/miniconda3/envs/fractal-server-v1-a15/lib/python3.8/site-packages/fractal_server/app/runner/_parsl/__init__.py", line 141, in recursive_task_assembly
    raise ValueError(msg)
ValueError: this_task.executor='gpu' is not in data_flow_kernel.executors.keys()=dict_keys(['cpu-low', 'cpu-mid', 'cpu-high', '_parsl_internal'])

When checking the workflow, the relevant workflow task looks like this:
   {
      "workflow_id": 2,
      "task_id": 5,
      "order": 5,
      "meta": null,
      "args": {
        "args": {
          "labeling_channel": "A01_C01",
          "labeling_level": 2,
          "ROI_table_name": "well_ROI_table"
        },
        "meta": {
          "executor": "cpu-low"
        }
      },
      "id": 16,
      "task": {
        "name": "Cellpose Segmentation",
        "command": "/opt/fmi/gliberal/fractal/fractal-demos/examples-v1/server/FRACTAL_ROOT/.fractal/fractal-tasks-core0.4.6/venv/bin/python /opt/fmi/gliberal/fractal/fractal-demos/examples-v1/server/FRAC
TAL_ROOT/.fractal/fractal-tasks-core0.4.6/venv/lib/python3.8/site-packages/fractal_tasks_core/cellpose_segmentation.py",
        "source": "pip:fractal-tasks-core==0.4.6",
        "input_type": "zarr",
        "output_type": "zarr",
        "default_args": {
          "labeling_channel": "A01_C01"
        },
        "meta": {
          "executor": "gpu",
          "parallelization_level": "image"
        },
        "id": 5
      }
    },

It appears the recursive_task_assembly of parsl uses the default executor, not the one provided to the workflow task. If this is just a parsl issue, then the new slurm executors should solve it. But would be good to have a test in place to check that a task that defaults to have a gpu executor can be used in a setup without gpu executors when the executor is overwritten in the workflow task to something that is available.

Yes, this is a bug. In principle we should be loading the WorkflowTask.executor property (see https://github.com/fractal-analytics-platform/fractal-server/blob/main/fractal_server/app/runner/_parsl/init.py#L133), which is correctly set to use "executor": "cpu-low" in your case.
I cannot say whether the new backend fixes it, but for sure a test is due.

When we define a project&workflow (for server 1.0.0b1 and slurm backend) as in
LABEL="2"

###############################################################################
# THINGS TO BE CHANGED BY THE USER
# Adapt to settings to where you run the example
BASE_FOLDER_EXAMPLE=`pwd`/..
###############################################################################

# Set useful variables
PRJ_NAME="proj-$LABEL"
DS_IN_NAME="input-ds-$LABEL"
DS_OUT_NAME="output-ds-$LABEL"
WF_NAME="Workflow $LABEL"

# Set cache path and remove any previous file from there
export FRACTAL_CACHE_PATH=`pwd`/".cache"
rm -rv ${FRACTAL_CACHE_PATH}

# Define/initialize empty project folder and temporary file
PROJ_DIR=`pwd`/tmp_${LABEL}
rm -r $PROJ_DIR
mkdir $PROJ_DIR

###############################################################################
# IMPORTANT: modify the following lines so that they point to absolute paths
INPUT_PATH=$BASE_FOLDER_EXAMPLE/images/10.5281_zenodo.7059515
OUTPUT_PATH=${PROJ_DIR}/output
###############################################################################

# Create project
OUTPUT=`fractal --batch project new $PRJ_NAME $PROJ_DIR`
PRJ_ID=`echo $OUTPUT | cut -d ' ' -f1`
DS_IN_ID=`echo $OUTPUT | cut -d ' ' -f2`
echo "PRJ_ID: $PRJ_ID"
echo "DS_IN_ID: $DS_IN_ID"

# Update dataset name/type, and add a resource
fractal dataset edit --name "$DS_IN_NAME" -t image --read-only $PRJ_ID $DS_IN_ID
fractal dataset add-resource -g "*.png" $PRJ_ID $DS_IN_ID $INPUT_PATH

# Add output dataset, and add a resource to it
DS_OUT_ID=`fractal --batch project add-dataset $PRJ_ID "$DS_OUT_NAME"`
echo "DS_OUT_ID: $DS_IN_ID"

fractal dataset edit -t zarr --read-write $PRJ_ID $DS_OUT_ID
fractal dataset add-resource -g "*.zarr" $PRJ_ID $DS_OUT_ID $OUTPUT_PATH

# Create workflow
WF_ID=`fractal --batch workflow new "$WF_NAME" $PRJ_ID`
echo "WF_ID: $WF_ID"

# Mapping of tasks to their IDs (until we add the by-name addressing)
#    "Create OME-ZARR structure": 1,
#    "Yokogawa to Zarr": 2,
#    "Replicate Zarr structure": 3,
#    "Maximum Intensity Projection": 4,
#    "Cellpose Segmentation": 5,
#    "Measurement": 6,
#    "Illumination correction": 7,
#    "Napari workflows wrapper": 8,
#    "Create OME-ZARR structure (multiplexing)": 9

# Add tasks to workflow
fractal workflow add-task $WF_ID 1 --args-file Parameters/args_create_ome_zarr.json
fractal workflow add-task $WF_ID 2
fractal workflow add-task $WF_ID 3
fractal workflow add-task $WF_ID 4
fractal workflow add-task $WF_ID 5 --args-file Parameters/args_cellpose_segmentation.json

# Look at the current workflows
fractal workflow show $WF_ID
echo

and then modify a workflow task as
$ fractal workflow edit-task 1 5  --json-file Parameters/cellpose_segmentation.json

where
$ cat Parameters/cellpose_segmentation.json
{
   "args": {
        "labeling_level": 2
    },
    "meta": {
        "executor": "cpu-low"
    }
}

and if we then apply the workflow with #fractal workflow apply -o $DS_OUT_ID -p $PRJ_ID $WF_ID $DS_IN_ID , then the task is submitted to the correct executor. This we can confirm by looking at the partition field in the submission script
#!/bin/sh
#SBATCH --output=/net/nfs4/pelkmanslab-fileserver-common/data/homes/fractal/fractal-demos/examples-v1/server/artifacts/workflow_000001_job_000001/slurmpy.stdout.%j.log
#SBATCH --partition=main
#SBATCH --mem=1g
#SBATCH --nodes=1
#SBATCH --job-name Cellpose_Segmentation
export CFUT_DIR=/net/nfs4/pelkmanslab-fileserver-common/data/homes/fractal/fractal-demos/examples-v1/server/artifacts/workflow_000001_job_000001
srun /data/homes/fractal/.conda/envs/fractal-server-v1/bin/python -m cfut.remote FMwWTeSpyEyLiXQBH7ZxIG2xf8EfmcSx

This suggests that the current version does not have this bug any more.

Should it make a difference whether we have the executor in the parameter of the task when it's added or when it's modified? Why did you add it with the parameters and then modify it with the same parameters above?

I then manually did the fractal workflow apply -o 2 -p 1 1 1 --worker-init "$WORKER_INIT" and now my workflow task looks like this:
{
  "workflow_id": 1,
  "task_id": 5,
  "order": 4,
  "meta": {
    "executor": "cpu-low"
  },
  "args": {
    "args": {
      "labeling_channel": "A01_C01",
      "labeling_level": 3,
      "ROI_table_name": "well_ROI_table",
      "label_name": "nuclei"
    },
    "meta": {
      "executor": "cpu-low"
    },
    "labeling_channel": "A01_C01",
    "labeling_level": 3,
    "ROI_table_name": "well_ROI_table",
    "label_name": "nuclei"
  },
  "id": 5,
  "task": {
    "name": "Cellpose Segmentation",
    "command": "/opt/fmi/gliberal/fractal/fractal-demos/examples-v1/server/FRACTAL_ROOT/.fractal/fractal-tasks-core0.5.1/venv/bin/python /opt/fmi/gliberal/fractal/fractal-demos/examples-v1/server/FRACTAL_ROOT/.fractal/fractal-tasks-core0.5.1/venv/lib/python3.8/site-packages/fractal_tasks_core/cellpose_segmentation.py",
    "source": "pip:fractal-tasks-core==0.5.1",
    "input_type": "zarr",
    "output_type": "zarr",
    "default_args": {
      "labeling_channel": "A01_C01"
    },
    "meta": {
      "executor": "gpu",
      "parallelization_level": "image"
    },
    "id": 5
  }
}

Now it works. And now I understand what broke before: It put the args & meta into an "args" keyword when I used it in add-task.
So, the old way of adding both args & meta as part of adding a task doesn't work anymore then, correct? Is there a way to still do it in one as part of fractal workflow add-task?

Yes, there is a difference (maybe counterintuitive one, but there is one). Let's look at the help messages:
$ fractal workflow add-task -h
usage: fractal workflow add-task [-h] [--order ORDER] [--args-file ARGS_FILE] id task_id
positional arguments:
  id                    Workflow id
  task_id               Task id
optional arguments:
  -h, --help            show this help message and exit
  --order ORDER         Order of this task within the workflow's task list
  --args-file ARGS_FILE
                        Path to a json serialised file containing the arguments ovverrides of the task

$ fractal workflow edit-task -h
usage: fractal workflow edit-task [-h] [--json-file JSON_FILE] id workflow_task_id
positional arguments:
  id                    Workflow id
  workflow_task_id      Workflow task Id, the Id of a task inside the list of tasks
optional arguments:
  -h, --help            show this help message and exit
  --json-file JSON_FILE
                        Path to a json serialised file containing the attribute ovverrides of the task

ARGS_FILE and JSON_FILE have different meanings. The former is only for args, while the latter is both for args and meta. The first one should have no special key, while the second should have args and/or meta keys.
In fact in your logs you see:
"args": {
        "args": {
          "labeling_channel": "A01_C01",
          "labeling_level": 3,
          "ROI_table_name": "well_ROI_table",
          "label_name": "nuclei"
        },

which is incorrect. This would mean that the task has an argument called args which takes value {"labeling_channel": "A01_C01", ...}.

TL;DR
In the current version, the add-task can only modify the args, and not meta. To get it work with the current version, you should only write the args in ARGS_FILE, namely:
{
          "labeling_channel": "A01_C01",
          "labeling_level": 3,
          "ROI_table_name": "well_ROI_table",
          "label_name": "nuclei"
        }

We  can then make things more uniform

Ok, thanks!


Country change event should strip off URL hooks

Viewport should be the country bounds after change.

dun!


Update .gitmodules with submodule branch information

Changelog-entry: Update .gitmodules with submodule branch information

I self-certify!


chore(pkg/cmd/report): address style changes

Some suggestions I got from @jadoonf

[x] I have read the contributing guidelines
[x] I have followed the coding guidelines
[x] I have written unit tests
[x] I have made sure that the pull request is of reasonable size and can be easily reviewed

LGTM

I need an approval for this @leodido


Uml 3297 GOVUK One Login header

Purpose
Migrates service front to the new One Login header when the allow_gov_one_login feature flag is active.
Fixes UML-3297
Checklist

[x] I have performed a self-review of my own code
I have added relevant logging with appropriate levels to my code
New event_codes have been documented on the wiki page
I have updated documentation (Confluence/GitHub wiki/tech debt doc) where relevant
[x] I have added tests to prove my work
[x] I have added welsh translation tags and updated translation files
[x] I have run an accessibility tool on any pages I have made changes to and fixed any issues found
[ ] I have notified the Interaction Designer of any content changes so that appropriate screenshots/flow diagram changes can be made
[ ] The product team have tested these changes

Headers currently broken due to the frontend upgrade.


Disable "download" button and show message when too many/few tiles in selected area

The "download" button on the offline area selector map should only be enabled when there are tiles to download, and when the resulting down is below a certain size. Toasts should be shown to indicate each respective state:

"Selected area to large. Zoom in further to download a smaller area."
"No imagery available for the selected area."

@amegantz for exact CTA text.

First toast options:

"Select a smaller area to download" (preferred)
"Zoom in to download a smaller area"

Second toast:
"No imagery to download in the selected area"


use futures::Poll (or clone of it) instead of nb::Error

Prefix and Apology
I went down the wrong track in #2 and japaric/embedded-hal#13. My error was because I was not familiar enough with the inner workings of futures and I used the incorrect terminology.
I do NOT think we should return a Future, but rather that our asyncronous functions should return a type identical to futures::Poll (or possibly futures::Poll itself)
Issue at hand
I suggest that although this is not futures, we should try and keep the API as similar as possible to futures. Future::poll returns the type Poll which is just Result<Async<T>, E>. See: Async.
These types are mathematically identical:
futures:
type Poll = Result<Async<T>, E>;

pub enum Async<T> {
    Ready(T),
    NotReady,
}

nb:
type MyResult = Result<T, nb::Error<E>>;
enum Error<E> {
    Other(E),
    WouldBlock,
}

We have simply moved the WouldBlock/NotReady around from Ok to Err.
However, Result<V, nb::Error<E>> is much less ergonomic for a function that is supposed to be asyncronous:

When someone creates a function that is non blocking, they should be returning a type that specifies the function as non-blocking. That type is the Poll type (although it doesn't have to come from futures).
Having the function return a Result that could have an Error::WouldBlock is much more difficult to follow -- you have to dig into what the error type is to understand that this is an async function!

Summary
An async function should return Poll type -- not have an "async error." In general, it should mimick the API of Future::poll. Fundamentally this is a usability/readability question, but it will also make it easier to reason about what futures::Poll::from::<nb::Poll>() does, as well as allow us to more easily leverage futures.
I posit that it would be better if we used the futures::Poll type directly, as that type really is zero cost by definition (futures::Poll<T, E> is mathenatically identical to Result<T, nb::Error<E>>) and would make converting async functions to futures much more readable and streamlined (they would automatically implement Future::poll!)

Also, according to the trait documentation, Future::poll is the only required method to implement a Future! If we simply return future::Poll for our functions, we will have already implemented them as futures (we will just have to have a struct which just returns them for the poll method).

I actually don't feel like returning Result<Async<T>, E> is the best idea. When playing with futures and trying to implement my own version of them, I realised Async<Result<T, E>> would be better approach.
When writing combinators, most of them don't care whether it's Ok or Err. They just return when Async::NotReady is encountered. So instead of doing this:
let foo = match async_op() {
    Ok(Async::Ready(val)) => val,
    Ok(Async::NotReady) => return Ok(Async::NotReady),
    Err(e) => return Err(e),
};

One would write:
let foo = match async_op() {
    Async::Ready(val) => val,
    Async::NotReady => return Ok(Async::NotReady),
};

It's usually done in macros, but those macros generate the code. Also it avoids type conversion.
Secondly, this allows one to cleanly express operations that can't fail. (Although, I have nothing against Result<T, !>, I think it shouldn't be over-used.)
Thirdly, I guess that would play nice with ? operator.
Finally, I suppose returning Async::NotReady is the common case. Err is probably least common case.
Disclaimer: I haven't explored all uses of Futures, so maybe I'm missing something.

I also like Async as a type much better, but I REALLY don't think it's worth having an API different than Future::poll here. Futures are the standard in simple async handling, we should maintain as similar an API as we can to make conversion simple and learning straightforward.

Maybe open an issue there? While Futures is becoming standard, it's not developed yet and may change.

The reason Poll is the type it is is so that futures can use and_then, etc (i.e. They behave like iterators). So it's a pretty decent reason.
It's also a reason why putting WouldBlock/NotReady in the Error is not a good idea, Result already has pretty much this exact same api, but it only works if you stop on the first error.

I don't think implementing and_then etc on Future would be a problem. Quite the opposite.

Sorry, I confused a few different topics there... In order for Future to have and_then, Poll has to have an Error type. That is why just using Async doesn't work.
Having Poll just be a Result means it has roughly the same API, which is a benefit as well

That is solvable:
trait Future {
    type Item; // Just the item, no Result here.

    // ...

    fn and_then(self) -> AndThen where Self: Future<Item=Result> {
        // ...
    }
}

I was actually trying to do something similar with `where Self::Item: SomeTrait`, `impl<T, E> SomeTrait for Result<T, E>`, `impl<T> SomeTriat for Option<T>`. Unfortunately it collided in some way (Rust is currently unable to understand that `Foo<Bar=Result<T, E>>` and `Foo<Bar=Option<T>>` are different traits.)

but Futre::poll has type:
fn poll(&mut self) -> Poll<Self::Item, Self::Error>;

That is why Item doesn't have to be a Result... because Error is defined spearately!
AndThen is then implemented like this:

impl<A, B, F> Future for AndThen<A, B, F>
    where A: Future,
          B: IntoFuture<Error=A::Error>,
          F: FnOnce(A::Item) -> B,
{
    type Item = B::Item;
    type Error = B::Error;

    fn poll(&mut self) -> Poll<B::Item, B::Error> {
        self.state.poll(|result, f| {
            result.map(|e| {
                Err(f(e).into_future())
            })
        })
    }
}

So again, poll has to return a Result in order for AndThen to know how to do result.map in it's poll() method. I'm pretty sure there is no way around that.

That is why Item doesn't have to be a Result... because Error is defined spearately!

I know. I suggest removing Error associated type and instead use Result as item for cases when the Future can fail. Don't use Result if future is infallible. Option is another option. ;)

but if you use a Result as the Item (and don't have an assigned Error) then and_then and related functions don't work. You have to just set Error = ! if future is infallible.

I have already mentioned this but when writing RTFM tasks / callback style
code WouldBlock as an error is the right type. The API in this crate is meant
to be usable in blocking mode, with futures, with the future #[async] /
await! (generators) and with callback code and the current API fits the bill
for all of them.
Neither Result<Async<T, E>> or Async<Result<T, E>> has an advantage over
nb::Result<T, E> in terms of pattern matching, but both are misleading in
callback style code. The former because serial.write(0).unwrap() would not
check if the operation completed or not, and the latter because
serial.write(0) does not even raise a warning about the operation not even
having occurred.

When someone creates a function that is non blocking, they should be returning
a type that specifies the function as non-blocking.

The standard library Read / Write traits when used in non-blocking mode
return a Result and use the ErrorKind::WouldBlock variant to signal that the
operation needs to block to complete. So in a sense we are already using the
standard return type for this operation.

Having the function return a Result that could have an Error::WouldBlock is
much more difficult to follow

This doesn't matter at all when writing blocking / futures-based code as you are
either using block!, try_nb!, hal-blocking or hal-futures and neither of
those returns nb::Result; they return a plain Result (or a future that will
converge to a Result).
For callback style code this is not as ergonomic as it should be but the
alternatives Result<Async<T, E>> and Async<Result<T, E>> are worse (see the
second paragraph).
(nb would be nicer if we had "untagged enums" as in Result<T, MyError | WouldBlock>; that would reduce one layer from the pattern matching:
match serial.write(byte) {
    Ok(()) => ..
    Err(MyError::Noise) => ..
    Err(MyError::Overrun) => ..
    Err(WouldBlock) => ..
    // ...
}

but that doesn't exist today)

An async function should return Poll type -- not have an "async error."

The Poll type was created to be the return type of polling a future not to
be the return type of an async function. Futures based async functions return
impl Future (a type that implements the Future trait). Proper #[async]
functions will return an impl Generator (a type that implements the
Generator trait)
The only nonblocking functions I have seen in practice are the ones in the
standard library Read / Write traits and those return a Result and use the
WouldBlock variant to indicate "this operation needs to block to complete".
nb is modeled after those functions.

but it will also make it easier to reason about what
futures::Poll::from::nb::Poll() does, as well as allow us to more easily
leverage futures.

So is the whole point to simplify writing futures based APIs on top of nb? You
want to go from:
fn write<S>(byte: u8) -> impl Future<Item = (), Error = Error>
where
    S: hal::Serial,
{
    future::poll_fn(|| {
        Ok(Async::Ready(try_nb!(serial.write(byte))))
    })
}

to:
fn write<S>(byte: u8) -> impl Future<Item = (), Error = Error>
where
    S: hal::Serial,
{
    future::poll_fn(|| serial.write(byte).into())
}

? Well, the required From implementation to make that possible requires (a)
that the nb crate depends on the futures crate, or (b) that the futures
crate depends on the nb crate. (a) is against the goal of the nb crate and
(b) seems unlikely so I don't think either is going to happen.

@vitiral why and_then couldn't be implemented for F: Future<Item=Result<T, E>>?
I know one can use !. I just found it cleaner to not differentiate between Ok and Err in then() implementation (and others as well).

and the latter because
serial.write(0) does not even raise a warning about the operation not even
having occurred.

The solution is obviously very simple: #[must_use = "Operation might not completed"]

The standard library Read / Write traits when used in non-blocking mode
return a Result and use the ErrorKind::WouldBlock variant to signal that the
operation needs to block to complete. So in a sense we are already using the
standard return type for this operation.

This breaks when read_exact() and write_all() are used. I find it quite unfortunate.

@Kixunil if you have a chain like this:
my_future.and_then(|r| ...).and_then(|r| ...)

If the first and_then fails you must not call the others. Therefore there has to be a "quit" type as part of Poll. That type is simply Error. For nb that type is Error::Other.
If you still don't understand, I would recommend copy/pasting the code that I linked and try to implement what you are saying. However, the point is semi-irrelevant because of my reply to japaric.
@japaric

The former because serial.write(0).unwrap() would not
check if the operation completed or not, and the latter because
serial.write(0) does not even raise a warning about the operation not even having occurred.

Ah, this (along with io::ErrorKind::WouldBlock) is the missing piece for me. This api makes much more sense now! I will re-look at the crate and see if I can get a PR that better documents the API. I'll have the PR close this issue.
Thanks for your patience and excellent explanation.

@vitiral I'm sure that is implementable. I don't have much time for it now though. Basic idea is to provide and_then() only for fallible operations (Item = Result).

@Kixunil

The solution is obviously very simple: #[must_use = "Operation might not completed"]

Yeah but even with that Async<Result<T, E>> is not as ergonomic as nb::Result<T, E>. With the later I can write serial.write(byte).unwrap_or_else(|| /* something */) (in callback-style code), but with the former I have to sort of unwrap twice: once to remove the Async layer and another one to remove the Result layer.

This breaks when read_exact() and write_all() are used. I find it quite unfortunate.

Are you referring to the standard library version of those functions? nb::Result and the WouldBlock idea is only applicable to operations that don't have internal state. Remember that the core operations here are read and write which map to syscalls. read_exact and write_all are stdlib specific constructs on top of read / write that contain internal state; WouldBlock is not applicable in that case. Tokio deals with those two functions differently as well.

I have to sort of unwrap twice: once to remove the Async layer and another one to remove the Result layer.

This depends heavily on how one uses it.

Are you referring to the standard library version of those functions?

Yes, I mean that standard library is broken in some sense. I say it to avoid this in other crates.
BTW Tokio had it incorrect too - it assumed that everything was written with async in mind. (If someone used read_exact or write_all, the code was broken.) They changed it to require AsyncRead and AsyncWrite marker traits (right decision IMHO).


casting of buggy embeds does not fail properly

Environment

Elixir version (elixir -v): 1.7.4
Ecto version (mix deps): 3.0.1
Operating system: macOS

Current behavior
If a schema uses embeds but the embedded schema is buggy (and does not import Ecto.Changeset) it will still compile but then the parent changeset function will be buggy and never casting the embed.
For Example:
defmodule Parent do
  use Ecto.Schema
  import Ecto.Changeset

  @primary_key false
  schema "parent" do
    embeds_one :child, __MODULE__.Child
  end

  defmodule Child do
    use Ecto.Schema
    # <---- missing import Ecto.Changeset here

    @primary_key false
    embedded_schema do
      field :name, :string
    end

    def changeset(schema, params \\ %{}) do
      schema
        |> cast(params, ~w(name)a) # <- this should fail as function should be undefined
        |> validate_required(~w(name)a) # <- this should fail as well
    end
  end

  def changeset(schema, params \\ %{}) do
    schema
      |> cast(params, [])
      |> cast_embed(:child, required: true)
  end
end

In this case:
iex(1)> Parent.changeset(%Parent{}, %{})
#Ecto.Changeset<
  action: nil,
  changes: %{},
  errors: [child: {"can't be blank", [validation: :required]}],
  data: #Parent<>,
  valid?: false
>

But it should really not even compile IMO
Expected behaviour
I am not sure what is expected but current behaviour seems wrong. And if I move the child module out then the code rightfully does not compile. This might actually be an Elixir issue.

It compiles because inner module gets aliases and imports from the outer module, probably what is happening is they are carried over in macro env when expanding the modules. Coincidently it was just mentioned on forum as undocumented behaviour: https://elixirforum.com/t/cultivates-elixir-posts/17948/2

This is "documented" when we say that alias, import and require are lexical. It means they do not care about functions, modules, conditionals, cases. At the moment they are defined, everything below the declaration in the same code branch will have the same aliases, imports, etc.
Of course, saying that the documentation is in the word "lexical" is a bit cryptic, so improvements to the Elixir documentation with examples are very welcome. Thank you! :heart:

I did notice though, that although the code still did compile, the behaviour was wrong. So perhaps we should not close this so quickly.

Thanks @evadne for the follow up. Reopening now. Can you please let us know which behaviour is wrong?

Will check this again shortly, I might be totally wrong though so please do not feel alerted yet 😯

@josevalim sorry, I am a mong and the bug was in my test.


Checboxes/radio buttons/switch not working in mobile chrome

https://css-tricks.com/forums/topic/checkbox-hack-on-mobile-webkit/

solution https://github.com/Teleopti/styleguide/pull/333

merged and handled on the teams


better filename handler

youtube-dl has restrict filename method, find a way to grab the final name and write it to song_title.txt file


Clarify filters for Razor components

Fixes #16780

@Rick-Anderson @scottaddie @wadepickett Did you change over to single digit ms.date month/day entries? It was leading zeros, but I see singles in some of the topics (e.g., this one).

I use single digits all the time. They ask for two digits, but single doesn't cause problems.


Use readable-stream

This is causing breakage on 0.8:
$ npm install
util.js:538
  ctor.prototype = Object.create(superCtor.prototype, {
                                          ^
TypeError: Cannot read property 'prototype' of undefined
    at Object.exports.inherits (util.js:538:43)
    at Object.<anonymous> (/home/travis/.nvm/v0.8.28/lib/node_modules/npm/node_modules/npmlog/node_modules/are-we-there-yet/index.js:115:6)
    at Module._compile (module.js:449:26)
    at Object.Module._extensions..js (module.js:467:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:362:17)
    at require (module.js:378:17)
    at Object.<anonymous> (/home/travis/.nvm/v0.8.28/lib/node_modules/npm/node_modules/npmlog/log.js:2:16)
    at Module._compile (module.js:449:26)

Ah, seems this was fixed in #2, but not yet published. Can we publish this so we can update the version in npm@2?

nathan7: That was landed in 1f380f6 and published in<EMAIL_ADDRESS>which is currently npm@next. It will be pushed to npm@latest this week per the usual release schedule.


/usr/local/bundle/gems/rest-client-1.8.0/lib/restclient/abstract_response.rb:74:in `return!': 422 Unprocessable Entity (RestClient::UnprocessableEntity)

First off thanks for the great module, but unfortunately I'm encountering a problem I don't quite understand when using your API wrapper (automated deployment via self-hosted gitlab-ce instance):
This is the error I'm getting:
$ gem install vagrant_cloud
Building native extensions.  This could take a while...
Successfully installed unf_ext-<IP_ADDRESS>
Successfully installed unf-0.1.4
Successfully installed domain_name-0.5.20161021
Successfully installed http-cookie-1.0.3
Successfully installed mime-types-2.99.3
Successfully installed netrc-0.11.0
Successfully installed rest-client-1.8.0
Successfully installed vagrant_cloud-0.4.2
8 gems installed
$ chmod a+x atlas-deploy.rb
$ ruby ./atlas-deploy.rb $ATLAS_USERNAME $ATLAS_TOKEN $BOX_NAME $BOX_VERSION $BOX_PROVIDER
/usr/local/bundle/gems/rest-client-1.8.0/lib/restclient/abstract_response.rb:74:in return!': 422 Unprocessable Entity (RestClient::UnprocessableEntity) 	from /usr/local/bundle/gems/rest-client-1.8.0/lib/restclient/request.rb:495:in process_result'
from /usr/local/bundle/gems/rest-client-1.8.0/lib/restclient/request.rb:421:in block in transmit' 	from /usr/local/lib/ruby/2.1.0/net/http.rb:853:in start'
from /usr/local/bundle/gems/rest-client-1.8.0/lib/restclient/request.rb:413:in transmit' 	from /usr/local/bundle/gems/rest-client-1.8.0/lib/restclient/request.rb:176:in execute'
from /usr/local/bundle/gems/rest-client-1.8.0/lib/restclient/request.rb:41:in execute' 	from /usr/local/bundle/gems/vagrant_cloud-0.4.2/lib/vagrant_cloud/account.rb:64:in request'
from /usr/local/bundle/gems/vagrant_cloud-0.4.2/lib/vagrant_cloud/box.rb:64:in create_version' 	from /usr/local/bundle/gems/vagrant_cloud-0.4.2/lib/vagrant_cloud/box.rb:73:in ensure_version'
from ./atlas-deploy.rb:14:in `'
ERROR: Build failed: exit code 1
This is my ruby deployment script:
# -*- mode: ruby -*-
# vi: set ft=ruby :

require 'vagrant_cloud'

atlas_username = ARGV[0]
atlas_token = ARGV[1]
box_name = ARGV[2]
box_version = ARGV[3]
box_provider = ARGV[4]

account = VagrantCloud::Account.new(atlas_username, atlas_token)
box = account.ensure_box(box_name)
version = box.ensure_version(box_version)
provider = version.ensure_provider(box_provider)

version.relaese
puts provider.download_url

any ideas where it goes so wrong? and what I can do do fix the issue?

The server responded with a non-200 status when trying to create a new box version. It's definitely not obvious from the stack trace, but you can see it here:

422 Unprocessable Entity (RestClient::UnprocessableEntity)

What data are you sending? What's your ARGV?
I would suggest that you output what is sent and received from the Atlas server in request in account.rb.

My call is this:
ruby ./atlas-deploy.rb $ATLAS_USERNAME $ATLAS_TOKEN $BOX_NAME $BOX_VERSION $BOX_PROVIDER
$ATLAS_USERNAME: my username
$ATLAS_TOKEN: the token I created
$BOX_NAME:  "vagrant-deploy-test"
$BOX_VERSION: "0.0.1"
$BOX_PROVIDER: "virtualbox"

My bad.  This can be closed


nats: no servers available for connection

Hi, I have problem starting the docker services. When I try to launch the services, it gives me
Starting ion_nats_1  ... done
Starting ion_redis_1 ... done
Recreating ion_islb_1 ... done
Recreating ion_app-biz_1 ... done
Recreating ion_sfu_1     ... done
Recreating ion_signal_1  ... done
Attaching to ion_nats_1, ion_redis_1, ion_islb_1, ion_sfu_1, ion_app-biz_1, ion_signal_1
redis_1    | 1:C 14 Aug 2021 12:25:43.980 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1    | 1:C 14 Aug 2021 12:25:43.980 # Redis version=6.0.9, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1    | 1:C 14 Aug 2021 12:25:43.980 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
nats_1     | [1] 2021/08/14 12:25:43.786132 [INF] Starting nats-server
nats_1     | [1] 2021/08/14 12:25:43.786168 [INF]   Version:  2.3.4
nats_1     | [1] 2021/08/14 12:25:43.786171 [INF]   Git:      [7112ae0]
nats_1     | [1] 2021/08/14 12:25:43.786175 [INF]   Name:     NBQDXCNWUH2MMTGLLR5IGU34RD3ZOVRBMK3DYVW3PU2AVLUHDWFSBQSJ
nats_1     | [1] 2021/08/14 12:25:43.786181 [INF]   ID:       NBQDXCNWUH2MMTGLLR5IGU34RD3ZOVRBMK3DYVW3PU2AVLUHDWFSBQSJ
nats_1     | [1] 2021/08/14 12:25:43.786186 [INF] Using configuration file: nats-server.conf
islb_1     | config /configs/islb.toml load ok!
islb_1     | time="2021-08-14 12:25:44.468" msg="--- starting islb node ---"
islb_1     | time="2021-08-14 12:25:44.468" msg="start pprof on :6061"
islb_1     | time="2021-08-14 12:25:44.469" msg="new nats conn error nats: no servers available for connection"
islb_1     | time="2021-08-14 12:25:44.469" msg="islb start error: nats: no servers available for connection"
redis_1    | 1:M 14 Aug 2021 12:25:43.981 * Running mode=standalone, port=6379.
redis_1    | 1:M 14 Aug 2021 12:25:43.981 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1    | 1:M 14 Aug 2021 12:25:43.981 # Server initialized
redis_1    | 1:M 14 Aug 2021 12:25:43.981 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
sfu_1      | config /configs/sfu.toml load ok!
nats_1     | [1] 2021/08/14 12:25:43.787187 [INF] Starting http monitor on <IP_ADDRESS>:8222
sfu_1      | time="2021-08-14 12:25:45.528" msg="--- starting sfu node ---"
sfu_1      | time="2021-08-14 12:25:45.528" msg="start pprof on :6062"
nats_1     | [1] 2021/08/14 12:25:43.787241 [INF] Listening for client connections on <IP_ADDRESS>:4222
redis_1    | 1:M 14 Aug 2021 12:25:43.981 * Loading RDB produced by version 6.0.9
redis_1    | 1:M 14 Aug 2021 12:25:43.981 * RDB age 6 seconds
redis_1    | 1:M 14 Aug 2021 12:25:43.981 * RDB memory usage when created 0.77 Mb
redis_1    | 1:M 14 Aug 2021 12:25:43.981 * DB loaded from disk: 0.000 seconds
redis_1    | 1:M 14 Aug 2021 12:25:43.981 * Ready to accept connections
sfu_1      | time="2021-08-14 12:25:45.531" msg="new nats conn error nats: no servers available for connection"
sfu_1      | time="2021-08-14 12:25:45.531" msg="sfu init start: nats: no servers available for connection"
signal_1   | config /configs/signal.toml load ok!
signal_1   | time="2021-08-14 12:25:47.128" msg="--- Starting Signal (gRPC + gRPC-Web) Server ---"
signal_1   | time="2021-08-14 12:25:47.128" msg="--- Bind to <IP_ADDRESS>:5551, NID = sig01 ---"
signal_1   | time="2021-08-14 12:25:47.129" msg="new nats conn error nats: no servers available for connection"
signal_1   | time="2021-08-14 12:25:47.129" msg="new signal: nats: no servers available for connection"
nats_1     | [1] 2021/08/14 12:25:43.787652 [INF] Server is ready
ion_islb_1 exited with code 255
nats_1     | [1] 2021/08/14 12:25:43.787675 [INF] Cluster name is oFGqGnOCPjNueMxldiybHE
nats_1     | [1] 2021/08/14 12:25:43.787678 [WRN] Cluster name was dynamically generated, consider setting one
nats_1     | [1] 2021/08/14 12:25:43.787700 [INF] Listening for route connections on <IP_ADDRESS>:6222
app-biz_1  | config /configs/app-biz.toml load ok!
app-biz_1  | time="2021-08-14 12:25:45.633" msg="--- [mengyao_debug] Starting Biz Node ---"
ion_sfu_1 exited with code 255
app-biz_1  | time="2021-08-14 12:25:45.633" msg="start pprof on :6060"
app-biz_1  | time="2021-08-14 12:25:45.635" msg="new nats conn error nats: no servers available for connection"
app-biz_1  | time="2021-08-14 12:25:45.635" msg="nats: no servers available for connection"
app-biz_1  | time="2021-08-14 12:25:45.635" msg="nats://nats:4222"
app-biz_1  | panic: runtime error: invalid memory address or nil pointer dereference
app-biz_1  | [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x9ec88d]
It seems that it cannot connect to nats:4222, while that the nat docker is already started, and 4222 port is opened.

please try new version


Feat  my skills

@brandonetter
I can't seem to get the my skills text to center. I think the breakpoints are looking better but on smaller devices the components look to be off to the left. Not sure what i'm doing wrong, probably not the right breakpoints. How do I figure out which breakpoints to use and how to center on smaller devices?

👍


Added German translation

I hope that the name of the file fits. Is there something else which needs to be done or will the translation now work out of the box?

Thx for this PR. Normally it should be enough. You didn't try locally?

I tried, but I have never worked with remote containers before. So I have no glue what I have to do ...

Thanks for your first PR, @azrael783! I will have a look, probably after this weekend, but it looks good.
Normally, you wouldn't have to translate all strings, but some of them will be 'translated' by Home Assistant Lokalise. We use certain keys in strings.json (see https://github.com/iMicknl/ha-tahoma/blob/master/custom_components/tahoma/strings.json) and they will be automatically translated to match the other Home Assistant integrations.
This is no issue for this custom integration, but eventually it would be nice to generate them with Hassfest or something and just translate the custom strings. But let's discuss on Discord sometime :-).

Thanks!


can now use opencv 3.0

Two small changes that allows OpenCV 3.0 to be used. The strings defining which type of detector and descriptor to use will be ignored when using OpenCV 3.0, since the needed creator functions are missing.

Thanks, I will look into this and see if its possible to change descriptors and detectors in opencv3 as well


Unexpected freezing and infinite loop when using localhost instead of <IP_ADDRESS>

As requested by @bahmutov @MikeMcC399 - this is the new issue related to https://github.com/bahmutov/start-server-and-test/issues/333 and was meant to be resolved by start-server-and-test ^2.0.0
This issue seems to impact mostly macOS version 13+ (I'm on macOS 13.2.1)
How to reproduce:

Clone https://github.com/Avansai/next-multilingual
Open package.json and make sure that all the <IP_ADDRESS> are replaced by localhost
run npm run install
Add the new script: "tmp-test": "cross-env DEBUG=start-server-and-test npm run e2e-build-headless"
run npm run tmp-test

You will see the following debug logs:
next-multilingual git:(main) ✗ npm run tmp-test

><EMAIL_ADDRESS>tmp-test
> cross-env DEBUG=start-server-and-test npm run e2e-build-headless

><EMAIL_ADDRESS>e2e-build-headless
> cross-env CYPRESS_isProd=true start-server-and-test start-example-build http://localhost:3000 cypress-headless

  start-server-and-test initial parsed arguments { _: [ 'start-example-build', 'http://localhost:3000', 'cypress-headless' ] } +0ms
  start-server-and-test named arguments: { expect: undefined, '--expected': '--expect' } +0ms
  start-server-and-test initial parsed arguments { _: [ 'start-example-build', 'http://localhost:3000', 'cypress-headless' ] } +2ms
  start-server-and-test parsing CLI arguments: [ 'start-example-build', 'http://localhost:3000', 'cypress-headless' ] +0ms
  start-server-and-test parsed args: { services: [ { start: 'npm run start-example-build', url: [Array] } ], test: 'npm run cypress-headless' } +0ms
1: starting server using command "npm run start-example-build"
and when url "[ 'http://localhost:3000' ]" is responding with HTTP status code 200
running tests using command "npm run cypress-headless"

  start-server-and-test single service "npm run start-example-build" to run and test +0ms
  start-server-and-test starting server with command "npm run start-example-build", verbose mode? true +0ms
  start-server-and-test starting waitOn [ 'http://localhost:3000' ] +3ms
  start-server-and-test wait-on options { resources: [ 'http://localhost:3000' ], interval: 2000, window: 1000, timeout: 300000, verbose: true, strictSSL: true, log: true, headers: { Accept: 'text/html, application/json, text/plain, */*' }, validateStatus: [Function (anonymous)] } +0ms
waiting for 1 resources: http://localhost:3000
making HTTP(S) head request to  url:http://localhost:3000 ...
  HTTP(S) error for http://localhost:3000 Error: connect ECONNREFUSED ::1:3000

><EMAIL_ADDRESS>start-example-build
> cd example && rm -Rf .next && npm run build && npm run start

> build
> cross-env ../node_modules/.bin/next build

info  - Loaded env from /Users/nbouvrette/Projects/next-multilingual/example/.env.production
warn  - You have enabled experimental feature (esmExternals) in next.config.mjs.
warn  - Experimental features are not covered by semver, and may cause unexpected or broken application behavior. Use at your own risk.

info  - Linting and checking validity of types .making HTTP(S) head request to  url:http://localhost:3000 ...
  HTTP(S) error for http://localhost:3000 Error: connect ECONNREFUSED ::1:3000
info  - Linting and checking validity of types .making HTTP(S) head request to  url:http://localhost:3000 ...
  HTTP(S) error for http://localhost:3000 Error: connect ECONNREFUSED ::1:3000
info  - Linting and checking validity of types
info  - Disabled SWC as replacement for Babel because of custom Babel configuration ".babelrc" https://nextjs.org/docs/messages/swc-disabled
info  - Using external babel configuration from /Users/nbouvrette/Projects/next-multilingual/example/.babelrc
info  - Creating an optimized production build .making HTTP(S) head request to  url:http://localhost:3000 ...
  HTTP(S) error for http://localhost:3000 Error: connect ECONNREFUSED ::1:3000
info  - Creating an optimized production build ...making HTTP(S) head request to  url:http://localhost:3000 ...
  HTTP(S) error for http://localhost:3000 Error: connect ECONNREFUSED ::1:3000
info  - Creating an optimized production build
info  - Compiled successfully
info  - Collecting page data
info  - Generating static pages (201/201)
info  - Finalizing page optimization

Route (pages)                                                                                Size     First Load JS
┌ λ /                                                                                2.15 kB         132 kB
├   /_app                                                                                0 B            99.1 kB
├ ○ /404                                                                                489 B           130 kB
├ ○ /500                                                                                496 B           130 kB
├ ○ /about-us                                                                                611 B           130 kB
├ λ /api/hello                                                                                0 B            99.1 kB
├ ○ /contact-us                                                                                818 B           130 kB
├   └ css/ce62caf524188c44.css                                                                                826 B
├ ○ /contact-us/message-sent                                                                                464 B           130 kB
├ ○ /tests/anchor-links                                                                                2.5 kB          132 kB
├ ○ /tests/anchor-links/long-page                                                                                2.31 kB         132 kB
├ λ /tests/custom-error-page                                                                                543 B           130 kB
├ ○ /tests/dynamic-routes                                                                                892 B           130 kB
├   └ css/800f675d941c42ed.css                                                                                808 B
├ ○ /tests/dynamic-routes/catch-all                                                                                1.54 kB         131 kB
├   └ css/bf17edfca951b198.css                                                                                768 B
├ ● /tests/dynamic-routes/catch-all/[...country] (389 ms)                                                           1.14 kB         131 kB
├   └ css/a40c443a74b4f904.css                                                                                730 B
├   ├ /en-us/tests/dynamic-routes/catch-all/united-states-of-america
├   ├ /en-us/tests/dynamic-routes/catch-all/united-states-of-america/test-page-for-catch-all-dynamic-routes
├   ├ /en-us/tests/dynamic-routes/catch-all/united-states-of-america/this-is-the-catch-all-dynamic-route-test-page
├   └ [+51 more paths]
├ ● /tests/dynamic-routes/catch-all/category/[[...category]] (541 ms)                                               1.16 kB         131 kB
├   └ css/e6d9ae7b90b9e8af.css                                                                                732 B
├   ├ /en-us/tests/dynamic-routes/catch-all/category
├   ├ /en-us/tests/dynamic-routes/catch-all/category/family
├   ├ /en-us/tests/dynamic-routes/catch-all/category/family/category
├   └ [+65 more paths]
├ ○ /tests/dynamic-routes/identifier                                                                                1.32 kB         131 kB
├   └ css/02a1dd1ba9487ab1.css                                                                                808 B
├ ● /tests/dynamic-routes/identifier/[id]                                                                           1.04 kB         131 kB
├   └ css/d32221824cb8cb43.css                                                                                872 B
├   └ /mul/tests/dynamic-routes/identifier/123
├ ○ /tests/dynamic-routes/text                                                                                1.53 kB         131 kB
├   └ css/f63d3992eea5ba32.css                                                                                810 B
├ ● /tests/dynamic-routes/text/[cityName]                                                                           1.14 kB         131 kB
├   └ css/15932671a57453ef.css                                                                                877 B
├   ├ /en-us/tests/dynamic-routes/text/montreal
├   ├ /en-us/tests/dynamic-routes/text/london
├   ├ /en-us/tests/dynamic-routes/text/shanghai
├   └ [+3 more paths]
├ ● /tests/dynamic-routes/text/[cityName]/point-of-interest                                                         2.24 kB         132 kB
├   └ css/8acedaf7c00f3d81.css                                                                                813 B
├   ├ /en-us/tests/dynamic-routes/text/montreal/point-of-interest
├   ├ /en-us/tests/dynamic-routes/text/london/point-of-interest
├   ├ /en-us/tests/dynamic-routes/text/shanghai/point-of-interest
├   └ [+3 more paths]
├ ● /tests/dynamic-routes/text/[cityName]/point-of-interest/[poi]                                                   1.08 kB         131 kB
├   └ css/8b7c5bd085a21450.css                                                                                873 B
├   ├ /en-us/tests/dynamic-routes/text/montreal/point-of-interest/bonsecours-market
├   ├ /en-us/tests/dynamic-routes/text/montreal/point-of-interest/mount-royal-park
├   ├ /en-us/tests/dynamic-routes/text/montreal/point-of-interest/old-port
├   └ [+15 more paths]
└ ○ /tests/jsx-injection                                                                                2.37 kB         132 kB
    └ css/8657a83eb2aec1ee.css                                                                                743 B
+ First Load JS shared by all                                                                                99.2 kB
  ├ chunks/framework-af64bd368ed34feb.js                                                                            45.2 kB
  ├ chunks/main-28cb1db53890d295.js                                                                                36.7 kB
  ├ chunks/pages/_app-6dfa98c01a87d206.js                                                                           15 kB
  ├ chunks/webpack-c42a931f31636e75.js                                                                              2.09 kB
  └ css/7789a9bd44560554.css                                                                                120 B

λ  (Server)  server-side renders at runtime (uses getInitialProps or getServerSideProps)
○  (Static)  automatically rendered as static HTML (uses no initial props)
●  (SSG)     automatically generated as static HTML + JSON (uses getStaticProps)

> start
> cross-env ../node_modules/.bin/next start

making HTTP(S) head request to  url:http://localhost:3000 ...
  HTTP(S) error for http://localhost:3000 Error: connect ECONNREFUSED ::1:3000
ready - started server on <IP_ADDRESS>:3000, url: http://localhost:3000
info  - Loaded env from /Users/nbouvrette/Projects/next-multilingual/example/.env.production
warn  - You have enabled experimental feature (esmExternals) in next.config.mjs.
warn  - Experimental features are not covered by semver, and may cause unexpected or broken application behavior. Use at your own risk.

making HTTP(S) head request to  url:http://localhost:3000 ...
  HTTP(S) error for http://localhost:3000 Error: connect ECONNREFUSED ::1:3000
making HTTP(S) head request to  url:http://localhost:3000 ...
  HTTP(S) error for http://localhost:3000 Error: connect ECONNREFUSED ::1:3000
making HTTP(S) head request to  url:http://localhost:3000 ...
  HTTP(S) error for http://localhost:3000 Error: connect ECONNREFUSED ::1:3000
^C%

@nbouvrette
Thanks for the logs and information!
The logs show:
ready - started server on <IP_ADDRESS>:3000, url: http://localhost:3000

making HTTP(S) head request to  url:http://localhost:3000 ...
  HTTP(S) error for http://localhost:3000 Error: connect ECONNREFUSED ::1:3000

For comparison:
In my own different test Next.js environment on Ubuntu (simple starter). I see it is successfully trying to contact http://localhost:3000 via ::1:3000.
On ubuntu
netstat -lnt | grep 3000

showed
tcp6       0      0 :::3000                 :::*                    LISTEN

curl -I http://localhost:3000 was successful HTTP/1.1 200 OK
curl -I http://[::1]:3000 was successful HTTP/1.1 200 OK

I suspect on your system that you may get a different result for netstat -lnt | grep 3000 and that curl -I http://[::1]:3000 will fail. If that is the case you might be able to get help from your Next.js community or Stackoverflow to get the server to also listen on the IPv6 address ::1.
(I cloned your repository and tried to follow your instructions in an Ubuntu environment, however that didn't work for me. I don't want to spend time now getting that to work as your debug logs showed enough of the problem. Also I don't have a macOS system so even if it did work for me I can't be sure that I am emulating your environment.)

On macOS 13.2.1, netstat -lnt | grep 3000:
tcp6       0      0  ::1.3000                                      ::1.61369                                     ESTABLISHED
tcp6       0      0  ::1.61369                                     ::1.3000                                      ESTABLISHED
tcp6       0      0  ::1.3000                                      ::1.61368                                     ESTABLISHED
tcp6       0      0  ::1.61368                                     ::1.3000                                      ESTABLISHED

And

curl -I http://localhost:3000 was successful HTTP/1.1 200 OK
curl -I http://[::1]:3000 was successful HTTP/1.1 200 OK

@nbouvrette
Thanks for the additional debug information. I don't understand in that case why
HTTP(S) error for http://localhost:3000 Error: connect ECONNREFUSED ::1:3000
is happening. It probably needs somebody with better network know-how than I have to work out why it's failing.

@nbouvrette
There is a possibility that you could start your server on host <IP_ADDRESS> and that your results might be different doing that. Check your documentation. -h <IP_ADDRESS> might do it. I have had mixed results with similar experiments. On Vite --host made a positive difference, with react HOST=<IP_ADDRESS> didn't help.

@nbouvrette
Regarding your repository https://github.com/Avansai/next-multilingual , if you want to make this better to use, I would suggest to create a new branch with your changes in it, then test it by cloning to a clean system. That way you can see if there is anything missing. I suspect that you may have some modules globally installed.
I cloned https://github.com/Avansai/next-multilingual
I guessed you meant npm install not npm run install
npm run e2e for instance gives
Error [ERR_MODULE_NOT_FOUND]: Cannot find package '@next/bundle-analyzer'

@nbouvrette
Thanks very much for your detailed answer! I do understand that you are looking for a "proper" fix! I logged a separate issue https://github.com/jeffbski/wait-on/issues/137 regarding the underlying technology in an attempt to get a proper fix. From my point of view there isn't a need to research your issue any deeper, since it is already clear.

The simplest outcome would be if the issue https://github.com/jeffbski/wait-on/issues/137 were to be resolved with a new release of wait-on. Hopefully there will be a response from the maintainer there soon.
Another theoretical possibility would be to move start-server-and-test to use the fork https://github.com/metcoder95/wait-on where the issue is resolved, however it seems that the creator of the fork doesn't intend it to be a long-term solution.

You don't need to do anything more on your repo for my benefit. Thanks for correcting the typo about the npm install command in any case!


RFC - Custom IAM Policies for Lambda and API Containers

RFC - Custom IAM Policies for Lambda and API Containers
This is a Request For Comments (RFC). RFCs are intended to elicit feedback regarding a proposed change to the Amplify Framework. Please feel free to post comments or questions here.
Summary
Amplify CLI provisions IAM roles with policies to access different resources. Currently, adding custom IAM policies requires the customer to manually edit the CFN template every time they run amplify push. There are no higher-level methods to provide custom IAM policies to attach to the provisioned IAM roles. Our aim is to provide an easier way within the CLI to custom IAM policies.
GitHub Issues:

How to give serverless container access to non-amplify-managed resourc...
How to add SES permission to LambdaExecutionRole · Issue #5432 · aws-a...
Allow functions to have additional policies · Issue #5142 · aws-amplif...
Give a Lambda function access to RDS · Issue #6324 · aws-amplify/ampli...
allow lambda function to call itself · Issue #7667 · aws-amplify/ampli...

Proposed Solutions
There are two proposed solutions. One is a file based option, customers can edit a file to provide their policies directly and the CLI incorporates the policy in the stack. The other is a CLI based workflow option, which will prompt customers to answer questions. The CLI performs actions to generate the required policies from the input.
Proposal 1: File based
To define a custom policy, the customer would need to create a file 'custom-policies.json' and define it using the structure below. Customers can specify multiple actions for multiple resources in the same service. If customers want to grant access to more than one service, they can add additional blocks to the file. The file is optional if customers don't need any custom policies for their project.
The structure of the file would be:
{
   "version": "1.0",
   "policies": [
       {
            "Resources": [
                "<enter resource ARN here>"
            ],
            "Action": [
                "<enter actions here>"
            ]
        },
        {
            "Resources": [
                "<enter resource ARN here>",
                "<enter resource ARN here>"
            ],
            "Action": [
                "<enter actions here>",
                "<enter actions here>"
            ]
        }
    ]
}

The ‘Effect’ will be “Allow” as default. If the ‘Effect’ field is defined, the default will be overridden.
Customers can run amplify push to get the policies updated.
Proposal 2: CLI based Workflow
The customers will be prompted for the policies in the following workflows:
amplify add function
amplify update function
amplify add api
amplify update api

By answering questions on the CLI, customers can add and edit custom policies, actions and resource.
Workflow
amplify add function
Customers can add or attach the custom policies to the new function they just create.
The customers will be prompted with the question
Do you want to configure advanced settings?(y/N)

After choosing yes, customers will be asked
Do you want to access other resources in this project from your Lambda function?(Y/n)

If choosing yes, customers will be prompted
Select the categories you want this function to have access to.
(Press <space> to select, <a> to toggle all, <i> to invert selection)
o api
o storage
o ...
o external resource

Customers can choose external resource to add or edit custom policies
Select the operation you would like:(Use arrow key)
>add new external resources(add custom policies)
 edit external resources(edit custom policies)

After choosing to add new custom policies, customers will be prompted to enter resource ARN
Enter the resource ARN:

Customers will be prompted if they want to add more resources
Do you want to add more resource (in the same service)? (Y/n)

If customers choose yes, they will be asked to enter another resource. If they choose no, then proceed to next step.
Customers will be prompted actions they want to add to corresponding resource
Enter an action you want to add:

Customers will be prompted if they wanted to add more actions
Do you want to add more actions? (Y/n)

If customers choose yes, they will be prompted to enter an action again, if they choose no, then proceed to next step.
If customers choose to edit custom policies, they will be ask to select the resource they want to edit
Select the resource you want this function to have access to:(Use arrow key)
...
...

After that, customers will be ask if they want to add new or delete actions, add new or delete resources:
Which operation do you want to perform on` `<service name>?(Use arrow key)
> add new actions
  delete actions
  add new resources
  delete resources

If customers want to delete resources:
Which resources you want to delete on <service name>?(Press <space> to select,
<a> to toggle all, <i> to invert selection)
o ...
o ...

If customers want to delete actions:
Which actions you want to delete on <service name>?(Press <space> to select,
<a> to toggle all, <i> to invert selection)
o ...
o ...

If customers want to add more actions or resources, they will go over the same workflow as adding custom policies.
After adding all the desired actions, customers will be prompted to change default setting of ‘Effect’:
Do you want to change the 'Effect', ‘Effect’ is “Allow” by default? (y/N)

If customers choose yes, they will be prompted to enter the effect:
Enter the 'Effect' you want to have for the resources:

amplify update function:
When customers update functions, they can also add or edit custom policies.
For amplify update function workflow, the customers will be prompted with the question
Which setting do you want to update? (Resource access permissions)

Select the categories you want this function to have access to.
(Press <space> to select, <a> to toggle all, <i> to invert selection)
o api
o storage
o ...
o external resource

Customers can choose external resource to add or edit external resources, or proceed to other options.
When they choose external resource, they will experience the same process as for amplify add function.
amplify add api
Customers can add or edit custom policies when they create a new container. After customers create the container, they can add or edit custom policies to attach to the container they just create.
After customers prompted with question:
Do you want to access other resources in this project from your api?(Y/n)

Customers will be prompted with the question which resource they want to continue with:
Select the categories you want this function to have access to.
(Press <space> to select, <a> to toggle all, <i> to invert selection)
o api
o storage
o ...
o external resource

If they choose external resource, they will experience the same process as for amplify add function.
amplify update api
Customers can add or attach custom policies when they update API. After selecting the API they want to update, they can choose to update the resource or not. If customers choose yes, they want to access other resources, they will be prompted with the question:
Select the categories you want this function to have access to.
(Press <space> to select, <a> to toggle all, <i> to invert selection)
o api
o storage
o ...
o external resource

Customers can choose external resource to continue with adding or editing custom policies, they will experience the same process as for amplify add function.
Validation
File based workflow: after executing amplify push, the format will be validated before merging to the corresponding CFN template.
CLI based workflow: after entering actions for custom policies, the format of actions will be first validated before adding to the *function-parameter.json.
*
The resource will be verified against the regex:
arn:(aws|aws-cn|aws-us-gov):([a-z0-9])*:(([a-z]{2})-(east|west|south|north|northwest|northeast|southeast|central)-[1-3]){0,1}:([0-9]{12}){0,1}:([a-z|A-Z|0-9|_|:|/|*]+)*

The actions will be verified using the following regex:
([a-z0-9])*:([a-z|A-Z|0-9|*]+)*

If what customers enter is not the correct format, they will have to correct it before proceeding.
Multi-Environment Workflow
When the customers checkout a new environment, all the policies are copied over and customers can update the setting using the amplify update <category> workflow.
When customers enter the resource name, they can specify which environment they are in optionally. For example:
`arn:aws:iam::123456789012:user`-{env}

If customer is in the dev environment, then the resource name should look like:
`arn:aws:iam::123456789012:user`-dev

Question for the community

Is the structure of the file for proposal 1 easy enough for you to maintain? How can we improve it?
Is the field ‘Effect’ needed in custom policies? Does it make sense to remove it, and consider all the ‘Effect’ would be ‘allow’.
How would you like to use custom policies across environments? Do you want to keep the same custom policies across environments?

Not one of the questions listed at the bottom, but if it's between proposal 1 and 2, as a user of the CLI I'd prefer 2 since it fits into the already existing workflows and it would be a natural place to look for tapping into this functionality.
Proposal 1 would require further reading of the documentation and potential visits back if you forget where the custom policies are kept.

It should be possible to add a Condition clause as well (such as s3 prefix partitions linked to auth groups/env), override the Effect to deny part of a resource and define arrays instead of strings (for instance by typing a hyphen). At least in a later version.
A further improvement in any case is to introduce (recursive) resource groups as a category, so they can be attached or edited as one. This would greatly reduce unnecessary CLI repetitiveness. It should be trivial to propagate across branches.
It should be possible to attach custom policies to (un)auth roles/groups. Last time I checked this was not reasonably possible at all.
Ideally there are 3 ways:

The CLI to attract a starter community to quickly get up to speed with typical scenarios.
(My preference) A minimal definition file for the entire project similar to AWS SAM, from which cloudformation files are generated. The CLI should generate such a file but the developer can choose to write it manually. This attracts developers who need the flexibility but still a reliable ready-to-go boilerplate application basis (i.e. Amplify).
The generated cloudformation files per function, which the developer can choose to generate or edit manually for tricky situations.

(Both YAML and JSON should be universally supported)
This intermediate approach allows quicker development of Amplify, since new/advanced features don't require writing CLI extensions right away, and the CLI code is separated from cloudformation complexity. It also allows rapid development of different CLI flavors by the community, since the Amplify team cannot anticipate every use case and workflow into a single one, and AWS ultimately wants to sell as much metal as possible.
It would really help to test a resource configuration quickly rather than having to push all the resources for every little edit, and waiting for it to rollback on every typo. It should also be possible to have the Amplify project as a single source of truth (at least on one person branches) and only push directly what has changed rather than needlessly checking dependencies in the cloud.
Specifically, a change in code or policy is not a change in resources and should therefore not trigger a deletion and regeneration of resources, only a quick update in content, similar to the Lambda console. A webpack-like mode could be added for near-instant lambda updates.
For real success, it should be easy to map custom resources to the DataStore/Storage/PubSub/Hub interfaces (e.g. send/receive SES). Authentication schemes of Amplify are currently too limited for serious use (RBAC, shared resources, dynamic groups, private subscriptions, s3 prefixed single buckets, SNS/SES, etc.), rendering DataStore and Storage useless/unreliable for most developers.
It should also be easy to use a single bucket for all deployments (in our case we branch per customer).

I have been adding custom policies into associated templates. Questions:

Are you planning for only one custom-policies.json? Or would there be one for each function, each hosting bucket, etc. I would hope at the least each function would get one to help with organization.
Would there be a way to pull out custom policies from existing templates etc. or would we have to manually move them over?
CLI method is great as long as it can remember the settings between envs etc.

Great addition!

Could it be a workflow writing into a file? Sometimes it's easier to update the file to re-generate policies without going through a workflow again?
I'd leave the Effect for both: deny and allow

Hi @GeorgeBellTMH,
Thanks for your comment!

Are you planning for only one custom-policies.json? Or would there be one for each function, each hosting bucket, etc. I would hope at the least each function would get one to help with organization.

There would be one custom-policies.json for each function for file based workflow. The file is optional and added manually following the template.

Would there be a way to pull out custom policies from existing templates etc. or would we have to manually move them over?

For now, we have to manually move the custom policies over.

The question "Do you want to access other resources in this project from your Lambda function?" is not clear, I would rather prompt something like "Do you want to access other resources (inside or oustide this project) from your Lambda function?"

It seems support custom-policies.json now exists. What is less clear is what variable placeholders can be used and this RFC seems the most comprehensive on this topic.
It seems that ${env} works but not much else. It would be great to also be able to include
${region}
${account}
${amplifyAppId} (in other places it gets used as secretsPathAmplifyAppId)


[enhancement] add ClusterMixin and TransformerMixin to onedal/

Description
This fixes an inconsistency in Mixin inheritance in onedal estimators.  Classifiers and Regressors inherit onedal mixins, while clustering and transformers inherit sklearn mixins. This PR makes all mixins in the onedal/ folder onedal mixins.
Secondly it makes uses of check_array use the onedal/ standard _check_array instead.

/intelci: run

/intelci: run

/intelci: run

/intelci: run

@icfaust please rebase you branch
I think there is only CI check is required

/intelci: run

/intelci: run


delete local storage on node drain

This should always happen, and I believe it blocks draining - closes #4

@matthope , any chance you could look this over?

I don't think there's any point - the spot node will die, might as well try to clean up in all instances...


Move pip upgrade from post_compile to pre_compile

Move pip upgrade from post_compile to pre_compile so it is available earlier in the process.

The buildpack is installing its own version of pip after the pre-compile hook. So then the post-compile hook fails like this:
-----> Running post-compile hook
SFDO_METACI_DEPLOY=SFDO_METACI_DEPLOY
Installing from sfdo-metaci.txt
Command "git clone -q<EMAIL_ADDRESS>/tmp/pip-m4gxk70p-build" failed with error code 128 in None
You are using pip version 9.0.2, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
/app/.heroku/python/bin/python: Error while finding module specification for 'snowfakery.cli' (ModuleNotFoundError: No module named 'snowfakery')
Warning: Snowfakery is not installed. If you don't use it, that is not a problem.
Ran hook from bin/post_compile

Closing this since it looks like it won't work after all.


Set Ethernet pin Nbr

Dear Espressif team
In earlier AT-Lib versions it was possible to change the Ethernet Pins in menuconfig

Old options in Menuconfig
We have a hardware which takes advantage of this feature. With current AT-Stack I can only set RMII clock number but not the other pins (Phy Power GPIO, SMI MDC and SMI MDIO).

Currently implemented options in Menuconfig
It would be great if you can bring back this feature because otherways we can't update our hardware with new firmware versions from you.
When starting the firmware on our hardware there is an endless restart loop because pins are incorrect. Could you please change the behaviour to start AT-Firmware properly, even the Ethernet could not be initialized (as it was in earlier AT-Versions)? Our hardware has optional Ethernet and we don't want to program different ESP-Images for the one with and without Ethernet.
We would really appreciate it if you could fix this problems. Thank you in advance.
Best regards

@ESP-Coco Please give some advice.

HI @ubichsel  in IDF release/v4.0, the ethernet config has changed

Thank you for your response.
Is there anything special I have to do? I completely downloaded the AT-Repository new as proposed here: https://github.com/espressif/esp-at/blob/master/docs/ESP_AT_Get_Started.md And all menupoints I see is

Even though I still miss the Phy Power Pin in your screenshot.

@ubichsel Are you sure you are currently using the master branch?
Looking at your configuration, it seems that you are not using IDF release/v4.0,

@ESP-Coco I'm not able to see the menu you have. What I do:

git clone --recursive https://github.com/espressif/esp-at.git
rm sdkconfig
rm -rf esp-idf
make defconfig
make menuconfig

If I have to do anything different please tell me how to do it.

Hi @ubichsel  I am very sorry that I found the same problem with you when I used the latest master version,  IDF removed these ethernet configuration at commit 42a462d.
Now you can make changes in the main/at_eth_init.c .
eth_mac_config_t mac_config = ETH_MAC_DEFAULT_CONFIG();  // apply default MAC configuration
mac_config.smi_mdc_gpio_num = 23;  // alter the GPIO used for MDC signal
mac_config.smi_mdio_gpio_num = 18; // alter the GPIO used for MDIO signal
esp_eth_mac_t *mac = esp_eth_mac_new_esp32(&mac_config); // create MAC instance

eth_phy_config_t phy_config = ETH_PHY_DEFAULT_CONFIG(); // apply default PHY configuration
phy_config.phy_addr = 1;       // alter the PHY address according to your board design
phy_config.reset_gpio_num = 5; // alter the GPIO used for PHY reset
esp_eth_phy_t *phy = esp_eth_phy_new_ip101(&phy_config); // create PHY instance

Hi ESP-Coco
Thanks for the clarification, I'm happy you could reproduce it yourself.
One more question. Is it still possible to set somewhere the phy Power GPIO? We have such a pin in our hardware and in eralier times it was possible to configure it.

And how can I set the clock output pin?
In menuconfig there is an option Output RMII clock from internal but which pin does it take? We have it on IO17 which was possible to set in earlier times.

@ubichsel

Poweron GPIO is a  workaround way in  previous IDF, GPIO0 is a straping pin (not only the pin of control download, but also the RMII clock). Our previous Ethernet hardware design needed to avoid the effect of GPIO0 on boot. In the new Ethernet development board, we don't need this workaround way. In Ethernet standard drive, there should be no Poweron GPIO.
In the new version of the driver, if your board needs a separate GPIO to control the power of the PHY crystal oscillator, it needs to be handled by yourself in the on_lowlevel_init_done callback function of esp_eth_config_t ( pull up or pull down this GPIO).
If the RMII clock mode is selected to CONFIG_ETH_RMII_CLK_OUTPUT, then GPIO0 can be used to output the REF_CLK signal. if you're not using PSRAM in your design, GPIO16 and GPIO17 are also available to output the reference clock. you can config it in Component config -> Ethernet -> Support ESP32 internal EMAC controller

Sorry for the long break, I had to work on a different Project.
I was able to set the pins as you suggested in the previouse post. Thank you for that. With the logic analyzer I can see the clock line, mdio, mdc and power line working correctly. On console I see no error messages regarding initialization.
Unfortunately then it does noting more. As of my point of view it should check on mdio every 4 seconds (default setting) for a link on cable. This is not done, I clearly see that with my logic analyzer when sniffing MDIO and MDC pins on RMII interface (no action after init is done). Can you give me any suggestion what I can do to bring Ethernet with LAN8720 to work? Is there any AT-Command which is not in documentation where I have to activate Ethernet or something like that?
Thank you in advance.

This should be the driver doesn't work, because esp_eth_start is not called after esp_eth_driver_install, which will start a timer to check the connection status of the network cable.

Dear ESP-Coco
Thank you very much for the response. My problem is I can't debug deeper. The function esp_eth_driver_install at the end of at_eth_init is successfull but then nothing more happens. I guess this is somewhere in AT-FW which is closed source. Can you give me further instructions how to debug this error since esp_eth_driver_install was successfull (no errors printed on monitor console)?

Hi @ubichsel  Can you replicate this problem using an Ethernet demo?
I think it is more convenient for us to analyze.

Hi @ESP-Coco
Today I got a step further. I think in the AT-Firmware the function "esp_eth_start" is not called after the "at_eth_init" function in at_eth_init.c. If I call esp_eth_start(eth_handle) in at_eth_init function at the end I get a "+ETH_GOT_IP:<IP_ADDRESS>" event when ethernet cable is connected.
The only thing missing are the following events:

"+ETH_CONNECTED"
"+ETH_DISCONNECTED"

Can you have a look if you miss this events in AT-Code and enable it again? Then I'm happy again with Ethernet.
Thank you again for the support.

Hi @ubichsel Thank you for reminding me this problem. You are right. Event processing does need to be added here. This problem arises because in v4.x version, IDF do not use the old ethernet event. This needs to be adapted by AT.  I'll get back to you soon.

I implemented the events in at_eth_init.c by my self and so it is working. It would be nice if you implement it into the regular AT-Stack (if others have same issue).
For me it works right now so I close the issue.


FastHttpLocust is very slow at returning the response text

Getting resp.text when using FastHttpLocust does a string decode (fasthttp.py:257) and it is crazy slow for large responses (~12s for a 1.5MB payload).
    with self.client.request(method, f"https://{host}{url}", name, catch_response=True) as resp:
        t = resp.text: # this takes ages
        ....
        return resp

Oh, and also it doesnt cache the value, so every call to .text will need to decode the string again.
I'm looking in to it...

Oh, the actual problem is the character decoding in apparent_encoding()

I can solve this by using cchardet instead of chardet. Encoding detection time drops from 12 seconds to 0.03s for me.
@heyman ok to add this as a dependency?

Or we could just read the Content-Type: charset header?

Or we could just read the Content-Type: charset header?

Yeah, maybe that's best. And assume ISO-8859-1 if none is specified. I guess it could cause problems when the server is setting incorrect chaset headers, though it shouldn't be that big of a problem since users of a load testing tool usually have control over both client and server side.

I'll have a go at it


RFC: Genesis Quorum

There is a chicken and egg scenario, where in order to "inaugurate" the first quorums, we need to have the first election, which means we need to have the an election seed, and we need to have quorum public keys for the first quorums distributed to the entire network. In the protocol this occurs via convergence block certificates, but convergence block certificates are produced by a Harvester quorum.
There are a few solutions to this, however, just spit balling with @mikehostetler helped us land on one that will work really well.
Since the genesis validators need to be whitelisted, we can accomplish this through an off-network formation of a Distributed Key, in accordance with our protocol. That distributed key will be used to mine and sign certify the genesis block, and will be broadcasted to the network. As part of the genesis block mining, the stake required will be built into the claims to be confirmed by the genesis block, and effectively will establish the first state snapshot of the network. After this the network can move forward as it is intended, and even this process follows the protocol of the network, the only difference being that the formation of the first quorum to certify the genesis block will be determined via a whitelist as opposed to being determined using PoC election. We could, in theory, even run an off-chain PoC election using the genesis seed, and make it fully compliant with the protocol.
Effectively the steps that this would require are as follows:

Generation of node claims in accordance to protocol
Whitelisting of the node claims
Some adjustments to the Genesis Block struct to include said whitelisted node claims, and the embedding of stake into the claims.
Distributed Key Generation for a minimum of 3 quorums (1 farmer, one active harvester, 1 redundancy harvester) off-chain, and the creation of node secret shares and quorum public keys.
Inclusion of these first quorum pubkeys in the genesis block.
Mining of the genesis block by a VRRB controlled node, with a claim that is also included in the block (obviously).
Certification of the genesis block by the active harvester quorum.
Appending of the genesis block to each nodes local copy of the DAG
Writing of the genesis claims, token distribution transactions, etc. to the Claims DB, Accounts DB, Transactions DB, State Trie and Transactions Trie.

This will solve the chicken and egg scenario, and should be fairly easy to coordinate, as we can use a simple program to generate the secret shares and public key off chain, have the public key set formed as part of the node config and have the genesis mining node (controlled by VRRB) add it to the genesis block upon mining of the genesis block.
We are opening this up for comment, and would appreciate  your weigh in on it. We will need to begin implementing this towards the end of next week, so please get your comments in.

@ASmithOWL can this issue be closed?


[Audit][SPARK-40819][SQL][3.3] Timestamp nanos behaviour regression (parquet reader)

Is your feature request related to a problem? Please describe.
https://github.com/apache/spark/commit/3ec9b05a23 adds a new config setting spark.sql.legacy.parquet.nanosAsLong.
Describe the solution you'd like
We should respect this setting, either by returning as the expected type, or by falling back to CPU if we cannot support it.
Describe alternatives you've considered
Additional context

This new config has been present since:

3.2.4 (unreleased)
3.3.2
3.4.0 (unreleased)

Fixed in https://github.com/NVIDIA/spark-rapids/pull/8044


Support New Body Mode GraphQL for Postman Collections

Using Postman Version 7.2.2 with the new graphql body mode and ran into:
Error: Unrecognized body mode: graphql

Looks like postman-collection version 3.4.7 supports the new graphql body mode.
Would love for it to be supported here! Let me know if I can assist with a PR.

Thanks for this. Really interesting new feature. Going to have a look when I can.

any chance there will be support for graphQL in the future?


Eslint rules for no-shadow and fixes required.

Eslint rule:
"no-shadow": [2, {"builtinGlobals": false, "hoist": "all"}],
...with all the fixes necessary to make it actually build.

@chrisprice is this eslint rule what you were looking for? If so I can fix it so it merges, otherwise we can change the rule or close the PR.

Thanks @bjedrzejewski, these look like valuable changes but #613 was meant to refer to enforcing a naming convention of -
// If there are no naming conflicts, just the imported filename
import dataJoin from '../util/dataJoin';
// To avoid potential naming conflicts, the imported filename suffixed by its containing package name
import dataJoinUtil from '../util/dataJoin';

As such I've edited the PR to not close the issue but I'd still like to see this one merged. Are you ok to fix the merge conflicts?

Just spotted that you've referenced #613 in the commit message as well, please could you rebase that out.

@chrisprice I merged and rebased off the the close comment.

@chrisprice are you ok to take a look at this one?

@bjedrzejewski Thanks, sorry I've only just gotten around to looking at this. There's a change from the property name root to rootValue. Obviously, if we make this change then we need to release this as a breaking change. Just checking if it was intentional?

@chrisprice good spot, this was unnecessary! I rebased it out now.

Great thanks! Looks like we're having intermittent build issues :unamused:

@chrisprice yeah there is something wrong with the scatterplot example...


Consider documenting a security policy

The Drupal framework is looking to adopt Revolt as its event loop. Since this would become a very integral part of Drupal (with some existing systems already moving to async), one of the evaluation criteria is that the package has a security policy that outlines how to properly report security issues.
From one of Drupal's security team members on the issue discussing Revolt's adoption:

Can we ask the maintainers if they are willing to publish a security policy? Given that this is a low level runtime dependency it seems quite important that if there is a security issue the maintainers are prepared to fix it within a reasonable timescale.

Would it be possible to clarify Revolt's security policy and document it in a SECURITY.md file so it shows up in GitHub's security tab? This can also help other larger frameworks in adopting Revolt :)

Good point, it's indeed missing here. I've just enabled private vulnerability reporting and added a SECURITY.md file.


Ability to specify the ticks shown on an axis

https://groups.google.com/forum/#!topic/d3plus/hKH-9DD8zMA

.x() or .y()
{'ticks': {'number':5}} // default is 10

In the next release, you'll be able to explicitly set the visible ticks on either axis. For example:
.y({"ticks": [0, 5, 10, 50]})