New feature: los_intersect

This function calculates the latitude and longitude where a line-of-sight, defined by azimuth and tilt (off-nadir) angles from a starting point, intersects with the Earth ellipsoid. It also returns the slant range from the starting point to the intersection point. It has similar functionality to the lookatSpheroid routine in the Matlab mapping toolbox but the algorithm is based on https://gist.github.com/stephenHartzell/00e812ceef6901f30290203945e2cf8b#file-los_intersection-py by @StephenHartzell.
BTW, I think there may be a small inaccuracy at high latitudes in the enu2uvw function.

Thanks @rpavlick , would be interested to hear more about possible enu2uvw issue

Never mind, it was actually the other library I was testing it against that had the problem. Thanks for putting together pymap3d, it is very useful.


Prepare for 3.1.1

Gotta fix those import dangerfiles

          1 Warning

  :warning:
  Tests were not updated

  Generated by :no_entry_sign: danger


Menu in footer might be confusing

It might be confusing having menu down in the footer, I think that pages like FAQ should appear in top right corner as for other platform in our ecosystem. Here we can discuss about it.
@libremente what do you think?

Do you have a draft mockup for this? I think that for the moment we could leave them there but in the future we could foresee a better menu.


Aohri resource health update 1

Contributing to the Azure SDK
Please see our CONTRIBUTING.md if you are not familiar with contributing to this repository or have questions.
For specific information about pull request etiquette and best practices, see this section.

API change check
APIView has identified API level changes in this PR and created following API reviews.
Azure.ResourceManager.ResourceHealth

@abhayohri:
Our automation is identifying you as a community contributor.  Please review the Azure SDK onboarding documentation and ensure that your GitHub organization membership and permissions are correct.

Azure SDK onboarding (Microsoft internal)
Azure SDK onboarding assistance (Microsoft internal)

Please also be sure to add yourself to CODEOWNERS for this library, if you will be maintaining it going forward.

@abhayohri:
Our automation is identifying you as a community contributor. Please review the Azure SDK onboarding documentation and ensure that your GitHub organization membership and permissions are correct.

Azure SDK onboarding (Microsoft internal)
Azure SDK onboarding assistance (Microsoft internal)

Please also be sure to add yourself to CODEOWNERS for this library, if you will be maintaining it going forward.

requested access for:
https://repos.opensource.microsoft.com/orgs/Azure/teams/azure-sdk-read/join/
and
https://github.com/orgs/Azure/teams/azure-sdk-write-cxp
do i need to request access to devops?
pr for adding self to CODEOWNERS: https://github.com/Azure/azure-sdk-for-net/pull/39517


New version: SaltStack.SaltMinion version 30.07.1

Checklist for Pull Requests

[x] Have you signed the Contributor License Agreement?
Resolve #163087
Related #163002

Manifests

[x] Have you checked that there aren't other open pull requests for the same manifest update/change?
[x] This PR only modifies one (1) manifest
[x] Have you validated your manifest locally with winget validate --manifest <path>?

[x] Does your manifest conform to the 1.6 schema?

Note: <path> is the directory's name containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

[!NOTE]
Seems already fixed by #163419, but it tagged the version as 3007.1 instead of 30.07.1.

[!Note]
Seems already fixed by #163419, but it tagged the version as 3007.1 instead of 30.07.1.

Hi @Trenly,
Can you help us to find the correct version number for this package version?

[!Note]
Seems already fixed by #163419, but it tagged the version as 3007.1 instead of 30.07.1.

Hi @Trenly,
Can you help us to find the correct version number for this package version?

@stephengillie - It writes 3007.1 to the registry
PS C:\Users\WDAGUtilityAccount\Desktop\winget-pkgs> winget install Saltstack.saltminion -s winget
Found Salt Minion [SaltStack.SaltMinion] Version 3007.1
This application is licensed to you by its owner.
Microsoft is not responsible for, nor does it grant any licenses to, third-party packages.
Downloading https://repo.saltproject.io/salt/py3/windows/latest/Salt-Minion-3007.1-Py3-AMD64-Setup.exe
  ██████████████████████████████  60.7 MB / 60.7 MB
Successfully verified installer hash
Starting package install...
Successfully installed
PS C:\Users\WDAGUtilityAccount\Desktop\winget-pkgs> Get-ARPTable | ft

DisplayName                                                        DisplayVersion Publisher             ProductCode                            Scope
-----------                                                        -------------- ---------             -----------                            -----
Notepad++ (64-bit x64)                                             8.6.9          Notepad++ Team        Notepad++                              Mac...
Salt Minion 3007.1 (Python 3)                                      3007.1         SaltStack, Inc        Salt Minion                            Mac...

Close with reason: Duplicate of #163419;


script fails on an album

Running python imguralbum.py https://imgur.com/gallery/wfMp9 ~/Desktop gives me this error :
Error: URL must be a valid Imgur Album

How to use
=============

However the URL is correct and leads to an album. Not really sure why it fails.

That command works for me. I can't see why this would fail, are you using the latest version?
The only lines that can cause that error are these:
# Check the URL is actually imgur:
        match = re.match("(https?)\:\/\/(www\.)?(?:m\.)?imgur\.com/(a|gallery)/([a-zA-Z0-9]+)(#[0-9]+)?", album_url)
        if not match:
            raise ImgurAlbumException("URL must be a valid Imgur Album")

Not working for me either with the latest version

Hello @nikitavoloboev , @Kaligule and @Coaggly  please check my pull request and fork of this program: #26


Issue with k8s.io/docs/tasks/

This is a Bug Report

Problem:
Proposed Solution:
Page to Update:
https://kubernetes.io/...

Where is information or tasks ?

This sounds like a duplicate of issue #16067
@MRDO5 was that what you wanted to see fixed?
/triage needs-information

pls feel free to open with actionable issue details or file a new one.


How to migrate from 1.13.1 to latest version , Getting following error

WebpackOptionsValidationError: Invalid configuration object. Webpack has been initialised using a configuration object that does not match the API schema.

configuration has an unknown property 'tslint'. These properties are valid:
object { amd?, bail?, cache?, context?, dependencies?, devServer?, devtool?, entry, externals?, loader?, module?, name?, node?, output?, parallelism?, performance?, plugins?, profile?, recordsInputPath?, recordsOutputPath?, recordsPath?, resolve?, resolveLoader?, stats?, target?, watch?, watchOptions? }

1.13.1 configuration  and help me same configuration for latest version
'use strict';

const path = require('path');
const webpack = require('webpack');

module.exports = {
    devtool: 'inline-source-map',
    module: {
        preLoaders: [
            { exclude: /node_modules/, loader: 'tslint', test: /\.ts$/ }
        ],
        loaders: [
            { loader: 'raw', test: /\.(css|html)$/ },
            { exclude: /node_modules/, loader: 'ts', test: /\.ts$/ }
        ]
    },
    resolve: {
        extensions: ['', '.js', '.ts'],
        modulesDirectories: ['node_modules'],
        root: path.resolve('.', 'src')
    },
    tslint: {
        emitErrors: true
    }
};

You need to move tslint to a webpack.LoaderOptionsPlugin:
module.exports = {
  // ... snip ...
  plugins: [
    new webpack.LoaderOptionsPlugin({
      tslint: {
        emitErrors: true
      }
    });
  ]
}

There is a migration guide: https://webpack.js.org/guides/migrating/

Can any provide me webpack configuration for running unit test cases for Angular 4 spec files using Karma, Jasmine


First stab at error page for non-running environments

it's butt ugly, but it should work - only way to find out is to run the terraform and update the admin site.
Maybe @RabsRincon can take a stab at the web design part of this once we know it works?

Error page as it stands can be seen here: https://d3e106rr72m90r.cloudfront.net/
The page has 3 main layouts:

503 from production

just displays some contact info

Down environment index page

only displays "Continue to Live Site" button

Down environment direct link

displays "Follow Link" and "Go Home" buttons

I love this.

Just want to get the logo designs signed off by @mattgodbolt and then it's good to go.

I love it! Ship it!


Issue (modules/docs): lib.mdDoc

nixpkgs lib now have lib.mdDoc, used to defined... to be honest I don't know why exists at all, devshell-files default interpret it as MD, maybe historical reasons, who knows.
Anyway, when the module use lib.mdDoc, the documentation module fails, requiring .nix file patching.
Exemple:
https://github.com/hugosenari/friendly-modules/commit/42ab8a4b4fbd0ab6108e7fa26ef12f50cc19f6c1

Thank to @TinHead for the first use case


build-storybook differences?

I'm trying to figure out a weird edge case I'm having, using  mobx for store management inside stories.
Take this (highly convoluted) story:
import React from 'react';
import { observable } from 'mobx';
import { observer } from 'mobx-react';

// Create a new store
const store = observable({
  toggle: true,
});

// Toggle the value inside the store
function toggle() {
  store.toggle = !store.toggle;
}

// Test component -- uses the store above, but NOT a direct observer
const Headline = props => (

{props.store.toggle.toString()}

);

Headline.propTypes = {
  store: React.PropTypes.shape({
    toggle: React.PropTypes.bool.isRequired,
  }),
};

// Generator func to return x number of <Headline> components, fed the store
function* getComponents(count) {
  for (let i = 1; i <= count; i++) {
    yield (
      <Headline key={i} store={store} />
    );
  }
}

// Wrap THIS one in an observer, and go down the chain
const Toggle = observer(() => (

    {[...getComponents(5)]}
    <button
      onClick={toggle}>Toggle</button>

));

// Export as an instantiated JSX comp.
export default <Toggle />;

When running in dev (npm run dev), clicking 'toggle' works fine...

However, when running in production (npm run build-storybook and starting a local server), toggle suddenly stops working...

My gut reaction is that it has something to do with transpilation differences. Stores built using MobX's observer function work differently depending on whether there's a prototype chain (if there is, properties on the object are observable, but not the parent object itself). But that doesn't seem to be happening because in both cases, there's no prototype chain, and in other examples where the parent store is not fed in but used directly, it works as expected.
It seems the observer chain is 'lost' somehow when the generator passes it to it.
I know it's a totally weird edge case, but are there any known differences between how code is rendered in dev vs. bundling?

Would have to take a closer look, but my gut feeling is that it might relate to this: https://github.com/mobxjs/mobx-react/issues/56. Could you remove the propTypes and check whether the dev setup is not "accidentally" working correctly as a result of propTypes?

Not that MobX-react only supports direct observers, all components that use observable data, should be observers. Because react might schedule re-rendering both on the same stack or a separate stack, and in the latter case accessing an observable won't be tracked, hence not triggering a re-render

@mweststrate - thanks for looking. I tried removing .propTypes; same result. The only 'fix' I tried that worked is to wrap the final component in an observer, i.e.:
const Headline = observer(props => (          // <-- now wrapped in 'observer'

{props.store.toggle.toString()}

));

Headline.propTypes = {
  store: React.PropTypes.shape({
    toggle: React.PropTypes.bool.isRequired,
  }),
};

The problem with this approach is that I'm using lots of third-party components in production, so I can't guarantee each will implement observer. My expectation is that I can wrap the first (higher-order) component, and that it will render anywhere in the chain that a change might occur.
What I'm trying to figure out is why it works with npm run storybook (running via Webpack Dev Server ) but not with npm build-storybook (which produces an identical preview.bundle.js to dev, save for different module numbers assigned by webpack.) - is there any obvious reason why React would re-schedule rendering in a different stack in the latter but not the former?
Thanks for your help.

If you have third party components that don't support observer, don't pass them observables, but convert the observables to plain objects in your observer based parent component. That will make the parent responsible for tracking the data instead of the child.

yeah, that's actually what I'm currently doing.
I guess knowing that React could re-schedule off-stack, the question is now why it does that in one environment, but not another... any ideas what triggers it?

Something else that is a bit freaky too (maybe related to mobxjs/mobx-react#56)
Under npm run storybook, dropping .propTypes actually prevents it from observing:
const Headline = props => (

{props.store.toggle.toString()}

);

/* commenting out .propTypes *stops* it working in dev */
// Headline.propTypes = {
//   store: React.PropTypes.shape({
//     toggle: React.PropTypes.bool.isRequired,
//   }),
// };

So I guess that's exactly what mobxjs/mobx-react#56 relates to -- adding a .propTypes definition causes a re-render (but only in dev; not in production), thus making the component update via npm run storybook but NOT npm build-storybook.
interesting!

d'oh, it suddenly makes sense.
.propTypes isn't evaluated in production (it's a dev-only check, right?) So it wouldn't 'observe' the underlying value, and thus wouldn't re-calculate.
So, in this case, .propTypes was causing a side-effect that unintentionally made the component appear to be observing (I guess inside the original call stack passing the prop?), but breaks as soon as .propTypes is dropped in production -- which is what it should do, because the final component isn't an observer.
Makes sense now!

I'll go ahead and close this and the related issue, since this seems to be the intentional behaviour and a user-land issue.


[Question] Reading the .har recording response body content

I use pollyJS extensively, and from time to time I need to examine the previously recorded responses by themselves.
I have the persisted HAR files, and they contain everything as they should. However, the response body seems to be a hex buffer, that I cannot convert back to utf8 format.
I tried reading it as hex, base64, and piping it through zlib inflate for good measure, but I keep getting a non-utf8 unworkable content (same result when loading the HAR files to many HAR viewers).
How can the content be read?
Thanks.

If the response's body content is encoded, we store the chunks as an array of hex values. When replaying, we then convert each hex value back into a buffer and write it to a Readable Stream which will then be read by the consumer.
Converting the array of hex strings back into buffer chunks:
https://github.com/Netflix/pollyjs/blob/5b4edf3702b3e3a1c4cbd0651de8f1e4b45bdd42/packages/%40pollyjs/adapter-node-http/src/index.js#L277-L283
Writing each chunk to the stream:
https://github.com/Netflix/pollyjs/blob/5b4edf3702b3e3a1c4cbd0651de8f1e4b45bdd42/packages/%40pollyjs/adapter-node-http/src/index.js#L212-L219

Thanks, after fiddling with it, the HAR response contented turned out to be a gzipped chunked hex array.
So with your help, I managed to get the original response by:
import {Readable} from "stream";
import {createGunzip} from "zlib";
import {createWriteStream} from "fs";

const responseContent = ["The hex array copied from the HAR file"];

const chunks = responseContent.map(chunk => Buffer.from(chunk, 'hex'));

const contentStream = new Readable();

const zipMiddleware = createGunzip();
contentStream.pipe(zipMiddleware);

const output = createWriteStream('./output.json');
zipMiddleware.pipe(output);

chunks.forEach(chunk => contentStream.push(chunk));
contentStream.push(null);

Sorry it took me so long to implement your information - didn't have the time for it till now.


feat: support kubernetes operators (#73)

This pull request improves go-make by adding support for Kubernetes operators. Besides it improves some variable default handling as well as the naming of variables.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>4
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
310

Relevant Lines:
310

💛  - Coveralls


Expect newlines after linting "Merge branch ..." message in output

When I try to merge more than two branches, commitlint will print its output before git's diff info. Then git will continue to print its output but without a newline.
Current Behavior

▶ git merge dev_name_A dev_name_B
Fast-forwarding to: dev_name_A
Trying simple merge with dev_name_B
Simple merge did not work, trying automatic merge.
Auto-merging package-lock.json
Auto-merging package.json
husky > commit-msg (node v10.12.0)

⧗   input: Merge branches 'dev_2.4.2' and 'webpack4' into dev_2.4.3
✔   found 0 problems, 0 warningsMerge made by the 'octopus' strategy.
 build/prod.js                                                         |   2 +-

 ...any other diff info

Expected Behavior

⧗   input: Merge branches 'dev_2.4.2' and 'webpack4' into dev_2.4.3
✔   found 0 problems, 0 warnings

Merge made by the 'octopus' strategy.
 build/prod.js                                                         |   2 +-

 ...any other diff info

Affected packages

[x] cli
[ ] core
[ ] prompt
[ ] config-angular

Possible Solution

Print additional newlines after printing commitlint's output.
Steps to Reproduce (for bugs)

Create three git chunks.
Merge two of those chunks into another chunk.

commitlint.config.js

module.exports = {
    extends: ['@commitlint/config-conventional'],
    rules: {
        'scope-empty': [2, 'never'],
        'scope-enum': [2, 'always', [
            'scope-A',
            'scope-B',
        ]],
        'subject-empty': [2, 'never'],
        'subject-case': [0, 'never'],
    },
};

Context

Your Environment

Executable
Version

commitlint --version
7.2.1

git --version
2.19.2

node --version
v10.12.0

In master we have newline in code. Problem could be in not published package @commitlint/format, cause in npm we have version not from master branch

@RoyalHunt No, I look into the locally installed package source code, and also found the newline.
So I test in another way:
▶ git commit -m "test: 123" > out
husky > commit-msg (node v10.12.0)

⧗   input: test: 123
✔   found 0 problems, 0 warnings%

and then:
▶ cat out
[v1 36ae863] test: 123
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 b

Seems they were output into different stream.


i need to upload a image using angularjs..but i can't able to get that..can you give me some idea to that..

i need to upload a image using angularjs..but i can't able to get that..any one can give me some idea to that..
thanks advance..

Please direct support questions like this one to one of the support channels


[Bug]: Autogen Studio: [Errno 2] No such file or directory: 'python', where only python3 and pip3 are available

Describe the bug
When running Autogen Studio, on macOS, where usually Python is only available as python3 locally in the command line, Function Calling does not work, as when an Agent is trying to execute Python code, the following error occurs: FileNotFoundError: [Errno 2] No such file or directory: 'python'
This make sense, as python is not available in the $PATH of this machine, only python3. Same for pip and pip3.
Steps to reproduce

Run Autogen Studio on a Mac autogenstudio ui --port 8081
Click on Workflows -> Default Workflow
Click on Test Workflow and choose the Stock Price example
Watch the Logs

Model Used
gpt-4-1106-preview
Expected Behavior
Code should be executed or path to python and pip should be configurable.
Screenshots and logs

2024-07-01 15:16:35.243 | INFO     | autogenstudio.web.app:message_handler:38 - ** Processing Agent Message on Queue: Active Connections: ['6c348afa-edfa-494b-ae5c-2ec8e62ddacf'] **
user_proxy (to default_assistant):

Plot a chart of NVDA and TESLA stock price for 2023. Save the result to a file named nvda_tesla.png

--------------------------------------------------------------------------------
2024-07-01 15:16:35.244 | INFO     | autogenstudio.web.app:message_handler:45 - Sending message to connection_id: 6c348afa-edfa-494b-ae5c-2ec8e62ddacf. Connection ID: 6c348afa-edfa-494b-ae5c-2ec8e62ddacf
2024-07-01 15:16:47.488 | INFO     | autogenstudio.web.app:message_handler:38 - ** Processing Agent Message on Queue: Active Connections: ['6c348afa-edfa-494b-ae5c-2ec8e62ddacf'] **
default_assistant (to user_proxy):

2024-07-01 15:16:47.489 | INFO     | autogenstudio.web.app:message_handler:45 - Sending message to connection_id: 6c348afa-edfa-494b-ae5c-2ec8e62ddacf. Connection ID: 6c348afa-edfa-494b-ae5c-2ec8e62ddacf
To plot a chart of NVDA (NVIDIA Corporation) and TESLA stock prices for 2023, we will need to perform the following steps:

1. Collect stock price data for NVDA and TESLA for the year 2023.
2. Plot the data on a chart.
3. Save the chart to a file named `nvda_tesla.png`.

For step 1, we can use a financial data provider like Yahoo Finance. We can access this data using the `yfinance` Python package. If `yfinance` is not installed, we will need to install it using pip.

For step 2 and 3, we will use the `matplotlib` library to plot the chart and save it to a file. If `matplotlib` is not installed, we will also need to install it using pip.

Let's start by installing the required packages using shell script:

```sh
# Install the required Python packages
pip install yfinance --quiet
pip install matplotlib --quiet
```

After installing the necessary packages, we will write a Python script to fetch the stock data, plot the chart, and save it to a file. Please execute the shell script above to install the required packages before running the Python script that I will provide next.

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING CODE BLOCK (inferred language is sh)...
user_proxy (to default_assistant):
2024-07-01 15:16:47.522 | INFO     | autogenstudio.web.app:message_handler:38 - ** Processing Agent Message on Queue: Active Connections: ['6c348afa-edfa-494b-ae5c-2ec8e62ddacf'] **

2024-07-01 15:16:47.522 | INFO     | autogenstudio.web.app:message_handler:45 - Sending message to connection_id: 6c348afa-edfa-494b-ae5c-2ec8e62ddacf. Connection ID: 6c348afa-edfa-494b-ae5c-2ec8e62ddacf
exitcode: 127 (execution failed)
Code output: /Users/robinmanuelthiel/.autogenstudio/files/user/198fb9b77fb10d10bae092eea5789295/None/20240701_15-16-34/Install the required Python packages: line 2: pip: command not found
/Users/robinmanuelthiel/.autogenstudio/files/user/198fb9b77fb10d10bae092eea5789295/None/20240701_15-16-34/Install the required Python packages: line 3: pip: command not found

--------------------------------------------------------------------------------
default_assistant (to user_proxy):
2024-07-01 15:16:54.581 | INFO     | autogenstudio.web.app:message_handler:38 - ** Processing Agent Message on Queue: Active Connections: ['6c348afa-edfa-494b-ae5c-2ec8e62ddacf'] **

It seems that the `pip` command is not found in your environment, which could indicate that Python is not installed or `pip` is not in your PATH. To resolve this, we can try using `python -m pip` instead, which should work if Python is installed and accessible.

Let's try installing the required packages using the `python -m pip` command:

```sh
# Install the required Python packages using python -m pip
python -m pip install yfinance --quiet
python -m pip install matplotlib --quiet
```

Please run the above shell script to install the necessary Python packages. If Python is not installed or if you encounter any further issues, please ensure that Python and pip are correctly installed on your system.
2024-07-01 15:16:54.582 | INFO     | autogenstudio.web.app:message_handler:45 - Sending message to connection_id: 6c348afa-edfa-494b-ae5c-2ec8e62ddacf. Connection ID: 6c348afa-edfa-494b-ae5c-2ec8e62ddacf

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING CODE BLOCK (inferred language is sh)...
user_proxy (to default_assistant):
2024-07-01 15:16:54.617 | INFO     | autogenstudio.web.app:message_handler:38 - ** Processing Agent Message on Queue: Active Connections: ['6c348afa-edfa-494b-ae5c-2ec8e62ddacf'] **

2024-07-01 15:16:54.617 | INFO     | autogenstudio.web.app:message_handler:45 - Sending message to connection_id: 6c348afa-edfa-494b-ae5c-2ec8e62ddacf. Connection ID: 6c348afa-edfa-494b-ae5c-2ec8e62ddacf
exitcode: 127 (execution failed)
Code output: /Users/robinmanuelthiel/.autogenstudio/files/user/198fb9b77fb10d10bae092eea5789295/None/20240701_15-16-34/Install the required Python packages using python -m pip: line 2: python: command not found
/Users/robinmanuelthiel/.autogenstudio/files/user/198fb9b77fb10d10bae092eea5789295/None/20240701_15-16-34/Install the required Python packages using python -m pip: line 3: python: command not found

--------------------------------------------------------------------------------
2024-07-01 15:17:00.044 | INFO     | autogenstudio.web.app:message_handler:38 - ** Processing Agent Message on Queue: Active Connections: ['6c348afa-edfa-494b-ae5c-2ec8e62ddacf'] **
default_assistant (to user_proxy):
2024-07-01 15:17:00.044 | INFO     | autogenstudio.web.app:message_handler:45 - Sending message to connection_id: 6c348afa-edfa-494b-ae5c-2ec8e62ddacf. Connection ID: 6c348afa-edfa-494b-ae5c-2ec8e62ddacf

It appears that the `python` command is not recognized in your environment. This could mean that Python is not installed, or it might be installed under a different name such as `python3`. Let's try using `python3 -m pip` to install the packages:

```sh
# Install the required Python packages using python3 -m pip
python3 -m pip install yfinance --quiet
python3 -m pip install matplotlib --quiet
```

Please run the above shell script to attempt the installation of the necessary Python packages. If this still does not work, it may be necessary to verify that Python is correctly installed on your system and accessible from the command line.

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING CODE BLOCK (inferred language is sh)...
user_proxy (to default_assistant):
2024-07-01 15:17:03.460 | INFO     | autogenstudio.web.app:message_handler:38 - ** Processing Agent Message on Queue: Active Connections: ['6c348afa-edfa-494b-ae5c-2ec8e62ddacf'] **

2024-07-01 15:17:03.460 | INFO     | autogenstudio.web.app:message_handler:45 - Sending message to connection_id: 6c348afa-edfa-494b-ae5c-2ec8e62ddacf. Connection ID: 6c348afa-edfa-494b-ae5c-2ec8e62ddacf
exitcode: 1 (execution failed)
Code output: error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
    xyz, where xyz is the package you are trying to
    install.

    If you wish to install a Python library that isn't in Homebrew,
    use a virtual environment:

    python3 -m venv path/to/venv
    source path/to/venv/bin/activate
    python3 -m pip install xyz

    If you wish to install a Python application that isn't in Homebrew,
    it may be easiest to use 'pipx install xyz', which will manage a
    virtual environment for you. You can install pipx with

    brew install pipx

    You may restore the old behavior of pip by passing
    the '--break-system-packages' flag to pip, or by adding
    'break-system-packages = true' to your pip.conf file. The latter
    will permanently disable this error.

    If you disable this error, we STRONGLY recommend that you additionally
    pass the '--user' flag to pip, or set 'user = true' in your pip.conf
    file. Failure to do this can result in a broken Homebrew installation.

    Read more about this behavior here:

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

[notice] A new release of pip is available: 24.0 -> 24.1.1
[notice] To update, run: python3.12 -m pip install --upgrade pip
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
    xyz, where xyz is the package you are trying to
    install.

    If you wish to install a Python library that isn't in Homebrew,
    use a virtual environment:

    python3 -m venv path/to/venv
    source path/to/venv/bin/activate
    python3 -m pip install xyz

    If you wish to install a Python application that isn't in Homebrew,
    it may be easiest to use 'pipx install xyz', which will manage a
    virtual environment for you. You can install pipx with

    brew install pipx

    You may restore the old behavior of pip by passing
    the '--break-system-packages' flag to pip, or by adding
    'break-system-packages = true' to your pip.conf file. The latter
    will permanently disable this error.

    If you disable this error, we STRONGLY recommend that you additionally
    pass the '--user' flag to pip, or set 'user = true' in your pip.conf
    file. Failure to do this can result in a broken Homebrew installation.

    Read more about this behavior here:

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

[notice] A new release of pip is available: 24.0 -> 24.1.1
[notice] To update, run: python3.12 -m pip install --upgrade pip

--------------------------------------------------------------------------------
default_assistant (to user_proxy):
2024-07-01 15:17:24.982 | INFO     | autogenstudio.web.app:message_handler:38 - ** Processing Agent Message on Queue: Active Connections: ['6c348afa-edfa-494b-ae5c-2ec8e62ddacf'] **
2024-07-01 15:17:24.982 | INFO     | autogenstudio.web.app:message_handler:45 - Sending message to connection_id: 6c348afa-edfa-494b-ae5c-2ec8e62ddacf. Connection ID: 6c348afa-edfa-494b-ae5c-2ec8e62ddacf

The error message indicates that the environment is externally managed and suggests using Homebrew to install system-wide Python packages or using a virtual environment for Python libraries that are not available in Homebrew.

Since we cannot proceed with the installation of the required packages in the current environment, I will provide you with a Python script that assumes the necessary packages (`yfinance` and `matplotlib`) are already installed. You can run this script in an environment where you have the required permissions to install packages or where the packages are already available.

Here is the Python script that fetches the stock data for NVDA and TESLA for the year 2023, plots the chart, and saves it to a file named `nvda_tesla.png`:

```python
# filename: plot_stocks.py
import yfinance as yf
import matplotlib.pyplot as plt

# Define the ticker symbols for NVIDIA and Tesla
nvda_ticker = 'NVDA'
tesla_ticker = 'TSLA'

# Define the start and end dates for the year 2023
start_date = '2023-01-01'
end_date = '2023-12-31'

# Fetch the historical data for both stocks
nvda_data = yf.download(nvda_ticker, start=start_date, end=end_date)
tesla_data = yf.download(tesla_ticker, start=start_date, end=end_date)

# Plot the closing prices of both stocks
plt.figure(figsize=(14, 7))
plt.plot(nvda_data['Close'], label='NVDA')
plt.plot(tesla_data['Close'], label='TESLA')

# Add title and labels to the plot
plt.title('NVDA and TESLA Stock Prices for 2023')
plt.xlabel('Date')
plt.ylabel('Price (USD)')

# Add a legend to the plot
plt.legend()

# Save the plot to a file
plt.savefig('nvda_tesla.png')
print('The chart has been saved as nvda_tesla.png')

# Show the plot
plt.show()
```

Please save the above script to a file named `plot_stocks.py` and run it in an environment where the necessary Python packages are installed. If you are unable to run this script due to the environment restrictions, you may need to set up a proper Python environment with the required permissions first.

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING CODE BLOCK (inferred language is python)...
Traceback (most recent call last):
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogenstudio/web/app.py", line 453, in run_session_workflow
    agent_response: Message = managers["chat"].chat(
                              ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogenstudio/chatmanager.py", line 79, in chat
    result_message: Message = workflow_manager.run(message=f"{message_text}", clear_history=False, history=history)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogenstudio/workflowmanager.py", line 400, in run
    self._run_workflow(message=message, history=history,
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogenstudio/workflowmanager.py", line 104, in _run_workflow
    self.sender.initiate_chat(
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogen/agentchat/conversable_agent.py", line 1018, in initiate_chat
    self.send(msg2send, recipient, silent=silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogen/agentchat/conversable_agent.py", line 655, in send
    recipient.receive(message, self, request_reply, silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogenstudio/workflowmanager.py", line 679, in receive
    super().receive(message, sender, request_reply, silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogen/agentchat/conversable_agent.py", line 820, in receive
    self.send(reply, sender, silent=silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogen/agentchat/conversable_agent.py", line 655, in send
    recipient.receive(message, self, request_reply, silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogenstudio/workflowmanager.py", line 679, in receive
    super().receive(message, sender, request_reply, silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogen/agentchat/conversable_agent.py", line 820, in receive
    self.send(reply, sender, silent=silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogen/agentchat/conversable_agent.py", line 655, in send
    recipient.receive(message, self, request_reply, silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogenstudio/workflowmanager.py", line 679, in receive
    super().receive(message, sender, request_reply, silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogen/agentchat/conversable_agent.py", line 820, in receive
    self.send(reply, sender, silent=silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogen/agentchat/conversable_agent.py", line 655, in send
    recipient.receive(message, self, request_reply, silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogenstudio/workflowmanager.py", line 679, in receive
    super().receive(message, sender, request_reply, silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogen/agentchat/conversable_agent.py", line 820, in receive
    self.send(reply, sender, silent=silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogen/agentchat/conversable_agent.py", line 655, in send
    recipient.receive(message, self, request_reply, silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogenstudio/workflowmanager.py", line 679, in receive
    super().receive(message, sender, request_reply, silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogen/agentchat/conversable_agent.py", line 820, in receive
    self.send(reply, sender, silent=silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogen/agentchat/conversable_agent.py", line 655, in send
    recipient.receive(message, self, request_reply, silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogenstudio/workflowmanager.py", line 679, in receive
    super().receive(message, sender, request_reply, silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogen/agentchat/conversable_agent.py", line 820, in receive
    self.send(reply, sender, silent=silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogen/agentchat/conversable_agent.py", line 655, in send
    recipient.receive(message, self, request_reply, silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogenstudio/workflowmanager.py", line 679, in receive
    super().receive(message, sender, request_reply, silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogen/agentchat/conversable_agent.py", line 820, in receive
    self.send(reply, sender, silent=silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogen/agentchat/conversable_agent.py", line 655, in send
    recipient.receive(message, self, request_reply, silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogenstudio/workflowmanager.py", line 679, in receive
    super().receive(message, sender, request_reply, silent)
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogen/agentchat/conversable_agent.py", line 818, in receive
    reply = self.generate_reply(messages=self.chat_messages[sender], sender=sender)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogen/agentchat/conversable_agent.py", line 1972, in generate_reply
    final, reply = reply_func(self, messages=messages, sender=sender, config=reply_func_tuple["config"])
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogen/agentchat/conversable_agent.py", line 1470, in _generate_code_execution_reply_using_executor
    code_result = self._code_executor.execute_code_blocks(code_blocks)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogen/coding/local_commandline_code_executor.py", line 246, in execute_code_blocks
    return self._execute_code_dont_check_setup(code_blocks)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/robinmanuelthiel/.local/pipx/venvs/autogenstudio/lib/python3.12/site-packages/autogen/coding/local_commandline_code_executor.py", line 304, in _execute_code_dont_check_setup
    result = subprocess.run(
             ^^^^^^^^^^^^^^^
  File<EMAIL_ADDRESS>line 548, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File<EMAIL_ADDRESS>line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File<EMAIL_ADDRESS>line 1955, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'python'

When running autogenstudio ui --port 8081 in a venv, where python and pip (without the 3-suffix) are available, it works.

Thanks for sharing your solution.


Genie 4 Documentation issue : dead link

Source Page: https://netflix.github.io/genie/docs/4.0.2/reference/#_concepts

Under section 7.2 Data model there is a link Genie 3 data model. When i follow this link it shows me  404 error.
Dead link address: https://netflix.github.io/genie/concepts/3/DataModel.html

Thanks for heads up, not sure when they get lost but the newer Data Model section https://netflix.github.io/genie/docs/4.0.2/reference/#_data_model covers everything you should need to know anyway. I'll probably remove the link.

It should link here https://netflix.github.io/genie/concepts/2/DataModel.html


proxy not working

I used with follwoing syntax on my authenticated proxy enabled network

Ouput:
Traceback (most recent call last):
  File "/usr/local/bin/gdown", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/gdown/cli.py", line 105, in main
    use_cookies=not args.no_cookies,
  File "/usr/local/lib/python3.7/site-packages/gdown/download.py", line 110, in download
    res = sess.get(url, stream=True)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 665, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 665, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 166, in resolve_redirects
    raise TooManyRedirects('Exceeded {} redirects.'.format(self.max_redirects), response=resp)
requests.exceptions.TooManyRedirects: Exceeded 30 redirects.

My proxy username and password doesn't contain any special character. curlworks

Sign in</nobr>

Sorry, you can't view or download this file at this time.

Too many users have viewed or downloaded this file recently. Please try accessing the file again later. If the file you are trying to access is particularly large or is shared with many people, it may take up to 24 hours to be able to view or download the file. If you still can't access a file after 24 hours, contact your domain administrator.

© 2021 Google - Help - Privacy & Terms

ProxyChains-3.1 (http://proxychains.sf.net)
|S-chain|-<>-<IP_ADDRESS>:9050-<--timeout
An error has occurred using proxy: None
HTTPSConnectionPool(host='drive.google.com', port=443): Max retries exceeded with url: /uc?id=1iwOO60H_1DdcU1zSIqBv_y5PwXbJHjQ2 (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f14384b53c8>: Failed to establish a new connection: [Errno 111] Connection refused')))

The problem is not related to gdown and proxy-chains. The error says: "Too many users have viewed or downloaded this file recently. Please try accessing the file again later. If the file you are trying to access is particularly large or is shared with many people, it may take up to 24 hours to be able to view or download the file. If you still can't access a file after 24 hours, contact your domain administrator."


Remove hCaptcha from recommended

hCaptcha is NOT a privacy-friendly solution. It violates GDPR, uses third-party tracking and advertising cookies. See report: https://tosdr.org/en/service/2207

added since conflicts appeared from other merges


Implement linear downsampling via resizing

Hey @dstansby, I had a go at implementing the downsampling bit, with a lot of help from @IgorTatarnikov.
Opening a draft PR to get early feedback on this.
We decided to go with an existing dask-ified implementations of resampling rather than coming up with something from scratch. We ended up using ome_zarr.dask_utils.resize which is ome-zarr's wrapper around skimage.transform.resize, and it seems to appropriately handle the chunking for us.
The advantange of ultimately relying on skimage.transform.resize is that we could flexibly rescale each axis by any factor, and choose the order of interpolation (e.g. 0 for 'nearest', 1 for 'linear', etc). Currently, I've hard-coded a factor of 2 and an order of 1 (linear), but these parameters could be easily exposed, if you wish them to. Other parameters, like anti-aliasing can be similarly configured/exposed if needed. For our use-cases, we may end up using different factors per-axis, because much of our data is non-isotropic (thicker in z) and we may want to make it isotropic at downsampled levels.
An alternative approach, which seems to work similarly, is to rely on dask_image.ndinterp.affine_transform, using a scaling matrix.
We also had to fiddle with the metadata a bit, specifically we added the coordinate transforms for each level, which I think is necessary based on visual checks with napari (using the built-in zarr readers).
I've included some rudimentary testing (by adding a few lines to your existing smoke tests).
If you decide to go forward with this implementation, docs will have to be updated as well.
One current limitation is that I've not benchmarked this. It worked fast on the data I had, but this is not a formal peformance guarantee. I saw you have started toying with benchmarking here, so maybe a similar approach can be used for this.
Let me know what you think, and I'll be happy to follow-up on any requested changes.

Thanks for the review. I should be able to do all that next week.

Could you copy the implementation of ome_zarr.dask_utils.resize, instead of depending on ome_zarr?

Yes, I can do that. I guess we'd put that in a separate module. Any preferences regarding that module's name?

Can you put the changes to pyproject.toml in a separate PR (I guess you ran a linter on it)? Would make this PR easier to review

Oops, sorry about that. I think that's because of a VSCode plugin that automatically applied those upon save. will take care of it.

I guess either downsample.py or rebin.py for the new module? I never know which word to use to use 😆 , but thinking a bit more probably downsample is more general so maybe go with that one?

A random thought that just popped into my head - I'm not sure we can (/I want to...) enable anything fancier than binning by two and taking the mean. Because I want to treat the x/y/z dimensions of the output the same, and e.g. not only antialias in the x/y plane, I think it's impossible without reading the whole dataset into memory (which we definitely can't do!) to do anything apart from binning-by-two (or another integer number) and then taking the mean in that bin.

A random thought that just popped into my head - I'm not sure we can (/I want to...) enable anything fancier than binning by two and taking the mean. Because I want to treat the x/y/z dimensions of the output the same, and e.g. not only antialias in the x/y plane, I think it's impossible without reading the whole dataset into memory (which we definitely can't do!) to do anything apart from binning-by-two (or another integer number) and then taking the mean in that bin.

Hmm, we may want to do fancier things. How about we still allow the resize() utility to do the fancier stuff (by exposing all arguments), but within the add_downsample_level() method we hardcode it to use a factor of 2, order=1 (linear) and turn off anti-aliasing? That way we can still re-use the resize() utility in other contexts.
Or do you mean to completely abandon the skimage resize approach?

From a practical point of view I think it's possible to use resize, but from a philosophical point of view I am wary about enabling any processing that treats some image axes different to the others. If you're anti-aliasing in the x-y plane, shouldn't you also be antialiasing in the x-z and y-z planes (or anti-aliasing in 3D, if that's even a thing??)?
I want to keep this package a simple as possible, mainly to keep its maintenance burden as low as possible. So I think we should at least start with just downscaling by binning-by-2 and taking the mean in this PR, and get that right. Once we have that implemented, I'm open to thinking about adding other methods to downsample, but I think worth splitting discussion of that into other issues.

Hey @dstansby, I started implementing some of your suggestions here, but I just saw that you also started working on this problem in the main branch.
I basically copied, modified and refactored ome-zarr's resize function and created a downsample.py module. The underlying code still uses skimage.transform.resize() chunk-wise, but with anti-aliasing turned off. I've also hard-coded a factor of 2 everywhere,.
Do take a look and let me know if it's worth for me to continue down this line, or if you've come up with a better solution meanwhile.

👋 a bit of an update:

I needed to do some downsampling myself last week, so I threw together an option over at https://github.com/HiPCTProject/stack-to-chunk/tree/downsample
I got a bit confused as to how to limit memory properly with dask last week, so decided to ditch dask for everything but storing the data in a dask.array.Array (https://github.com/HiPCTProject/stack-to-chunk/pull/32). With the new approach it's possible to accurately forecast how much memory is going to be used per process (https://github.com/HiPCTProject/stack-to-chunk/pull/33)
I also did a bunch of metadata fixes. The first commit here is useful however downsampling happens (thanks!), so I merged it over at https://github.com/HiPCTProject/stack-to-chunk/pull/38
The updates to the tutorial are also good here, so I'll cherry-pick them in whatever happens.

My current thinking is that https://github.com/HiPCTProject/stack-to-chunk/tree/downsample has two advantages over this PR:

The approach is simpler (it's just manually implemented bin and take the mean), and doesn't rely on scipy.
The approach to multiprocessing is manual, which gives a way to manually limit memory use. In this PR dask is used, which I'm a bit wary of now in terms of keeping memory use low at any one time.

If this PR is faster it might be a better option than https://github.com/HiPCTProject/stack-to-chunk/tree/downsample. I tried to run it on a modest dataset (5GB), but it didn't work 😢 . It created the new zarr dataset, but didn't seem to actually write anything out to the chunks. Which is weird, since it seems to work in the documentation tutorial... have you tried this on local data? Does it work for you?
For a way forward I'm happy to still consider this approach, but I think it's still likely I'd go with https://github.com/HiPCTProject/stack-to-chunk/tree/downsample for the reasons above. So I think the best thing for me to do I turn that into a PR that you folks can try and review?

For a way forward I'm happy to still consider this approach, but I think it's still likely I'd go with downsample for the reasons above. So I think the best thing for me to do I turn that into a PR that you folks can try and review?

That sounds like a good plan. I agree your approach is much simpler and it's nice not to rely on scipy for this.
I'm happy to review it by running it on some of our data.
As for this PR, feel free to cherry-pick whatever bits you like and close it when no longer needed.

I tried to run it on a modest dataset (5GB), but it didn't work

Weird, I haven't tried on "real" data after the latest commits. It should though, since it worked on the cat images and it's basically a refactoring of just using ome-zarr's resize (which I had verified). I might investigate, but looks like it's not high priority given that we're going with a different approach.


Poster/Slide introduction typo

Hi, interesting work!
I think there is a little typo in the "Introduction" part of the poster:
It says "pushs the easier samples away", but should presumably say "pushes the harder samples away".
Edit: This also applies to the "Introduction — Goals" slide.
Edit 2: Never mind, I just misread the sentence. :)
There still is a typo with "pushs" -> "pushes", but that's unimportant.

Will fix the typo. Thanks! :)


How to zoom image when i click button?

I have a button, so i want to zoom image when i click a button. How can i do this? Thank you very much!

Please view the wiki, which has comprehensive guides. https://github.com/davemorrissey/subsampling-scale-image-view/wiki


How can the output data be converted to bvh format?

I'm trying to convert the rotations and joint position into bvh format so that I can do better visualization. I can see that there is an IK method in the motion_process.py file which might help me to get the local rotation information. But it turns out that it's not correct. I fed in the joint positions returned by recover_from_ric method.
Would be appreciate if any help or hint is provided.

Thank you very much for the reply! Quick question on the example data, is it the global or local position of the joints? What would the first entry be like?

Hi, the example data use global positions of joints. It is in the same format as what you obtained through recover_from_ric method. You may also need to add these two lines:
n_raw_offsets = torch.from_numpy(t2m_raw_offsets)
kinematic_chain = t2m_kinematic_chain
So this quat_params give you the local rotations of each bone with respect to the pre-defined offsets.

Hello! has this problem been solved? Can you share the method with me?

Converting the SMPL pose to BVH was easy, and I got that done yesterday.

Converting the SMPL pose to BVH was easy, and I got that done yesterday.

Hello! Could you share the method with me? Thank you very much!

Converting the SMPL pose to BVH was easy, and I got that done yesterday.

Hi, if possible, could you please send the code or github repos? I am very interested. Thanks.

For whom are interested in converting the positions into bvh format, this script can help a lot. https://github.com/DeepMotionEditing/deep-motion-editing/blob/master/utils/InverseKinematics.py What you need to do is 1) finding a smpl template bvh file that contains offset information, 2) using their bvh loader to load bvh file into an Animation object,  3) passing the animation object and positions into this inverse kinematics and get the rotations, 4) write the animation into a bvh file. I have succeeded using other data in a similar way. You may contact me if you have question.

The BasicInverseKinematics should suffice already. For initial rotations at the beginning of optimization, you could just use identity transformation.

Hi i am looking for solution that  converting result npy files to bvh files
so anyone who give me some detail guides and sample codes?
I tried many things but finally failed.

Hi, I got a lot of comments that our current rotation representation seems not compatible to other 3D softwares like blender. I kind of get the reason. In IK/FK in skeleton.py, for i_th bone, we are calculating the rotations for itself. While in bvh, actually we should get the rotations of it parent instead. Therefore, in line 91, you could try to use its parent bone, instead of the bone itself. I am not sure if it works. Here I attach the codes of our FK and bvh FK, you may see the difference while obtaining global positions:
Our FK:
for i in range(1, len(chain)):
      R = qmul(R, quat_params[:, chain[i]])
      offset_vec = offsets[:, chain[i]]
      joints[:, chain[i]] = qrot(R, offset_vec) + joints[:, chain[i-1]]

BVH FK:
for i in range(1, len(self.parents)):
    global_quats[:, i] = qmul(global_quats[:, self.parents[i]], local_quats[:, i])
    global_pos[:, i] = qrot(global_quats[:, self.parents[i]], offsets[:, i]) + global_pos[:, self.parents[i]]

Hope this helps you. I do not have time to validate this idea. But if anyone figure it out in this or any other ways, I would appreciate so much if you could let me know. If it does not work, I know the recent work ReMoDiffuse managed to use the rotation representation in their demo. You may refer to them.

any way to output the resulting animation as an fbx file, or do you know of  way that the npy can be used in a 3d software (ie, blender)?

To get further on this, I tried to modify recover_from_rot to return the local quaternion rotations instead of positions (via FK), using the following code
def recover_from_rot(data, joints_num, skeleton):
    r_rot_quat, r_pos = recover_root_rot_pos(data)

    r_rot_cont6d = quaternion_to_cont6d(r_rot_quat)

    start_indx = 1 + 2 + 1 + (joints_num - 1) * 3
    end_indx = start_indx + (joints_num - 1) * 6
    cont6d_params = data[..., start_indx:end_indx]
    #     print(r_rot_cont6d.shape, cont6d_params.shape, r_pos.shape)
    cont6d_params = torch.cat([r_rot_cont6d, cont6d_params], dim=-1)
    cont6d_params = cont6d_params.view(-1, joints_num, 6)

    rotations = cont6d_to_quat(cont6d_params)
    # Don't calculate positions, return the rotations directly instead
    # positions = skeleton.forward_kinematics_cont6d(cont6d_params, r_pos)

    return r_pos.numpy(), r_rot_quat.numpy(), rotations.numpy()

However, this produces weird rotations when I assume that each of the rotations is per-frame, per-bone local rotations (with shape <frame_count>, <bone_count>, 4>)

Could you help me with this?


@name stays after picking from list

When writing @name and then followed by space and then picking the name the text (@name) sticks. See screenshot

i also got the same issue but only on android can anyone help me


Problems when run on cordova-android@7

hi,
I'm trying to make the test project work with<EMAIL_ADDRESS>but when I run on Android device, the test app starts but outputs only deviceready.
In the Android Studio Logcat this warn comes back:

Rejecting re-init on previously-failed class java.lang.Class<com.example.HelloCJni>: java.lang.UnsatisfiedLinkError: dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/io.cordova.hellocordova-uUJEPTqhZpGH28gw-fLRPg==/base.apk"],nativeLibraryDirectories=[/data/app/io.cordova.hellocordova-uUJEPTqhZpGH28gw-fLRPg==/lib/arm64, /system/lib64, /system/vendor/lib64]]] couldn't find "libhelloc.so"
java.lang.NoClassDefFoundError: com.example.HelloCJni
at com.example.HelloCJni.calculate(Native Method)
at com.example.HelloCPlugin.execute(HelloCPlugin.java:35)
at org.apache.cordova.CordovaPlugin.execute(CordovaPlugin.java:98)
at org.apache.cordova.PluginManager.exec(PluginManager.java:132)
at org.apache.cordova.CordovaBridge.jsExec(CordovaBridge.java:59)
at org.apache.cordova.engine.SystemExposedJsApi.exec(SystemExposedJsApi.java:41)
at android.os.MessageQueue.nativePollOnce(Native Method)
at android.os.MessageQueue.next(MessageQueue.java:325)
at android.os.Looper.loop(Looper.java:142)
at android.os.HandlerThread.run(HandlerThread.java:65)
Caused by: java.lang.UnsatisfiedLinkError: dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/io.cordova.hellocordova-uUJEPTqhZpGH28gw-fLRPg==/base.apk"],nativeLibraryDirectories=[/data/app/io.cordova.hellocordova-uUJEPTqhZpGH28gw-fLRPg==/lib/arm64, /system/lib64, /system/vendor/lib64]]] couldn't find "libhelloc.so"
at java.lang.Runtime.loadLibrary0(Runtime.java:1011)
at java.lang.System.loadLibrary(System.java:1657)
at com.example.HelloCJni.(HelloCJni.java:12)
at com.example.HelloCJni.getArch(Native Method)
at com.example.HelloCPlugin.execute(HelloCPlugin.java:28)

I do not know if this is the problem but before the support at cordova-android@7, the test app worked with cordova-platform@6.
Thanks

Looks like the shared object libraries didn't get deployed to the Android project.
Try updating you CLI to cordova@8.

I updated the Cordova CLI to the latest version (8.1.2) and re-added the cordova-android@7 but the situation is the same.

OK, I'll take a look. Maybe the SO's are not getting deployed into the right place for cordova-android@7 since I updated the platform version...

The Android libs path has changed in cordova-android@7 - the above commit updates this plugin to use the new path so it should now run OK with cordova-android@7

@brodybits yeah, in a "real" plugin it would - cordova-android@7 has caused some really breaking changes.
But since this one is just an example plugin and isn't published to npm, the semantic versioning is redundant so I won't bother here.
BTW nice idea though to workaround it by wrapping all the .so files in a .jar

One more thing that is not clear to me. I would like to add my .c and .h files and the general flow would be:

replace by adding the C source codes to src/common/
adapt the HelloCPlugin.java and HelloCJni.java files
and recompile the new libraries plugins/cordova-plugin-hello-c/compile-android
It's correct?

yep, but if you're doing this with the plugin in-place inside an app project (i.e. myapp/plugins/cordova-plugin-hello-c), then you'll need to remove/re-add the platform to make the changes take effect:
cordova platform rm android --nosave && cordova platform add android --nosave

Otherwise, if your plugin lives outside of the your app directory, you can do:
cordova plugin rm cordova-plugin-hello-c --nosave && cordova plugin add /local/path/to/cordova-plugin-hello-c --nosave

Okay thank you

I just raised https://github.com/apache/cordova-docs/issues/902 to get this documented properly.

FYI I just raised https://github.com/apache/cordova-android/issues/547 to deal with the large number of plugins not working on cordova-android@7.

@brodybits thanks, I have had compatibility problems reported against<EMAIL_ADDRESS>which appears to make some breaking changes, e.g https://github.com/dpa99c/cordova-android-support-gradle-release/issues/32#issuecomment-437342785
I've worked around that particular issue (change in deployment location of Gradle files) in a new plugin version.
As for this example plugin, it's now the new version 18 of the Android NDK which is causing me issues - although I've updated the plugin to deploy the so files to the new location used by cordova-android@7 the C implementations of the JNI interface functions are not being found at runtime. Hopefully it's just a question of updating the C function signatures.


fix: accepting only maxConnections > 0

The package @grpc/grpc-js different from grpc, doesn't work properly with transient instances, it has a memory leak when it is used like that, so, it's important to avoid using maxConnections = 0.
In that way, this PR makes it forbidden

That is a breaking change, and you can alert first or up release to next major release.


External vulnerability scan results for review

Vulnerability Scan Report
@johnmod3 sent an email in with a couple of attachments concerning vulnerabilities scanned in this repository.
I've take a quick look and these seem to be a subset of our dependabot and CodeQL findings.   I believe that most of these dependency bumps are going to be addressed in the next release.  Some have been captured in the current crop of dependabot PRs.
I'm attaching for further review by the CSET development team.
Thank you @johnmod3 for your contribution.  Having extra sets of eyes is why we're here!  👀
Attachments:

Project Analysis Summary _ Ion Console.pdf
CISA_1dad9f26-ca07-4a29-9e74-5c648280e9f0.json

Updated all


Feature/setup knowledge graph component

You can test the component at: http://localhost:3000/testGraph
It uses mock data for now.
You can emulate the error handling by providing a true value to the setError method and clicking on a node.
The testing page will be removed once this component can be added to the sidebar.

Since it is mostly UI stuff, I'd let others to provide approval. One thing I felt I had to zoom too much to get to a particular node, in case the graph is very big. Can we make it so, it automatically zooms in on the main paper and user can navigate from there?

Thanks for the feedback! Maybe we can ask the customers about this, I think that makes the most sense here. It may not be super-intuitive to immediately zoom into the middle node, the user could feel a bit confused. It may, however,  be possible to zoom into a node immediately after it is selected. But I could take a look at this in the future if it seems like something we need.


add random play

Use Math.floor(Math.random() * player.queue.length) to specify the index of the song to achieve random play

I feel like this is one of those things that users should implement themselves or make a plugin for

Agreed, If I am understanding this correctly, this is technically just an implementation of the shuffle method on the Queue class with the ability to restore the original order of the queue. Probably wouldn't be used very often.


Icon for the Most Used category

Fixes #6

https://mybinder.org/v2/gh/fcollonval/jlab-enhanced-launcher/fcollonval%2Fissue6?urlpath=lab


Editor Plugin fails to reference a singleton that is intended to load with itself.

Godot version:
3.1.1 stable official x64
OS/device including version:
Windows 10 1709 x64
Issue description:
Singleton, spawned in editor plugin main file with add_autoload_singleton(), and deleted with remove_autoload_singleton(), prevents to reactivate plugin if it's use was implemented to code in-place and then project reopened.

Steps to reproduce:

Create an EditorPlugin.
Create singleton file with some variable.
Write add_autoload_singleton("SingletonName", "res://singleton_file.gd") to EditorPlugin's _enter_tree().
Write remove_autoload_singleton("SingletonName") to EditorPlugin's _exit_tree().
Go to Project Settings -> Plugins and activate plugin.
Add some use of singleton variables to EditorPlugin's main file, like
var a = SingletonName.test_var
Go to Project Settings -> Plugins and disable plugin.
Return to Project List and reopen project.
Go to Project Settings -> Plugins and try to activate plugin.
See a warning window and error of absent singleton that is intended to load.

Workaround:
Create needed singleton with needed name yourself and THEN activate plugin. It will work.
Minimal reproduction project:
http://www.mediafire.com/file/o0qjswb0r074zik/PluginAutoload.zip/file

Still valid in 3.2.3 rc1
The way I can think of to fix this is to initially ignore the error when enabling the plugin and then check again if it works (in case of the autoload the error would be gone)
There's however a workaround: store the singleton in a variable by using var singleton = OS.get_singleton("SingletonName"). It's probably less convenient though.

hi try to use singleton (auto load) inside a plugin but it still fails.
I add it by following code
tool
extends EditorPlugin
func _init():
    add_autoload_singleton("Global", "res://Global.gd")
    prints("singleton", Engine.has_singleton("Global"), Engine.get_singleton("Global"))

The Global.gd script
extends Node

func _init():
    prints("auto load succes")

The error output is
Autoload hinzufügen
 Failed to retrieve non-existent singleton 'Global'.
singleton False [Object:null]

When i look under project settings the singleton is correct loaded, but i can't access from plugin context

The hacky solution that seems to work for me is:

The plugin adds itself to a group called "Plugin", then add_autoload_singleton(mySingleton)
the singleton has a _enter_tree() function that  uses get_tree().call_group("Plugin", initSingleton, self)
the plugin has a function initSingleton(who) that assigns the singleton to a variable that it can then use later

Not sure if there's a more elegant hack, or a correct way to deal with this?

Simpler workaround:
Don't write the autoload's name directly. Use get_node("/root/NameOfTheAutoload"). Autoloads are children of root so they can be accessed that way if the script using them can be loaded before autoloads exist.

Simpler workaround (also a legit one, not a hack): Don't write the autoload's name directly. Use get_node("/root/NameOfTheAutoload"). Autoloads are children of root so they can be accessed that way if the script using them can be loaded before the autoload exists. If the script wants to access the autoload before it exists though, it can use has_node to check it. But in this case that would seem like a design issue.

This has issues in some cases. An autoload can be called from the Object class, but you can't use get_node(). There would have to be a way to turn off reference errors when the plugin is disabled.


Add socialite provider for gitea

http://gitea.io/ is a self-hosted Git service like GitLab, but it uses slightly different OAuth2 endpoints. This PR makes it possible for us at Codeberg.org (which is the reason why I have created this PR) as well as many other people, organizations and companies to use BookStack together with their Gitea instances.

Thanks for offering this @moqmar, But I've come to the decision I don't want to grow out the supported third-party social provides much further, at least without a high level of benefit/request from the user-base. It's not a large amount of work to initially implement but it gets troublesome attempting to support people here when issues eventually arise.
Within the latest release, v21.04, I've added a system where you can perform some level of back-end customization. Details on this system can be seen here: https://github.com/BookStackApp/BookStack/blob/release/dev/docs/logical-theme-system.md
It's still in the testing stages, and I'm looking for feedback on it, but it should be possible to use additional socialite providers via this system, An example of this is shown at the bottom using Reddit which I tested during development of that system.


steal-stache/can missing after add component

if having generator add dependencies is out of scope at least edit the guide https://donejs.com/plugin.html#show-it-off
or have the plugin generator add these dependencies

Thanks for filing @janat08 !  I don't quite understand the problem. Are you saying that steal-stache and can are removed as dependencies after running donejs add component my-component?

No, nothing ever adds those dependencies.

@janat08 Do you mean the guide should show how to add canjs dependencies, or install those dependencies when generating a plugin?
If yes, what if the plugin doesn't use canjs?

I don't know anymore.

I think what we should confirm is you can:

donejs add plugin
donejs add component my-component
confirm the package.json has can and steal-stache as dependencies
confirm the package.json has steal-stache as a steal plugin like https://github.com/canjs/canjs/blob/175036d7376a88eb5fc5e6f550066d64c52493ad/package.json#L219-L221

If those are true, we can re-close.


Create codeql.yml

The issue was with java code not being built, fixed it in a PR - https://github.com/Azure/azure-functions-openai-extension/pull/62
Hence, closing this one.


Add unit test on pkg/chaosdaemon

What problem does this PR solve?
Add unit test
What is changed and how does it work?
Add unit test
Check List
Tests

Unit test

Code changes

Has Go code change

Side effects

Breaking backward compatibility

Related changes

Need to update the documentation

Does this PR introduce a user-facing change?:

NONE

/test

/test

Codecov Report

Merging #271 into master will increase coverage by 6.59%.
The diff coverage is 92.3%.

@@            Coverage Diff             @@
##           master     #271      +/-   ##
==========================================
+ Coverage   44.17%   50.77%   +6.59%
==========================================
  Files          33       33
  Lines        1469     1479      +10
==========================================
+ Hits          649      751     +102
+ Misses        740      647      -93
- Partials       80       81       +1

Impacted Files
Coverage Δ

pkg/chaosdaemon/util.go
76.25% <ø> (+2.5%)
:arrow_up:

pkg/chaosdaemon/netem_linux.go
17.5% <100%> (+17.5%)
:arrow_up:

pkg/chaosdaemon/iptables_server.go
100% <100%> (+100%)
:arrow_up:

pkg/chaosdaemon/server.go
44.44% <50%> (+44.44%)
:arrow_up:

pkg/chaosdaemon/netem_server.go
100% <0%> (+100%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a4b9f26...c7ad63d. Read the comment docs.

/merge


The TextWriter writer should not be closed in the WriteStream method

The WriteStream in the SrtWriter method might called multiple times. I think the TextWriter writer should not be closed early in this method.
https://github.com/AlexPoint/SubtitlesParser/blob/19ab8d31859e4556ee8adbe4649fe837d4522228/SubtitlesParser/Classes/Writers/SrtWriter.cs#L58-L72

Good catch!
Indeed, the TextWriter object should be closed to avoid any issue.
Could you submit a fix so that I can package a corrective version?


Reading S3 files becomes slow after 1.5.4

As mentioned earlier in #74, it appears that the reading speed is very slow after 1.5.4.
$ pyvenv-3.4 env
$ source env/bin/activate
$ pip install smart_open==1.5.3 tqdm ipython
$ ipython

from tqdm import tqdm
from smart_open import smart_open
for _ in tqdm(smart_open('s3://xxxxx', 'rb')):
    pass

2868923it [00:53, 53888.94it/s]
$ pyvenv-3.4 env
$ source env/bin/activate
$ pip install smart_open==1.5.4 tqdm ipython
$ ipython

from tqdm import tqdm
from smart_open import smart_open
for _ in tqdm(smart_open('s3://xxxxx', 'rb')):
    pass

8401it [00:18, 442.64it/s] (too slow so I could not wait for it to finish.)

Thanks for report @appierys, that's really suspicious, we'll investigate it, related issue #151.

I'm looking into this now.

I've reproduced the issue, but the performance gap between 1.5.3 and .4 wasn't as large as originally reported. @appierys, could you tell us the extension of the S3 file you were trying to open? The extension influences how smart_open actually opens the file.
Here are my efforts to reproduce the bug:
(so1.5.3)sergeyich:issue152 misha$ time python reproduce.py s3://commoncrawl/cc-index/collections/CC-MAIN-2013-20/indexes/cluster.idx
705it [00:01, 383.09it/s]

real    0m3.395s
user    0m0.233s
sys     0m0.044s
(so1.5.3)sergeyich:issue152 misha$ deactivate
sergeyich:issue152 misha$ source so1.5.4/bin/activate
(so1.5.4)sergeyich:issue152 misha$ time python reproduce.py s3://commoncrawl/cc-index/collections/CC-MAIN-2013-20/indexes/cluster.idx
977it [00:15, 64.74it/s]
real    0m18.065s
user    0m0.813s
sys     0m0.164s
(so1.5.4)sergeyich:issue152 misha$

1.5.4 appears to be around 6 times slower than 1.5.3.
reproduce.py:
import sys

from tqdm import tqdm
from smart_open import smart_open
for i, _ in enumerate(tqdm(smart_open(sys.argv[1], 'rb'))):
    if i > 1e3:
        break

Some basic profiling:

Examining the profiles:
(so1.5.3)sergeyich:issue152 misha$ python -m pstats 153.out
Welcome to the profile statistics browser.
153.out% sort time
153.out% stats 10
Sat Dec  2 00:14:37 2017    153.out

         160797 function calls (156627 primitive calls) in 7.134 seconds

   Ordered by: internal time
   List reduced from 2097 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       32    5.513    0.172    5.513    0.172 {method 'read' of '_ssl._SSLSocket' objects}
        1    0.921    0.921    0.921    0.921 {method 'do_handshake' of '_ssl._SSLSocket' objects}
        1    0.250    0.250    0.250    0.250 {method 'connect' of '_socket.socket' objects}
      250    0.057    0.000    0.057    0.000 {method 'read' of '_io.FileIO' objects}
      250    0.041    0.000    0.041    0.000 {built-in method marshal.loads}
        1    0.037    0.037    0.037    0.037 {built-in method _socket.getaddrinfo}
     1464    0.029    0.000    0.029    0.000 {built-in method posix.stat}
  846/845    0.024    0.000    0.045    0.000 {built-in method builtins.__build_class__}
      250    0.012    0.000    0.069    0.000 <frozen importlib._bootstrap_external>:830(get_data)
       25    0.011    0.000    0.011    0.000 {built-in method _imp.create_dynamic}

153.out%
Goodbye.
(so1.5.3)sergeyich:issue152 misha$ python -m pstats 154.out
Welcome to the profile statistics browser.
154.out% sort time
154.out% stats 10
Sat Dec  2 00:14:06 2017    154.out

         777545 function calls (770324 primitive calls) in 9.819 seconds

   Ordered by: internal time
   List reduced from 3531 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       47    7.066    0.150    7.066    0.150 {method 'read' of '_ssl._SSLSocket' objects}
        2    0.932    0.466    0.932    0.466 {method 'do_handshake' of '_ssl._SSLSocket' objects}
        2    0.403    0.202    0.403    0.202 {method 'connect' of '_socket.socket' objects}
      415    0.158    0.000    0.158    0.000 {method 'read' of '_io.FileIO' objects}
   118917    0.135    0.000    0.145    0.000 /Users/misha/git/smart_open/issue152/so1.5.4/lib/python3.6/site-packages/smart_open/s3.py:228(_read_from_buffer)
   118917    0.082    0.000    6.312    0.000 /Users/misha/git/smart_open/issue152/so1.5.4/lib/python3.6/site-packages/smart_open/s3.py:172(read)
      415    0.081    0.000    0.081    0.000 {built-in method marshal.loads}
     2828    0.058    0.000    0.058    0.000 {built-in method posix.stat}
     1003    0.055    0.000    6.369    0.006 /Users/misha/git/smart_open/issue152/so1.5.4/lib/python3.6/site-packages/tqdm/_tqdm.py:923(__iter__)
1412/1408    0.053    0.000    0.086    0.000 {built-in method builtins.__build_class__}

Interestingly, 1.5.4 calls functions like _SSLSocket.read and _FileIO.read more often than 1.5.3. The former also appears to be opening two SSL connections (connect, handshake).
1.5.4 spends the majority of the time in a read() method inside s3.py. The number of calls is quite large: perhaps the buffer size is not being set correctly?

153.out.gz
154.out.gz

Hi,
I also investigated this issue, as I encountered the performance problem too.
Here is my investigation.
I used reproduce.py written by @mpenkov .
Confirming 1.5.3 and 1.5.4 performance:
tamaki@tamaki-ThinkPad-X201 ~/proj/smart_open (master)
$ virtualenv venv
tamaki@tamaki-ThinkPad-X201 ~/proj/smart_open (master)
$ . ./venv/bin/activate
(venv) tamaki@tamaki-ThinkPad-X201 ~/proj/smart_open (master)
$ pip install -e . tqdm
(venv) tamaki@tamaki-ThinkPad-X201 ~/proj/smart_open (master)
$ git checkout 1.5.3
(venv) tamaki@tamaki-ThinkPad-X201 ~/proj/smart_open ((1.5.3))
$ time python reproduce.py s3://otamachan-test-us-east-1/test.tgz
620it [00:01, 500.94it/s]

real    0m2.447s
user    0m0.156s
sys     0m0.016s
(venv) tamaki@tamaki-ThinkPad-X201 ~/proj/smart_open ((1.5.3))
$ git checkout 1.5.4
(venv) tamaki@tamaki-ThinkPad-X201 ~/proj/smart_open ((1.5.4))
$ time python reproduce.py s3://otamachan-test-us-east-1/test.tgz
976it [00:07, 135.19it/s]

real    0m8.429s
user    0m0.892s
sys     0m0.064s

After increasing the buffer size by 10, the performance looks better.
(venv) tamaki@tamaki-ThinkPad-X201 ~/proj/smart_open ((1.5.4))
$ git diff
diff --git a/smart_open/s3.py b/smart_open/s3.py
index 420afbd..2b0fcae 100644
--- a/smart_open/s3.py
+++ b/smart_open/s3.py
@@ -196,7 +196,7 @@ class BufferedInputBase(io.BufferedIOBase):
         #
         # logger.debug('filling %r byte-long buffer up to %r bytes', len(self._buffer), size)
         while len(self._buffer) < size and not self._eof:
-            raw = self._raw_reader.read(size=io.DEFAULT_BUFFER_SIZE)
+            raw = self._raw_reader.read(size=io.DEFAULT_BUFFER_SIZE*10)
             if len(raw):
                 self._buffer += raw
             else:
$ time python reproduce.py s3://otamachan-test-us-east-1/test.tgz
976it [00:03, 287.71it/s]

real    0m4.676s
user    0m1.260s
sys     0m0.040s

However, when I increase the buffer size by 100, the performance gets worse.
(For a larger file such as 1G file, I think the buffer size should be increased to dozens of mega bytes.)
(venv) tamaki@tamaki-ThinkPad-X201 ~/proj/smart_open ((1.5.4))
$ git diff
diff --git a/smart_open/s3.py b/smart_open/s3.py
index 420afbd..d87881e 100644
--- a/smart_open/s3.py
+++ b/smart_open/s3.py
@@ -196,7 +196,7 @@ class BufferedInputBase(io.BufferedIOBase):
         #
         # logger.debug('filling %r byte-long buffer up to %r bytes', len(self._buffer), size)
         while len(self._buffer) < size and not self._eof:
-            raw = self._raw_reader.read(size=io.DEFAULT_BUFFER_SIZE)
+            raw = self._raw_reader.read(size=io.DEFAULT_BUFFER_SIZE*100)
             if len(raw):
                 self._buffer += raw
             else:
(venv) tamaki@tamaki-ThinkPad-X201 ~/proj/smart_open ((1.5.4))
$ time python reproduce.py s3://otamachan-test-us-east-1/test.tgz
997it [00:17, 56.48it/s]
real    0m18.878s
user    0m16.272s
sys     0m0.072s

I assumed this is because the byte string slicing is becoming the bottleneck, I refactored the code without using slicing(https://github.com/otamachan/smart_open/commit/6989112bab1b79d507185644f5946320b30324c4)
The result:
(venv) tamaki@tamaki-ThinkPad-X201 ~/proj/smart_open ((1.5.4))
$ git remote add otamachan https://github.com/otamachan/smart_open.git
(venv) tamaki@tamaki-ThinkPad-X201 ~/proj/smart_open ((1.5.4))
$ git fetch otamachan
(venv) tamaki@tamaki-ThinkPad-X201 ~/proj/smart_open ((1.5.4))
$ git checkout improve-read-performance
(venv) tamaki@tamaki-ThinkPad-X201 ~/proj/smart_open (improve-read-performance)
$ time python reproduce.py s3://otamachan-test-us-east-1/test.tgz
997it [00:02, 419.11it/s]

real    0m3.613s
user    0m0.596s
sys     0m0.092s

Looks better but still worse than 1.5.3 .
Hope this helps.

Hi @mpenkov, perhaps the gap is larger for me because I am accessing the bucket from a different region while the bucket is in us-east-1.
In addition, the speed of 1.5.3 is initially slow, but gets faster over time, so if you only test 1K lines, the difference would be smaller.
The file extension is .json, but it's actually many lines of jsons, and the total file size is 920M.

Some statistics some 1e3 is changed to 1e6:
Results from 1.5.3:
(1.5.3) $ for i in 1 2 3 4 5; do time python reproduce.py s3://commoncrawl/cc-index/collections/CC-MAIN-2013-20/indexes/cluster.idx ; done
598857it [00:43, 13639.56it/s]

real    0m45.683s
user    0m4.916s
sys     0m0.876s
598857it [00:42, 14215.39it/s]

real    0m43.882s
user    0m4.688s
sys     0m0.784s
598857it [00:35, 16742.60it/s]

real    0m37.726s
user    0m4.744s
sys     0m0.836s
598857it [01:35, 6293.18it/s]

real    1m36.958s
user    0m5.036s
sys     0m0.784s
598857it [00:50, 11935.76it/s]

real    0m52.096s
user    0m5.020s
sys     0m0.772s

For 1.5.4, it takes much time to complete the test, but I will update this comment once I have the results.

@appierys thank you for providing more details.
I identified the problem as a buffering issue. It’s addressed in this branch: https://github.com/mpenkov/smart_open
Could you try it and let me know if it resolves your issue?

Hi @mpenkov,
Thanks. The results from your master branch is as below, it's faster than before but still slower than 1.5.3 when 600K lines are read:
(master) $ for i in 1 2 3 4 5; do time python reproduce.py s3://commoncrawl/cc-index/collections/CC-MAIN-2013-20/indexes/cluster.idx ; done
598857it [02:00, 4958.64it/s]

real    2m3.100s
user    0m16.044s
sys     0m0.760s
598857it [03:02, 3284.05it/s]

real    3m4.428s
user    0m16.248s
sys     0m0.764s
598857it [03:15, 3064.41it/s]

real    3m17.652s
user    0m16.532s
sys     0m0.532s
598857it [02:56, 3389.60it/s]

real    2m58.812s
user    0m16.764s
sys     0m0.696s
598857it [02:40, 3727.32it/s]

real    2m43.201s
user    0m14.988s
sys     0m0.844s

@appierys OK, I can see that it is still several times slower. This matches the results on my side as well.  Will continue looking into it. Thank you.

Hi,
I also implemented the readline method and made the raw reader threaded.(https://github.com/otamachan/smart_open/commit/569169d2c800094a4b5cfed868cac07c1090b272)
When I increase the buffer size to 20 Mbytes, it looks more better but still worse than 1.5.3.
(venv) tamaki@tamaki-ThinkPad-X201 ~/proj/smart_open (master)
$ git checkout 1.5.3
(venv) tamaki@tamaki-ThinkPad-X201 ~/proj/smart_open ((1.5.3))
$ for i in 1 2 3 4 5; do time python reproduce.py s3://commoncrawl/cc-index/collections/CC-MAIN-2013-20/indexes/cluster.idx ; done598857it [00:32, 18355.81it/s]

real    0m33.760s
user    0m2.904s
sys     0m0.444s
598857it [00:24, 24435.69it/s]

real    0m25.990s
user    0m2.724s
sys     0m0.332s
598857it [00:27, 22121.49it/s]

real    0m28.339s
user    0m2.944s
sys     0m0.276s
598857it [00:25, 23430.73it/s]

real    0m26.758s
user    0m2.684s
sys     0m0.392s
598857it [00:34, 17357.63it/s]

real    0m35.697s
user    0m2.848s
sys     0m0.364s
$ git checkout improve-read-performance
(venv) tamaki@tamaki-ThinkPad-X201 ~/proj/smart_open (improve-read-performance)
$ for i in 1 2 3 4 5; do time python reproduce.py s3://commoncrawl/cc-index/collections/CC-MAIN-2013-20/indexes/cluster.idx ; done
598857it [00:35, 16928.79it/s]

real    0m36.654s
user    0m2.252s
sys     0m0.604s
598857it [00:36, 16548.59it/s]

real    0m37.339s
user    0m2.488s
sys     0m0.732s
598857it [00:32, 18385.05it/s]

real    0m33.761s
user    0m2.244s
sys     0m0.580s
598857it [00:28, 20821.97it/s]

real    0m29.974s
user    0m2.216s
sys     0m0.716s
598857it [00:24, 24546.06it/s]

real    0m25.625s
user    0m2.184s
sys     0m0.616s

Thanks.

What's behind this decrease in performance, despite buffering? What changed?

I assume that
1.5.3 uses https://github.com/boto/boto/blob/develop/boto/s3/key.py#L308 .
This is one GET request and reads the body asynchronously. But this is not seekable.
On the other hand,
1.5.4 uses https://github.com/RaRe-Technologies/smart_open/blob/master/smart_open/s3.py#L89  .
This separates the GET request into several parts and waits until the reading is done. This decreases the throughput but is seekable.

@piskvorky Tamaki is right. I implemented the new seekable S3 reader based on boto3’s ranged GET queries, as we discussed in Issue #42. It’s possible that boto3 introduced some overhead, but I’m surprised it’s this much.
I’m still hunting around with a profiler.

@otamachan Threading is a good idea for some applications, but I don't think introducing it is appropriate, because it limits our (and our users') options with regards to multiprocessing. More specifically, once you start up threads, you can't fork your process (because the that will copy synchronization primitives like mutexes to the child process).
This means people who already use smart_open in a multiprocessing-based application will experience problems.

@mpenkov Thanks for your comment. I agree with you. I just wanted to confirm whether threading could improve the performance or not. For the reproduce.py test, the performance was not be so improved even when threading is introduced. It should be avoided for multiprocessing applications as you said.

Here are some more profiling results. They were obtained by:
$ time python -m cProfile -o master.out reproduce.py s3://commoncrawl/cc-index/collections/CC-MAIN-2017-34/indexes/cluster.idx 100000
$ time python -m cProfile -o master.out reproduce.py s3://commoncrawl/cc-index/collections/CC-MAIN-2017-39/indexes/cluster.idx 100000

using theh 6c22ed8 commit.
I used this reproduce.py:
import sys
import logging

from tqdm import tqdm
from smart_open import smart_open

logging.basicConfig(level=logging.CRITICAL)
length_sum = 0
limit = int(sys.argv[2])
for i, line in enumerate(tqdm(smart_open(sys.argv[1], 'rb'))):
    if i > limit:
        break
    length_sum += len(line)
print(length_sum / i)

I used different input files from the same bucket because I noticed there is caching behavior that confounds tests. So, for timing, it makes sense to change the input file (but keep the same number of iterations) between tests.
I found that for both 1.5.3 and the current master branch (mine), the majority of the time is being spent in SSLSocket.read. (top: 1.5.3, bottom: master)
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  2679   71.599    0.027   71.599    0.027 {method 'read' of '_ssl._SSLSocket' objects

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  1428  110.426    0.077  110.426    0.077 {method 'read' of '_ssl._SSLSocket' objects}

1.5.3 calls the method 2679 times, taking 0.027s per call. master calls the method 1428 times, taking 0.077 per call. This is rather odd given that they're reading approximately the same amounts of data.
Perhaps master has a buffer that is twice the size of 1.5.3's counterpart somewhere? That would explain the larger number of calls, but not why they are taking so long. (2679/1428 ~= 2, but 0.027 / 0.077 ~= 3). I tried adjusting the buffer in the s3 module, but it appears to have no effect.
153.out.gz
master.out.gz

Need to continue investigation here (but @mpenkov fix it partially in #155)

Hi,
I finally found a way to get not only seekability but also almost same performance as 1.5.3 .
https://github.com/otamachan/smart_open/commit/8089460fc3bddfb325b3ae8026df6a1ca0b95a4b
Does this work?
(venv) tamaki@tamaki-ThinkPad-X201 ~/proj/smart_open (improve-read-performance2)
$ git checkout 1.5.3
(venv) tamaki@tamaki-ThinkPad-X201 ~/proj/smart_open ((1.5.3))
$ for i in 1 2 3 4 5; do time python reproduce.py s3://commoncrawl/cc-index/collections/CC-MAIN-2013-20/indexes/cluster.idx ; done
598857it [00:45, 13091.32it/s]

real    0m46.958s
user    0m3.056s
sys     0m0.376s
598857it [00:26, 22613.33it/s]

real    0m27.647s
user    0m2.720s
sys     0m0.364s
598857it [00:25, 23576.58it/s]

real    0m26.655s
user    0m2.800s
sys     0m0.340s
598857it [00:21, 27806.55it/s]

real    0m22.771s
user    0m2.848s
sys     0m0.288s
598857it [00:35, 16639.95it/s]

real    0m37.176s
user    0m2.952s
sys     0m0.352s
$ git checkout improve-read-performance2
(venv) tamaki@tamaki-ThinkPad-X201 ~/proj/smart_open (improve-read-performance2)
$ for i in 1 2 3 4 5; do time python reproduce.py s3://commoncrawl/cc-index/collections/CC-MAIN-2013-20/indexes/cluster.idx ; done
598857it [00:31, 19086.47it/s]

real    0m34.027s
user    0m9.868s
sys     0m0.184s
598857it [00:26, 22517.10it/s]

real    0m29.215s
user    0m10.084s
sys     0m0.184s
598857it [00:26, 22240.85it/s]

real    0m29.382s
user    0m9.848s
sys     0m0.224s
598857it [00:37, 15814.44it/s]

real    0m40.821s
user    0m10.348s
sys     0m0.244s
598857it [00:26, 22711.12it/s]

real    0m28.834s
user    0m9.936s
sys     0m0.224s

@otamachan Thank you for your suggestion! I had a closer look at your implementation and finally realized what the remaining problem was. It isn't necessary to go back to boto to achieve the same performance: we can do the same thing with the newer boto3.
https://github.com/RaRe-Technologies/smart_open/pull/157
Thank you for pointing me the right way. どうもありがとうございました！

@mpenkov I'm happy that the performance is now back by your right implementation and I could be of help.  I really appreciate your great effort to improve this wonderful library!
こちらこそありがとうございます！

Not sure if it's just me, but I tested 1.5.5 and obtained the following results:
for i in 1 2 3 4 5; do time python reproduce.py s3://commoncrawl/cc-index/collections/CC-MAIN-2013-20/indexes/cluster.idx ; done
598857it [03:12, 3103.18it/s]

real    3m15.232s
user    0m17.400s
sys     0m0.800s
598857it [03:14, 3081.01it/s]

real    3m16.483s
user    0m16.724s
sys     0m0.812s
598857it [03:12, 3115.35it/s]

@appierys This was fixed by fbc82cc04bf92e8c588c49c43ff5aff8234ea87d and the commit is only in the master branch. Please try the master branch.
@menshikh-iv I would appreciate if you would release 1.5.6 because the performance issue is not fully resolved by 1.5.5 . Thanks in advance.

@otamachan Thanks~ It works.

@otamachan I'll release 1.5.6 but slightly later (after setup integration testing contour).


Map /dls/science/users/ space

If it is set up for current user. This is an additional area
used for storing stuff and development.

Why is .devcontainer.json a hidden file in this dir and not in dev-all?

That is because dev-all has a .devcontainer hidden folder. In this instance the folder also containers the Dockerfile to build the image that the devcontainer.json uses.
for dev-c7 the .devcontainer file is standalone and refers to the image that we would have already built and published to GHCR.
In reality the .devcontainer file does not really belong in dev-c7, it is the file that you would extract and drop into your folder that you want to automatically launch dev-c7 inside of vscode.  Hopefully this will become clearer when I document dev-c7 soon.
Feel free to come and discuss if this does not make sense!


Move the MC implementation to the common library

This avoids code duplication.
Review at https://rbcommons.com/s/OpenH264/r/1102/.

I'll try to have a look to see why the travis build failed - it does seem like something is crashing, but it does seem to run clean in valgrind locally, so I'm not sure if it depends on the compiler version...

I will restart the failed runs now.

I finally found a compiler config locally where I can reproduce the issue, I'll try to look into it...

Ok, finally found the bug. I had missed a small detail in one of the MC functions when replacing the encoder functions with the ones from the decoder (that are a bit more generic than the special cased encoder ones, but one of the encoder functions required more scratch space in a temp buffer).