Actually Additions flax not working with blazebloom soil

Just Dire Things version
1.4.6
Minecraft Version
1.21.1
(Neo)Forge Version
21.1.72
Modpack & Version
Craftoria 1.6.1
Do you have optifine or Rubidium installed?
I have other rendering based mods installed
Describe the issue
When planting flax seeds from Actually Additions on blazebloom soil the mature crop is replanted as string instead of the flax seed resulting in tripwires on the soil.
Steps to reproduce

Have blazebloom soil
Plant a flax seed from Actually Additions on it and wait for it to mature
When automatically harvested, it will drop flax and string
The blazebloom soil is attempting to replant the string instead of the flax and you get tripwires on the soil

Expected behaviour
It should replant ther flax instead of the string
Screenshots
No response
Log files
No response
Additional information
Actually additions has just been released

Unfortunately theres no generic way to check if a crop is a 'two high' tall crop.  I'll consider some kind of block tagging system to support this in the future.


[BUG]: pilot\scene\base_chat.py line 148 中，ModelMessage這個類型不能被json序列化

上述對象ModelMessage由 pilot\scene\base_chat.py中line 262的def generate_llm_messages(self) 返回

Yes, the chatglm-6b model currently does not have a high accuracy rate in generating Json format. You can change to vicuna-13b, which has a higher accuracy rate.

But as you can see, it has yet come to the model generation part. Additionally, the ModelMessage object that raised the json-serialization-related error was produced by ( pilot\scene\base_chat.py generate_llm_messages(self) ) while the payload was returned by self.__call_base() in nostrum_call(self).

But as you can see, it has yet come to the model generation part. Additionally, the ModelMessage object that raised the json-serialization-related error was produced by ( pilot\scene\base_chat.py generate_llm_messages(self) ) while the payload was returned by self.__call_base() in stream_call(self).

Thanks for your reply, we will fix it later, currently, you can directly use our new ui through python pilot/server/dbgpt_server.py to temporarily avoid this problem.

Thanks for your reply, I’ll have a look.
On Thu, 27 Jul 2023 at 2:13 PM, FangYin Cheng @.***>
wrote:

But as you can see, it has yet come to the model generation part.
Additionally, the ModelMessage object that raised the
json-serialization-related error was produced by ( pilot\scene\base_chat.py
generate_llm_messages(self) ) while the payload was returned by
self.__call_base() in stream_call(self).
Thanks for your reply, we will fix it later, currently, you can directly
use our new ui through python pilot/server/dbgpt_server.py to temporarily
avoid this problem.
—
Reply to this email directly, view it on GitHub
https://github.com/eosphoros-ai/DB-GPT/issues/367#issuecomment-1652971909,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ARJZ5AGIXAMSPAFMLXURRNLXSIBG5ANCNFSM6AAAAAA2YC5QXQ
.
You are receiving this because you authored the thread.Message ID:
@.***>


TestRPC / HD Wallet signing differences

Signing things is harder than it should be... I'd love a better interface for this that works with testrpc and real nodes....
const sign = (address, message) => {
    return new Promise((resolve, reject) => {
        // console.log(message.length)
        // THIS IS LAME... testrpc... I hate you....
        web3.eth.sign(address, message, (err, signature) => {
            if (err) {
                web3.eth.sign(address, web3.sha3(message), (err, signature) => {
                    if (err) {
                        throw err;
                    }
                    resolve(signature)
                })
            } else {
                resolve(signature)
            }
        })
    })
}

const recover = (address, message, signature) => {
    return new Promise((resolve, reject) => {
        var r = util.toBuffer(signature.slice(0, 66))
        var s = util.toBuffer('0x' + signature.slice(66, 130))
        var v = parseInt(signature.slice(130, 132), 16)
        if (v !== 27) {
            v += 27
        }
        var m = util.toBuffer(web3.sha3(message));
        var pub = util.ecrecover(m, v, r, s)
        var recovered_address = '0x' + util.pubToAddress(pub).toString('hex')
        resolve(recovered_address)
    })
}

These were improved here: https://github.com/transmute-industries/transmute-framework/pull/82
still not happy with the way it looks...

resolved in https://github.com/transmute-industries/transmute-framework/pulls


where does the log files stored for logstash and elasticsearch

Hi, I am wondering where are the log files for logstash and elasticsearch stored in the docker? I couldn't find them anywhere. There is no elasticsearch/logs.
cheers

@daiyueweng what do you want the log files for? Do you know that you can access the logs using docker-compose logs ?
Or if you want you can also start the stack in the foreground using docker-compose up and the logs will be streamed.

thanks, docker-compose up is very helpful.


Add SpotifyGPX to showcase

See the project here.
Implementation of SpotifyAPI can be found in 2024.36

Nice project!


LASB-1745 Update Rails to ver.7

Link to story
What
This PR:

update to Rails 7.
replaces many single quotes with double quotes, as per guide style: https://github.com/rails/rails/pull/42064
remove various empty initializers

Checklist
Before you ask people to review this PR:

[x] Tests and linters should be passing
[x ] Github should not be reporting conflicts; you should have recently run git rebase master.
[x] Avoid mixing whitespace changes with code changes in the same commit. These make diffs harder to read and conflicts more likely.
[x] You should have looked at the diff against master and ensured that nothing unexpected is included in your changes.
[x] You should have checked that the commit messages say why the change was made.

This is one of the deprecation fixed:


Bump v0.5.0

feature - image push
feature - container commit
using images.GetImage function for image inspect
show confirmation dialog in the center of different views
disk usage dialog table headers color update
Esc key shall not close dialogs if its dropdown widgets has focus
infobar + help color update for headers
image history dialog update
update button labels based on their function
code coverage for network and volume create dialogs
code coverage for ui/utils
makefile update - darwin build
docs: adding templates for bug report and features
docs: adding security policy
docs: Mac build
Bump github.com/containers/podman/v4 from 4.1.0 to 4.1.1
Bump github.com/docker/docker
Bump github.com/rs/zerolog from 1.26.1 to 1.27.0
Bump github.com/spf13/cobra from 1.4.0 to 1.5.0

LGTM


ModuleNotFoundError: No module named 'alexapy.aiohttp'

When trying to install the Alexa media component I am getting an error in the log "ModuleNotFoundError: No module named 'alexapy.aiohttp'
I then went to the CLI and just ran python.  I did an import of alexapy and I get this error just from trying to import the module.
Before this I also ran pip3 install alexapy.  It said all requirements are already met.
Full Traceback

import alexapy
Traceback (most recent call last):
File "", line 1, in
File "/srv/homeassistant/lib/python3.8/site-packages/alexapy/init.py", line 11, in
from .alexaapi import AlexaAPI
File "/srv/homeassistant/lib/python3.8/site-packages/alexapy/alexaapi.py", line 16, in
from alexapy.aiohttp import ClientConnectionError, ClientResponse
ModuleNotFoundError: No module named 'alexapy.aiohttp'

After digging into the lib directory in python and looking in the alexapy directory, I found the aiohttp directory was missing.  I did a pip3 uninstall alexapy then reinstalled it which fixed the problem.


Error creating new Weave Project in version 2.0.139

Hi,
I am trying to use Mulesoft: DataWeave RAML RestSdk Verion 2.0.139
When I create a new Weave Project and chose all default values, then in the end I get the dialog show below. If I chose "Load File System Changes" I get an empty pom.xml. If I chose "Keep Memory Changes" I get a basic pom.xml with no dependencies.
/Michael

I have the same problem, the pom.xml file is empty. I copied the pom.xml file from a project created with the Visual Studio Dataweave extension in the IntelliJ project, and the project runs fine in IntelliJ with the valid pom.xml.


Add both CODE128B and CODE128C to demo page

Please add both CODE128B and CODE128C to the demo page. I was looking for CODE128C generator, but the demo page only allows me to select CODE128 which is actually CODE128B.
Great library by the way!

Going through old issues. This one was fixed a while ago :)
http://lindell.me/JsBarcode/


Improve language packs

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 15.95%. Comparing base (ae4b797) to head (5c366f0).
Report is 2 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #605   +/-   ##
=======================================
  Coverage   15.95%   15.95%
=======================================
  Files          23       23
  Lines         909      909
  Branches      114      114
=======================================
  Hits          145      145
  Misses        764      764

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


sage required php 8.2

Version
latest
What did you expect to happen?
Sage theme auto setup
What actually happens?
At first it's return error that php is currently 8.0 and error message seems reads php 8.2 is minimum requirement
Steps to reproduce
Just install roots/acorn inside bedrock
System info
n/a
Log output
n/a

Please confirm this isn't a support request.
No

Here is how I fix it. Please add it to your "bedrock.dockerfile" file:
FROM php:8.2-fpm as base
LABEL name=bedrock
LABEL intermediate=true
Install essential packages
RUN apt-get update
&& apt-get install -y
build-essential
curl
git
gnupg
less
nano
vim
unzip
zip
&& apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false
&& rm -rf /var/lib/apt/lists/*
&& apt-get clean
FROM base as php
LABEL name=bedrock
LABEL intermediate=true
Install php extensions and related packages
ADD https://github.com/mlocati/docker-php-extension-installer/releases/latest/download/install-php-extensions /usr/local/bin/
RUN chmod +x /usr/local/bin/install-php-extensions && sync
&& install-php-extensions
@composer
exif
gd
memcached
mysqli
pcntl
pdo_mysql
zip
&& apt-get update
&& apt-get install -y
gifsicle
jpegoptim
libpng-dev
libjpeg62-turbo-dev
libfreetype6-dev
libmemcached-dev
locales
lua-zlib-dev
optipng
pngquant
&& apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false
&& rm -rf /var/lib/apt/lists/*
&& apt-get clean
FROM php as bedrock
LABEL name=bedrock
Install nginx & supervisor
RUN curl -sL https://deb.nodesource.com/setup_16.x | bash
&& apt-get update
&& apt-get install -y
nginx
nodejs
supervisor
npm
&& apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false
&& rm -rf /var/lib/apt/lists/*
&& apt-get clean
&& npm install -g yarn
Configure nginx, php-fpm and supervisor
COPY ./build/nginx/nginx.conf /etc/nginx/nginx.conf
COPY ./build/nginx/sites-enabled /etc/nginx/conf.d
COPY ./build/nginx/sites-enabled /etc/nginx/sites-enabled
COPY ./build/php/8.0/fpm/pool.d /etc/php/8.0/fpm/pool.d
COPY ./build/supervisor/supervisord.conf /etc/supervisord.conf
WordPress CLI
RUN curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar
&& chmod +x wp-cli.phar
&& mv wp-cli.phar /usr/bin/_wp;
COPY ./build/bin/wp.sh /srv/wp.sh
RUN chmod +x /srv/wp.sh
&& mv /srv/wp.sh /usr/bin/wp
Installation helper
COPY ./build/bin/bedrock-install.sh /srv/bedrock-install.sh
RUN chmod +x /srv/bedrock-install.sh
WORKDIR /srv/bedrock
CMD ["/srv/bedrock-install.sh"]

Bedrock requires PHP 8.0 as a minimum which is why this setup uses it.
Bedrock is now bumping the requirements to PHP 8.1, along with moving to a dev container setup contained in the repo. This repo will soon be archived.


TFA Recovery codes discussion

Hello,
my Version
<PackageReference Include="Microsoft.AspNetCore.All" Version="2.0.6" />
I created the latest startup project of asp.net core with Identity (local stored account in my sql). I created a new user and went to enable TFA. I got the recovery code and tryied the TFA. Login wokrs fine, but LoginWithRecoveryCode always fail with message "Invalid recovery code entered".
My recovery code is as i copied it from the webpage.
ef3899d4 97bf06ad
b8927df8 6ad00f89
ec5e6e30 17872076
1ac5ca54 c8ec38a6
2fed51c8 c7a063e9

I try to past it in 4 ways
//1 as i copied it
//2 ef3899d4 97bf06ad b8927df8 6ad00f89 ec5e6e30 17872076 1ac5ca54 c8ec38a6 2fed51c8 c7a063e9
//3 ef3899d497bf06adb8927df86ad00f89ec5e6e30178720761ac5ca54c8ec38a62fed51c8c7a063e9
//4 ef3899d4;97bf06ad;b8927df8;6ad00f89;ec5e6e30;17872076;1ac5ca54;c8ec38a6;2fed51c8;c7a063e9

The 4th way is as copied from the database.
none of this wokrs.
thank you.

I think that 1,2,3 ways should all work

Did you try to just enter one of the codes, instead all of them?

Encrypt the codes

yes, i opened an issue for that a while ago

#836 #1341

What do you think encrypting the codes would give you here? You still need a valid password, they're for use when you don't have your 2fa app.

What do you think encrypting the codes would give you here? You still need a valid password, they're for use when you don't have your 2fa app.

I always try to protect the identity system with the assumption an attacker has read-only access to the DB. Thus, protecting access tokens, 2FA codes, 2FA reset codes, email confirmation codes, etc. In ASP.NET Identity some of these values are protected by virtue of data protection and not storing them in the DB. But many other values are in the DB... in plain text.

Reading the data from the db would give you access to any account with TFA enabled. If the codes are one-way encrypted that adds an extra layer of security.

No it wouldn't allow any access with just the codes. Those recovery codes still need a correct username and password, they're for occasions when you don't have access to your 2fa generator.


Recursive Mapped Type Narrows Branded Property to Never

TypeScript Version:  nightly, 3.7.2, 3.6.3, 3.5.1 (issue occurs in all versions except 3.5.1)

Search Terms:
recursive, mapped, brand, branded, nominal
Code
/**
 * A simplified version of<EMAIL_ADDRESS>`PreloadedState<S>`. Simplified to
 * produce a minimal repro.
 */
export type PreloadedState<S> = {
    [K in keyof S]: S[K] extends object ? PreloadedState<S[K]> : S[K]
};

export type Branded<K, T> = T & {
    __BRAND__: K; // (or K & void)
};

type InitialState = PreloadedState<{
    someString: Branded<'Foo', string>;
}>; // evaluates to { someString: never }

Expected behavior:
InitialState is typed as { someString: Branded<'Foo', string> }.
Actual behavior:
InitialState is typed as { someString: never }.
Playground Link:
Here
Context:
This issue affects usage of<EMAIL_ADDRESS>when calling Redux.createStore. Redux.createStore uses a recursive mapped type for the preloadedState. In my case, PreloadedState<S> is incompatible with S, because PreloadedState<S> contains never types and S does not.

I would argue that string & { x : "foo" } is an object type.

The produced output is correct. Effectively you've written a type that produces a string & object, which is uninhabitable.
You could consider reversing the conditional:
export type PreloadedState<S> = {
    [K in keyof S]: S[K] extends (string | number | boolean) ? S[K] : PreloadedState<S[K]>;
};

export type Branded<K, T> = T & {
    __BRAND__: K;
};

type InitialState = PreloadedState<{
    someString: Branded<'Foo', string>;
}>;

but ultimately Branded produces contradictory types and you're going to encounter problems sooner or later.

@RyanCavanaugh thanks for the response! Reversing the conditional seems like a good solution, as well as updating consumers to accept PreloadedState<S> | S. It turns out that the redux folks went with the latter solution for 5.x.x.
I am curious to know a couple things...

Am I implementing Branded optimally? Is there a better approach at present for hacking in nominal typing?
In an ideal world, would Branded<'Foo', string> resolve to never immediately? It's a bit confusing to me that its type is non-never prior to its acquaintance with PreloadedState. I feel like this is either a) a tsc implementation detail, or b) something I'm missing about how PreloadedState is operating here. My feeling is that b is more likely :)

It won't resolve to never immediately because TS' own source code uses brands on primitives

@AnyhowStep I see. Is there a general rule about when resolution to never occurs? Is this behavior documented? The reason I ask is this: as a consumer of branded types, I'd like to understand where they will fail. Thanks!

Ummmmm........
Not as far as I know.
For example, in 3.5.1, true & number is not immediately reduced to never.
But in 3.7.2, it is immediately reduced to never

Branding with a primitive an object type is an at-your-own-risk endeavor. Basically you can do this as long as the type never goes through a simplifying transform (i.e. never gets produced from a higher-order operation). That said, I'm describing the current behavior, not outlining any promises of future behavior. The "best" way to do this, if you feel you have to, is to just fully replace the primitive with an opaque object type; this is "guaranteed" to not encounter problems at the expensive of losing access to the underlying primitive's behavior.
We do use some branding internally but don't file bugs against ourselves when we encounter weirdness 😉


Change usage of CURL to ansible uri module

we need to wait on Ansible 2.1 (so the dependency on httplib is removed for CoreOS) -
this code needs updated - https://github.com/Capgemini/kubeform/blob/master/roles/k8s-master/tasks/main.yml#L88
and we need to modify wercker.yml
http://docs.ansible.com/ansible/uri_module.html

Closing - no longer relevant.


Set default stdout of wasm to console.log

Fix issue #1610
The print function of wasm will be

console.log by default or when giving undefined or "console"
a dumb function when giving null
the function when giving a function

The behaviors in firefox 70.0.1 are listed as following.
>> rp.pyEval('print("a")',{stdout: 'console'})
"a"
← undefined
>> rp.pyEval('print("a")',{stdout: null})
← undefined
>> rp.pyEval('print("a")')
"a"
← undefined

Thanks for your review.


Change use of WHAT to ^name

It's the recommended way. WHAT is used to actually get a type object, which isn't what we want. We want the name of the class the object was instantiated from.

The equivalent of typeof is actually .WHAT.
This is why it has :ok-test<WHAT>, which opts it out of the test to make sure .WHAT isn't used when it isn't needed.

El mar., 29 may. 2018 a las 21:55, Brad Gilbert<EMAIL_ADDRESS>escribió:

The equivalent of typeof is actually .WHAT.
This is why it has :ok-test, which opts it out of the test to make
sure .WHAT isn't used when it isn't needed.

What's happened to this comment?

Thanks @JJ For the improvement.
Teaching people to use .NAME is not good, even if that's what JS actually does, because in many places it won't work. Since this is an introductory tutorial, I think it's better to give them the tool they should be using anyway:
https://irclog.perlgeek.de/perl6/2018-05-29#i_16222140


Last bits of cleanup from the git meltdown. Added the WelcomeFooter

Some changes got lost from the meltdown, and are now restored.

👍


Update Jenkins Operator chart for support for Jenkins Configuration As Code

Signed-off-by: Marat Garafutdinov<EMAIL_ADDRESS>
What this PR does / why we need it:
Updates Jenkins Operator chart with latest jenkins-operator version with support for JASC plugin
Which issue this PR fixes
Special notes for your reviewer:
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[x] Variables are documented in the README.md

/ok-to-test

@maratoid What happens on an upgrade for someone? With the CRD fields changing but the version not some things will break.

Ok, will do. Thanks!


option to select the ruff version on the playground

currently the version is displayed here but there's no option to change it (also it's currently outdated since the latest version is 0.0.292)

it would be nice if there was an option to pick the version, like in the mypy and typescript playgrounds

We're interested but this is relatively low priority for us. We normally update it automatically on release at https://github.com/astral-sh/ruff/actions/workflows/playground.yaml looks like something is broken.

Duplicate of https://github.com/astral-sh/ruff/issues/1386.


loot names containing commas with following spaces break stuff

example: "Deep, Deep, Deep egg" breaks because it tries to separate on ,
Need to figure out another way of doing it, or just make exception cases for this.

Could look at the loot string and find if 3 or more items were dropped:

\d+ would match 3 or more times,
" and " would be present,
", \d" would be present.

This regex uses lookaheads in a list of three or more items to return the number, the item, and the ", " at the end, which can be stripped off:
(\d+.*?, (?=\d+))|(\d+.* and \d+.*)

The last match item in the array has two items separated by " and ".
For two items being dropped we can look for " and \d+", use the index of that, and get two loot strings regardless of commas.
The last case should be a single item being dropped and we don't care about commas.

An actual string we're working with:
"text": "I sounded the Hunter's Horn and was successful in the hunt! I caught a 10 oz. Elite Guardian Mouse<\/a> worth 25,500 points and 4,205 gold.

The mouse also dropped the following loot:<\/b>
1 Nightshade<\/a>, and 2 Oxygen Canisters<\/a>"

we now separate the items by using  link and get loot item_id and plural name from another part of response object


bad Emoji input on macOS

I'm using IME to input some emoji characters, but the screen shows two squares.
after dig into glfw source code, i just found that there may be a bug at https://github.com/glfw/glfw/blob/master/src/cocoa_window.m#L726-L734, since NSString is stored as UTF-16 encoding.
And i found a solution at https://stackoverflow.com/questions/20537706/how-to-handle-32bit-unicode-characters-in-a-nsstring, it may helps, thanks.

I haven't tested on Linux and Windows yet,  so i don't know if there are same problem.

Thank you for reporting this! It should be fixed now with ad9eb768c9518dbe94232e5201d318d512ad9beb, and on Windows with 6ce207039240a63120a91ee4ba4d39bd0a3ef8c8.


add OWNERS for new ci-tooling area label

What this PR does / why we need it:
add OWNERS for new ci-tooling area label
Which issue(s) this PR fixes:
N/A

/approve

/area ci-tooling

/lgtm


switch_case_semicolon_to_colon fixer breaks match expressions

Bug report
PHP CS Fixer 2.18.0 on PHP 8.0.1
If I run PHP CS Fixer on a file containing a match expression, that file is listed under
Files that were not fixed due to errors reported during linting after fixing

In verbose mode, I get the following error message:
[PhpCsFixer\Linter\LintingException]
Parse error: syntax error, unexpected token ":", expecting "," or ";" on line 7.

Applied fixers: switch_case_semicolon_to_colon

If I disable the mentioned switch_case_semicolon_to_colon fixer, the error is gone.
Code snippet that reproduces the problem
echo match (\PHP_MAJOR_VERSION) {
    8 => 'eight',
    9 => 'nine',
    default => 'unknown',
};

hey @derrabus , thanks for the report!
do you want to give it a try and raise PR fixing it?

hey @derrabus , thanks for the report!
do you want to give it a try and raise PR fixing it?

Sure, I can try. 😃 #5447

Sure, I can try. 😃 #5447


Need option to not verify filters upon registration

Here is my scenario:
I have an application that accepts telemtry from various sources. The telemetry consists of 1000's of data fields.
My clients need the ability to subscribe to a specific field's received event, ie. the "FirstName" field was received.
Since, as mentioned above, there are 1000's of fields, I don't want to create 1000's of filters ahead.
What I would like is some kind of Ad-Hoc Filters, meaning - the client can subscribe to a non-exist events, and get notified when they occur.
Now, there is no problem in the Publish phase - the NotifyAsync  method can accept non-exist filter names and send them to the subscribers.
The problem lies in the Registration phase. The WebHookRegistrationsController specifically validates the Filters collection of the WebHook, and makes sure the filters specified there exist.
I would like to have an option to override this validation, and allow the subscribers to subscribe to non-existent filters.
Thanks!

I think the right solution here is for me to make it easier to write your own controller and then have the validation happen on the side., i.e. have something you can call into. This will also address #88. I will send out an update when ready.

Thanks!
In the meantime, how can I substitue the current controller with a custom one? Is it possible?

Yes, it is straight forward to creating your own controller -- you can copy most of the existing controller  [1]and then modify as you see fit.
[1] https://github.com/aspnet/WebHooks/blob/master/src/Microsoft.AspNet.WebHooks.Custom.Api/Controllers/WebHookFiltersController.cs

I just committed 757530d2af40da1d51e0cf387595561e1e5cebf0 which makes it much simpler to write your own WebHook registration mechanism whether that is a separate API controller, an MVC controller, or something else entirely.  Please see the commit for details -- you might be able to look at the existing WebHookRegistrationsController.cs for an example of how it is used.


Scrollbar for inline chat is off the screen which makes it confusing

Gif upload isn't working so sadly have to upload a zip...
Recording 2023-11-03 at 15.25.21.gif.zip
Anyway steps are:

Get Copilot to output something big like another test
be confused that the visible scrollbar doesn't apply to the diff but rather the editor

This is a UI bug so I don't think you need the logs but lmk if you do.

cc @alexdima again so that it doesn't go unnoticed

Closing because the new live3 mode will take care of this because we won't show editors inside editors anymore.


Add README.txt or something for locally cloned wikis

Discussed in https://github.com/spenserblack/actions-wiki/discussions/46

Originally posted by jcbhmr March 22, 2023
I just had the random thought to include some kind of a file that would not be visible on github but would be if you cloned the wiki locally to say "heyyyy checkout the github repo instead, don't put stuff in the wiki here!"
similar idea i had just now while writing this would be to prepend a <!-- DO NOT EDIT ON PUBLISHED WIKI, ONLY EDIT IN SOURCE REPO -->\n or similar comment to all .md files to discourage editing the published version?
these ideas are kinda redundant if the user sees the "github-actions[bot] edited this" thing though... 🤷♀

This seems like a potentially valid idea, but right now it's not a high priority IMO, so I'm adding the "icebox" label. If this gets interest from the community, I'll remove the label.

btw i think github projects was designed for this idea of "priority" stuff, while labels are intended to "categorize"?
maybe not tho cause here's a demo that uses projects to categorize. https://github.com/git-lfs/git-lfs/projects/3
but wait! here's the builtin template that is for priority (ish) since it uses a "backlog" and then other actively-working-on categories

Yeah, I was considering creating a project, but then I felt that it was a bit of overkill for this repo. Right now a simple issue list is plenty IMO.
But, you're right, projects are great for this sort of thing :slightly_smiling_face:

now that im thinking about it from like 3 weeks in the future; i think this is exactly what the ideas discussions are for! you can leave a discussion "open" and it's like a proposal idea in stasis for an indefinite time (until a robot github workflow action closes it after 30 days or whatever if that's configured)
just a thought


leaking file descriptors

I believe either the scheduler or the workers are not closing file descriptors appropriately after calling scatter. Sample code:
import psutil
print(psutil.Process().num_fds())

from distributed import Executor
executor = Executor()

import numpy as np
from dask import delayed, compute

for i in range(0,10):
  #data = np.random.randint(0, 1000, size=(145, 174, 145, 288))
  data = np.random.randint(0, 1000, size=(145, 174, 145, 2))
  print(psutil.Process().num_fds())

  remote_slices = executor.scatter([data[..., i] for i in range(data.shape[-1])])
  print(psutil.Process().num_fds())

  jobs = [delayed(len)(slice) for slice in remote_slices]
  print(psutil.Process().num_fds())

  j = compute(*jobs, get=executor.get)
  print(psutil.Process().num_fds())

This prints
4
751
803
803
805
805
822
822
824
824
831
831
833
833
852
852
854
854
871
871
873
873
875
875
877
877
879
879
881
881
883
883
885
885
887
887
889
889
891
891
893

And if you use the commented out line to generate data then the program crashes with Too many open files on my machine.

Here it is running on Python 3.4 on a 4 core machine running Ubuntu 16.04
In [1]: paste
import psutil
print(psutil.Process().num_fds())

from distributed import Executor
executor = Executor()

import numpy as np
from dask import delayed, compute

for i in range(0,10):
  #data = np.random.randint(0, 1000, size=(145, 174, 145, 288))
  data = np.random.randint(0, 1000, size=(145, 174, 145, 2))
  print(psutil.Process().num_fds())

  remote_slices = executor.scatter([data[..., i] for i in range(data.shape[-1])])
  print(psutil.Process().num_fds())

  jobs = [delayed(len)(slice) for slice in remote_slices]
  print(psutil.Process().num_fds())

  j = compute(*jobs, get=executor.get)
  print(psutil.Process().num_fds())
## -- End pasted text --
9
42
50
50
52
54
54
54
56
56
58
58
58
58
58
58
58
58
58
58
58
58
58
58
58
58
58
58
58
58
58
58
58
58
58
58
58
58
70
70
70

Can you say a bit about your hardware and software setup?

Increasing the last dimension of the numpy array doesn't seem to affect things:
import psutil
print(psutil.Process().num_fds())

from distributed import Executor
executor = Executor()

import numpy as np
from dask import delayed, compute

for i in range(0,10):
  #data = np.random.randint(0, 1000, size=(145, 174, 145, 288))
  data = np.random.randint(0, 1000, size=(145, 174, 145, 2))
  print(psutil.Process().num_fds())

  remote_slices = executor.scatter([data[..., i] for i in range(data.shape[-1])])
  print(psutil.Process().num_fds())

  jobs = [delayed(len)(slice) for slice in remote_slices]
  print(psutil.Process().num_fds())

  j = compute(*jobs, get=executor.get)
  print(psutil.Process().num_fds())

I ran mine on Python 3.4 on a 120 core server running Ubuntu 14.04 with 1TB of ram (hence I suppose the worker will be using many more threads)
When I increased the data array from (145, 174, 145, 2) to (145, 174, 145, 288) I got:
4
764
distributed.utils - ERROR - [Errno 24] Too many open files
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/distributed/utils.py", line 102, in f
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
  File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
  File "<string>", line 3, in raise_exc_info
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1021, in run
  File "/usr/local/lib/python3.4/dist-packages/distributed/executor.py", line 783, in _scatter
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
  File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
  File "<string>", line 3, in raise_exc_info
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1021, in run
  File "/usr/local/lib/python3.4/dist-packages/distributed/core.py", line 438, in send_recv_from_rpc
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
  File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
  File "<string>", line 3, in raise_exc_info
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1024, in run
  File "/usr/local/lib/python3.4/dist-packages/distributed/core.py", line 340, in send_recv
  File "/usr/local/lib/python3.4/dist-packages/six.py", line 685, in reraise
  File "/usr/local/lib/python3.4/dist-packages/distributed/core.py", line 206, in handle_stream
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
  File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
  File "<string>", line 3, in raise_exc_info
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1021, in run
  File "/usr/local/lib/python3.4/dist-packages/distributed/scheduler.py", line 1577, in scatter
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
  File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
  File "<string>", line 3, in raise_exc_info
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1021, in run
  File "/usr/local/lib/python3.4/dist-packages/distributed/client.py", line 229, in scatter_to_workers
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
  File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
  File "<string>", line 3, in raise_exc_info
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1021, in run
  File "/usr/local/lib/python3.4/dist-packages/distributed/utils.py", line 80, in All
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
  File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
  File "<string>", line 3, in raise_exc_info
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1021, in run
  File "/usr/local/lib/python3.4/dist-packages/distributed/core.py", line 437, in send_recv_from_rpc
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
  File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
  File "<string>", line 3, in raise_exc_info
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1021, in run
  File "/usr/local/lib/python3.4/dist-packages/distributed/core.py", line 421, in live_stream
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
  File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
  File "<string>", line 3, in raise_exc_info
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1021, in run
  File "/usr/local/lib/python3.4/dist-packages/distributed/core.py", line 295, in connect
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
  File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
  File "<string>", line 3, in raise_exc_info
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1024, in run
  File "/usr/local/lib/python3.4/dist-packages/tornado/tcpclient.py", line 168, in connect
  File "/usr/local/lib/python3.4/dist-packages/tornado/tcpclient.py", line 81, in start
  File "/usr/local/lib/python3.4/dist-packages/tornado/tcpclient.py", line 96, in try_connect
  File "/usr/local/lib/python3.4/dist-packages/tornado/tcpclient.py", line 180, in _create_stream
  File "/usr/lib/python3.4/socket.py", line 126, in __init__
OSError: [Errno 24] Too many open files

OK the 120 core server makes things a lot more clear here.  It looks like each connection from scheduler to worker takes on the order of 10 connections.  If you have 120 logical cores then it will try to open 1200 files, which, with default linux settings, will blow past the default limit of 1024 open files.
We could shrink this down a bit, but not by more than a factor of two in the near future.  There is a decent amount of asynchronous communication back and forth between everyone.
I think I recall your e-mail saying that you had set the open file limit.  Is it possible that this process wasn't completed somehow?  I usually do this by appending the following lines to /etc/security/limits.conf
my_username soft nofile 102400
my_username hard nofile 102400

I then make sure to restart the terminal from which I launch things to make sure it gets these new limits.
http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

Indeed setting the file limit helped. But I am still observing an increasing number of file handles:
4
783
8480
8502
9042
11901
17966
18049
18228
20275
26722
26746
26905
27511
28157
28159
28159
28159
28260
28260

Is there a reason for this?
Two other observations:
Our actual production setting uses 16 ec2 m3.2xlarge machines, and running on that cluster still returns Too many opened files error even after setting ulimit -n to be 1000000 on all machines.
And when I ran the test code on one of the m3.2xlarge machines locally with data = np.random.randint(0, 1000, size=(145, 174, 145, 288)) I got:
4
127
Traceback (most recent call last):
  File "t.py", line 17, in <module>
    remote_slices = executor.scatter([data[..., i] for i in range(data.shape[-1])])
  File "/usr/local/lib/python3.4/dist-packages/distributed/executor.py", line 885, in scatter
    broadcast=broadcast)
  File "/usr/local/lib/python3.4/dist-packages/distributed/utils.py", line 116, in sync
    six.reraise(type(error[0]), error[0], traceback[0])
  File "/usr/lib/python3/dist-packages/six.py", line 536, in reraise
    raise value
  File "/usr/local/lib/python3.4/dist-packages/distributed/utils.py", line 102, in f
    result[0] = yield gen.maybe_future(func(*args, **kwargs))
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1021, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/local/lib/python3.4/dist-packages/distributed/executor.py", line 783, in _scatter
    client=self.id, broadcast=broadcast)
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1021, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/local/lib/python3.4/dist-packages/distributed/core.py", line 438, in send_recv_from_rpc
    result = yield send_recv(stream=stream, op=key, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1021, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/local/lib/python3.4/dist-packages/distributed/core.py", line 335, in send_recv
    yield write(stream, msg)
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 285, in wrapper
    yielded = next(result)
  File "/usr/local/lib/python3.4/dist-packages/distributed/core.py", line 269, in write
    frames = protocol.dumps(msg)
  File "/usr/local/lib/python3.4/dist-packages/distributed/protocol.py", line 81, in dumps
    frames = dumps_msgpack(small)
  File "/usr/local/lib/python3.4/dist-packages/distributed/protocol.py", line 153, in dumps_msgpack
    payload = msgpack.dumps(msg, use_bin_type=True)
  File "/usr/local/lib/python3.4/dist-packages/msgpack/__init__.py", line 47, in packb
    return Packer(**kwargs).pack(o)
SystemError: error return without exception set

Not sure if the two errors are related.

The other error that you've reported in #1412 is, I think, unrelated.

Any suggestions to work around the too many files opened error? (which I guess was a work around for #1412)

20000 open file handles for 120 cores definitely sounds like far too many.  Here is my experience with a 100 process local cluster on my laptop:

In [1]: from distributed import Executor, LocalCluster

In [2]: c = LocalCluster(100)

In [3]: c
Out[3]: LocalCluster("<IP_ADDRESS>:8786", workers=100, ncores=400)

In [4]: executor = Executor(c)

In [5]: exec
exec      executor

In [5]: executor
Out[5]: <Executor: scheduler="<IP_ADDRESS>:8786" processes=100 cores=400>

In [6]: import psutil

In [7]: print(psutil.Process().num_fds())
621

In [8]: import numpy as np

In [9]: from dask import delayed, compute

In [10]: for i in range(0,10):
    data = np.random.randint(0, 1000, size=(174, 145, 200))
    print(psutil.Process().num_fds())
    remote_slices = executor.scatter([data[..., i] for i in range(data.shape[-1])])
    print(psutil.Process().num_fds())
    jobs = [delayed(len)(slice) for slice in remote_slices]
    print(psutil.Process().num_fds())
    j = compute(*jobs, get=executor.get)
    print(psutil.Process().num_fds())
   ....:
620
620
620
670
670
720
720
770
770
916
916
916
916
916
916
916
916
916
916
916
916
916
916
916
916
916
916
916
916
916
916
916
916
916
916
916
916
916
916
916

What operating system are you running?  What hardware do you have that has 120 cores on one machine?

Any suggestions to work around the too many files opened error?

I'm inclined to figure out what's going on and fix it.

which I guess was a work around for #1412

I'm not sure I understand this comment.  Can you clarify?

Also how did you install things?  What version of Tornado are you using?

What operating system are you running? What hardware do you have that has 120 cores on one machine?

This was a high-end server running ubuntu 14.04 that does have 120 cores. That aside, even when running the code I pasted at the top on an aws machine with 8 cores the # of files grew to 643 when finished (the # of files didn't stay constant as in your case running on the laptop). Is that behavior intended? I imagine it will run out of handles eventually regardless of the machine settings.

which I guess was a work around for #1412

As you recall using scatter was your suggested workaround for the out of memory error I raised earlier. Seems like I am hitting the file handle issue when using that workaround.

Also how did you install things? What version of Tornado are you using?

Everything was installed using pip3:
pip3 show tornado dask distributed
---
Name: tornado
Version: 4.4
Location: /usr/local/lib/python3.4/dist-packages
Requires: backports-abc
---
Name: dask
Version: 0.10.1
Location: /usr/local/lib/python3.4/dist-packages
Requires:
ubuntu@ip-172-31-4-204:~$ pip3 show tornado dask distributed
---
Name: distributed
Version: 1.11.2
Location: /usr/local/lib/python3.4/dist-packages
Requires: cloudpickle, dask, locket, toolz, msgpack-python, psutil, boto3, six, tblib, click, s3fs, tornado

This was a high-end server running ubuntu 14.04 that does have 120 cores. That aside, even when running the code I pasted at the top on an aws machine with 8 cores the # of files grew to 643 when finished (the # of files didn't stay constant as in your case running on the laptop). Is that behavior intended? I imagine it will run out of handles eventually regardless of the machine settings.

We definitely expect a near constant number of file descriptors.  The system is decently intelligent about pooling connections and only opening new streams if more concurrent connections are required.  This is pretty rare.  Scattering shouldn't have a significant effect.  It doesn't consume a significant number of connections.
I've tried installing a similar software environment to yours tools and still get my normal results.
(fds) mrocklin@notebook:~$ pip freeze
backports-abc==0.4
boto3==1.3.1
botocore==1.4.39
click==6.6
cloudpickle==0.2.1
dask==0.10.1
decorator==4.0.10
distributed==1.11.2
docutils==0.12
ipython==5.0.0
ipython-genutils==0.1.0
jmespath==0.9.0
locket==0.2.0
msgpack-python==0.4.7
numpy==1.11.1
pexpect==4.2.0
pickleshare==0.7.3
prompt-toolkit==1.0.3
psutil==4.3.0
ptyprocess==0.5.1
Pygments==2.1.3
python-dateutil==2.5.3
s3fs==0.0.6
simplegeneric==0.8.1
six==1.10.0
tblib==1.3.0
toolz==0.8.0
tornado==4.4.1
traitlets==4.2.2
wcwidth==0.1.7

I can try some EC2 hardware tomorrow.  Anything special I should consider?

OK, I'm running on some hardware generously provided by @akcheung
OK, so I am seeing an interesting correlation that, as the size of the data increases so too does the number of file descriptors.  This appears to be sub-linear, but is fairly interesting.  Perhaps there is some OS-level caching going on that keeps old sockets alive?
Regardless, I'm not seeing a leaking effect, the number remains constant.
dasktest@ip-172-31-4-204:~$ python3 mrocklin-test.py
4
<Executor: scheduler="<IP_ADDRESS>:9999" processes=8 cores=8>
66
217
217
217
225
268
268
268
268
268
268
268
268
268
268
268
268
268
268
268
268

When I tone the size of the data down significantly it seems to level out at 70 open files, which is at our same 10x nworkers from before.
Something that may have been screwing with numbers before is that there are a lot of dask-worker processes somewhere that are actively trying to connect to the 8786 port.  So when we start up a simple Executor() with a local cluster, he immediately gets swarmed by twenty other dask-workers around.  I don't have permissions to kill these so I moved to a new port.
I've tried this with the system installation and a user-side miniconda installation and they both perform about the same.

Thanks @mrocklin . We just shutdown the other dask-workers so 8786 port should work now. Any idea what is going on with the full (145, 174, 145, 288) dataset? Right now it just crashes.

At the moment, no, I don't know what's going on.  There is a clear dependence on the amount of data sent with the number of file descriptors.  Nothing in the dask.distributed codebase handles anything like this though.  We've just opened up a small handful of Tornado IOStreams.  I'll look a bit upstream and see if anything is going on with Tornado or any of the compression libraries.

I'm probably not going to get anything else done today though.  if you'd like to shut down your EC2 machine now would be a good time.

Great. Thanks @mrocklin for looking into this.

I believe that this was caused by heartbeats piling up due to heavy network traffic.  The system was clever enough to notice that existing channels were full and so kept opening up new ones, causing the issue here.
I believe that this is fixed in https://github.com/dask/distributed/pull/388

Just pulled the latest from the repo but I'm still getting a SystemError in Packer when running the full (145, 174, 145, 288) dataset test case. Did I miss a step somewhere?
$ pip3 show distributed
---
Name: distributed
Version: 1.11.3
Location: /usr/local/lib/python3.4/dist-packages

Test output:
4
63
distributed.utils - ERROR - error return without exception set
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/distributed/utils.py", line 102, in f
    result[0] = yield gen.maybe_future(func(*args, **kwargs))
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1021, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/local/lib/python3.4/dist-packages/distributed/executor.py", line 787, in _scatter
    client=self.id, broadcast=broadcast)
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1021, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/local/lib/python3.4/dist-packages/distributed/core.py", line 445, in send_recv_from_rpc
    result = yield send_recv(stream=stream, op=key, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1021, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/local/lib/python3.4/dist-packages/distributed/core.py", line 339, in send_recv
    yield write(stream, msg)
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 285, in wrapper
    yielded = next(result)
  File "/usr/local/lib/python3.4/dist-packages/distributed/core.py", line 273, in write
    frames = protocol.dumps(msg)
  File "/usr/local/lib/python3.4/dist-packages/distributed/protocol.py", line 81, in dumps
    frames = dumps_msgpack(small)
  File "/usr/local/lib/python3.4/dist-packages/distributed/protocol.py", line 155, in dumps_msgpack
    payload = msgpack.dumps(msg, use_bin_type=True)
  File "/usr/local/lib/python3.4/dist-packages/msgpack/__init__.py", line 47, in packb
    return Packer(**kwargs).pack(o)
SystemError: error return without exception set

I haven't resolved the msgpack issue https://github.com/dask/dask/issues/1412.
I think that I have resolved the file descriptor issue.
Does your computation still use significantly more than 10x the number of workers fail when using scatter?

For the small dataset the number of files stayed constant after a few iterations. Thanks for looking into that. For the large dataset I wasn't able to tell as it crashed after 2 iterations (see above).

I wrote the following before looking more deeply at your traceback.  It's still correct so I'm going to send it, however I'm not confident that it is the source of your problem.
The error above means that you have large datasets within the delayed values that you submit up to the scheduler.  This would happen if you did the following:
x = np.random.random(size=large)
v = delayed(f)(x)
x = e.persist(x)

It should not happen if you do the following:
x = np.random.random(size=large)
[x] = e.scatter([x])
v = delayed(f)(x)
x = e.persist(x)

It looks like this is actually happening within scatter, which is odd.  I'll take a look when I have time (maybe later today if other things go well)

Here's the full trace:
Traceback (most recent call last):
  File "test.py", line 17, in <module>
    remote_slices = executor.scatter([data[..., i] for i in range(data.shape[-1])])
  File "/usr/local/lib/python3.4/dist-packages/distributed/executor.py", line 889, in scatter
    broadcast=broadcast)
  File "/usr/local/lib/python3.4/dist-packages/distributed/utils.py", line 116, in sync
    six.reraise(type(error[0]), error[0], traceback[0])
  File "/usr/local/lib/python3.4/dist-packages/six.py", line 686, in reraise
    raise value
  File "/usr/local/lib/python3.4/dist-packages/distributed/utils.py", line 102, in f
    result[0] = yield gen.maybe_future(func(*args, **kwargs))
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1021, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/local/lib/python3.4/dist-packages/distributed/executor.py", line 787, in _scatter
    client=self.id, broadcast=broadcast)
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1021, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/local/lib/python3.4/dist-packages/distributed/core.py", line 438, in send_recv_from_rpc
    result = yield send_recv(stream=stream, op=key, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1021, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/local/lib/python3.4/dist-packages/distributed/core.py", line 335, in send_recv
    yield write(stream, msg)
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 1015, in run
    value = future.result()
  File "/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py", line 237, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 285, in wrapper
    yielded = next(result)
  File "/usr/local/lib/python3.4/dist-packages/distributed/core.py", line 269, in write
    frames = protocol.dumps(msg)
  File "/usr/local/lib/python3.4/dist-packages/distributed/protocol.py", line 81, in dumps
    frames = dumps_msgpack(small)
  File "/usr/local/lib/python3.4/dist-packages/distributed/protocol.py", line 155, in dumps_msgpack
    payload = msgpack.dumps(msg, use_bin_type=True)
  File "/usr/local/lib/python3.4/dist-packages/msgpack/__init__.py", line 47, in packb
    return Packer(**kwargs).pack(o)
SystemError: error return without exception set

I think I've identified the issue.  It might be a day or two before it gets fixed.  We're going to use it as a training issue.  In the mean time you should be able to get by by sending data up in smaller batches.  This is somewhat inefficient but will get you started
futures = list(e.scatter((data[..., i] for i in range(data.shape[-1]))))

Notice that we're passing an iterator, not a list, to scatter.  This will pass data up one at a time.

Thanks. FYI I'm still getting the same error after switching to iterators.

Also @akcheung, you're probably already aware of this, but just in case, it's rare to use scatter on large datasets.  It's far more common to have workers load data themselves.  The reason why we've never run into this issue before is because using scatter in this way is something of an anti-pattern.
It's far more common to, say, create slices on the workers
slices = [delayed(np.random.random)((145, 174, 145)) for i in range(288)]
slices = e.persist(*slices)

Or, if what you actually want is a random nd-array use dask.array to construct it for you.
import dask.array as da
x = da.random.random((145, 174, 145, 288), chunks=(145, 174, 145, 1))
x = e.persist(x)

Our code has to first download 2GB worth of data and then compute on it. We can't split the download across workers but can distribute it afterwards to workers. Which code do you think will be more efficient?
slices = partition(download())
slices = e.scatter(slices)
jobs = e.map(runOnSlice, slices)
e.compute(jobs)

or
slices = partition(download())
slices = e.scatter(slices)
ds = da.concatenate(futures_to_dask_arrays(slices))
f = delayed(runOnDaskArray)(ds)
e.compute(f)

Not being able to split data ahead of time is unfortunate.  Are there perhaps many 2GB-sized datasets that you need to manage?  Is there any reason to call download on the workers themselves?

You might also want to try out https://github.com/dask/distributed/pull/397

Yes we do have many 2GB datasets to process, but each one can be processed independently so I don't see the need to concatenate them together into a single dask array. Our current strategy is to ask each worker to simultaneously download one or more of the datasets, and then split it across other workers to be processed in parallel. Unless you think it is better to download each dataset on the master sequentially, and then split each one across the workers?

I agree that it's probably much nicer to download on the workers and split from there.  It's worth noting that large data transfer between workers is a much more commonly used code path, and so is unlikely to suffer from the sorts of issues you were seeing in scatter.

Can you suggest how to scatter data from the workers w/o collecting it on the master? Tried this code but didn't work:
def download():
  return np.random.random(10)

e = Executor()
dFuture = e.submit(download)
e.scatter(futures_to_dask_arrays([dF]))

Got TypeError: Cannot serialize socket object

This should work:
def download():
    return np.random.random((10, 10, 10))

x = delayed(download)()
slices = [x[:, :, i] for i in range(10)]

slices = e.persist(*slices)
e.rebalance(slices)

However given your recent questions I suspect that you would be best served by taking a step back and asking more generally what you're trying to do.  There are many possible approaches to using Dask.


AdaptiveVector isNonZero is broken when sparseval == Monoid[T].zero

Here's an example of the breakage:
import com.twitter.algebird._
val a = SparseVector(Map.empty, 0, 5)
val b = AdaptiveVector.fromVector(Vector(1), 0)
Monoid.zeroEquiv[AdaptiveVector[Int]].equiv(Monoid.plus(a, b), b)
// => false

Monoid.plus returns SparseVector(1, 0, 0, 0, 0), extending the first vector. We want to be able to add zero-only vectors of any length and treat them as identity.

This is only a problem when the sparse value is the same as the zero of the underlying monoid.

this is still flakey:
AdaptiveVector[Int] has a monoid

Hi all, two questions

Could we add two more cases to AVSemigroup.plus to check if either argument isZero

def plus(left: AdaptiveVector[V], right: AdaptiveVector[V]) = {
...
  (left, right) match {
    case _ if isZero(left) => right
    case _ if isZero(right) => left
    case (DenseVector(lv, ls, ld), DenseVector(rv, rs, rd)) => ...
}

This bug is happening because the size of the result is getting set to the maxSize of the two arguments which is the right thing to do as long as neither argument isZero.

While we're at it, could we simplify AVMonoid.isZero to

def isZero(v: AdaptiveVector[V]) = (v.size == 0) || v.denseIterator.isEmpty

Whoops! Scratch (2), I get it now: we want to consider a vector where the sparseValue is not the Monoid.zero, but with dense values which are Monoid.zero to still be zero. Got it.

Now that I've poked around, a couple qs about what the semantics should be.

Is denseEquiv--having the same non-sparse values at the same indices, but being of possibly different lengths--the right notion of equivalence? This makes sense algebraically so I argue: yes. If you are doing machine learning, though, different-length vectors mean different things--they have different features--but given that we already allow adding vectors of different lengths here, if the user doesn't want vectors of different lengths to interact, they should make sure they don't already.

Currently, for sparseValue != monoid.zero, plus adds sparse values

val v = AdaptiveVector.fill[Int](10)(2)
// => v: com.twitter.algebird.AdaptiveVector[Int] = SparseVector(2, 2, 2, 2, 2, 2, 2, 2, 2, 2)
Semigroup.plus(v,v)
// => twoV: com.twitter.algebird.AdaptiveVector[Int] = DenseVector(4, 4, 4, 4, 4, 4, 4, 4, 4, 4)

I think we should only be adding non-sparse values so this looks like a bug to me, but want to make sure this is not, in fact, desired behavior.

I think the only use of AdaptiveVector is for sketchmap. I think we never change the size of a densevector.
I'm happy adding some better constraints and saying that two densevectors are only equal if they have the same length. I think we should do the more complex thing for sparse vs dense.

Same length for dense, not-necessarily the same length for sparse makes perfect sense so I'll do that and address the adding sparse values issue, too.
Okay, I see, AdaptiveVector is used in AdaptiveMatrix (and related things), which then gets used in SketchMap and that's about it.


Work with modals and use provider to config css positions

Hello @flocca
We have recently used your directive for some of our projects, but the directive wont work with modals and the position of the x is variable, then we decide to make some changes, we create a provider for the directive and the config for the css positions is provided here, also we have fixed the issue with the modals and now works perfectly.
If you consider use our work, you can check in our fork:
https://github.com/zgbjgg/angular-clearable/tree/0.1.3
Thank you!!!

Hi @zgbjgg and thank you for your work.
I can manually update my code with your changes or if you prefer you can do a Pull Request.
Please tell me what you prefer.
Thanks again,
Giacomo

Hello,
We could open a PR, but we've find another issue, when input is hidden by ng-show and then show the input, the directive doesnt work, let us check the issue to complete the directive and then open the PR.

Hello @flocca
We fixed the issue, but the question is that the master code updated, we used the code in tag 0.1.2 since is the bower version that install in our project, mmm how we can made a PR, directly to the master and then you resolve the conflict?


Update codebase and CI to use node v18

Node v16 has reached its end of life past September, this PR updates node version to v18.
Changes not included in the diff are:

Update Vercel from node v16 to node v18.
Update Netlify from v16 to v18.

Edit: Looks like updating node to v18 is breaking Storybook.

Updating node to v18 breaks Storybook V6, we must upgrade storybook to V7.
To resolve this, we need to upgrade Storybook to V7. Previous attempts to upgrade to Storybook V7 resulted in compatibility issues that prevented Storybook from building. I will give it another try and see if those compatibility arise again.


How can I send message to a specific client

Problem description:
You know, There may be many connections at the same time. But I just want to sand specific message to specific client. What shoule I do? Thanks
What I'm doing:
It is a chat room, One user can send messages to another user …… It's just like send_to([id...])

It depends.
If you never intend to run more than one process, you can just store a mapping of "client identifier" => "websocket connection" in a dict:
    connected = {}

    async def handler(websocket):
        # Register.
        connected[<identifier>] = websocket
        try:
            # Do stuff...
        finally:
            # Unregister.
            del connected[<identifier>]

If you may run more than one process, then you need some sort of interprocess communication, maybe based on a publish / subscribe system.

Marking as a documentation issue because this should be addressed in the FAQ, in common patterns, or maybe with a full example.

Updating from aagustin's example, you need to use the await .send(message). If you need to send to all clients in a list/dict use <client_list/dict>.broadcast(message)


Cant create ScanamoAlpakka with DynamoClient

Hi, I have an issue when trying to create a simple request
 val client = DynamoClient(DynamoSettings(system))
 ScanamoAlpakka.scan[String](client)(tableName)

An exception or error caused a run to abort: Found interface akka.stream.alpakka.dynamodb.scaladsl.DynamoClient, but class was expected
java.lang.IncompatibleClassChangeError: Found interface akka.stream.alpakka.dynamodb.scaladsl.DynamoClient, but class was expected
at com.gu.scanamo.ops.AlpakkaInterpreter$$anon$1.apply(AlpakkaInterpreter.scala:21)
at com.gu.scanamo.ops.AlpakkaInterpreter$$anon$1.apply(AlpakkaInterpreter.scala:13)
at cats.free.Free$$anonfun$foldMap$1.apply(Free.scala:155)
at cats.free.Free$$anonfun$foldMap$1.apply(Free.scala:153)
at cats.StackSafeMonad$class.tailRecM(StackSafeMonad.scala:15)
at cats.instances.FutureInstances$$anon$1.tailRecM(future.scala:10)
at cats.free.Free.foldMap(Free.scala:153)
at cats.free.Free$$anonfun$foldMap$1.apply(Free.scala:156)
at cats.free.Free$$anonfun$foldMap$1.apply(Free.scala:153)
at cats.StackSafeMonad$class.tailRecM(StackSafeMonad.scala:15)
at cats.instances.FutureInstances$$anon$1.tailRecM(future.scala:10)
at cats.free.Free.foldMap(Free.scala:153)
at com.gu.scanamo.ScanamoAlpakka$.exec(ScanamoAlpakka.scala:21)
at com.gu.scanamo.ScanamoAlpakka$.scan(ScanamoAlpakka.scala:61)

thanks

Looks like this could be due to an alpakka version incompatability. Scanamo uses 0.15.1. It looks like DynamoClient has been changed from a class to a trait in the latest version https://github.com/akka/alpakka/pull/1057 .


Build: add a make target to build the openssl application [1.1.1]

This commit adds a build_the_app make target, which essentially
builds apps/openssl, respecting all dependencies.
This target is mainly intended to be used by developers in order
to speed up compilation time.
Fixes #11932
(Backport of #11946)

I dislike this as much here as in #11946...  however, because the alternative I proposed there depends on new building features, the alternative I proposed there can't be applied in 1.1.1.
Also, considering that 1.1.1 isn't in development in the same manner as master, I don't see much value in this backport.

Ok. The motivation was consistency, so people wouldn't have to remember which targets are valid on which branch. But I get your point.


Cannot use "Api" as folder name when require modules

I have a viewModel which need to do http request and an endpoint configured in Api/endpoint.js
If I use var api = require("../Api/endpoint.js"), then I will get
Unable to resolve module.. error.
But if I rename the "Api" folder to "Endpoint" or something else, the error disappears.
I'm working on iOS project.

@facebook-github-bot stack-overflow


user resource doesn't say what changed

Same as #10640 but for the user resource
Description
Just says "user altered" which is not useful, it should say what changed as other resources do.
Chef Version
16.6.14
Platform Version
Windows, Linux

you may be able to pull something from my discarded pull/7716 .

Thanks @bby-bishopclark looks like you did similar to what I did for the group resource in #10642 ... so yeah I imagine fixing user will be similar.

Yeah.  Sorry it didn't get anywhere back then.

I'll have a PR today

that's great news!  I can't use 15+, but I'm glad it'll be fixed somewhere.

@bby-bishopclark you can use cinc if the licensing is an issue. Same code.

@bby-bishopclark you can use cinc if the licensing is an issue. Same code.

Yep.  As mentioned elsewhere, that's the plan once they figure out yum repos and if they don't have the same trapdoor.  Thanks for the suggestion, though.

Yep. As mentioned elsewhere, that's the plan once they figure out yum repos and if they don't have the same trapdoor.
Thanks for the suggestion, though.

Huh? It doesn't require accepting EULAs, if that's what you mean by "trap door"... been using it for over a year now.

We're getting off-topic, but that's great news if Cinc avoids the trapdoor.  Now, if they can look back in time and recover the lost art of yum repos, it'll achieve minimal viability.


0.15 Regression of border rendering when rendering a filled ImageNode

Bevy version
0.15
[Optional] Relevant system information
Linux pop!OS, bevy 0.15, cargo --version returns cargo 1.83.0 (5ffbef321 2024-10-29), using stable
`AdapterInfo { name: "AMD Radeon RX 7800 XT (RADV NAVI32)", vendor: 4098, device: 29822, device_type: DiscreteGpu, driver: "radv", driver_info: "Mesa 24.0.3-1pop1~1711635559~22.04~7a9f319", backend: Vulkan }`

What you did
Applied a border in the Node component that was UiRect::all(Val::Px(10.)) and added the BorderColor component to the same entity.
This rendered the border fine.
What went wrong
I then added an ImageNode to the same entity, with a Handle<Image> that took up all the space available.
The border disappeared.
I didn't expect the border to disappear, in bevy 0.14 the border was not overridden. Its not the end of the world for my current projects that this behavior exists, but it is a regression from 0.14.
Additional information
0.15 code:
use bevy::prelude::*;

fn main() -> impl std::process::Termination {
    App::new()
        .add_plugins(DefaultPlugins)
        .add_systems(Startup, bug)
        .run()
}

fn bug(mut commands: Commands) {
    commands.spawn(Camera2d);
    commands.spawn((
        ImageNode::solid_color(bevy::color::palettes::css::BLANCHED_ALMOND.into()), // un/comment to observe border bug
        Node {
            border: UiRect::all(Val::Px(10.)),
            width: Val::Percent(100.),
            height: Val::Percent(100.),
            ..default()
        },
        // using green to show that the camera does indeed show the (transparent) ImageNode
        BackgroundColor(bevy::color::palettes::css::GREEN.into()),
        // should display a yellow border
        BorderColor(bevy::color::palettes::css::YELLOW.into()),
    ));
}

With ImageNode commented out:
(Yellow border renders fine)

With ImageNode included (not commended out):
(Yellow border and green background do not render at all)

I would write some 0.14 code to demonstrate this same behaviour, but I can't work out how to construct a solid Handle<Image> like the convenient ImageNode::solid_color call above does. I'm currently porting bevy_cosmic_edit from 0.14 to 0.15 which has its own code to render an Image of an exact size which is where I originally noticed this regression (in example every_option).
Also, I know that ImageNode.color exists and that ImageNode::solid_color doesn't construct a non-default Image<Handle>, so this problem may be with ImageNode.color

I'm pretty sure it's not a bug. You need to spawn everything except the ImageNode, ant then the ImageNode as a children.

This is a bug, it's my fault probably, all the changes with required components got pretty confusing. It's easy to fix, I'll put in a pr later this evening.

Ahh thanks very much
I read through the bevy 0.15 blog post about required components, my only uncertain aspect is the expected behaviour of stacking multiple 'driving components' together. What if you put a ImageNode and a Mesh3d together? Anyway thats a larger more open question

Ahh thanks very much I read through the bevy 0.15 blog post about required components, my only uncertain aspect is the expected behaviour of stacking multiple 'driving components' together. What if you put a ImageNode and a Mesh3d together? Anyway thats a larger more open question

Mixing together different driving components in unintended ways will probably result in an unpedictable mess or panic and it's not a good idea unless you have a deep knowledge of all the possible interactions of the components.

Right looked at this properly now, doesn't look like a bug after all.
It's a measure func thing. You need to set NodeImageMode::Stretch otherwise it'll override the border width.
Modified your example so it draws the border over the solid image color:
use bevy::prelude::*;

fn main() -> impl std::process::Termination {
    App::new()
        .add_plugins(DefaultPlugins)
        .add_systems(Startup, bug)
        .run()
}

fn bug(mut commands: Commands) {
    commands.spawn(Camera2d);
    commands.spawn((
        ImageNode::solid_color(bevy::color::palettes::css::BLANCHED_ALMOND.into())
            .with_mode(NodeImageMode::Stretch),
        Node {
            border: UiRect::all(Val::Px(10.)),
            width: Val::Percent(100.),
            height: Val::Percent(100.),
            ..default()
        },
        // using green to show that the camera does indeed show the (transparent) ImageNode
        BackgroundColor(bevy::color::palettes::css::GREEN.into()),
        // should display a yellow border
        BorderColor(bevy::color::palettes::css::YELLOW.into()),
    ));
}

The documentation about this is really poor. I'm sure we used to have a UI image example, but it seems to have got lost somewhere?

Oh OK thanks
Maybe some version of my example code should be added to that enum variant for future travelers? Anyway this fixed my issue thanks :)


add access token method

There aren't fucntion for generate authorize url and get access token.
But, oauth2 gem provide these functions, so I implemented it for fitbit.

We need refresh access token method.
So I create it.


[Package Request]: Update VaclavSlavik.Poedit to 3.2

Package Requested

[X] I would like someone else to build the manifest.
[ ] I would like help so I can submit the manifest.
[ ] I have performed a search and couldn't find this package.
[X] I have checked that the installer for this package is not a script file
[X] I think there is a new version available and I have provided the URL.

Please provide the following information
Publisher: VaclavSlavik
Publisher Url: https://github.com/vslavik
Package Name: VaclavSlavik.Poedit
Description: Translations editor for Mac, Windows and Unix
Package Version: 3.2
Installer URL: https://download.poedit.net/Poedit-3.2-setup.exe
Current Package Identifier (for new versions): VaclavSlavik.Poedit

Resolved by #131661 ?


add open-closed eye state model

Reopened pr due to fix merge conflicts

Jenkins ok to test

@ygnn123 please fix warnings detected by AccuracyChecker and Demos precommit CI

Please address my comments form #983

Your comments were about the interactive_face_demo. I excluded oc eye model from a demo. And integrated to gaze_estimation_demo

Guys, the link checker in Documentation (Ubuntu 16.04) build found 7 broken links. But I don't know how i can fix links...

Guys, the link checker in Documentation (Ubuntu 16.04) build found 7 broken links. But I don't know how i can fix links...

@ygnn123 I looked on it, it is not caused of your changes, looks like some temporal problems with Pascal VOC server... So, you should not fix them

Your comments were about the interactive_face_demo. I excluded oc eye model from a demo. And integrated to gaze_estimation_demo

Yes, but most of them are still applicable for that pull request like https://github.com/opencv/open_model_zoo/pull/983#discussion_r407952421 or the request to update models.lst.

Your comments were about the interactive_face_demo. I excluded oc eye model from a demo. And integrated to gaze_estimation_demo

Yes, but most of them are still applicable for that pull request like #983 (comment) or the request to update models.lst.
EDIT: actually not yes. My comments are about gaze_estimation_demo. Unresolved ones.

Hi. I've fixed that. Could you check? All auto checks are passed.

Jenkins please retry a build

There are still other comments to apply. At least #983 (comment) is not applied yet.

All comments are applied now. I think)))

MRL Eye Dataset is ok, DSI-797

Jenkins: test this please


Crash App When set Url  Platform IOS

Could you help me? When i get url in sever and set url on uri react native video, it be crashed app
https://user-images.githubusercontent.com/110653846/220851237-991e004c-60d3-4652-9d13-efbb28a0beef.mov

Could you print the 'url' to check is it valid or not. Also 'onLoad', 'onProgress' callback could cause your crash. Could you share those as well?

No version provided, no sample, no reason to incriminate RNV, let's close this ticket

No versio

n provided, no sample, no reason to incriminate RNV, let's close this ticket

This is my onProgress and onLoad @freeboub

Please provide the crash description. The logcat or the js callstack...

please, help me :(

@memberwebm Just think about it. How we would help you without knowing your whole code.
Please provide it and I will try to figure out what is wrong.
Also, if you start your application from xCode, after the crash you have some logs in your xCode.

FIRST: I get url from sever,

Can you provide the url to check wether is it valid or not?
Can you provide the whole code? It looks like you have a couple of other callbacks, maybe it's easier to debug.
Can you reproduce this issue in a simplified public repository?

this is url video : https://rr3---sn-npoeenll.googlevideo.com/videoplayback?expire=1677679026&ei=UgX_Y4-zNqae4t4P04WdwA8&ip=<IP_ADDRESS>&id=o-AGyH3zxCyOu-TSc9XrxZqGdo2QDYYMgWwMC7a2-V5lXe&itag=18&source=youtube&requiressl=yes&mh=A0&mm=31%2C26&mn=sn-npoeenll%2Csn-30a7rne6&ms=au%2Conr&mv=m&mvi=3&pl=23&ctier=SH&initcwndbps=703750&spc=H3gIhkvVLOLinFsuxY35p6pnR7L45jA&vprv=1&svpuc=1&mime=video%2Fmp4&gir=yes&clen=2412863&ratebypass=yes&dur=32.320&lmt=1677576004044210&mt=1677657213&fvip=2&fexp=24007246&c=ANDROID&txp=5530434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cctier%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cratebypass%2Cdur%2Clmt&sig=AOq0QJ8wRgIhANoHI_7uMIG2leWfZN2tr7IIPpv_Hox1Ue044zaiDrFPAiEAp9WcoxxK1vyTMuEeSK_nakV73_b08dmZ5WoDXHHT6lQ%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIgAr6fuhIw7ZiQ4Hwrn04CPsYfQFdtY3rY_fIFgAYZrjgCIQChyCs-bWsNX7P0k1XVauQ7DklYp8V1Xj8bPs0RHtmw0w%3D%3D

As I saw, this url use redirection. Maybe, it could cause your error. Could you hardcode this url into and example repository?

yes i harcore, it work but when use url from sever i been crashed app

@vargajacint  I edited the code to be more concise, can you help me?

Okey, let's start from the beginning.

Which version of react-native-video are you using?
Check your code without any business logics. Particularly render just the Video component with a hardcoded video url. Is it still crashing?
Wrap your async code with try...catch block, and log the caught error.

For testing purpose use this video url: http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4


0.9.4 broken? Validation checks for the firstOfWeek in localizer

I'm using the 0.9.4 release and I'm seeing an assertion error that I wasn't seeing in 0.9.2.
The failure is around finding the function firstOfWeek in the localizer.
I'm using moment to initialize:
BigCalendar.setLocalizer(
  BigCalendar.momentLocalizer(moment)
);

not broken for me, can you be a bit more specific about the error?

This invariant is failing: https://github.com/intljusticemission/react-big-calendar/blob/master/src/localizer.js#L27
This is running with Moment 2.10.6.  Moment has startOf("week") - I think startOfWeek is deprecated.

Just checking through the code in more detail, and I see it's using the locale.firstDayOfWeek. I think this must just be a initialization problem and the momentLocalizer isn't being returned properly. I'll keep investigating.  Thanks.

Ahh I think the problem is I made a change to the API but didn't document it.
you don't need setLocalizer just use BigCalendar.momentLocalizer(moment) it will set itself

Ah! That's fixed it. Thanks!


Go vet: possible misuse of unsafe.Pointer

Problem: many (by measures of "awesome-go") go vet warnings.
There are many integer to pointer casts rule triggers in go vet report for Yanff.
Most (all?) cases go from pointer->uintptr->pointer, so it should not be unsafe as lint
assumes (it references Go 1.3a changes to GC).
What do you think about these solutions ("workarounds", actually):

Add more methods like unparsed, which return unsafe.Pointer with offset calculations applied, without additional type cast to uintptr. This way, 3-level casts will go away and related lint warnings will go avay.
Note: go vet allows this constructions: unsafe.Pointer(uintptr(p) + offset).

Move this kind of data casts (maywith struct itself) in separate package. Go can do cross-package inlining, so no performance will be lost, right? Report on Yanff will increase.

Related to adding YANFF to awesome-go.

This is a feature of the project and not a flaw or some mistake. We are using unsafe memory to manipulate C memory from Go, modifying what pointer refers to without copying for performance reasons. It cannot be done without copying and any other method also implies some copy. We know exactly what we are doing. And go_vet is kind of recommendation and not a rule. So if go_vet is a stopper for being included - fine.

Iskander's suggestion makes Go vet much more happy: for packet/packet.go it gives only one warning:
packet\packet.go:355: possible misuse of unsafe.Pointer
https://github.com/intel-go/yanff/commit/1d70bd23b30a7dc5f057d75a1df6168f9037ab17

Leaving proposal 20171 link for the reference.

proposal: spec: disallow T<->uintptr conversion for type T unsafe.Pointer

Deeper investigation is needed, but at a first glance it seems if this ever accepted, yanff code will break.

proposal 20171  is  about  type aliases/other stuff that hides  unsafe.Pointer. I don;t think YANFF uses type aliases, so it shouldn't be affected

@TocarIP, seems like you are right.
But in this case, #20171 looks awfully confusing.

Suggestion is accepted, fixed in #150


severity levels

Two things:

I have cleaned a bit the task logic because it was messy. Now the task logic is only in the start method. There is still some room for cleaning some logic but I will leave it for now.
I have added the concept of "severity levels". This way you can mark tasks and subtasks with different severity levels for various purposes. The most obvious ones are for the UI and the logs. You can use the following script to see how this works:

import logging

from brigade.core import Brigade
from brigade.plugins.functions.text import print_result
from brigade.plugins.inventory.simple import SimpleInventory
from brigade.plugins.tasks import commands

def another_group_of_tasks(task):
    task.run(task=commands.command,
             command="echo 1. {}".format(task.host),
             severity=logging.CRITICAL,
             name="Task A")
    task.run(task=commands.command,
             command="echo 1. {}".format(task.host),
             name="Task B")

def task_fails_for_some(task):
    task.run(task=commands.command,
             command="echo 1. {}".format(task.host),
             name="Task 1")
    task.run(task=commands.command,
             command="echo 2. {}".format(task.host),
             name="Task 2")
    task.run(another_group_of_tasks)
    if task.host.name == "dev3.group_2":
        # let's hardcode a failure
        task.run(task=commands.command,
                 command="sasdasdasd",
                 name="Task 3")
    else:
        task.run(task=commands.command,
                 command="echo 2. {}".format(task.host),
                 severity=logging.DEBUG,
                 name="Task 3")

brg = Brigade(
    inventory=SimpleInventory("tests/inventory_data/hosts.yaml",
                              "tests/inventory_data/groups.yaml"),
    dry_run=True,
)
result = brg.run(task_fails_for_some)
print_result(result, severity_level=logging.INFO, vars=['stdout'])

Just change the severity_level in the print_result method to WARN, DEBUG, etc... and see how tasks come and go from the UI.

For the test script provided it might be nice to have print_result optionally show the tracebacks or exception message for failed hosts.

That should happen already unless you are filtering the vars:
---- Task 3 ** changed : False ------------------------------------------------- ERROR
Traceback (most recent call last):
  File "/Users/dbarroso/workspace/brigade/brigade/core/task.py", line 59, in start
    r = self.task(self, **self.params)
  File "/Users/dbarroso/workspace/brigade/brigade/plugins/tasks/commands/command.py", line 29, in command
    shell=False)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 1344, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'sasdasdasd': 'sasdasdasd'


merging svaba

i was unable to merge SvABA calls using SURVIVOR and I don't see any issues regarding SvABA, so this might be a new topic for this tool. My command is:
SURVIVOR merge sample_files 100 1 0 0 0 50 sample_merged.vcf
i am allowing a minimum of one caller but SURVIVOR seems to simply skip svaba ones, probably not recognizing them as SVs.
and i have generated a simple comparison matrix (the third vcf is svaba):
$ cat sample_merged.mat.txt
152	122	0	125
122	545	0	406
0	0	0	0
125	406	0	770

a few sample lines from the svaba vcf include:
$ grep -v ^# svaba.somatic.sv.vcf | head
1	15142819	9184:1	T	[1:15145551[T	33	PASS	DISC_MAPQ=60;EVDNC=ASDIS;HOMSEQ=TTG;MAPQ=60;MATEID=9184:2;MATENM=0;NM=0;NUMPARTS=2;SCTG=c_1_15141001_15166001_51C;SPAN=2732;SVTYPE=BND	GT:AD:DP:GQ:PL:SR:DR:LR:LO	0/0:0:40:10.8:0,10.8,118.8:0:0:10.93:0	0/1:13:36:33.2:33.2,0,53:6:10:-33.08:33.38
1	15143384	9186:1	C	C]1:15144465]	22	PASS	DISC_MAPQ=60;EVDNC=ASDIS;HOMSEQ=GT;MAPQ=60;MATEID=9186:2;MATENM=0;NM=0;NUMPARTS=2;SCTG=c_1_15141001_15166001_132C;SPAN=1081;SVTYPE=BND	GT:AD:DP:GQ:PL:SR:DR:LR:LO	0/0:0:42:11.4:0,11.4,125.4:0:0:11.47:0	0/1:9:26:22.7:22.7,0,39.2:3:8:-22.61:22.9
1	15144465	9186:2	T	T]1:15143384]	22	PASS	DISC_MAPQ=57;EVDNC=ASDIS;HOMSEQ=GT;MAPQ=60;MATEID=9186:1;MATENM=0;NM=0;NUMPARTS=2;SCTG=c_1_15141001_15166001_132C;SPAN=1081;SVTYPE=BND	GT:AD:DP:GQ:PL:SR:DR:LR:LO	0/0:0:42:11.4:0,11.4,125.4:0:0:11.47:0	0/1:9:26:22.7:22.7,0,39.2:3:8:-22.61:22.9

Svaba seems to only return BND, not sure if that's an issue. my merge command allows a minimum of one caller and ignores conflicting types.

here's a related example. we have two vcfs that are both called by SvABA on the same sample but vcf A was run by us and vcf B was called by another organization. I was able to find two calls that are likely referring to the same SV but vcf B was included but vcf A was excluded.
calls from vcf A:
1	15142819	9184:1	T	[1:15145551[T	33	PASS	DISC_MAPQ=60;EVDNC=ASDIS;HOMSEQ=TTG;MAPQ=60;MATEID=9184:2;MATENM=0;NM=0;NUMPARTS=2;SCTG=c_1_15141001_15166001_51C;SPAN=2732;SVTYPE=BND	GT:AD:DP:GQ:PL:SR:DR:LR:LO	0/0:0:40:10.8:0,10.8,118.8:0:0:10.93:0	0/1:13:36:33.2:33.2,0,53:6:10:-33.08:33.38
1	15145551	9184:2	C	[1:15142819[C	33	PASS	DISC_MAPQ=60;EVDNC=ASDIS;HOMSEQ=TTG;MAPQ=60;MATEID=9184:1;MATENM=0;NM=0;NUMPARTS=2;SCTG=c_1_15141001_15166001_51C;SPAN=2732;SVTYPE=BND	GT:AD:DP:GQ:PL:SR:DR:LR:LO	0/0:0:40:10.8:0,10.8,118.8:0:0:10.93:0	0/1:13:36:33.2:33.2,0,53:6:10:-33.08:33.38

calls from vcf B:
1	15142891	3476:no_id:1	G	[1:15145551[G	.	PASS	BLACKLIST=0;EVDNC=ASDIS;HOMSEQ=CAA;MAPQ=60;MATEID=3476:no_id:2;MATEMAPQ=60;NDISC=0;NSPLIT=0;NUMPARTS=2;SCTG=c_1_15142501_15148501_16;SPAN=2660;SVTYPE=BND;TDISC=8;TSPLIT=4	NALT:NALT_RP:NALT_SR:READ_ID	0:0:0:.	11:8:4:HS7_5723-2-46-16007-41348,HS14_5615-1-63-4915-9134,IL34_5198-4-118-3337-18138,IL34_5198-7-27-17628-16584,IL34_5198-8-6-16510-14867,HS17_5841-7-1201-14741-147696,IL34_5198-2-106-13813-18791,IL34_5198-3-107-18322-17933,HS14_5615-2-3-9163-127602,HS14_5615-2-4-15789-93334,IL34_5198-2-72-5044-15786
1	15145551	3476:no_id:2	C	[1:15142891[C	.	PASS	BLACKLIST=0;EVDNC=ASDIS;HOMSEQ=CAA;MAPQ=60;MATEID=3476:no_id:1;MATEMAPQ=60;NDISC=0;NSPLIT=0;NUMPARTS=2;SCTG=c_1_15142501_15148501_16;SPAN=2660;SVTYPE=BND;TDISC=8;TSPLIT=4	NALT:NALT_RP:NALT_SR:READ_ID	0:0:0:.	11:8:4:HS7_5723-2-46-16007-41348,HS14_5615-1-63-4915-9134,IL34_5198-4-118-3337-18138,IL34_5198-7-27-17628-16584,IL34_5198-8-6-16510-14867,HS17_5841-7-1201-14741-147696,IL34_5198-2-106-13813-18791,IL34_5198-3-107-18322-17933,HS14_5615-2-3-9163-127602,HS14_5615-2-4-15789-93334,IL34_5198-2-72-5044-15786

Thinking that my minimum length parameter was excluding some of the calls, changed minimum length to 0. This did not solve the issue -- 100% of entries had SUPP_VEC=10.

well i think i figured it out. SURVIVOR does not like the GT field, but this is (somewhat) remedied by specifying minimum caller to be 0 instead of 1. Then all the calls accepted from vcf A say SUPP_VEC=00. not ideal, so instead I'll just have to clean out GT from the vcf before merging.


Minecraft 1.19 support

Add 1.19 version support

This factor depends only on Mineflayer lib update, which don't have 1.19 support.


Flow_refresh.sh Script failed in Sidecar Container

What happened?
I try to import a flow from Config File via the Sidecar Container but the flow_refresh.sh script is failing. The flow will not imported via the API Refresh.
Actually I am using pv/pvc and install flows and nodes manually via web interface. Thought I would be a good way to use a sidecar like kubernetes microservice style.
k logs node-red-.... node-red-sidecar
{"time": "2023-04-26T12:17:07.098556+00:00", "msg": "Starting collector", "level": "INFO"}
{"time": "2023-04-26T12:17:07.100418+00:00", "msg": "No folder annotation was provided, defaulting to k8s-sidecar-target-directory", "level": "WARNING"}
{"time": "2023-04-26T12:17:07.100861+00:00", "msg": "Loading incluster config ...", "level": "INFO"}
{"time": "2023-04-26T12:17:07.101686+00:00", "msg": "Config for cluster api at 'https://<IP_ADDRESS>:443' loaded...", "level": "INFO"}
{"time": "2023-04-26T12:17:07.102336+00:00", "msg": "Unique filenames will not be enforced.", "level": "INFO"}
{"time": "2023-04-26T12:17:07.102516+00:00", "msg": "5xx response content will not be enabled.", "level": "INFO"}
{"time": "2023-04-26T12:17:17.129633+00:00", "msg": "Script failed with error: Command '['sh', '/app/flow_refresh.sh']' returned non-zero exit status 127.", "level": "ERROR"}

How can we reproduce this?
Connecting to the sidecar shell and checking the flow_refresh script:
k exec  node-red-..... -it -c node-red-sidecar -- sh
/app $ sh flow_refresh.sh
node-red flow refresh api
flow_refresh.sh: line 4: curl: not found
flow_refresh.sh: line 4: jq: not found
flow_refresh.sh: line 5: curl: not found

The k8s-sidecar container is missing curl and jq binary!
Is this the common behaviour?
Helm Chart version
0.23.1

Search

[X] I did search for other open and closed issues before opening this.

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Additional context
No response

Yepp @hinrichd, thats what we got asked here too: https://github.com/SchwarzIT/node-red-chart/issues/149
The issue is, I am not sure if this project should take the semi-ownership of the https://github.com/kiwigrid/k8s-sidecar project.
Maybe worth to file a feature-request there and ask to add curl and jq to the base image?

Hey @dirien
I have opened a ticket in the k8s-sidecar project .
Let´s see what they can do.

awesome @hinrichd! Let's see what the answer is!

Hi @dirien,
here is the answer for the sidecar project...

Adding more binaries to the image does not really match the philosophy of the project having a minimal image and hence minimal attack surface.
But we're building upon a generic python image and already do have the python requests package installed. So I think it should be possible to use python instead of separate binaries to perform the requests and parse JSON responses.

I totally agree with this comment. To fit best into the sidecar the flowrefresh script should be using python and the request package/module as dependency.
I am only the mac and bash guy and not python. Will have a look at psf requests in some days, it says it is for humans...
✌

UPDATE: Now, there is a new version of the k8s-sidecar release called 1.24.0. In this version the SCRIPT Environment Variable will execute python scripts as well by respecting the script´s shebang. Before, the script was directly called by sh command and you had to  run python from within the script as subprocess. This has some downsides to stdin/out. See this FR related to the original issue.
Meanwhile I rewrite the flow_refresh.sh script to flow_refresh.py due to dependencies of the python based image.
Executing manually the flow_refresh.py with .\flow_refresh.py inside the sidecar will reload the node-red flows:
--> Script Output from Sidecar Container
/tmp $ hostname
node-red-54d6679cb-m9qfr
/tmp $ vi flow_refresh.py
/tmp $ chmod 755 flow_refresh.py
/tmp $ ./flow_refresh.py
node-red flow refresh api via k8s-sidecar
Sidecar sleeping for 5 seconds...
node-red flow refresh api response code = 200

--> Node-Red Log Output
8 May 12:47:30 - [warn] Verschlüsselte Credentials nicht gefunden
8 May 12:47:30 - [info] Flows werden gestoppt
8 May 12:47:30 - [info] Flows sind gestoppt
8 May 12:47:30 - [info] Flows werden gestartet
8 May 12:47:30 - [info] Flows sind gestartet

flow_refresh.py

Default Chart SLEEP_TIME_SIDECAR must be integer not string (5s)
retry and retry delay not supported
flow_refresh.py has to be executable, otherwise it will be executed be sh an will fail.

#!/usr/bin/env python

import time
import os
import requests
import json
import sys

# SET VARIABLES FROM CONTAINER ENVIRONMENT
SLEEP_TIME_SIDECAR = 5 if os.getenv("SLEEP_TIME_SIDECAR") is None else int(os.getenv("SLEEP_TIME_SIDECAR"))
USERNAME = os.getenv("USERNAME")
PASSWORD = os.getenv("PASSWORD")
URL = os.getenv("URL")

print('node-red flow refresh api via k8s-sidecar')
print('Sidecar sleeping for', SLEEP_TIME_SIDECAR, 'seconds...')
time.sleep(SLEEP_TIME_SIDECAR)

# GET NODE RED BEARER TOKEN
PAYLOAD_TOKEN = {"client_id": "node-red-admin", "grant_type": "password", "scope": "*", "username": USERNAME, "password": PASSWORD}
r_token = requests.post(URL + '/auth/token', data=PAYLOAD_TOKEN, timeout=30,  verify=False)
token = (json.loads(r_token.text)["access_token"])

# FLOW REFRESH/RELOAD FLOWS FROM SECRET/CONFIGMAP
PAYLOAD_FLOW_REFRESH = "{\"flows\": [{\"type\": \"tab\"}]}"
HEADERS_FLOW_REFRESH={
  'Authorization': 'Bearer' + ' ' + token,
  'content-type': 'application/json; charset=utf-8',
  'Node-RED-Deployment-Type': 'reload',
  'Node-RED-API-Version': 'v2'
}

r_flow_refresh = requests.post(URL + '/flows', headers=HEADERS_FLOW_REFRESH, data=PAYLOAD_FLOW_REFRESH, timeout=30,  verify=False)

if r_flow_refresh.status_code == requests.codes.ok:
    print('node-red flow refresh api respoonse code =', r_flow_refresh.status_code)
    sys.exit(0)
else:
    sys.exit('Error-Code', r_flow_refresh.status_code)

@hinrichd many thanks for the contribution!


⚠ Bofrost.at has degraded performance

In 5424cbc, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 949 ms

Resolved: Bofrost.at performance has improved in 8b461e5 after 6 minutes.


PR: Harden Runner Test

PR to test Harden Runner by accessing a restricted URL using curl.

Just a test to show how Harden Runner works.


feature: remove unused vendor packages

Signed-off-by: fengzixu<EMAIL_ADDRESS>
Ⅰ. Describe what this PR did
According to #2399 , we should clear unused packages located in the vendor dir.
Ⅱ. Does this pull request fix one issue?

#2399
Ⅲ. Why don't you add test cases (unit test/integration test)? (你真的觉得不需要加测试吗？)
Just clearing.
Ⅳ. Describe how to verify it
I run the command make && make install && make test to verify that removing packages doesn't influence the pouch.
Ⅴ. Special notes for reviews
Removed packages

github.com/go-resty/resty
github.com/magiconair/properties
github.com/go-ini/ini
github.com/containerd/containerd/version
github.com/alibaba/pouch/libcontainerd
github.com/Microsoft/go-winio/vhd

@fengzixu could you remove the two useless packages in vendor.json by
govendor remove github.com/alibaba/pouch/libcontainerd
govendor remove github.com/Microsoft/go-winio/vhd

These two pkgs are useless.

Thank @fengzixu


xNN - output column in tab "Neural Network Define" tab

Is your feature request related to a problem? Please describe.
-The output column in the Neural Network Define tab is not required.
Describe the solution you'd like

The second column can be deleted.

already fixed


cla: accepted by oscvr (recover original commit signature)

Thanks @adrianbenavides backed up bdb8daae57444c6eba7e248922d0dedc5f6aac93 in https://github.com/build-trust/ockam/tree/cla/oscvr


upload user feedback

Provide user feedback about uploads :

if file > 5.000 lines recommend splitting the file but allow to proceed
provide live feedback (flash messages) while treating if file > 1000 lines

duplicate of issue #17


check details.response.statusCode when error is falsy

Description
If response status is 200, minimal-request returns null as error. In order to have explicit feedback that a resource that should be forbidden is exposed instead, we need to check against the details.response.statusCode.
Before

After

Thanks!


[Task] Add unit test cases for NotificationButton component

Task Description:
Currently, we run unit tests using jest which can be found here.
However, the test cases are not yet comprehensive enough and work can be done to cover more grounds. Specifically for this issue, we hope to add test cases for the NotificationButton component.
Deliverable(s):
New unit test cases that cover the testing of the NotificationButton component.
Additional Context:

Button components can be found here
An example for AudioButton can be found here which may serve as a good source for reference

Completed with #144


[Feature]: more control over logging

Contact Details
richard.vandenberg (at) NCSC-NL
What should we build?
When the connection to the database fails, Orchestrator logs the connection string including password to stderr. We have no control over this because of this line:
https://github.com/workfloworchestrator/orchestrator-core/blob/4a3e438bfed77e6db8b2e06f3d7a9ae8e4b3ada6/orchestrator/app.py#L262
We would like to be able to set pretty_exceptions_show_locals=False and other Typer() options.
Additionally this line sets up structlog in a very static way, causing all sorts of problems. See https://github.com/workfloworchestrator/nwa-stdlib/pull/32
It would be great to be able to set our own handlers and formatting.
https://github.com/workfloworchestrator/orchestrator-core/blob/4a3e438bfed77e6db8b2e06f3d7a9ae8e4b3ada6/orchestrator/app.py#L117
Relevant pseudo code
No response

It would be very handy if we could pass some extra info to OrchestratorCore that allows one to configure/use the additional loggers config dict.
Then we can use it like it was supposed:
implementation of initialise_logging()

Hi Guys, thanks for the issue. Logging is something that could definitely receive some TLC, perhaps you would be willing to make a PR?