jest-editor-support: Add support for Windows Subsystem for Linux in Runner

Add useWsl configuration flag to project worksapce
Translate windows paths in call args to POSIX paths valid in WSL

see https://github.com/jest-community/vscode-jest/issues/331

Summary
See https://github.com/jest-community/vscode-jest/issues/331 for motivation.
When running vscode in Windows but executing jest in WSL (WIndows Subsystem for Linux), additional steps have to be performed:

project_workspace now has a useWsl flag (like the built-in flag for vscode launch options) which can be set to true
The wsl command has to be prepended to the shell command if useWsl is true
The jest output path is a windows path, but wsl is running in a different filesystem with POSIX paths (usually C:/ is /mnt/c), this has to be translated.
The last step is done in a own package wsl-path, as this is also required in vscode-jest and I don't really see it in the scope of either jest or vscode jest (see https://github.com/mojadev/wsl-path)

Test plan
See the following command being executed on a Windows 10 Host - normally the path would be C:..

Codecov Report

Merging #6941 into master will decrease coverage by <.01%.
The diff coverage is 80%.

@@            Coverage Diff             @@
##           master    #6941      +/-   ##
==========================================
- Coverage   66.97%   66.96%   -0.01%
==========================================
  Files         250      250
  Lines       10379    10373       -6
  Branches        4        4
==========================================
- Hits         6951     6946       -5
+ Misses       3427     3426       -1
  Partials        1        1

Impacted Files
Coverage Δ

...kages/jest-editor-support/src/project_workspace.js
88.88% <ø> (+1.38%)
:arrow_up:

packages/jest-editor-support/src/Process.js
62.16% <80%> (-4.51%)
:arrow_down:

packages/jest-circus/src/index.js
57.69% <0%> (-8.98%)
:arrow_down:

packages/expect/src/jest_matchers_object.js
84.61% <0%> (-1.1%)
:arrow_down:

packages/jest-config/src/normalize.js
91.33% <0%> (-0.11%)
:arrow_down:

packages/jest-each/src/index.js
77.77% <0%> (ø)
:arrow_up:

packages/jest-jasmine2/src/each.js
0% <0%> (ø)
:arrow_up:

...ackages/jest-jasmine2/src/jasmine/jasmine_light.js
0% <0%> (ø)
:arrow_up:

packages/expect/src/matchers.js
97.45% <0%> (ø)
:arrow_up:

... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 790abe7...e46ab29. Read the comment docs.

I'll have some time towards the end of the week to really get into it if the other two don't. Looks great at a glance though. Nicely done Jannis!

@orta @seanpoulter @connectdotz ping 🙂

I don't have a window machine to verify this, if @seanpoulter didn't have time to review it, I suggest @stephtr who actually reviewed the original jest-community/vscode-jest#331

At first thank you for your work!
One thing I noticed is that if you have several Linux distros installed in parallel (which is perfectly supported by Microsoft), you have no chance of specifying in which one the command should be executed. One way of solving it would be to not automatically prepend wsl as a command but have the user to explicitly set for example bash npm test --.
Beside that, is it intended that we transform windows -> linux paths in jest-editor-support but do the backtransformation in vscode-jest instead of within Runner.js?

Hey,
very good points - I'll rework the PR this week.

Regarding multiple wsl environments -> I basically mirrored the behaviour of  the launch.json configuration, which always uses the default distro. Originally I also didn't have the prepend command magic in there, but I didn't like having to set two settings (one useWsl + an updated path) - detecting the environment from the path is not really an option, I think. How about useWsl: string | boolean, e.g.

    useWsl: true // default wsl env
    useWsl: 'ubuntu' // use ubuntu
    useWsl: 'bash;  // use bash

as possible config options?

Having the backtransformation in vscode-jest would be definetly best (also because I don't need two PRs :) ), but where would you see that happening right? When I get it right, editor-support is only providing the output as is, reading the actual results is done in vscode-jest. I could see two options to support mapping only in one place:

Move reading the json file to editor-support and make an implicit mapping when in wsl
Move wsl support deeper into jest, so the file is already mapped, which I would not really consider an option
I didn't want to change too much in the existing code for this PR, but I'm willing to :)

How about useWsl: string | boolean

In my opinion that would be perfect!

When I get it right, editor-support is only providing the output as is, reading the actual results is done in vscode-jest.

You initially added the translateWslPathsToWindowsPaths to updateWithData, which itself is called within jestProcess.onJestEditorSupportEvent('executableJSON', (data: JestTotalResults) => {this.updateWithData(data)}), which in turn gets triggered by _parseOutput within jest-editor-support/Runner.js.

You're right, I somehow had in mind vscode-jest reads the file by itself. Thanks!
Then I'll

update wsl-path to support multiple environments
update useWsl to (string | boolean)
move the translate logic out of vscode-jest and into _parseOuptut

I hope that I manage to do it in the next few days

Sounds good to me.
@connectdotz or @seanpoulter: is that ok for you?

I'm unable to comment for a few weeks, sorry.

@stephtr : Sorry for the delay here (I didn't find too much time for reworking it, and my main system is Fedora again). One question regarding file naming: I will add a result_reader module which contains a readTestResults. Now I'm a bit confused because some modules are uppercase (Process.js) and others are lower / snakecase (test_reconciler). Is there some kind of pattern behind this? Should I use result_reader.js or ResultReader.js?

See #4969

@stephtr I updated the PR as discussed, maybe you find some time to look over it

@mojadev could you open this up against https://github.com/jest-community/jest-editor-support (See #7232).
Thanks, and sorry for the inconvenience!

No problem! Actually the big jest setup causes real issues for me under WSL (mostly yarn and npm related),  so I appreciate having a dedicated repo.
From: Simen Bekkhus
Sent: Monday, October 22, 2018 1:30 PM
To: facebook/jest
Cc: Jannis Moßhammer; Mention
Subject: Re: [facebook/jest] jest-editor-support: Add support for WindowsSubsystem for Linux in Runner (#6941)
@mojadev could you open this up against https://github.com/jest-community/jest-editor-support (See #7232).
Thanks, and sorry for the inconvenience!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.


rust toolchain version update

What this PR does / why we need it:
Updates the toolchain version. This is required to ingest new updated dependencies that requires a more recent version of rustc.

Looks like the tarpaulin version may need to be increased or parameterized to use the correct rust version: https://github.com/xd009642/tarpaulin


TKCalendarMonthTiles __unsafe_unretained

Issue with selecting dates from the calendar.
When selecting a date, Zombies states that 'monthDate' has been previously released.
Falling over in this function of TKCalendarMonthView.m on the last line highlighted here:

(NSDate*) dateSelected{
if(selectedDay < 1 || selectedPortion != 1) return nil;
TKDateInformation info = [monthDate dateInformationWithTimeZone:[NSTimeZone timeZoneForSecondsFromGMT:0]];

Removed the '__unsafe_unretained' declaration and all was well once more. It is possible that i'm doing something wrong but it was a little suspect that this only occured after the introduction of ARC.

This issue should be fixed with latest commit.


GPA-7871: Updated Regular Font

Updating the PDF library revealed an issue with the Regular font.  I went ahead and grabbed an updated version of the Roboto library (which had alot more Lookups in it) and added back our glyphs and that fixed the issue.
Not sure what happened to our font along the way but this font definitely prints nicer in the pdfs
Here is the glyphs I added back (I went through the pull requests - hopefully I didn't miss one)
Big X			uni2715
Composition		uni25E6
Angle			uni2220
Triangle_Symbol		uni25B3
Square			uni25A1
Congruent		uni2245
Left_Ceiling_Bracket	uni23BE
Left_Floor_Bracket	uni23BF
Right_Celing_Bracket	uni23CB
Right_Floor_Bracket	uni23CC
Right_Arrow		uni279C
Union			uni222A
Intersection		uni2229
Bullet_Dot		uni26AB
Black Circle		uni25CF
Black Large Circle	uni2B24

Let's us a font comparison tool to ensure all glyphs are present. Additionally we can determine why this font is so much larger and reduce it.

Good call on the diff tool, turned out I was missing uni2218.  I used fontdiffenator (looks like we aren't the only ones with great naming of tools) https://github.com/googlefonts/fontdiffenator
Here is the summary of the results (keep in mind I updated the font to use the latest repo)
Summary.txt
Here is the gifs it spit out:
result_gifs.zip

One more note - in the summary it is stating a missing glyph is Tcommaaccent but it is in this version of the font.  Here is a screenshot:

Here is the results from the base Roboto font compiled vs this font
Summary.txt
Along with the gifs:
gifs-roboto-gmm.zip

There has been alot more glyphs added glyphs new: 2119

Although now I am wondering how we went from 381 KB   to 523 KB by pulling in the latest version of Roboto - I may need to look into this some more

Well I know why it is 523 KB now.  When I pulled down the public repo instead of setting up the whole build process (which is quite a nightmare) I opted to go with their pre-compiled ttf files under the src/hinted folder and that is already at 504 KB, so if we want this file down to the 300 range I will need to work on building it again with their process and then convert that one over again.

The font was most likely messed up because getting the roboto source to build on a Windows box is a horrible nightmare - outdated 3rd party dependencies and still using python 2.7.  I have spent a couple hours on this and am still coming up with errors.  I am not sure how much time we want to spend on this


(Feature) Adds OtpInput and fixes some styles

This PR closes #166574011, by:

Adding OTP input with custom styles
Fixes from issue comment:
• lines are not same width
• asterix missing as fillers (don't do it if its not simple)
• checkmark after code was sent again should be with transparent background and blue text color

This looks awesome
I've tested it and worked great. Only thing I've found was the "send again" button showing up even if the number was never sent for the first time (by pressing delete on the last field)
It's not a big deal I think but it should be easily fixable by keeping track on when API is fired

just to clarify, @dariominones and @kbardi found out that this is not an actual bug. It's the intended behavior.


Conditional Statements in C++

Bhai accept kor, jama chai
pune r treat ta noi naa e dili

Hey thanks for your contribution ❗. Keep up the good job and leave a star if you liked the repo .
Happy hacking 😉


Implement Mocha-based test suite

Looks awesome. Does this have to be in individual files? Seems like it'd get too spread out for my tastes.

We could compact them, but notice that tests for a single move (Roost) are already a hundred lines (and that's not all we should test about Roost).

Hooray, I can finally test changes at work!

Good job @Slayer95! :+1:


Model loading became very slow after #11001

Description
The PR (#11001) introduced a checked in SymbolBlock.imports that makes the loading of large graph takes very long time (it used to be instantaneous):
The issue is the code checking if any symbol is row_sparse, removing this check allows to load large model instantaneously again.
Environment info (Required)
----------Python Info----------
Version      : 3.6.5
Compiler     : GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-9<IP_ADDRESS>)
Build        : ('default', 'Jun 17 2018 12:26:58')
Arch         : ('64bit', '')
------------Pip Info-----------
Version      : 18.0
Directory    : /usr/local/lib/python3.6/site-packages/pip
----------MXNet Info-----------
/usr/local/lib/python3.6/site-packages/requests/init.py:80: RequestsDependencyWarning: urllib3 (1.24) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
Version      : 1.3.0
Directory    : /usr/local/lib/python3.6/site-packages/mxnet
Commit Hash   : b3be92f4a48bce62a5a8424271871c2f81c8f7f1
----------System Info----------
Platform     : Darwin-16.7.0-x86_64-i386-64bit
system       : Darwin
node         : 186590d6796f.ant.amazon.com
release      : 16.7.0
version      : Darwin Kernel Version 16.7.0: Wed Oct 10 20:06:00 PDT 2018; root:xnu-3789.73.24~1/RELEASE_X86_64
----------Hardware Info----------
machine      : x86_64
processor    : i386
b'machdep.cpu.extfeatures: SYSCALL XD 1GBPAGE EM64T LAHF LZCNT PREFETCHW RDTSCP TSCI'
b'machdep.cpu.leaf7_features: SMEP ERMS RDWRFSGS TSC_THREAD_OFFSET BMI1 AVX2 BMI2 INVPCID SMAP RDSEED ADX IPT FPU_CSDS'
b'machdep.cpu.features: FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE SSE3 PCLMULQDQ DTES64 MON DSCPL VMX EST TM2 SSSE3 FMA CX16 TPR PDCM SSE4.1 SSE4.2 x2APIC MOVBE POPCNT AES PCID XSAVE OSXSAVE SEGLIM64 TSCTMR AVX1.0 RDRAND F16C'
b'machdep.cpu.brand_string: Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz'
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0543 sec, LOAD: 1.2793 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0628 sec, LOAD: 0.9555 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.0802 sec, LOAD: 0.8386 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0073 sec, LOAD: 1.2169 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0521 sec, LOAD: 1.1819 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0419 sec, LOAD: 0.3027 sec.
Minimum reproducible example
(If you are using your own code, please provide a short script that reproduces the error. Otherwise, please provide link to the existing example.)
Steps to reproduce
(Paste the commands you ran that produced the error.)

download model attached and script to load it
model-loading-issue.zip
python slow_model_loading.py

What have you tried to solve it?

comment check that symbol is row_sparse

It will be great to test loading a large graph to avoid such regression (the model I attached could serve as a unit-test).

@geoalgo Thank you for reporting the performance regression in , model loading.

@mxnet-label-bot add [Performance]

Looks like there might be some issue with the .get_internals() function which is the true bottle neck, some quadratic complexity maybe.

meet the same problem too.


Failing tests somehow related to CloudFirestoreModularAdapter.isFastBoot

Hi, I'm refactoring a project to use the latest version of this addon. My test suite is failing when the isFastBoot getter in the adapter is called, though I can't work out why.
Here is an example of a test that fails, when run in isolation:
import { module, test } from 'qunit';
import { setupRenderingTest } from 'ember-qunit';
import { render, settled } from '@ember/test-helpers';
import { hbs } from 'ember-cli-htmlbars';

import { signInWithEmailAndPassword } from 'ember-cloud-firestore-adapter/firebase/auth';

module('Integration | Component | video-list', function (hooks) {
  setupRenderingTest(hooks);

  hooks.beforeEach(async function () {
    const session = this.owner.lookup('service:session');
    await session.authenticate('authenticator:oneday-firebase', (auth) => {
      return signInWithEmailAndPassword(auth<EMAIL_ADDRESS>'123456');
    });
    await settled();
  });

  hooks.afterEach(async function () {
    const session = this.owner.lookup('service:session');
    if (session && session.isAuthenticated) {
      await session.invalidate();
    }
    await settled();
  });

  test.only('it renders', async function (assert) {
    let store = this.owner.lookup('service:store');
    store.generateStackTracesForTrackedRequests = true;
    const videos = await store.findAll('video', {});
    this.set('videos', videos);
    await render(hbs`<VideoList @videos={{this.videos}} />`);
    assert.dom('[data-test-video-list]').exists();
  });
});

Here is the error:
index.js:135 Uncaught (in promise) Error: Can not call `.lookup` after the owner has been destroyed
    at Container.lookup (index.js:135:1)
    at Class.lookup (container_proxy.js:81:1)
    at Class.get isFastBoot [as isFastBoot] (cloud-firestore-modular.js:101:99)
    at Array.eval (cloud-firestore-modular.js:211:54)
    at next (index.esm2017.js:6943:985)
    at eval (index.esm2017.js:5400:271)

Any thoughts on why this is destroyed by the time it gets into the adapter?

Can you point out in which part of the adapter code it's failing? And maybe a snippet of VideoList template as well.
My hunch is that you may be trying to render some relationships which isn't being waited for by the tests so while it's doing those, the test gets destroyed already.


[Ticket]: Translating our Documentation

Project
Website (potterdb.com)
Describe the ticket you want to create
To make our Potter DB accessible to a global audience, we're reaching out to contributors like you for help in translating our documentation into multiple languages.
Get started

Please drop a quick note below to let us know which language you're working on. This helps us keep track of ongoing translations and prevents duplication of efforts.
Clone the Repository and navigate to the docs folder within the repository.
Translate the *.en.mdx files within the /pages directory and start translating them into your desired language.

If you encounter any challenges or need clarification while translating, don't hesitate to reach out to us. We're here to assist you every step of the way.
Acceptance Criteria

[ ] Translate the documentation to your language
[ ] Add you language to the i18n section of docs/theme.config.tsx and docs/next.config.mjs
[ ] Make sure to translate the values (edit / feedback links, theme switch, toc etc.) in the docs/theme.config.tsx,

Additional Details
https://docs.potterdb.com

Hi 👋🏼 🪄
I'd like to participate to this exciting project 🧙🏼♀ ! I don't code Ruby and don't know much about GraphQL so I had a look at this issue : I can translate documentation in French.
Would it be interesting for the project?

Yes, translations for the documentation to all languages are welcome!

Great ! Thanks a lot ;)

Hello, I want to participate translating into Spanish.

Hello, I want to participate translating into Spanish.

Great, any translation is much appreciated!

Hi there, I would like to do Slovak.

Hi 👋🏼
I'm translating the Resource part in doc and I want to be sure : datas will be in english only (at least for the moment) so in the Resources/Books page, Attributes name must stay in English, right ?

Hi 👋🏼 I'm translating the Resource part in doc and I want to be sure : datas will be in english only (at least for the moment) so in the Resources/Books page, Attributes name must stay in English, right ?

Yes, exactly, those are the attributes / fields from the API and should not be translated.

@danielschuster-muc Can I update rest.en.mdx to apply the same style than in French version to the list of predicates ?
So this:

  Expand to see all available filter predicates- `eq`: equals - `eq_any`: equals
  any - `eq_all`: equals all - `not_eq`: not equal to - `not_eq_any`: not equal to any -
  `not_eq_all`: not equal to all - `matches`: matches - `matches_any`: matches any - `matches_all`:
  matches all - `does_not_match`: doesn't match - `does_not_match_any`: doesn't match any -
  `does_not_match_all`: doesn't match all - `lt`: less than - `lt_any`: less than any - `lt_all`:
  less than all - `lteq`: less than or equal to - `lteq_any`: less than or equal to any -
  `lteq_all`: less than or equal to all - `gt`: greater than - `gt_any`: greater than any -
  `gt_all`: greater than all - `gteq`: greater than or equal to - `gteq_any`: greater than or equal
  to any - `gteq_all`: greater than or equal to all - `in`: in - `in_any`: in any - `in_all`: in all
  - `not_in`: not in - `not_in_any`: not in any - `not_in_all`: not in all - `cont`: contains -
  `cont_any`: contains any - `cont_all`: contains all - `not_cont`: doesn't contain -
  `not_cont_any`: doesn't contain any - `not_cont_all`: doesn't contain all - `start`: starts with -
  `start_any`: starts with any - `start_all`: starts with all - `not_start`: doesn't start with -
  `not_start_any`: doesn't start with any - `not_start_all`: doesn't start with all - `end`: ends
  with - `end_any`: ends with any - `end_all`: ends with all - `not_end`: doesn't end with -
  `not_end_any`: doesn't end with any - `not_end_all`: doesn't end with all - `'true'`: is true -
  `'false'`: is false - `present`: is present - `blank`: is blank - `'null'`: is null - `not_null`:
  is not null

becoming this:

  Expand to see all available filter predicates
- `eq`: equals
- `eq_any`: equals any
- `eq_all`: equals all

...

@danielschuster-muc Can I update rest.en.mdx to apply the same style than in French version to the list of predicates ? So this:

  Expand to see all available filter predicates- `eq`: equals - `eq_any`: equals
  any - `eq_all`: equals all -
...

becoming this:

  Expand to see all available filter predicates
- `eq`: equals
- `eq_any`: equals any
- `eq_all`: equals all

...

Oh, you’re right, should be fixed. You can update those if you want.

Great, PR submitted 🚀
Thanks

hi @danielschuster-muc i dont know how to do my commits

Can it be translated into Arabic?

Can it be translated into Arabic?

sure! any language is appreciated.

Hi 👋 I'm also a Potterhead and I saw your project in Hacktoberfest discord server.
I would like to help translating in Filipino language.

Hi 👋 I'm also a Potterhead and I saw your project in Hacktoberfest discord server.
I would like to help translating in Filipino language.

Sure! I just assigned you.

Hello! I want to participate translating into Portuguese.

Happy to provide German translation ✌


Let material ThemeData dictate brightness if cupertinoOverrideTheme.brightness is null

Description
Make cupertino widgets interpret ThemeData.brightness as forcibly overriding the brightness of the subtree, if cupertinoOverrideTheme.brightness is null.
Fixes: https://github.com/flutter/flutter/issues/47255
Tests
I added the following tests:

MaterialTheme overrides the brightness

Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

[ ] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[ ] I signed the CLA.
[ ] I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
[ ] I read the Tree Hygiene wiki page, which explains my responsibilities.
[ ] I updated/added relevant documentation (doc comments with ///).
[ ] All existing and new tests are passing.
[ ] The analyzer (flutter analyze --flutter-repo) does not report any problems on my PR.
[ ] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Did any tests fail when you ran them? Please read Handling breaking changes.

[ ] No, no existing tests failed, so this is not a breaking change.
[ ] Yes, this is a breaking change. If not, delete the remainder of this section.

[ ] I wrote a design doc: https://flutter.dev/go/template Replace this with a link to your design doc's short link
[ ] I got input from the developer relations team, specifically from: Replace with the names of who gave advice
[ ] I wrote a migration guide: Replace with a link to your migration guide

@darrenaustin @HansMuller changed the approach. Ready for review.


BUG: passing in large data object to DataFrame apply() function resulting in SegFault

Modin version checks

[X] I have checked that this issue has not already been reported.

[X] I have confirmed this bug exists on the latest released version of Modin.

[ ] I have confirmed this bug exists on the main branch of Modin. (In order to do this you can follow this guide.)

Reproducible Example
import modin.pandas as pd
import numpy as np

n_features = 20000
n_samples = 20000
features = [f'f{i+1}' for i in range(n_features)]
samples = [f's{i+1}' for i in range(n_samples)]

df = pd.DataFrame(np.random.rand(n_samples, n_features), columns=features, index=samples).round(2)

# mat_df = pd.DataFrame(np.random.rand(n_features, n_features), columns=features, index=features).round(2)
# res_df = df.dot(mat_df)

# this can be simplified to:
mat_arr = np.random.rand(n_features, n_features)
res_df = df.dot(mat_arr)

Issue Description
Getting Segmentation fault (core dumped) when doing matrix multiplication on 20K x 20K matrices.
I have two servers, each with 16 threads, >50GB available RAM, >300GB available storage. Code tested on both py<EMAIL_ADDRESS>and py<EMAIL_ADDRESS>both showed same result.
The code is trying to do a dot() on two 20K x 20K matrices, each matrix is roughly 3GB (showing on object store memory in one of the nodes), the seg fault happens when calling df.dot(mat_df). However, the code will work on small matrices like 500 x 500.
There's not much to show on the terminal, basically it just says UserWarning: Ray Client is attempting to retrieve a 3.00 GiB object over the network, which may be slow. Consider serializing the object to a file and using S3 or rsync instead. and the program gets killed.
I have tried to increase my object store memory and also specify a plasma directory, and still getting seg fault, the problem seems to happen after the computation has finished on remote node and being transferred by to client, I saw an increase to 12GB on the object store memory , and around 3GB increase on the client's main memory (when the warning message shows up), right when the main memory increased to 3GB, the program gets killed.
Expected Behavior
should return the transformed df.
Error Logs

Replace this line with the error backtrace (if applicable).

Installed Versions

UserWarning: Setuptools is replacing distutils.
 Do you have a more detailed stack trace? I'm unable to reproduce the error locally, and I don't have access to the server hardware that you do.

@noloerino Hi, as I mentioned in the description, there's not much useful information on the terminal (userwarning + seg fault). But if you think logs from Ray could help, please let me know where and how I can pull for you, I am not very familiar with Ray clusters.

@SiRumCz could you run the following code and show output here?
import modin.pandas as pd
import numpy as np

n_features = 20000
n_samples = 20000
features = [f'f{i+1}' for i in range(n_features)]
samples = [f's{i+1}' for i in range(n_samples)]

df = pd.DataFrame(np.random.rand(n_samples, n_features), columns=features, index=samples).round(2)
print(f"{df._query_compiler._modin_frame._partitions.shape=}")

from modin.config import CpuCount, NPartitions
print(f"{CpuCount.get()=}, {NPartitions.get()=}")

@anmyachev Thanks for the update, here's the output:
>>> print(f"{df._query_compiler._modin_frame._partitions.shape=}")
df._query_compiler._modin_frame._partitions.shape=(48, 48)
>>> from modin.config import CpuCount, NPartitions
>>> print(f"{CpuCount.get()=}, {NPartitions.get()=}")
CpuCount.get()=16, NPartitions.get()=48

On both of my py3.8 and py3.10 venv, the version is 1.59.0.

@SiRumCz it would be great if you could also provide the following information:

Ray client logs can be found at /tmp/ray/session_latest/logs on the head node (by default). Please show if possible.
Could you also show the result of ray.cluster_resources() after connecting ray to the cluster from the main process?
What Modin environment variables did you set manually?
How did you set up the cluster, at least in general terms? Did you enable object spilling and so on?

@anmyachev I have an update, it seems like the problem is not from dot, it seems like the problem comes from my network. I have removed dot operation and the code will still fail with se fault:
import modin.pandas as pd
import numpy as np

n_features = 20000
n_samples = 20000
features = [f'f{i+1}' for i in range(n_features)]
samples = [f's{i+1}' for i in range(n_samples)]

df = pd.DataFrame(np.random.rand(n_samples, n_features), columns=features, index=samples).round(2)
mat_df = pd.DataFrame(np.random.rand(n_features, n_features), columns=features, index=features).round(2)
res_df = df.apply(lambda x,m: m.iloc[0], axis=1, m=mat_df) # this will still get seg fault

Error message:
>>> df.apply(lambda x,m: m.head(1), axis=1, m=mat_df)
UserWarning: Ray Client is attempting to retrieve a 2.99 GiB object over the network, which may be slow. Consider serializing the object to a file and using S3 or rsync instead.
Segmentation fault (core dumped)

@modin-project/modin-ray maybe you can tell us which direction to look in order to understand the reason for this segfault?

@anmyachev I can make it to work by not setting an RAY_ADDRESS, Modin/Ray seems to be able to pick up the address when I am running the client code on one of the Ray nodes. However, it is worth noting that I am seeing different ports after this change, before it was on port 10001, and now 6379 is being picked up.

@anmyachev I can make it to work by not setting an RAY_ADDRESS, Modin/Ray seems to be able to pick up the address when I am running the client code on one of the Ray nodes. However, it is worth noting that I am seeing different ports after this change, before it was on port 10001, and now 6379 is being picked up.

@SiRumCz good catch! Let's ensure (if needed) you are connecting to an existing cluster (instead of creating a new one), use ray.cluster_resources().
Thanks for all the information! I'm working to figure out what's wrong.

@anmyachev thanks for the advice. Just to add to the solutions, while it works, I am see some activities that's kind of confusing:

the memory usage on my nodes seems to be very high during the dataframe operation, there are moments that almost all the memory space are being filled with all cores running 100%.
during the operation, sometimes I am see messages that's related to OOM issue like 19 Workers (tasks / actors) killed due to memory pressure (OOM), and Some workers of the worker process(1964300) have not registered within the timeout. The process is still alive, probably it's hanging during start.

Although I am seeing these messages, my code would still be able to give me the result.

the memory usage on my nodes seems to be very high during the dataframe operation, there are moments that almost all the memory space are being filled with all cores running 100%.

Well round operation is highly parallel. It can use all available cores (partitions are not combined into columns or rows). For this operation, I would expect potentially two times as much memory usage as the dataframe occupies. However for dot it can be 3x (it should be decreased down to 2x by https://github.com/modin-project/modin/pull/6644)

@anmyachev I can make it to work by not setting an RAY_ADDRESS, Modin/Ray seems to be able to pick up the address when I am running the client code on one of the Ray nodes. However, it is worth noting that I am seeing different ports after this change, before it was on port 10001, and now 6379 is being picked up. Before:
2023-10-12 18:18:13,232 INFO worker.py:1329 -- Using address ray://<addr>:10001 set in the environment variable RAY_ADDRESS

After:
2023-10-13 10:06:44,844 INFO worker.py:1458 -- Connecting to existing Ray cluster at address: <addr>:6379...
2023-10-13 10:06:44,852 INFO worker.py:1633 -- Connected to Ray cluster. View the dashboard at http://<addr>:8265

@SiRumCz It turns out there are two modes of operation. Connecting  to a cluster through a Ray client and without a client. Perhaps connecting through the client does not work for Modin due to some architectural restrictions (which are mentioned in the note), but which exactly I don’t know yet.

@anmyachev thanks for the explanation, due to some architectural restrictions (which are mentioned in the note), could you please provide a bit more details so I can learn a bit more about it, thanks.

@anmyachev I mean this general note:
Ray Client has architectural limitations and may not work as expected when using Ray for ML workloads (like Ray Tune or Ray Train). Use [Ray Jobs API](https://docs.ray.io/en/latest/cluster/running-applications/job-submission/index.html#jobs-overview) for interactive development on ML projects.

I don’t know the details yet, but maybe @modin-project/modin-ray know.

The problem seems to be from a package grpc, upon checking my system, although the grpcio python package is the same version for both Ray nodes, they do have different Ubuntu versions, one has 22.04 and one has 20.04, and I am assuming one has higher grpc lib.

@SiRumCz can you install the same grpc versions for both systems?

@anmyachev by grpc, doyou mean grpcio for python? I actually have the same grpcio version (1.59.0) for both system.

and I am assuming one has higher grpc lib.

@SiRumCz not grpcio but grpc. You wrote that one of the Ubuntu systems may have a different version of grpc. If you see that Segmentation fault problem has gone away, we could try to do a reproducer for grpc or ray (because looks like a bug). However, upgrading package may not be easy and may take a lot of time, so don't do it if you don't have one :)

@anmyachev That was my thoughts but upon checking (grpc installation page)[https://grpc.io/blog/installation/], it seems to be language specific, and for python, it is grpcio, so I assume my environment is ok.
If I am to install grpc package, do you have any idea which one to be installed?
grpc package in Ubuntu
libgrpc++-dev: high performance general RPC framework (development)
libgrpc++1.51: high performance general RPC framework
libgrpc++1.51-dbgsym: debug symbols for libgrpc++1.51
libgrpc-dev: high performance general RPC framework (development)
libgrpc29: high performance general RPC framework
libgrpc29-dbgsym: debug symbols for libgrpc29
protobuf-compiler-grpc: high performance general RPC framework - protobuf plugin
protobuf-compiler-grpc-dbgsym: debug symbols for protobuf-compiler-grpc
python3-grpcio: GRPC system (Python 3)
python3-grpcio-dbgsym: debug symbols for python3-grpcio
ruby-grpc: GRPC system in Ruby
ruby-grpc-dbgsym: debug symbols for ruby-grpc
ruby-grpc-tools: Development tools for Ruby gRPC

@SiRumCz looks like I found pretty close issue in ray itself https://github.com/ray-project/ray/issues/38713. There's also a Segmentation Fault when one try to put a large object through ray.put (this is similar to what we see in your stack trace here) in Ray Client connection mode.

so I assume my environment is ok.

So let's assume that the environment is correct and just try to avoid connecting through the Ray Client for now.

@anmyachev good find!

Blocked by https://github.com/ray-project/ray/issues/38713


Update: Google component improvements

Added some improvements to google components:

Remove obsolete gmail references in google-calendar component
Avoid constants in interfaces
Remove unused oauth scopes configuration in gmail component

Thanks @christophd, merged on master


docformat sometimes produces output which it changes further on rerunning

Consider the following code:
class HealthCheck():
    """Classes of health check
    for use in :attr:`~hypothesis.settings.suppress_health_check`

    """

When run on this code, docformat produces the following:
class HealthCheck():
    """Classes of health check for use in
    :attr:`~hypothesis.settings.suppress_health_check`"""

Running docformat on this again produces the following:
class HealthCheck():
    """Classes of health check for use in.

    :attr:`~hypothesis.settings.suppress_health_check`

    """

Generally I would expect formatting tools to format to fixed points - i.e. once you've run them once, running them again should produce zero changes. Is it intentional that this is not the case here?

Thanks for the report! This is a bug. The first change you note is a line wrapping. When run again, a heuristic takes affect that didn't get triggered in the previous case. A temporary workaround would be to make is_probably_beginning_of_sentence() less zealous. But there may be a better approach to this.

Closed by #41.


Helm Chart for Argo-Workflows is Not Respecting Helm Values File

Pre-requisites

[X] I have double-checked my configuration
[X] I can confirm the issue exists when I tested with :latest
[X] I have searched existing issues and could not find a match for this bug
[ ] I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?
I tried deploying under a past version 0.33.1 and had a working configuration to deploy Argo-Workflows with a lot of values set up:

SSO Enabled
Metrics Enabled
Namespaced Enabled
ALB Integration Enabled
Ingress Enabled

It all worked at one point, now I can't get the same results when I try to redeploy it.
Version
latest
Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
Do the following, replacing respective values, and all you get is a 503 error code:

- Create an Oauth Application in Okta
- Create a Bookmark Application in Okta for a redirect
- Have an EKS Cluster
- Have an AWS Account
- You can use Terraform or Helm it doesn't matter
- Use a custom values file like this:

---
server:
  serviceType: "NodePort"
  authModes:
    - sso
  secure: true
  sso:
    enabled: true
    insecureSkipVerify: false
    clientId:
      key: "client_id"
      name: "argo-server-sso"
    clientSecret:
      key: "client_secret"
      name: "argo-server-sso"
    issuer: "<some_url>"
    redirectUrl: "<some_url>/oauth2/callback"
    sessionExpiry: "1h" #  1 Hour for Session Expiration
  ingress:
    annotations:
      alb.ingress.kubernetes.io/certificate-arn: <some_certificate_arn>
      alb.ingress.kubernetes.io/healthcheck-path: /
      alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
      alb.ingress.kubernetes.io/wafv2-acl-arn: <some_waf_arn>
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/ssl-redirect: "443"
      alb.ingress.kubernetes.io/subnets: <some_subnets>
      alb.ingress.kubernetes.io/tags: Environment=dev,Team=test
      alb.ingress.kubernetes.io/target-type: ip
      external-dns.alpha.kubernetes.io/hostname: <some_url>
    enabled: "true"
    ingressClassName: "alb"
    hosts:
      - <some_url>
    paths:
      - /*
    pathType: ImplementationSpecific
controller:
  logging:
    format: json
  metricsConfig:
    enabled: "true"
    serviceMonitor:
      enabled: "true"
    telemetryConfig:
      enabled: "true"
artifactRepositoryRef:
  artifact-repositories:
    annotations:
      workflows.argoproj.io/default-artifact-repository: default
    default:
      s3:
        bucket: irt-dl-us-argoworkflow-unrestricted-dev
        endpoint: s3.amazonaws.com
        region: us-east-1
        keyFormat: "workflow\
                    /{{workflow.creationTimestamp.Y}}\
                    /{{workflow.creationTimestamp.m}}\
                    /{{workflow.creationTimestamp.d}}\
                    /{{workflow.name}}\
                    /{{pod.name}}"
        useSDKCreds: true

Applying with:

helm install argo-workflows argo/argo-workflows --version 0.33.1 -n arrgo-workflows

### Logs from the workflow controller

```text
The controller works fine, it's the server pod that doesn't work.

Logs from in your workflow's wait container
I don't see a wait container.

The Helm Charts are a separately maintained repo, please redirect to https://github.com/argoproj/argo-helm

It all worked at one point, now I can't get the same results when I try to redeploy it.

Also, in any repo, you'll have to be a lot more specific about what is different and unexpected. Simply saying "It's different" is missing context and is not actionable.


Code folding does not support clicking left to the line numbers (editor)

In the code editor, if one left-clicks on left of the line number, this selects the line.
However if there is a folded block above, the line number is wrongly computed and the editor jump to another (wrong) location.

@rlaverde, another one for you ;-)


react-adal + axios

How to combine adalFetch with axios?

Its in the readme:
export const adalApiFetch = (fetch, url, options) =>
  adalFetch(authContext, adalConfig.endpoints.api, fetch, url, options);

The variable fetch passed here could be any lib you want.

import axios from 'axios';
import { adalApiFetch } from 'adalConfig;

const myApiCall = () => adalApiFetch(axios, 'www.yourdomain.com/api/endpoint', options)

There are a number of other examples in the issue tracker if you search axios

@mrcnw  => @Jonwheeler thank you.


Add support to projects that is using Firebase 7.0.0 > with SPM

Issue Description

The project is using SPM and Firebase 7.1.0. When I executed the fastlane command to upload symbols, fastlane crashs because it can't find the upload_symbols binary at /Applications/Fabric.app/**/upload-symbols or ./Pods/**/upload-symbols
Configuration

fastlane run upload_symbols_to_crashlytics
Complete verbose output

[⠼] 🚀 Ignoring digest-crc-0.6.1 because its extensions are not built. Try: gem pristine digest-crc --version 0.6.1
Ignoring unf_ext-<IP_ADDRESS> because its extensions are not built. Try: gem pristine unf_ext --version <IP_ADDRESS>
[✔] 🚀
[16:18:41]: fastlane detected a Gemfile in the current directory
[16:18:41]: However, it seems like you didn't use `bundle exec`
[16:18:41]: To launch fastlane faster, please use
[16:18:41]:
[16:18:41]: $ bundle exec fastlane run upload_symbols_to_crashlytics
[16:18:41]:
[16:18:41]: Get started using a Gemfile for fastlane https://docs.fastlane.tools/getting-started/ios/setup/#use-a-gemfile
Ignoring digest-crc-0.6.1 because its extensions are not built. Try: gem pristine digest-crc --version 0.6.1
Ignoring unf_ext-<IP_ADDRESS> because its extensions are not built. Try: gem pristine unf_ext --version <IP_ADDRESS>
[16:18:45]: Installing Ruby gem 'fastlane-plugin-firebase_app_distribution'...
Fetching fastlane-plugin-firebase_app_distribution-0.2.4.gem
[16:18:52]: Successfully installed 'fastlane-plugin-firebase_app_distribution'
+------------------------+---------+------------------------+
|                       Used plugins                        |
+------------------------+---------+------------------------+
| Plugin                 | Version | Action                 |
+------------------------+---------+------------------------+
| fastlane-plugin-fireb  | 0.2.4   | firebase_app_distribu  |
| ase_app_distribution   |         | tion_login             |
|                        |         | firebase_app_distribu  |
|                        |         | tion                   |
+------------------------+---------+------------------------+

[16:18:54]: -------------------------------------------
[16:18:54]: --- Step: upload_symbols_to_crashlytics ---
[16:18:54]: -------------------------------------------

[!] Failed to find Fabric's upload_symbols binary at /Applications/Fabric.app/**/upload-symbols or ./Pods/**/upload-symbols. Please specify the location of the binary explicitly by using the binary_path option

Environment

Library
Version

fastlane
2.167.0

fastlane-plugin-firebase_app_distribution
0.2.4

Hello @brennobemoura! Seems like this is an issue with the upload_symbols_to_crashlytics step which is not part of the App Distribution action. It doesn't seem like this issue is related to App Distribution. Could you possibly post this in the main fastlane repo to get support for the upload_symbols_to_crashlytics error?

All right, sorry 😉 I will open this issue there. Thanks!

No worries! Thanks!


Recyclerview Exemplar App ReadMe File Added

Description
Added README.md file for RecyclerView Exemplar App
Type of Change:

Application File added

Done added some more points into readme file to describe more about recyclerview and view binding + official docs/reference
Please add label of Hacktoberfest and also on RecyclerView exemplar app Pull Request too.
Thanks,

Thanks for your contribution @akshatbhuhagal 😃


Initialise OPENSSL_armcap_P to 0 before setting it based on capabilities, not after

In some builds, OPENSSL_armcap_P is set based on sysctl() results, but then cleared to 0 by an initialisation
that should really be done much earlier.

I think it's better to move this earlier to avoid future additions having the same issue.

Merged, thanks for the fix.


Oidc integration

It would be great to have oidc integration, so anything like Azure, Google, Dex could be used for SSO

Thanks for the suggestion! We've had a few users asking to expose more OIDC controls so I'll queue this into the backlog to work on next.

Fastest way to develop agains Generic OIDC provider would be a https://github.com/dexidp/dex

I wanted to weigh in on this, as I'm currently running a Dex setup. This would be very helpful for my usecase.
Thanks for the top notch work 👍

This will help us inmesely to get rid of openvpn..

#586 has been merged to support this and will be available in the next release. Please open another issue if you face any issues getting it set up!


Do we want to codegen pointer-providence-related casts differently?

In codegen_rvalue we used to handle two kinds of Rvalue::Cast: CastKind::Misc and CastKind::Pointer
Rust introduced in a couple PRs new variants that distinguish (from CastKind::Misc) casts that turn pointers into numbers or vice versa. This is likely due to the need to think specially about these, since they need to invent (or forget) pointer providence.
We currently do not take advantage of this. Should we?
Rust PRs:

https://github.com/rust-lang/rust/pull/97582
https://github.com/rust-lang/rust/pull/97653

I believe this got further broken out in a later rust toolchain update, and CastKind:Misc no longer exists, in favor of recording explicit kinds for everything. I don't think there's anything special to do here anymore. If we should be handling this differently, it'd be a different issue to support a specific aim.


Allow changing macro expansions on test actions.

There are cases where it's useful to specify an alternate target as macro expansion for tests instead of the default which it's currently the first buildable reference.

@yonaskolb would love your thoughts on this, or other alternatives! Tested the change on a big project generating a few hundred schemas. It does the job.

So basically, I'm trying to achieve a scheme where I can:

Run a full version of an app.
Build and launch mock app during tests action.
Be able to execute my tests.

It seems to work for the most part except I'm losing my test macroExpansions. The problem is that I'm relying on my tests to be able to expand macros to find some files.
When setting up a scheme like this, the action will pick up the first buildable reference it finds, and sometimes, for some reason, none. So, I would like to explicitly define which buildable reference to use for the test macro expansion.
One potential solution is to avoid relying on $(SOURCE_ROOT) in my test env vars and instead hardcode the paths. However, I'm inclined to explore a solution that utilizes macros. What are your thoughts on this?  @giginet @yonaskolb @freddi-kit

Maybe this instead? https://github.com/yonaskolb/XcodeGen/pull/1471

@yonaskolb I had some conflicting logic with the previous PR I had to fix - if you can give it another check 🙏🏻


citing vedo

Hi @marcomusy ,
I am trying to set up the bib entry for citing vedo in the following way,
@article{macro2021,
  author = {M. Musy et al.},
  title = {vedo, a python module for scientific analysis and visualization of 3D objects and point clouds},
  year = {2021},
  publisher= {Zenodo},
  doi = {10.5281/zenodo.4287635},
}

Is this ok?

Looks good! Thanks for using vedo and for all the feedback which helped on improving the package.


added option to receive amounts from the telegram bot in sats

Related to #219
Needs the merge of this pull request alexbosworth/ln-telegram#9

The ln-telegram reference on package.json can be changed after alexbosworth/ln-telegram#9 is merged

I've fixed the dependency to ln-telegram, I think that can be merged now


pascal bsp example

An example of using the pascal language in the compilation process

Codecov Report

Merging #40 into master will increase coverage by <.01%.
The diff coverage is 93.33%.

@@            Coverage Diff             @@
##           master      #40      +/-   ##
==========================================
+ Coverage   79.23%   79.24%   +<.01%
==========================================
  Files         326      326
  Lines       42481    42496      +15
==========================================
+ Hits        33660    33674      +14
- Misses       8821     8822       +1

Impacted Files
Coverage Δ

ppci/build/buildtasks.py
90.2% <93.33%> (+0.36%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0043061...3b5aa1a. Read the comment docs.

Nice job! Good example about using pascal. Unfortunately the pascal frontend is not as capable as the C frontend, a good example is always good to have! Thanks for contributing!


[go.mod]Bump from v0.3.0 to pseudoversion

Our intention is to track service operator and lib-common dependencies via pseudoversions. However renovate does not automatically bump from a tagged version (e.g. v0.3.0) to a newer but not tagged pseudoversion. So this patch does the manual bump. After this renovate will bump the newer pseduoversions.

/retest
fail looks like something not related to this bump

I think we need to manual adapt at least due to the need of the new RBAC rule for MariaDBAccount CR (see https://github.com/openstack-k8s-operators/nova-operator/pull/660)

/retest

We can do this bump and do a separate bump with adaptation to the latest mariadb


Possible to integrate SponsorBlock ?

Have you looked into this ? Though its already really, really good.

From the quick look I had right now, it seems that all you need to use SponsorBlock are the --sponsorblock- arguments for yt-dlp. I don't plan to explicitly integrate them into the interface, but the next version (coming soon) will allow the user to use custom arguments, so that should be enough I think.

Ok thats good to know.
Also for some reason when I choose webm it always downloads the avc stream instead of the vp9 one. This doesn't happen in normal yt-dlp for me.

Also for some reason when I choose webm it always downloads the avc stream instead of the vp9 one.

I just tested, and it seems the argument -S "res:1080,vcodec:webm,fps" causes yt-dlp to select the formats 137+251 (mp4/avc video + webm/opus audio). For -S "res:1080,vcodec:webm,acodec:webm,fps", it selects 137+140 (mp4/avc video + m4a audio). I don't know if I'm missing something, but that seems like a bug to me.
I guess the only way to avoid that, is to manually select the formats (go into settings and check "YouTube: show video info and list available formats").


ci: cleanup scripts for ci

Make the npm scripts more cross-platform friendly
Adds a buildcommand for CI

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 96.27%. Comparing base (561b0d7) to head (3838d6c).

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##             main      #36   +/-   ##
=======================================
  Coverage   96.27%   96.27%
=======================================
  Files          28       28
  Lines         403      403
  Branches       74       74
=======================================
  Hits          388      388
  Misses         13       13
  Partials        2        2

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Support directly connecting with tokens

Working on a webhook interaction with Feedly, each user will bring their own tokens. I would be remiss to hardcode it on the server. It'd be great if, instead of using a file to process the tokens, I could pass them directly into the Feedly constructor.

I like it.  Would you be willing to pass in an expiry date also?  I would throw an exception if you supplied any of (access_token, refresh_token, expires) without specifying all three.

Yeah, I'd pass in all three. Right now I'm manually setting these fields after construction:
f.state = {
    access_token: getAccessToken(),
    refresh_token: getRefreshToken(),
    expires: new Date("2018-12-08T01:54:19.154Z")
}


occurrences.get exception in demos documention

From
https://github.com/sckott/pygbif/blob/master/demos/pygbif-intro.ipynb
The example
from pygbif import occurrences
occurrences.get(key = 252408386)

Throws an exception
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/pygbif/occurrences/get.py", line 17, in get
    out = gbif_GET(url, {}, **kwargs)
  File "/Library/Python/2.7/site-packages/pygbif/gbifutils.py", line 19, in gbif_GET
    out.raise_for_status()
  File "/Library/Python/2.7/site-packages/requests/models.py", line 862, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://api.gbif.org/v1/occurrence/252408386

@tucotuco thanks for the heads up on this - that occurrence key does not exist anymore :(
I'll update the docs and make sure all egs work ... ⏲

Thank you!


[SDL 0232] Add Pushing Buffer Support to AudioStreamManager

Proposal: Add Pushing Buffer Support to AudioStreamManager

This proposal is to add the ability to push an audio buffer to the AudioStreamManager so it supports playing audio from raw data in addition to supporting playing audio from actual files.

Review: https://github.com/smartdevicelink/sdl_evolution/issues/732

I see the AudioManager only provides startAudioStream(boolean encrypted, final CompletionListener completionListener);
Is it possible to add startAudioStream(boolean encrypted, (boolean encrypted, final AudioStreamingCodec codec, final AudioStreamingParams params, final CompletionListener completionListener)?
So the app can specify the audio streaming parameters for the audio buffer.
Thanks


Fix wrong metadata parsing in CompactOnExpiredCollector

The bug only affects users who enable the option ENABLE_NEW_ENCODING.
We fixed the metadata parsing (on expire time field) in CompactOnExpiredCollector.
Besides, we change the type of some timestamp from int to int64.

Carry comment from @git-hulk in PragmaTwice/kvrocks#2 for better context:

RocksDB compaction allows to use of trivial move mode(move file into the next level instead of using compaction filter to generate the output file) when the compaction is not triggered by the manual way.

So after this fix, NeedCompact will return the correct result and then wait for the background compaction schedule(become no manual way so it allows the trivial move), so after we call the storage->Compact, it won't manually compact the moved file.

I fixed this issue by using force bottom-level compaction when triggering the manual compaction, it's our expected behavior.

See https://github.com/facebook/rocksdb/blob/ba597514309b686d8addb59616f067d5522186b7/db/db_impl/db_impl_compaction_flush.cc#L3358

Thanks all!


ImageUrl Array doesn't appear to be updating when array is updated (ImageUrl array reduced or added)

I've noticed that when I remove a photo from the imageUrl array that the array updates showing the proper array with a reduced length but that somehow when swiping to the very end of the array it get's passed undefined returning the error: Cannot read property 'url' of undefined
It's not exactly clear where the renderItem isn't updating because the index and totalIndex seem to update properly.

I have same issue. Any ideas how to fix it?


feat: add zookeeper 1.2.0 release

update zookeeper with release 1.2.0

:tada: This PR is included in version 1.71.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


feat: implement age values encryption

Implements:
https://jira.linode.com/browse/APL-184
Guidelines:
https://collaborate.akamai.com/confluence/display/APL/Age
Description:
This PR implements age values encryption, adds age into the tools image and updates bootstrap.ts for age key generation.
Is paired with: https://github.com/linode/apl-console/pull/432 and https://github.com/linode/apl-api/pull/553
NB: Following items will be done after reviews and testing

[ ] Tools image with v2.5.0
[ ] update versions.yaml

Checklist

[ ] Architecture Design Records have been added as adr/*.md and appended to list in adr/_index.md, if applicable.
[ ] The values-schema.yaml file and test/** fixtures have been updated to reflect code changes, if applicable.
[ ] The OpenApi Schema from redkubes/otomi-api project is compatible with definitions from values-schema.yaml file, if applicable.
[ ] Helm releases are meeting otomi's baseline security policies, if applicable.
[ ] Helm chart and helmfile changes are tested against upgrade scenario, if applicable.

I have tested this also on a cluster initialized without any SOPS configuration. Maybe not a common use case, but also works well.


Asciidoctor-vscode 0.3.3 won't install OS X 10.12.5 VSC 1.12.2

Both via the GUI and the cmd-line it won't install. cmd-line more helpful. Following these steps:
git clone https://github.com/joaompinto/asciidoctor-vscode
cd asciidoctor-vscode
npm install
sudo npm install -g vsce
vsce package
code --install-extension *.vsix
I get:
[ { Error: ENOENT: no such file or directory, open '/Users/tok/.vscode/extensions/joaompinto.asciidoctor-vscode-0.3.3/package.json'
errno: -2,
code: 'ENOENT',
syscall: 'open',
path: '/Users/tok/.vscode/extensions/joaompinto.asciidoctor-vscode-0.3.3/package.json' } ]
Now if I copy the package.json file in manually, I still get the same issue. I assume because code --install-extension *.vsix removes Users/tok/.vscode/extensions/joaompinto.asciidoctor-vscode-0.3.3 and starts over from scratch.
Vexing. FWIW this problem didn't manifest itself until I installed various other bits of the the whole asciidoc world (ascidoctor, plus some alignedstuff such as dblatex)

Could you please test with 0.3.4 which was published today ? Thanks

Sorry for the delay... Latest version 0.3.7 works with Code 1.18.
I'm going to clone the repo and in the unlikely event I have time will try to work on the code syntax highlighting e.g.

my code here
Then I can move from Atom to VS Code ;)


Update README.md for OSX setup

Common issue run into for first time setting up tool that uses pfctl, added instructions to address the problem in the README.

Thanks @keerthik ! So whats up with the settings, do you know? For me it works just out of the box.

I believe some recent OSX updates probably were increasing security measures, so disabled pfctl access to external calls until explicitly modified. At least that's my hunch. Did your /etc/pf.conf already have these lines or just never needed them?


SalesChannelContextRestorer::restoreByCustomer doesn't use context extensions

PHP Version
8.1
Shopware Version
<IP_ADDRESS>
Expected behaviour
The restoreByCustomer method uses the current context extensions like in assembleSalesChannelContext https://github.com/shopware/platform/blob/trunk/src/Core/Checkout/Cart/Order/OrderConverter.php#L301
Actual behaviour
The restoreByCustomer method only uses the context states, but not the extensions:
https://github.com/shopware/platform/blob/trunk/src/Core/System/SalesChannel/Context/SalesChannelContextRestorer.php#L162
How to reproduce

Create a customer via customerRepository and add the MailSendSubscriber::MAIL_CONFIG_EXTENSION as extension to the context and try to skip the registration email
The restoreByCustomer loses the extension and email will be sent 😢

Hi,
thx for your issue we will put it into our new evaluation process and have a look at it again.

[public] Automated response: This issue is linked to the internal issue https://shopware.atlassian.net/browse/NEXT-37960.
[created from NEXT-37960, comment 489036]


Feature Request: Add practical MPP support to SendToRouteV2

Background
Continued from #5733.
Currently, SendToRouteV2 only has very minimal and extremely impractical support for MPP: if just one of the shards fails, it cannot be replaced. A single shard failure causes the whole MPP to fail. This is problematic as the successful shards cannot be cancelled (or settled) and need to time out before their funds can be reused.
Your environment

lnd 0.13.1-beta

Steps to reproduce
Send MPP via SendToRouteV2 with two shards, one of them successful (reaching the destination), one of the failing (ending in some "temporary channel failure" due to lack of funds).
Expected behaviour
I can send a shard to replace the failing shard, (hopefully) completing the payment.
Actual behaviour
The payment cannot be re-used or completed. The successful shard blocks funds until it times out.

Very strong ACK! This would also allow others and me to test my results more easily on top of lnd by wrapping the simulation at https://github.com/renepickhardt/mpp-splitter/pull/11 I can do it if I collaborate with a given recipient if the recieving node allows me to send to various payment hashes and he keeps all of them and knows they belong to the same virtual MPP payment but this prevents me from conducting proper mainnet tests. As @Roasbeef mentioned in https://github.com/lightningnetwork/lnd/issues/5988#issuecomment-1083535587 that testing is probably of highest importance I would love to see this prioritized a bit higher.

Fyi in case Java is too much hustle I released a standalone python package of our "reference implementation" for optimally reliable payment flows at https://github.com/renepickhardt/pickhardtpayments
I think a very simple mainnet test can be done by
A) bring own oracleLightningNetwork class as a wrapper to the not existing and here requested mpp api. (msybe you could also include python to go for tests?)
B) make the pay command in the payment session class async
Ofc some other constraints (htlc limits are not considered so I would not run this in production) but it should be fairly straight forward to conduct your own mainnet tests with this.
Fyi @guggero

I had a look at the code and found something that could be a bug.
To me it sounds like a non-terminal failure is set with the intention of "it's just a bad route", but this failure is treated as terminal. As such, the whole MPP is "dead".
Sets generic "NO_ROUTE" error:
https://github.com/lightningnetwork/lnd/blob/master/routing/router.go#L2274
Comment just a few lines above indicating that NO_ROUTE is not a terminal error:

Process the error to check if it maps into a terminal error code, if not use a generic NO_ROUTE error.
https://github.com/lightningnetwork/lnd/blob/master/routing/router.go#L2266

MPP failure reason

NOTE: Will only be set once the daemon has given up on the payment altogether.
https://github.com/lightningnetwork/lnd/blob/master/channeldb/mp_payment.go#L168

"is terminal failure" treats ANY failure reason as terminal:
https://github.com/lightningnetwork/lnd/blob/master/channeldb/mp_payment.go#L179

Related explanation in linked issue: https://github.com/lightningnetwork/lnd/issues/5733#issuecomment-919985512


Question about installing against an external sqlite

Hi all, having a odd problem that I could use some eyes on.
version: 4.0.0
OS: Docker (Debian)
Node: 8.10
In order to get around the max expression depth of 1000 I needed to install sqlite manually with special compile time flag. The sqlite install and node-sqlite3 install both install correctly without any errors. I can enter the docker container, run sqlite, run the limit command and see the following...
sqlite> .limit
              length<PHONE_NUMBER>
          sql_length<PHONE_NUMBER>
              column 2000
          expr_depth 3000
     compound_select 500
             vdbe_op 250000000
        function_arg 127
            attached 10
 like_pattern_length 50000
     variable_number 999
       trigger_depth 1000
      worker_threads 0

You can see that the expr_depth is now set to 3000.
The issue is that when I run my code, I  get the following error...
{ Error: SQLITE_ERROR: Expression tree is too large (maximum depth 1000) errno: 1, code: 'SQLITE_ERROR' }

I'm confused as to why this module isn't respecting the new limit I'm setting.
I've attached the Dockerfile and relevant logs. If I could get some eyes on this that would be awesome. Maybe I'm missing something obvious.
FROM node:8.10
RUN apt-get update  \
    && apt-get -y install libsqlite3-dev

WORKDIR /usr/src/app

RUN wget https://www.sqlite.org/snapshot/sqlite-snapshot-201902021347.tar.gz \
    && tar xvfz sqlite-snapshot-201902021347.tar.gz

RUN cd sqlite-snapshot-201902021347 \
    && CFLAGS="-DSQLITE_THREADSAFE=1 -DHAVE_USLEEP=1 -DSQLITE_MAX_EXPR_DEPTH=3000" ./configure \
    && make \
    && make install

COPY ./rules-engine/package*.json ./

RUN npm install --build-from-source --sqlite=/usr/local  --verbose

COPY functions/models functions/models/
COPY rules-engine/ rules-engine/

CMD [ "npm", "start"]

Build logs: https://gist.github.com/jflewis/b71dbecaaf0b692494561ebc65c4c25c

Look into the node-gyp files, the additional "-DSQLITE_MAX_EXPR_DEPTH=3000" should be put there.
The docker example you provided will create a normal version of sqlite.
However, running this will do another build (the actual native .node file that will be included):
RUN npm install --build-from-source --sqlite=/usr/local  --verbose

Add the flags here (and in the other array):
https://github.com/mapbox/node-sqlite3/blob/master/deps/sqlite3.gyp#L80


list_logs and list_logs_with_pagination do not return logs when using a query containing specific values in @ prefixed fields

Describe the bug
I can run the following query in the DataDog UI and it successfully returns the log I'm looking for (as seen in screenshot 1):
@event:Processing\ completed\ for\ sample\ 05d5f80e-a512-47cc-9049-e19e1b710188*

When I construct the same request using this Python library, I get 0 logs returned.
To Reproduce
Steps to reproduce the behavior:

Code snippet to search logs using the list_logs function from the LogsApi class:

configuration = Configuration(host="https://api.datadoghq.eu", api_key={"apiKeyAuth": api_key, "appKeyAuth": app_key})
request_body = LogsListRequest(
    filter=LogsQueryFilter(
        query="@event:Processing\ completed\ for\ sample\ 05d5f80e-a512-47cc-9049-e19e1b710188*",
        indexes=["default"],
    ),
    sort=LogsSort.TIMESTAMP_ASCENDING,
    page=LogsListRequestPage(limit=500),
)
with ApiClient(configuration=configuration) as api_client:
    logs_api = LogsApi(api_client)
    logs = logs_api.list_logs(body=request_body)

The logs["data"] key will return an empty list ([])

I've confirmed my API and app keys are valid by using the validate() method from the AuthenticationApi class. I've also confirmed that the default index exists using the list_log_indexes() method from the LogsIndexesApi class.
I've tried lots of combinations of the @event: filter in the query and can confirm the following:

@event:* returns 1 or more logs
@event:Processing* returns 0 logs (any term following the :, even if it's wildcarded, doesn't seem to work)

It seems like any value added to the @ prefixed filters in searches do not seem to yield results? I've also tried to search for logs using other @ prefixed filters:

@dd.service:* returns 1 or more logs
@dd.service:worker returns 0 logs (any term following the :, even if it's wildcarded, doesn't seem to work)
@logger:* returns 1 or more logs
@logger:analysis* returns 0 logs (any term following the :, even if it's wildcarded, doesn't seem to work)

Is this a limitation of searching via this library?
Expected behavior
I would expect the same query in the DataDog UI to return the identical logs when using the same query in this Python library.
Screenshots
Screenshot 1

Environment and Versions (please complete the following information):
A clear and precise description of your setup:

We have a company account in the DataDog EU region
datadog-api-client version: 2.12.0

Adding this in case it helps. When I do a more wider search for log messages in the UI using the namespace only, I can see that it says there are 5,239 logs available between 7:02am and 7:22am this morning:

When I use list_logs_with_pagination instead of list_logs, the number of logs returned is much less:
configuration = Configuration(host="https://api.datadoghq.eu", api_key={"apiKeyAuth": api_key, "appKeyAuth": app_key})
request_body = LogsListRequest(
    filter=LogsQueryFilter(
        query="kube_namespace=psga-dev",
        indexes=["default"],
        _from="2023-06-23T07:02:24+00:00",
        to="2023-06-23T07:21:25.608470+00:00"
    ),
    sort=LogsSort.TIMESTAMP_ASCENDING,
    page=LogsListRequestPage(limit=500),
)

datadog_logs: list[dict[str, Any] = []
with ApiClient(configuration=configuration) as api_client:
    logs_api = LogsApi(api_client)
    logs = logs_api.list_logs_with_pagination(body=request_body)
    while True:
        try:
            for number, log in enumerate(logs):
                datadog_logs.append(log.to_dict())
            except ApiException as e:
                wait_for = int(e.headers["x-ratelimit-reset"])
                LOG.warning(f"Hit DataDog API rate limit, waiting for {wait_for} seconds for it to reset")
                time.sleep(wait_for)
            else:
                break

The datadog_logs list contains only 399 logs, not the 5,239 as seen in the UI.
Am I missing something on the pagination method above?

Hi,
Thanks for the report. I don't think this is an issue with the library, though it might be an issue with the API. Would you mind opening a ticket with our support team and linking this? Thanks.

Thanks Thomas. For clarity, I've raised a support ticket with DataDog and received the following automated response:

Your request (#1245578) has been received and will be reviewed by our team.

When I receive a response from them, I'll make sure this ticket is updated.

@therve I've been in touch with Joseph Beltran on the DataDog Support team and he's investigated this issue with the API but cannot find any faults with the endpoint that returns logs. Here's his reply to me as of this morning:

Following up on this case, strictly with regards to the use of the Get a list of logs API endpoint, our Logs team was unable to find the same inconsistencies you were reporting, with the number of logs being found and returned by the same query when being run in the UI vs. the API. This was also was true for the same cases I tested previously last Friday, when simply querying , or a prefixed value with wildcard (e.g. @event:Processing). This still leads me to believe that there may be an issue with the Python Library, and in general, the number of logs returned by these list_logs_with_pagination and list_logs functions, though I find it odd that I haven't been able to find any other recent reports of this behavior from other customers.

I don't have as much expertise or familiarity with the Python library (as opposed to the general Logs product and behavior with the UI/API), so I would encourage reaching back out in the GitHub issue you opened. In the meantime, I can also escalate this case internally with our engineering team for further investigation on this. I'll keep you posted here with any updates as soon as I receive them.

Therefore, I'm reaching back out to you to see if there's an issue with this Python library.
For what it's worth, I tried my code again this morning and seeing the same behaviour from last week.

Please also try with raw curl as well. Something like this:
curl -X POST "https://api.datadoghq.eu/api/v2/logs/events/search" -H "Accept: application/json" -H "Content-Type: application/json" -H "DD-API-KEY: ${DD_API_KEY}" -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" -d '{"filter": {"indexes": ["default"], "query": "@event:Processing\\ completed\\ for\\ sample\\ 05d5f80e-a512-47cc-9049-e19e1b710188*"}, "page": {"limit": 1000}, "sort": "timestamp"}' | jq '.data | length'

logs = logs_api.list_logs_with_pagination(body=request_body)
    for log in logs:
        datadog_logs.append(log.to_dict())

Apologies for the delay @therve, it's been a busy week.
Below is the LogsListRequest object that I'm building to send as the body of the request:
request_body = LogsListRequest(
    filter=LogsQueryFilter(
        query=str(query),
        indexes=[index],
        _from=_from.isoformat() if _from else unset,
        to=to.isoformat() if to else unset,
    ),
    sort=LogsSort.TIMESTAMP_ASCENDING,
    page=LogsListRequestPage(limit=500),
)

Where query is a class that I've built to use method chaining for ease of use building queries when interacting with DataDog. I've included it below for transparency:
class DataDogQuery:
    """
    A class using the method chaining design pattern to allow DataDog query strings to be built fluently.
    """

    def __init__(self) -> None:
        self._string = ""

    def __str__(self) -> str:
        """
        :return: The query as a string.
        """
        return self._string

    def event(self, event: str) -> "DataDogQuery":
        """
        Add a specific event to the query. Events containing words separated by a space automatically have a backslash
        appended to each word. The * character can be used as a wildcard to lazy match events.

        :param event: The event to append to the query.
        :return: The same instance of this object.
        """
        if " " in event:
            event = r"\ ".join(event.split(" "))
        self._append(query=f"@event:{event}")
        return self

    def kube_namespace(self, namespace: str) -> "DataDogQuery":
        """
        Add a specific Kubernetes namespace to the query.

        :param namespace: The name of the namespace to append to the query.
        :return: The same instance of this object.
        """
        self._append(query=f"kube_namespace:{namespace}")
        return self

    def request(self, method: str, relative_path: str) -> "DataDogQuery":
        """
        Add a specific request method and relative path to the query.

        :param method: The method used when making a request (e.g. GET/POST/PATCH/DELETE).
        :param relative_path: The relative path used when making a request (e.g. /api/path/endpoint/).
        :return: The same instance of this object.
        """
        self._append(query=f'@request:"{method} {relative_path}"')
        return self

    def status(self, status: str) -> "DataDogQuery":
        """
        Add a specific log status to the query.

        :param status: The log status to search for (e.g. critical, error, warn, info).
        :return: The same instance of this object.
        """
        self._append(query=f"status:{status}")
        return self

    def _append(self, query: str) -> None:
        """
        Appends the provided query to the string.

        :param query: The query to append to the existing string.
        """
        self._string += f" {query}" if self._string else query

Here's the output from the console when debugging locally (the \\ are expected as I'm escaping the spaces). It shows I'm not wrapping the string after @event: with quotes:
>>> request_body.filter.query
'kube_namespace:psga-dev @event:Processing\ completed*'

Please also try with raw curl as well. Something like this:
curl -X POST "https://api.datadoghq.eu/api/v2/logs/events/search" -H "Accept: application/json" -H "Content-Type: application/json" -H "DD-API-KEY: ${DD_API_KEY}" -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" -d '{"filter": {"indexes": ["default"], "query": "@event:Processing\\ completed\\ for\\ sample\\ 05d5f80e-a512-47cc-9049-e19e1b710188*"}, "page": {"limit": 1000}, "sort": "timestamp"}' | jq '.data | length'

Thanks for the example curl command. I've run the following one locally and you can see it's returning 19 logs:
>>> curl -X POST "https://api.datadoghq.eu/api/v2/logs/events/search" -H "Accept: application/json" -H "Content-Type: application/json" -H "DD-API-KEY: ${DD_API_KEY}" -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" -d '{"filter": {"indexes": ["default"], "query": "@event:Processing\\ completed\\ for\\ sample*", "from": "2023-06-29T11:00:00+01:00", "to": "2023-06-30T11:00:00+01:00"}, "page": {"limit": 1000}, "sort": "timestamp"}' | jq '.data | length'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 55836  100 55624  100   212   174k    679 --:--:-- --:--:-- --:--:--  174k
19

That marries up with the UI:

When I do the same search locally using this Python library:
configuration = Configuration(host="https://api.datadoghq.eu", api_key={"apiKeyAuth": api_key, "appKeyAuth": app_key})
request_body = LogsListRequest(
    filter=LogsQueryFilter(
        query="@event:Processing\ completed\ for\ sample*",
        indexes=[index],
        _from="2023-06-29T11:00:00+01:00",
        to="2023-06-30T11:00:00+01:00",
    ),
    sort=LogsSort.TIMESTAMP_ASCENDING,
    page=LogsListRequestPage(limit=500),
)
with ApiClient(configuration=configuration) as api_client:
    logs_api = LogsApi(api_client)
    logs = logs_api.list_logs(body=request_body)

I get the same response as len(logs.to_dict()["data"]) returns 19 as well. I've also tried with the list_logs_with_pagination method and that also gives me back 19 logs.
I'll keep digging on my side because something is clearly not working as I'd expect.

I've finally managed to understand why I'm not seeing the logs I'm looking for. The following example shows where I've been going wrong.
There are two logs I want to confirm exist in DataDog with messages:
f"Created sample run for sample {guid}*"
f"Processing completed for sample {guid}*"

These logs are emitted when a specific event happens in the product I'm testing. That event has a started and completed datetime field, which I'm using when searching for logs (as the _from and to respectively).
One of these objects has the following datetimes:
started: 2023-06-30T07:01:58+01:00
completed: 2023-06-30T07:21:25.657414+01:00

An example set of logs (just the event and timestamp fields) I'm looking for are:
{
    "event": "Created sample run for sample 0b6b2b33-6dfc-4afe-83f9-c044b1c77d53",
    "timestamp": "2023-06-30T06:01:57.758856Z"
},
{
    "event": "Processing completed for sample 0b6b2b33-6dfc-4afe-83f9-c044b1c77d53",
    "timestamp": "2023-06-30T06:21:28.596670Z"
}

Note that the first log has a timestamp of about 0.25 seconds before the _from filter and the second log has a timestamp of about 3.6 seconds after the to filter. That's entirely on me, apologies @therve.
Happy to close this bug and apologies for wasting your time.

Ah great thanks for the careful debugging. Much prefer that than an arcane bug in our code. Let us know if anything comes up again.